Skip to main content

Transactional Processing

How to limit transaction processing, e.g., when importing large XML and JSON files

Written by Lenka Haringerová

🚨 This page describes an advanced variant of XML and JSON import, which can lead to data inconsistency if used improperly!

By default, each import is performed as a single database transaction — meaning either everything is saved, or nothing is. This is usually the desired behavior, and there is typically no reason to change it.

However, there may be situations where transactional behavior is not necessary. In that case, this behavior can be changed using the atomic attribute.


XML

<?xml version="1.0"?>
<winstrom version="1.0" atomic="false">
<faktura-vydana>
<id>code:123</id> ...
<polozkyFaktury>
<faktura-vydana-polozka>...</faktura-vydana-polozka>
<faktura-vydana-polozka>...</faktura-vydana-polozka>
</polozkyFaktury>
</faktura-vydana>
<faktura-vydana>
<id>code:456</id> ...
<polozkyFaktury>
<faktura-vydana-polozka>...</faktura-vydana-polozka>
<faktura-vydana-polozka>...</faktura-vydana-polozka>
</polozkyFaktury>
</faktura-vydana>
</winstrom>

JSON

{
"winstrom": {
"@version": "1.0",
"@atomic": "false",
"faktura-vydana": [
{
"id": "code:123",
"polozkyFaktury": {
"faktura-vydana-polozka": [ "...", "..." ]
}
},
{
"id": "code:456",
"polozkyFaktury": {
"faktura-vydana-polozka": [ "...", "..." ]
}
}
]
}
}


How the import behaves

If you set the atomic attribute to false, each record is imported in a separate transaction. So in the example above, two database transactions take place — one for invoice 123, and another for invoice 456. Line items are part of the invoice and are therefore imported in the same transaction as the invoice itself.

⚠️ The server does not flag an invalid value for this attribute — it behaves as if the import were atomic. So make sure the attribute really contains false, and not, for example, a typo.


What benefits does this provide

When importing large XML files with many records, the transaction takes a long time and a lot of information needs to be kept in memory. Both of these negatively affect performance. However, if each record is independent and it doesn't matter if saving one of them fails (for example, if you run the import regularly and/or can intervene manually in case of issues), you can significantly reduce the memory demands of the import.

For really large imports, the memory requirements for holding not-yet-saved data become so large that a significant portion of CPU time starts being consumed by the garbage collector (this can be monitored, for example, with the jconsole tool, which is a standard part of the JDK development environment). In atomic="false" mode, in such cases, the time required can also be drastically reduced.


Related

Did this answer your question?