'Insufficient funds' error when doing multiple transfers at once

[michielbdejong]

The postgresql database has conditions on its tables which mean that if multiple requests try to increase the balance of the 'hold' account simultaneously, some of those requests will fail with an SQL error which is something like one of the following:

• { error: UPDATE "L_ACCOUNTS" SET "BALANCE" = "BALANCE" + $1 WHERE "NAME" = $2 - current transaction is aborted, commands ignored until end of transaction block
• { error: UPDATE "L_ACCOUNTS" SET "BALANCE" = "BALANCE" + $1 WHERE "NAME" = $2 - could not serialize access due to read/write dependencies among transactions
• ledger:error-handler error error: insert into "L_TRANSFERS" ("ADDITIONAL_INFO", "EXECUTION_CONDITION", "EXPIRES_DTTM", "LEDGER", "PROPOSED_DTTM", "STATUS_ID", "TRANSFER_UUID") values (DEFAULT, $1, $2, $3, $4, $5, $6) - could not serialize access due to read/write dependencies among transactions

This error is then "translated" to 'Sender has insufficient funds', even if the sender does have enough funds for the transfer they were attempting to send.

[michielbdejong]

It seems this brings the ledger into a broken state, and even after restarting the host server, users will incorrectly be told they have insufficient funds. I'm also seeing non-zero balance of the hold account. -> seems it will be OK as long a you wait a bit before sending the next payment through the same ledger.

[michielbdejong]

we talked about this at lunch, it would probably be more efficient to postpone any database writes until after a transfer has been fulfilled. If the ledger process crashes and comes back up, attempts to fulfill payments that were in flight since before the crash will fail, but this seems to be a small price to pay for getting way more ledger throughput and stability.

[michielbdejong]

It seems this bug can be circumvented by leaving roughly 150ms between one payment and the next.