Performance degradation with growing number of transaction

[eshavkun]

Case:

• 1 user comes every second to perform 1 transfer

Result:

1. Wait time for transaction to be included grows fast with number of users:
   • Up to 5 users: usually no delay
   • Up to 10: usually up to 1 second delay
   • Up to 20: usually up to 3 second delay
   • After 20th user even 3 seconds is not enough
2. Only first block contains more than 1 transaction, others have only 1 although the load is constant

Expected:

• More stable wait time (not that fast growing) for transaction to be included
• Blocks having more than 1 transaction when there is more than 1 in the queue

[johannbarbie]

@pinkiebell found an upstream issue in tendermint that might fix this

[johannbarbie]

depends on #184 ?

[pinkiebell]

Almost solved by #222

[eshavkun]

The issue still manifests itself on testnet. The following tests we done:

30 seconds, 10 arrivals per second (300 transactions total). Result:

• Blocks passed: 87
• Blocks with only 1 tx: 40
• Max txs per block: 11
  => Many blocks (every second basically) still have only on tx
• Number of times eth_getBlockByNumber returned null: 10
• Wait times for receipt: up to 10 sec, median: 6, p95: 9
  => Wait times grow from 1 to 10 seconds during the test run, becoming stable 8-10 seconds towards the end of the test

10 seconds, 50 arrivals per second (500 transactions total). Result:

• Blocks passed: unknown: more than 100 of last txs didn't receive a receipt within 10 seconds
• Blocks with only 1 tx: at least 12 (see above)
• Max txs per block: 63
• Number of times eth_getBlockByNumber returned null: 3
  => This fails do not seem to correlate with load
• Number of times user didn't receive receipt withing 10 seconds: 163 (~30%!)