[BUG] - missing resource isolation to prefer critical tasks over query processing

[andrejpodzimek]

Internal/External
External

Area
Other

Summary
Leader log queries impede critical validator processing and cause extreme numbers of missed slot leader checks.

Steps to reproduce

0. Watch the frequency of missed slot leader checks over time.
1. Run a demanding cardano-cli query in a loop against the validator (example below).
2. Watch the disaster unfold: In my case, there were 7% of missed slot leader checks due to a repeated query.
3. Repeat the test with a regular relay node. Tip differences will run sky high (>100) when queries are processed.

Expected behavior
Proper resource isolation.

• Ongoing query processing never delays inward tip propagation (“height”).
• Ongoing query processing on a validator never ever causes missed slot leader checks!
  The fact that one should not run queries against a validator is orthogonal; a validator should either process such queries gracefully, without impediment to critical operations, or outright reject them.
• Timing-critical tasks must take precedence. (They should not be timing-critical, but sadly are.)

System info (please complete the following information):

• OS Name: ArchLinux
• OS Version Only bad distros have this.
• Node version (output of cardano-node --version)

  cardano-node 9.1.1 - linux-x86_64 - ghc-8.10
  git rev 66dc08944479792b2823c9e1356914820c9ea059
  

• CLI version (output of cardano-cli --version)

  cardano-cli 9.2.1.0 - linux-x86_64 - ghc-8.10
  git rev 66dc08944479792b2823c9e1356914820c9ea059
  

Screenshots and attachments
An example query to expose resource isolation problems:

cardano-cli query leadership-schedule \
  --socket-path /run/cardano-validator/socket \
  --genesis config/mainnet-shelley-genesis.json \
  --mainnet \
  --vrf-signing-key-file keys/mainnet/vrf.skey \
  --stake-pool-id ... \
  --next

RTS options:

... +RTS -N -A64m -H -Iw59 --nonmoving-gc -RTS ...

Additional context
This case could be dismissed with “use a workaround”, i.e. “have a separate relay node for slot leader queries only”, i.e. not for routing to a validator. However, such an idea is suboptimal, increasing the amount of resources a pool operator must set aside by up to 50%, compared to the simplest relay + validator setup.

The lack of proper resource isolation may have been a contributing factor to my problem of never successfully validating a block, described in this post and above.