eth: refactor tx sending #2979
Draft
eth: refactor tx sending #2979
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactor sending evm txs so that we store the txs before sending and are accepting of send errors. This avoids a scenario where we send the tx, but still receive an error, perhaps because of a bad connection or other problem with the rpc provider. Implement a send queue so that the caller doesn't have to wait for the send. This should speed up ticks in core if the rpc provider is functional but slow to respond.
buck54321
force-pushed
the
eth-sendq
branch
from
61fa850
to
1791b16
Compare
JoeGruffins
reviewed
Sep 18, 2024
There was a problem hiding this comment.
Working well, but unsure if it solved the problem in core. I will still see the long delay warning with loadbot and some long blocking ticks.
A few errors while testing, but I dont know if because of the changes:
2024-09-18 11:01:47.309 [WRN] CORE: notify: |WARNING| (order) Match revoked - Match 47c635b0 has been revoked - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.309 [TRC] CORE: Not unlocking coins for order with status booked
2024-09-18 11:01:47.309 [WRN] CORE: Match 47c635b022252f880462a53c757a677dd1ef57131ab6189f06d4607aa4345d33 revoked in status MakerRedeemed for order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.313 [WRN] CORE: Revoking order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.314 [TRC] MUX: "POST http://127.0.0.3:5758/api/walletsettings HTTP/1.1" from 127.0.0.1:48026 - 200 40B in 71.033µs
2024-09-18 11:01:47.316 [ERR] CORE: notify: |ERROR| (order) Order revoked - Order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35 on market eth_btc at 127.0.0.1:17273 has been revoked by the server - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.330 [TRC] MUX: "POST http://127.0.0.3:5758/api/walletsettings HTTP/1.1" from 127.0.0.1:48026 - 200 40B in 52.198µs
2024-09-18 11:01:47.632 [ERR] CORE: notify: |ERROR| (order) Redeem reporting error - Error notifying DEX of redemption for match 39bbc6c9399f2c418c126acaf98bf546910e12325b02bbabfb64fac81ff3ff83: error sending 'redeem' message: timed out waiting for "redeem" response (timeout) - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.632 [ERR] CORE: notify: |ERROR| (order) Redeem reporting error - Error notifying DEX of redemption for match 8f8a30a12a924a142a8557132218975a5cec36703711ea0a0b7f3e097726ad6e: error sending 'redeem' message: timed out waiting for "redeem" response (timeout) - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.633 [ERR] CORE[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":87,"payload":{"result":null,"error":{"code":24,"message":"failed to find redeemed coin 0c8a532d34a2cd9e1f537b847d0b9ce0d99460cf12d3bd608b9629fe29d7dab4:0"}},"sig":""}
2024-09-18 11:01:47.633 [ERR] CORE[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":88,"payload":{"result":null,"error":{"code":24,"message":"failed to find redeemed coin 0c8a532d34a2cd9e1f537b847d0b9ce0d99460cf12d3bd608b9629fe29d7dab4:1"}},"sig":""}
2024-09-18 11:23:44.709 [TRC] CORE[btc][SPV]: PublishTransaction(87d1579da7113d02b6c965080a2ff8926059b2b7fdc1101dd978612f73641e09) completed in 50.992061935s
2024-09-18 11:23:44.932 [DBG] CORE: Waiting 3.7s to tick trade 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c with 17 active matches
2024-09-18 11:23:44.932 [INF] CORE: runJob(redemption) completed in 1.249934129s
2024-09-18 11:23:44.932 [INF] CORE: Notified of maker's redemption (eth: 0x469a81253e1db943f47102496030820a13b3e70e4afdfe33ddd78601fb501231) and validated secret for order 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c...
2024-09-18 11:23:44.937 [DBG] CORE: Unsubscribing from the eth_btc order book for 127.0.0.1:17273
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [INF] CORE: Notified of maker's redemption (eth: 0xa38a58763fbeb9e9f0b7cf07a83098e36e0a401d32d806a2c6a2629d2665336a) and validated secret for order 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c...
2024-09-18 20:23:51.520 [TRC] MANTLE:STACKER:1: book note received
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for ed8c399a6bc8fba32ff3740d66a698ed4c4366d9a53577bef68a3eadb1859646 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 55ac8998a3554c56f9b09638dd94ed09fc67738abbacddcf3ae73603c8b0b17b succeeded
2024-09-18 20:23:51.521 [TRC] MANTLE:STACKER:1: book note received
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for f377f9cae7a8b7b49ba140663d4286ca510f0c954e10d4119dda6892c490835d succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for b9dc2ebf43dbcc91799af18edd46b93ccb4254b964a8903c4e6ea687bfdfa025 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 06d0d98d75b7f043eda075116fe83e3781ba084bd6200042e4478f8b34f6e3ed succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 49949cc075b456d5f596c9a193cb036a490b0658d3dc03e4f9e7c9d0f827eb82 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for a86c81866a07465ab1e7e88a2e029d9a09eaebd738c6ac46dc416a070138870f succeeded
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33807,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33809,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33813,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33810,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [INF] CORE:STACKER:1: Received bondexpired notification from 127.0.0.1:17273 for account 40c3d950a8824878fcec1289d974e110359089dd08fbf640bff5fe37d22fa4c8...
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33820,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33815,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33808,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33818,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
....
2024-09-18 20:23:51.522 [WRN] CORE:STACKER:1: Revoking order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258
...
2024-09-18 20:23:53.572 [ERR] CORE:STACKER:1: notify: |ERROR| (order) Order revoked - Order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 on market eth_btc at 127.0.0.1:17273 has been revoked by the server - Order: 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258
2024-09-18 20:23:53.572 [TRC] MANTLE:STACKER:0: book note received
2024-09-18 20:23:53.573 [CRT] MANTLE:STACKER:1: Error note received: {"type":"order","topic":"OrderRevoked","subject":"Order revoked","details":"Order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 on market eth_btc at 127.0.0.1:17273 has been revoked by the server","severity":5,"stamp":1726658632756,"acked":false,"id":"295136668dd800f2","order":{"host":"127.0.0.1:17273","baseID":60,"baseSymbol":"eth","quoteID":0,"quoteSymbol":"btc","market":"eth_btc","type":1,"id":"4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258","stamp":1726658596325,"submitTime":1726658592070,"sig":"3044022046a7a895373d257d64dff05f5c2e19da3f8b769a8d27e894d2ef72bba37d0a16022014206f1b20b76890f5d7ff205c6fc9105c34110061bc00b68abd1a5efc7ad2b9","status":5,"epoch":115110573,"qty":100000000,"sell":false,"filled":0,"matches":null,"cancelling":false,"canceled":false,"feesPaid":{"swap":0,"redemption":0,"funding":0,"refund":0},"allFeesConfirmed":true,"fundingCoins":[{"id":"d4474ff18b0575a97c9c78b138e86ee8865fee106a9a0576a157d9b9f88d0e6400000001","stringID":"640e8df8b9d957a176059a6a10ee5f86e86ee838b1789c7ca975058bf14f47d4:1","assetID":0,"symbol":"btc"}],"lockedamt":0,"parentAssetLockedAmt":0,"redeemLockedAmt":0,"refundLockedAmt":0,"accelerationCoins":[],"rate":658000,"tif":1,"targetOrderID":"","readyToTick":true}}
Comment on lines
+1053
to
+1055
| select { | ||
| case qt := <-w.sendQ: | ||
| queued = append(queued, qt) |
There was a problem hiding this comment.
It looks like this only allows one qt to be queued at a time in the slice, so the length of queued will be 0 or 1 and you don't need a slice. Am I reading wrong? w.sendQ does still allow for a queue.
There was a problem hiding this comment.
Yeah I did realize that I just hadn't gotten around to fixing it yet. I think I'm gonna refactor to get rid of the atomic uint32 too and just fall back to synchronous sending if the buffered channel is blocking.
Ethiqque
reviewed
Sep 26, 2024
Comment on lines
+354
to
+358
| creds, err := walletCredentials(chainCfg.ChainID, walletDir, net) | ||
| if err != nil { | ||
| return nil, nil, nil, fmt.Errorf("error generating wallet credentials: %w", err) | ||
| } | ||
| cl, err := newMultiRPCClient(creds, providers, log, chainCfg, 3, net) |
|
Moving to draft for now. I think there are some good things here, but it doesn't clearly help the issues we see with long ticks in core and there are some potential drawbacks that need more consideration. One thing I'm very worried about is that we could successfully send a tx, but still see an error from the rpc server for whatever reason or just a poorly-timed disconnect. That would cause some issues up the line, and could be the cause of a couple of reported bugs where redemptions were not properly recorded in core. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor sending evm txs so that we store the txs before sending and are accepting of send errors. This avoids a scenario where we send the tx, but still receive an error, perhaps because of a bad connection or other problem with the rpc provider.
Implement a send queue so that the caller doesn't have to wait for the send. This should speed up ticks in core if the rpc provider is functional but slow to respond.