Kupo x Hydra #117

KtorZ · 2023-04-19T09:35:54Z

🐉 Adds --hydra-host and --hydra-port command line options to connect and index the head state of a hydra-node

🐉 Internally, Hydra is just another ChainProducer and HeadIsOpen and SnapshotConfirmed messages are mapped to rollforward of blocks.

TODO:

Finish Hydra decoders (missing datum, script and metadata)
Handle intersection negotiation by skipping blocks
Many tests assume a busy network and fail because of this. Make it possible to run them against a busy hydra-node easily (e.g. re-using the benchmarks in https://github.com/input-output-hk/hydra)
TBD: Clear index state when seeing a HeadIsClosed message? Use a rollbackward?

This way the default is still to compile with -Werror, but it allows a local override in a cabal.project.local file with: constraints: kupo -production

We need to keep the transactions around until a SnapshotConfirmed will yield a "block" containing them.

This should make much of a difference, but is a more idiomatic way to store things by key.

We are re-using the ServerOutput golden files from the hydra repository to test compatibility with the hydra-node websocket API.

- Stub for TxValid decoder by using empty values (apart from txId) - Add confirmedTransactionIds parsing for Snapshot

This should already give use enough information to index some UTxO in the head.

Simulates transactions for each UTxO in the initial snapshot to make kupo index this "genesis" block.

The decoder is copy & pasted from ogmios with only ada/lovelace needing replacing. We should DRY this (or change our apis to be compatible).

KtorZ · 2023-09-26T13:51:57Z

docs/api/latest.yaml

+      --since origin \
+      --match * \
+      --workdir ./db
+    ```


KtorZ · 2023-09-26T13:58:45Z

src/Kupo/App/ChainSync/Hydra.hs

+    -> [Point]
+    -> WS.Connection
+    -> m IntersectionNotFoundException
+runChainSyncClient mailbox beforeMainLoop _pts ws = do


We're still doing nothing with the intersection points here, which means that we'll constantly re-index everything from scratch. This might re-insert previously deleted UTxO.

If I recall correctly, we discussed that a while back and settled on simply "skipping" messages from Hydra here until the intersection point as a first 'naive' implementation and until Hydra provides more reliable mechanisms for negotiating an intersection.

KtorZ · 2023-09-26T14:02:38Z

src/Kupo/App/ChainSync/Hydra.hs

+            HeadIsOpen{genesisTxs} ->
+                atomically (putHighFrequencyMessage mailbox (mkHydraBlock 0 genesisTxs))
+            TxValid{tx} ->
+                pushTx tx


I suppose that this will work for lack of better options, but this will cause Kupo's memory to grow possibly unbounded.

Transactions are 'popped' on snapshot confirmed, which means that the memory should likely just grow and shrink on every snapshot, but that also means that any non-confirmed transaction will sit in the memory forever.

That's true. It's not ideal, but necessary given the current API. We should, however, record kupo's use case for a need of a more "complete" SnapshotConfirmed message.. there is an ADR about making the Hydra API more resource based: cardano-scaling/hydra#1028.

KtorZ · 2023-09-26T14:04:26Z

src/Kupo/App/ChainSync/Hydra.hs

+      pushTx :: PartialTransaction -> m ()
+    , -- | Resolves a transaction id and removes it. Throws
+      -- 'TransactionNotInStore' when not found.
+      popTxById :: MonadThrow m => TransactionId -> m PartialTransaction


Note: it might be better to take a Set TransactionId to avoid the repeated readTVar/writeTVar since transactions are always removed in batches from the store.

In practice, hydra will snapshot after every transaction. So rarely we would receive a SnapshotConfirmed with more than one tx inside. So batching things up is likely not going to be of big impact and I'd argue the singular API is a bit easier to use?

I'd argue the singular API is a bit easier to use?

I'd argue for the opposite; taking a foldable makes the API more generic and thus more easily composable.

I've solved your argument by accepting a [PartialTransaction] for popTxByIds ;)

KtorZ · 2023-09-26T14:05:49Z

src/Kupo/Data/Hydra.hs

I haven't looked at that module yet. But it seems like it's mostly about the Hydra's API and the underlying plumbing.

KtorZ · 2023-09-26T14:07:08Z

test/Test/Kupo/Data/HydraSpec.hs

+              tx' <- popTxById txId
+              tx' `shouldBe` tx
+              popTxById txId `shouldThrow` \case
+                  TransactionNotInStore txId' -> txId' == txId


Small note: this test is 'misplaced' as it's testing stuff from Kupo.App.ChainSync.Hydra, not from Kupo.Data.Hydra.

I can move things around eventually, so consider this is just a note for myself 😅

Let's have @v0d1ch do it? :P

Moved to appropriate place.

So that we remove many elements from the store in a single atomic transaction.

We do this by skipping blocks because we don't really have much alternative. This is therefore relatively unsound for very long running head. Though is it more sound that actually replaying all events from that head onto an existing database; so it's kind of a best effort solution.

KtorZ self-assigned this Apr 19, 2023

ch1bo force-pushed the ensemble/hydra-producer branch from 9a8bf3d to 418f735 Compare July 18, 2023 16:18

ch1bo and others added 7 commits September 12, 2023 15:59

Add a Hydra ChainProducer and a test to connect to Hydra

2821a87

Scaffold Hydra producer plumbing, actual syncing logic to be done.

628da34

Decode a hydra messages in the Hydra ChainSync

8b7858d

Fix incomplete pattern-match for FetchBlockClient

715dd00

Reduce test timeout for Hydra smoke test.

74fc7c4

Connect Hydra ChainSync to mailbox with hard-coded data

5fa1065

Actually (partially) parse confirmed snapshot from Hydra

c2c32af

ch1bo force-pushed the ensemble/hydra-producer branch from 418f735 to c2c32af Compare September 12, 2023 15:05

ch1bo self-assigned this Sep 19, 2023

ch1bo mentioned this pull request Sep 19, 2023

Basic indexer integration with Kupo cardano-scaling/hydra#1078

Closed

2 tasks

ch1bo linked an issue Sep 20, 2023 that may be closed by this pull request

Basic indexer integration with Kupo cardano-scaling/hydra#1078

Closed

2 tasks

ch1bo and others added 6 commits September 22, 2023 10:03

Decode HeadIsOpen message and interpret it as a roll forward

fe25c12

Flip cabal flag production to allow local override

3cab669

This way the default is still to compile with -Werror, but it allows a local override in a cabal.project.local file with: constraints: kupo -production

Draft a TransactionStore to handle TxValid messages

cbaf466

We need to keep the transactions around until a SnapshotConfirmed will yield a "block" containing them.

Add HydraSpec with a TransactionStore property test

197cf87

Implement TransactionStore handle

09494b3

Implement TransactionStore usind a Map

297df6d

This should make much of a difference, but is a more idiomatic way to store things by key.

ch1bo force-pushed the ensemble/hydra-producer branch 2 times, most recently from cc666a9 to 09e560e Compare September 25, 2023 11:06

ch1bo and others added 7 commits September 25, 2023 17:20

Add hydra submodule and test decoding of hydra messages

d45d143

We are re-using the ServerOutput golden files from the hydra repository to test compatibility with the hydra-node websocket API.

Implement more decoders for Hydra messages

83c687a

- Stub for TxValid decoder by using empty values (apart from txId) - Add confirmedTransactionIds parsing for Snapshot

Decode inputs and outputs partially

3ff12fd

This should already give use enough information to index some UTxO in the head.

Decode HeadIsOpen message as multiple "genesis" transactions

478046d

Simulates transactions for each UTxO in the initial snapshot to make kupo index this "genesis" block.

Decode values in Hydra messages

77b64aa

The decoder is copy & pasted from ogmios with only ada/lovelace needing replacing. We should DRY this (or change our apis to be compatible).

Add Hydra command line options

2adcece

Update changelog and add documentation about Hydra options

0d1009a

ch1bo force-pushed the ensemble/hydra-producer branch from b6efbd4 to 0d1009a Compare September 25, 2023 15:20

KtorZ commented Sep 26, 2023

View reviewed changes

KtorZ added 2 commits October 12, 2023 15:30

Rework Hydra transaction store API to use list.

35fe6c2

So that we remove many elements from the store in a single atomic transaction.

Implement remaining JSON decoders for Hydra.

6a6c767

KtorZ force-pushed the ensemble/hydra-producer branch from a259e6b to 397a4d8 Compare October 13, 2023 13:13

KtorZ force-pushed the ensemble/hydra-producer branch from 397a4d8 to f28b673 Compare October 13, 2023 15:16

KtorZ merged commit b126d23 into master Oct 13, 2023
1 check passed

ch1bo deleted the ensemble/hydra-producer branch October 30, 2023 11:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kupo x Hydra #117

Kupo x Hydra #117

KtorZ commented Apr 19, 2023 •

edited

Loading

KtorZ Sep 26, 2023

KtorZ Sep 26, 2023

KtorZ Sep 26, 2023

ch1bo Sep 26, 2023

KtorZ Sep 26, 2023

ch1bo Sep 26, 2023

KtorZ Sep 26, 2023

v0d1ch Oct 3, 2023

KtorZ Sep 26, 2023

KtorZ Sep 26, 2023

ch1bo Sep 26, 2023

v0d1ch Oct 3, 2023

Kupo x Hydra #117

Kupo x Hydra #117

Conversation

KtorZ commented Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KtorZ commented Apr 19, 2023 •

edited

Loading