feat: Provably build derived op blocks #78

hashcashier · 2024-01-19T12:21:40Z

This PR amends the optimism block derivation process to verifiably build (using composition) the block associated with each batch before choosing whether to drop or adopt the batch as the new op head. The transaction data extracted during derivation is used directly as input for the block building process, which is executed as its own zkvm session whose receipt is pulled as an assumption for derivation in via composition.

…l/derive-tx-exec

intoverflow · 2024-01-22T22:51:33Z

Do we want to execute Op transactions in the derive process? I was thinking that we'd use Zeth for that, as described here.

hashcashier · 2024-01-23T08:49:40Z

Yeah I don't think that the derive process makes sense on its own without checking for a valid execution as otherwise it could adopt a batch with an invalid chain state/transaction list, violating the spec here. Deferring that check to another program would add complexity imo.

But actually, now that I think about this a bit more in terms of the spec section above and using it as a fraud/validity proof, this needs some changes. It should immediately drop bad batches instead of the post-derive validation it does now.

…l/derive-tx-exec

intoverflow · 2024-01-23T18:39:09Z

Here's my only hesitation:

The derive process (without tx execution) is very expensive. In the linked Op Rfp comment, running derive for 10 blocks is >1B cycles. It's expensive for two reasons:

It must read a bunch of Eth blocks, and
Batch data tends to be "clumped up:" you can have several Eth blocks in a row with no useful data, then suddenly a bunch of batches all appear at the same time.

Because of this, it makes sense to derive several blocks in the same proof (as this re-uses the work needed to read so many Eth blocks).

But if the derive process also executes transactions, we run into problems with having several blocks derived in 1 proof. (At least, that's what happened when we tried this a few months ago; I will try again with this PR to see if the situation has changed.)

Tldr: including the tx execution in the derive process might increase our total cycle count, since it might restrict us to only deriving 1 block per proof, which prevents us from being able to amorize the costly derive process across several blocks.

…l/derive-tx-exec

lib/src/builder/execute/optimism.rs

lib/src/builder/mod.rs

lib/src/input.rs

Wollac · 2024-01-31T20:50:02Z

lib/src/builder/mod.rs

+        input: BlockBuildInput<Self::TxEssence>,
+    ) -> Result<BlockBuildOutput> {
+        // Database initialization failure does not mean the block is faulty
+        let input_hash = input.partial_hash();


Could we compute this hash only when a failure occurs? I don't think it is needed otherwise.

yeah just adding a todo because it's a bit of a refactor to not have the builder own the input or return that only on error. luckily it's a sha256

lib/src/host/preflight.rs

primitives/src/transactions/optimism.rs

primitives/src/trie/mpt.rs

guests/eth-block/src/main.rs

Wollac · 2024-01-31T22:14:31Z

lib/src/optimism/mod.rs

+                // The first transaction MUST be a L1 attributes deposited transaction,
+                // followed by an array of zero-or-more user-deposited transactions.
+                let l1_attributes_tx = self.derive_l1_attributes_deposited_tx(&op_batch);
+                // TODO: revise that skipping undecodable transactions is part of spec


Important point, we might want to check what OP-geth is doing with invalid transactions

OK, I did some research on this:

This is analog to Ethereum, the CL creates the PayloadAttribute and passes it to the EL to create the ExecutionPayload, essentially what we call block input (maybe we should consider renaming)

For OP the PayloadAttribute also contains transactions, that must then be included by the EL.

If the PayloadAttribute cannot be parsed it is ignored, and the next batch is selected.

If the state execution fails, we have a consensus failure and everything is reset.

This means that undecodable transactions are not skipped, but they must all be valid.

With this in mind, I am wondering wether we actually need to prove the block failure... This would also be some kind of consensus error and since we didn't implement the resetting mechanic, simply abort seems to be fine.

Funnily enough, op-geth neither returns a state reset nor an invalid attribute error, so i guess it would just stop in this case.

there could be other problems with the block that could lead to dropping its batch, which needs to be justified with a proof.

lib/src/optimism/mod.rs

…l/derive-tx-exec

lib/src/optimism/mod.rs

primitives/src/transactions/ethereum.rs

Wollac

The transaction RLP-Encoding/Decoding is getting more and more ugly. But we can address that in a different PR or hope that alloy will implement it for us.

lib/src/optimism/mod.rs

lib/src/builder/mod.rs

Wollac

Maybe we should discuss those considerations.

…ceipt cache (+misc) (#80) * update crates * prove on bonsai * verify bonsai receipts * repeat bonsai requests until success * wider bonsai loop * switch to async bonsai * patch blocking code * feat: Deterministic builds with receipt cache (+misc) (#80) * readme,cache,cuda,no-dev-mode,receipt uuid output, etc.. * logging nit * update readme * rpc_cache -> cache_rpc * det build, receipt cache, info->debug * fix nondet receipt label * logging and TransactionKind rlp decode fix * nit * move more info output to trace * fix CI

* feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80)

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * test for warnings * nits * docker in readme * create cache file if not found * update copyright * update copyright --------- Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * initial monolith * refactor prompt style * op-info support * delete old host binaries * nits * fix tests * help nit * short params * bump zkvm, fix CI * clippies * save receipts * refactor * move clap structs to cli module * refactor * refactor main monolith * fixes and changes * vs code change * default cache folder nit * build after derive * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * missing cache files for test * update crates * prove on bonsai * verify bonsai receipts * readme,cache,cuda,no-dev-mode,receipt uuid output, etc.. * logging nit * rlp decoding transactions * provably bad blocks * build on derive * derivation test fix * support erroneous preflights * host: derive with receipts * fix op head * verification nit * repeat bonsai requests until success * wider bonsai loop * switch to async bonsai * update readme * patch blocking code * patch blocking code * rpc_cache -> cache_rpc * det build, receipt cache, info->debug * fix nondet receipt label * logging and TransactionKind rlp decode fix * update risc0 to release v0.20 * cargo locks * nit * move more info output to trace * update GH action to 0.20 * use old actions * stark2snark * changes * refactor rlp & comments * update to v20.1 * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * fmt * SNARK Verification via RPC call * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * test for warnings * nits * docker in readme * remove env unset code because #91 --------- Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * deterministic cache * test for warnings * nits * docker in readme * create cache file if not found * fmt * update copyright * update copyright --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Rami <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * improve execution * test for warnings * nits * docker in readme * create cache file if not found * update copyright * update copyright --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Rami <[email protected]>

use timestamp based slot num to query blob

hashcashier added 2 commits January 19, 2024 13:09

build after derive

dc98f83

Merge branch 'rkhalil/tidy-host' of github.com:risc0/zeth into rkhali…

b77f900

…l/derive-tx-exec

hashcashier requested review from intoverflow and Wollac January 19, 2024 12:21

hashcashier self-assigned this Jan 19, 2024

missing cache files for test

bdf939f

hashcashier changed the title ~~Provably build derived op blocks~~ feat: Provably build derived op blocks Jan 22, 2024

hashcashier added 2 commits January 23, 2024 13:50

Merge branch 'rkhalil/tidy-host' of github.com:risc0/zeth into rkhali…

cbc65a1

…l/derive-tx-exec

rlp decoding transactions

7069754

provably bad blocks

e96c68f

hashcashier marked this pull request as draft January 24, 2024 09:36

hashcashier added 5 commits January 24, 2024 17:25

build on derive

f3cb39e

derivation test fix

8a7c1f6

support erroneous preflights

eda3d11

host: derive with receipts

bc125d3

fix op head

1665e74

hashcashier marked this pull request as ready for review January 25, 2024 14:52

hashcashier added 4 commits January 26, 2024 10:58

verification nit

3ef9e55

patch blocking code

b3b6b38

Merge branch 'rkhalil/tidy-host' of github.com:risc0/zeth into rkhali…

bf4cb50

…l/derive-tx-exec

cargo locks

148a089

Wollac reviewed Jan 31, 2024

View reviewed changes

hashcashier added 2 commits February 2, 2024 13:59

Merge branch 'rkhalil/tidy-host' of github.com:risc0/zeth into rkhali…

1bb1942

…l/derive-tx-exec

refactor rlp & comments

9fa10e9

hashcashier requested a review from Wollac February 2, 2024 15:43

Wollac added 2 commits February 4, 2024 22:27

fix transaction RLP decoding

763c20f

adapt to changes

a91897f

fix clippy warnings

dc1ef51

Wollac requested changes Feb 4, 2024

View reviewed changes

lib/src/optimism/mod.rs Outdated Show resolved Hide resolved

primitives/src/transactions/ethereum.rs Outdated Show resolved Hide resolved

Wollac approved these changes Feb 4, 2024

View reviewed changes

Wollac reviewed Feb 4, 2024

View reviewed changes

lib/src/optimism/mod.rs Outdated Show resolved Hide resolved

Wollac reviewed Feb 4, 2024

View reviewed changes

lib/src/builder/mod.rs Outdated Show resolved Hide resolved

Wollac reviewed Feb 4, 2024

View reviewed changes

hashcashier and others added 4 commits February 5, 2024 11:07

state input hash on success, and with rlp instead of bincode

850e44b

fix tests

6e93b09

panic to signal invalid block build inputs

2002bb6

hashcashier merged commit cf4be86 into rkhalil/tidy-host Feb 5, 2024
4 checks passed

hashcashier deleted the rkhalil/derive-tx-exec branch February 5, 2024 15:56

hashcashier added a commit that referenced this pull request Feb 5, 2024

refactor: Unified zeth utility (#72)

d2ab2f6

* feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80)

johntaiko pushed a commit to johntaiko/zeth that referenced this pull request Apr 10, 2024

Merge pull request risc0#78 from smtmfft/ts-to-slot

211093d

use timestamp based slot num to query blob

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Provably build derived op blocks #78

feat: Provably build derived op blocks #78

hashcashier commented Jan 19, 2024 •

edited

Loading

intoverflow commented Jan 22, 2024

hashcashier commented Jan 23, 2024

intoverflow commented Jan 23, 2024

Wollac Jan 31, 2024

hashcashier Feb 2, 2024

Wollac Jan 31, 2024

Wollac Feb 4, 2024 •

edited

Loading

Wollac Feb 5, 2024

hashcashier Feb 5, 2024

Wollac left a comment

Wollac left a comment

feat: Provably build derived op blocks #78

feat: Provably build derived op blocks #78

Conversation

hashcashier commented Jan 19, 2024 • edited Loading

intoverflow commented Jan 22, 2024

hashcashier commented Jan 23, 2024

intoverflow commented Jan 23, 2024

Wollac Jan 31, 2024

Choose a reason for hiding this comment

hashcashier Feb 2, 2024

Choose a reason for hiding this comment

Wollac Jan 31, 2024

Choose a reason for hiding this comment

Wollac Feb 4, 2024 • edited Loading

Choose a reason for hiding this comment

Wollac Feb 5, 2024

Choose a reason for hiding this comment

hashcashier Feb 5, 2024

Choose a reason for hiding this comment

Wollac left a comment

Choose a reason for hiding this comment

Wollac left a comment

Choose a reason for hiding this comment

hashcashier commented Jan 19, 2024 •

edited

Loading

Wollac Feb 4, 2024 •

edited

Loading