Optimism L1 -> L2 derivation #51

intoverflow · 2023-11-02T07:25:55Z

This PR builds on prior work by @Wollac and @hashcashier. It introduces a new tool: op-derive. This tool has two inputs:

block_no, an L2 block number (used as the initial "safe head")
blocks, a block count (indicating how many blocks should be derived)

The tool begins by reading the system transaction from the initial safe head. This allows the tool to determine which L1 block it should begin reading from.

It then applies the L1 -> L2 derivation process for the specified sequence of L2 blocks (block_no + 1, block_no + 2, ..., block_no + blocks).

Along the way, the tool verifies that each of the derived L2 blocks have the expected block number, parent block, and list of transactions.

This complements Zeth's existing op-block tool, which can prove that a given list of transactions generates a given block.

Direct links:

The derive logic is defined in the library; the host and guest both use this same function. The function fetches data using the BatcherDb trait. In the guest, we use an in-memory implementation; on the host, we use an implementation that fetches data using RPC, and which also saves this data to an in-memory implementation that can be later passed to the guest ("preflight data fetching").

Sample output

Starting from op block 110800000, we derive 10 blocks:

Eth tail: 18341377 0x4eb45301f17a8fcdbf65dd1122361db575792b517287121675bf856dfb8122f2
Op Head: 110800000 0x1545e3edbd5b7a5cb1ce231c9eabb6da6dd97596b8db21b81201187b5de6245e
Derived: 110800001 0x45eb3f54dfe4dfe462ebb5f13a4e48b4e3332f0ef48106f241f190dde7b68086
Derived: 110800002 0x89d059ee39f9d41560dbe7f658bb311628e04f7453b566aac5e48b9d01f5183a
Derived: 110800003 0x39c420afa9c7e36a5d2c90c1606035c5264fc3831e21b0fd9e2ae7c75d5e48fb
Derived: 110800004 0x562299262875e710d33b213408673094a81aad435093ee54dbe4eab87a71cb2c
Derived: 110800005 0xf399e43c3ed2d5a691202f059f497e20a1167403bb3eb85544269d194c17f2d2
Derived: 110800006 0xd9f3a0765136a70b2678d7b146323e34cc3321bd9052b45de07835ae0f0a32e1
Derived: 110800007 0xb2cbf289486819481d901c1ca3451db5d7b697a3643c9cccfbe4b3fe43d9583d
Derived: 110800008 0x28ee07c0cdc0ad79254affe94b025ac4585bf4159f1641eaf2ef4b10ac779b3b
Derived: 110800009 0x55cb0f317da978e69480ada142661741128b579f6d77abd40022b2d6113bdde0
Derived: 110800010 0x8684698f999dc866f034df4fcf954ce1e929dd2fd98c03399003039b5a773e18

Semantically, this output means: if L2 block 110800000 and L1 block 18341377 both have the correct hash, then L2 block 110800001 with the indicated hash has the correct block number, parent block, and list of transactions.

Additionally, if L2 block 110800001 has the correct hash, then L2 block 110800002 with the indicated hash has the correct block number, parent block, and list of transactions.

(And so on, inductively.)

Testing

Tested with by deriving 10,000 blocks starting from 110800000:

$ RUST_LOG=info ../zeth/target/release/op-derive \
        --eth-rpc-url="https://eth-mainnet.g.alchemy.com/v2/[my api key]" \
        --op-rpc-url="https://opt-mainnet.g.alchemy.com/v2/[my api key]" \
        --cache
        --block-no=110800000
        --blocks=10000
...
Eth tail: 18343028 0x053b8fbc01b52f4e8eda7539818b3f6c86a11ef1ceeecefbf73fe1930d3ec2e9
Op Head: 110800000 0x1545e3edbd5b7a5cb1ce231c9eabb6da6dd97596b8db21b81201187b5de6245e
Derived: 110800001 0x45eb3f54dfe4dfe462ebb5f13a4e48b4e3332f0ef48106f241f190dde7b68086
...
Derived: 110800010 0x8684698f999dc866f034df4fcf954ce1e929dd2fd98c03399003039b5a773e18
...
Derived: 110800100 0x31f3a2aef4d2bfad22ad8d75bc5e0e5f1f04bab51367bc9502712af72359d725
...
Derived: 110801000 0xea3dc1b2acdb188539fcaf165c51fc61f630f2a20bfeebd8da31528d7cccef9d
...
Derived: 110810000 0xe346f9d53c24d5214f6a8fcfb294e1319aee886be11eb02f2fe7e012436bdda4

…ens/op-derive

Wollac

It works for the happy world case, but there are a few open questions we should consider.

host/src/bin/op-derive.rs

lib/src/optimism/mod.rs

Wollac · 2023-12-14T21:00:00Z

lib/src/optimism/mod.rs

+                            let trie_key = tx_no.to_rlp();
+                            tx_trie.insert(&trie_key, tx)?;
+                        }
+                        if tx_trie.hash() != new_op_head.transactions_root {


Can't we validate more fields of the new_op_head ?

My thinking is that Zeth verifies the other fields. (Specifically: if Zeth agrees that the block hash of new_op_head is valid, then all of the fields are valid.) This, together with a derive proof that the transaction list is valid, completes the proof.

lib/src/consts.rs

lib/src/lib.rs

lib/src/optimism/batcher_db.rs

Wollac · 2023-12-14T21:26:38Z

We should also absolutely add an e2e test similar to block_cli_ethereum for the derivation data now in the repo

Wollac · 2023-12-15T19:08:07Z

We should also absolutely add an e2e test similar to block_cli_ethereum for the derivation data now in the repo

done

…tput when polling

intoverflow · 2023-12-16T04:12:52Z

lib/src/optimism/batcher.rs

+                hash: eth_block_hash,
+                timestamp: eth_block.block_header.timestamp.try_into().unwrap(),
+                base_fee_per_gas: eth_block.block_header.base_fee_per_gas,
+                deposits: deposits::extract_transactions(&config, eth_block)?,


I don't think we actually need to extract deposits here.

Batcher::new() is always given the L1 block corresponding to the Op head's epoch. In particular, if that block contains any deposits, they would have already been processed either in the Op head or one of its predecessors.

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * format guest code * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]>

commit 0aa4be1 Author: ceciliaz030 <[email protected]> Date: Mon Feb 12 18:42:01 2024 +0800 feature taiko compile commit f3b6a0f Author: ceciliaz030 <[email protected]> Date: Mon Feb 12 01:31:59 2024 +0800 TaikoProvider, TaikoSystemInfo, preflight commit 1e9ffe2 Author: ceciliaz030 <[email protected]> Date: Mon Feb 12 01:31:15 2024 +0800 host refactor commit 851b817 Author: ceciliaz030 <[email protected]> Date: Sat Feb 10 02:51:09 2024 +0800 fix commit 39d8654 Author: ceciliaz030 <[email protected]> Date: Sat Feb 10 02:35:08 2024 +0800 fix lib/hosts commit f87ffd7 Author: Cecilia Zhang <[email protected]> Date: Sat Feb 10 01:39:55 2024 +0800 refactor host Taiko API commit ef4172f Author: ceciliaz030 <[email protected]> Date: Wed Feb 7 21:54:13 2024 +0800 wip commit 6867af4 Author: Wolfgang Welz <[email protected]> Date: Tue Jan 23 11:43:23 2024 +0100 Fix OP block header validation (taikoxyz#74) * verify op block loaded from DB * improve errors * Simplify BatcherDb (taikoxyz#81) * fix warnings --------- Co-authored-by: Tim Carstens <[email protected]> commit 3a99983 Author: Wolfgang Welz <[email protected]> Date: Mon Jan 22 22:10:46 2024 +0100 Fix batch queue handling in Optimism derivation (taikoxyz#73) * sort batches in a multimap * update copyright commit 7280614 Author: Wolfgang Welz <[email protected]> Date: Mon Jan 22 21:21:48 2024 +0100 feat: Use revm Optimism execution (taikoxyz#44) * do not default source_hash and mint * use revm optimism execution * update testdata * add optimism tests * upgrade revm to latest commit * upgrade revm with latest op changes * Upgrade `revm` in Optimism execution (taikoxyz#76) * Update dependencies versions * fix revm commit * use correct tags * use EVM in comments * update copyright * register optimism revm handle * update revm * update k256 --------- Co-authored-by: john xu <[email protected]> --------- Co-authored-by: john xu <[email protected]> commit fbe4255 Author: Wolfgang Welz <[email protected]> Date: Fri Dec 22 15:52:52 2023 +0100 Fix channel bank handling in Optimism derivation (taikoxyz#71) * fix channel bank * channel_index is not mut commit 0b2efed Author: Wolfgang Welz <[email protected]> Date: Tue Dec 19 13:06:40 2023 +0100 Fix frame parsing and loading in Optimism derivation (taikoxyz#68) * cleanup frame parsing * fix frame loading * move byte wrapping into RLP encoding * improve documentation * fmt * use an actual Batch for the tests * log batcher transaction * add unit tests commit 4c0496c Author: Tim Carstens <[email protected]> Date: Mon Dec 18 04:25:01 2023 -0800 Optimism L1 -> L2 derivation (taikoxyz#51) * transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * format guest code * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]> commit 916f8c5 Author: Wolfgang Welz <[email protected]> Date: Tue Dec 12 19:39:39 2023 +0100 Update CI (taikoxyz#60) * Update and fix CI * fix cargo-install version * try without cache * Revert "try without cache" This reverts commit d91547ab1697989050cfdfce35229e8899b0b27f. * add GITHUB_TOKEN * review suggestions * make version env commit c24c7b8 Author: John Smith <[email protected]> Date: Tue Dec 12 22:02:16 2023 +0800 fix: use PathBuf instead of String to compatible different platforms and non-Unicode sequences in filepath (taikoxyz#65) * fix: use PathBuf and Path for compatibility * fix: use PathBuf as cache path type commit ee88f6d Author: John Smith <[email protected]> Date: Tue Dec 12 00:11:27 2023 +0800 fix: clippy complain (taikoxyz#64) commit 3e70593 Author: Wolfgang Welz <[email protected]> Date: Mon Nov 27 09:43:01 2023 +0100 Pre-flight and block building improvements (taikoxyz#55) * cleanup host crate * minor cleanups * fix trie creation * separate node maps * fix tests * fix proof generation * fix integration test * add documentation * inline public mpt methods * move mpt functions * fix verification of zero RPC accounts commit 59c23fd Author: Wolfgang Welz <[email protected]> Date: Tue Nov 21 17:23:03 2023 +0100 chore: MPT cleanups (taikoxyz#56) * MPT cleanups * avoid clone when calling hash * fix clippy commit 71135b7 Author: Tim Carstens <[email protected]> Date: Tue Nov 21 07:09:20 2023 -0800 Reintroduce data() on TxEssence (taikoxyz#57) commit 2290e8b Author: Wolfgang Welz <[email protected]> Date: Sat Nov 18 10:24:53 2023 +0100 cleanup transaction code (taikoxyz#53) commit 2dc0bc6 Author: Wolfgang Welz <[email protected]> Date: Sat Nov 18 10:23:30 2023 +0100 chore(deps): bump risc0 to v0.19.1 (taikoxyz#54) * upgrade dependencies * upgrade dependencies commit eed232b Author: Wolfgang Welz <[email protected]> Date: Wed Nov 1 19:05:41 2023 +0100 upgrade risc0 to v0.19 (taikoxyz#50) commit 69402de Author: Tim Carstens <[email protected]> Date: Thu Oct 26 13:07:33 2023 -0700 Introduce op-info tool (taikoxyz#48) * Introduce op-info tool * Whitespace * Update host/Cargo.toml Co-authored-by: Wolfgang Welz <[email protected]> * Update Cargo.lock --------- Co-authored-by: Wolfgang Welz <[email protected]> commit 2f12782 Author: Wolfgang Welz <[email protected]> Date: Thu Oct 19 23:10:56 2023 +0200 fix clippy (taikoxyz#45) commit 4db793c Author: John Smith <[email protected]> Date: Thu Oct 19 21:14:19 2023 +0800 fix: extra data can equal to MAX_EXTRA_DATA_BYTES (taikoxyz#43) commit ddb349b Author: Tim Carstens <[email protected]> Date: Thu Oct 19 04:35:16 2023 -0700 Enable RPC retries (taikoxyz#42)

hashcashier and others added 30 commits August 31, 2023 11:17

transactions module

7b08224

TxEssence trait

07c0335

Generic Transaction struct

ff74eeb

tx essence as a generic parameter

7fd240c

OptimismTxEssence

cb1b7d5

op chain spec

a3d826b

OpTxExecStrategy

c27cfcf

Merge branch 'main' of github.com:risc0/zeth into rkhalil/op

133e881

redundant block builder type params

b261dbf

strategy bundles

3dc1629

host binary parameters

ca30106

profiling flag

7e5ce52

optimism

b267f3f

new derivation binary skeleton, copy over libs

16db242

providers and conversion utils

dc4a251

epoch transitioning

2fa1919

basic derive flow

2ca612d

host-side derivation

2482e7d

read metadata from op head

b67f8ba

op-derive guest

c8fdfb3

disable guest memory leaks

2b1df5c

heapless batch derivation

25b97c0

Merge branch 'main' of github.com:risc0/zeth into rkhalil/op-derive

4839400

Pull from main

912786d

Merge with main

f2c8076

Add missing import

dd471d5

Remove heapless BinaryHeap

187cebc

Remove heapless

33df195

Merge from main

4805a4b

Merge branch 'rkhalil/op-derive' of github.com:risc0/zeth into tcarst…

83f482d

…ens/op-derive

format guest code

6d7364c

Wollac approved these changes Dec 14, 2023

View reviewed changes

Wollac added 3 commits December 15, 2023 12:08

add op-derive cmd test

1538b26

add cmd tests

0e08feb

cleanup optimism/mod

0cbb413

intoverflow added 14 commits December 15, 2023 12:03

Rename command line args

2b0ada8

Fix test arguments

3b166b5

Rename config field to max_channel_bank_size

bc1ec04

Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL

2ab54b2

Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity

2d007dd

Import from std instead of alloc/core

c81dd60

Re-enable core::mem::forget() optimization

2f6a865

Replace asserts with ensures; enforce absence of receipts for Op blocks

707b89b

Simplfy iteration through derived transactions

098d82d

Add Bonsai support to op-derive. Also add Bonsai session status to ou…

dae216a

…tput when polling

More println

040aa3a

Rework Batcher initialization

2fd7c89

More logging

a08b13d

More log output if Bonsai workflow fails

2b7186a

intoverflow commented Dec 16, 2023

View reviewed changes

hashcashier approved these changes Dec 18, 2023

View reviewed changes

hashcashier merged commit 4c0496c into main Dec 18, 2023
4 checks passed

hashcashier deleted the tcarstens/op-derive branch December 18, 2023 12:25

hashcashier restored the tcarstens/op-derive branch December 18, 2023 12:26

hashcashier deleted the tcarstens/op-derive branch December 18, 2023 12:40

Wollac mentioned this pull request Dec 18, 2023

Fix frame parsing and loading in Optimism derivation #68

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimism L1 -> L2 derivation #51

Optimism L1 -> L2 derivation #51

intoverflow commented Nov 2, 2023 •

edited

Loading

Wollac left a comment

Wollac Dec 14, 2023

intoverflow Dec 16, 2023

Wollac commented Dec 14, 2023

Wollac commented Dec 15, 2023

intoverflow Dec 16, 2023

Optimism L1 -> L2 derivation #51

Optimism L1 -> L2 derivation #51

Conversation

intoverflow commented Nov 2, 2023 • edited Loading

Sample output

Testing

Wollac left a comment

Choose a reason for hiding this comment

Wollac Dec 14, 2023

Choose a reason for hiding this comment

intoverflow Dec 16, 2023

Choose a reason for hiding this comment

Wollac commented Dec 14, 2023

Wollac commented Dec 15, 2023

intoverflow Dec 16, 2023

Choose a reason for hiding this comment

intoverflow commented Nov 2, 2023 •

edited

Loading