refactor: Unified zeth utility #72

hashcashier · 2024-01-10T13:13:06Z

This PR integrates all features into a single CLI binary with a command based usage pattern. The task to be performed by the binary (preflight/exec/prove/verify) is now explicit in the command called rather than implied by the provided arguments.

zeth --help

Usage: zeth <COMMAND>

Commands:
  build    Build blocks natively outside the zkVM
  run      Run the block creation process inside the executor
  prove    Provably create blocks inside the zkVM
  verify   Verify a block creation receipt
  op-info  Output debug information about an optimism block
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

For every command, the --network parameter can be set to either ethereum or optimism for provable derivation of single blocks from either chain. This provides the basic functionality of the original main binary.

zeth build --help

Build blocks natively outside the zkVM

Usage: zeth build [OPTIONS] --block-number=<BLOCK_NUMBER>

Options:
  -w, --network=<NETWORK>            Network name (ethereum/optimism/optimism-derived) [default: ethereum]
  -e, --eth-rpc-url=<ETH_RPC_URL>    URL of the Ethereum RPC node
  -o, --op-rpc-url=<OP_RPC_URL>      URL of the Optimism RPC node
  -c, --cache[=<CACHE>]              Use a local directory as a cache for RPC calls. Accepts a custom directory. [default: host/testdata]
  -b, --block-number=<BLOCK_NUMBER>  Block number to begin from
  -n, --block-count=<BLOCK_COUNT>    Number of blocks to provably derive [default: 1]
  -m, --composition[=<COMPOSITION>]  Compose separate block derivation proofs together. Accepts a custom number of blocks to process per derivation call. (optimism-derived network only) [default: 1]
  -h, --help                         Print help

With --network=optimism-derived, the optimism blocks requested are derived using the data available on the ethereum chain. The number of sequential op blocks to derive is set through the --block-count parameter. The derivation proof creation method is done without proof composition by default, requiring the derivation to fit inside a single zkVM execution.

zeth run --help

Run the block creation process inside the executor

Usage: zeth run [OPTIONS] --block-number=<BLOCK_NUMBER>

Options:
  -w, --network=<NETWORK>            Network name (ethereum/optimism/optimism-derived) [default: ethereum]
  -e, --eth-rpc-url=<ETH_RPC_URL>    URL of the Ethereum RPC node
  -o, --op-rpc-url=<OP_RPC_URL>      URL of the Optimism RPC node
  -c, --cache[=<CACHE>]              Use a local directory as a cache for RPC calls. Accepts a custom directory. [default: host/testdata]
  -b, --block-number=<BLOCK_NUMBER>  Block number to begin from
  -n, --block-count=<BLOCK_COUNT>    Number of blocks to provably derive [default: 1]
  -l, --local-exec=<LOCAL_EXEC>      The maximum segment cycle count as a power of 2 [default: 20]
  -p, --profile                      Whether to profile the zkVM execution
  -h, --help                         Print help

The derivation proof can alternatively be created using proof composition by setting the --composition parameter to the number of op blocks per rolled up proof. However, the run command does not support this parameter because receipts are required for the proof composition. Consequently, one has to call the prove command in dev mode for the same functionality.

zeth prove --help

Provably create blocks inside the zkVM

Usage: zeth prove [OPTIONS] --block-number=<BLOCK_NUMBER>

Options:
  -w, --network=<NETWORK>            Network name (ethereum/optimism/optimism-derived) [default: ethereum]
  -e, --eth-rpc-url=<ETH_RPC_URL>    URL of the Ethereum RPC node
  -o, --op-rpc-url=<OP_RPC_URL>      URL of the Optimism RPC node
  -c, --cache[=<CACHE>]              Use a local directory as a cache for RPC calls. Accepts a custom directory. [default: host/testdata]
  -b, --block-number=<BLOCK_NUMBER>  Block number to begin from
  -n, --block-count=<BLOCK_COUNT>    Number of blocks to provably derive [default: 1]
  -l, --local-exec=<LOCAL_EXEC>      The maximum segment cycle count as a power of 2 [default: 20]
  -p, --profile                      Whether to profile the zkVM execution
  -m, --composition[=<COMPOSITION>]  Compose separate block derivation proofs together. Accepts a custom number of blocks to process per derivation call. (optimism-derived network only) [default: 1]
  -s, --submit-to-bonsai             Prove remotely using Bonsai
  -h, --help                         Print help

To verify existing receipts:

zeth verify --help

Verify a block creation receipt

Usage: zeth verify [OPTIONS] --block-number=<BLOCK_NUMBER>

Options:
  -w, --network=<NETWORK>
          Network name (ethereum/optimism/optimism-derived) [default: ethereum]
  -e, --eth-rpc-url=<ETH_RPC_URL>
          URL of the Ethereum RPC node
  -o, --op-rpc-url=<OP_RPC_URL>
          URL of the Optimism RPC node
  -c, --cache[=<CACHE>]
          Use a local directory as a cache for RPC calls. Accepts a custom directory. [default: host/testdata]
  -b, --block-number=<BLOCK_NUMBER>
          Block number to begin from
  -n, --block-count=<BLOCK_COUNT>
          Number of blocks to provably derive [default: 1]
  -r, --receipt-bonsai-uuid=<RECEIPT_BONSAI_UUID>
          Verify the receipt from the provided Bonsai Session UUID
  -h, --help
          Print help

Lastly, the op-info binary has also been integrated as a stand alone command:

zeth op-info --help

Output debug information about an optimism block

Usage: zeth op-info [OPTIONS] --block-number=<BLOCK_NUMBER>

Options:
  -w, --network=<NETWORK>            Network name (ethereum/optimism/optimism-derived) [default: ethereum]
  -e, --eth-rpc-url=<ETH_RPC_URL>    URL of the Ethereum RPC node
  -o, --op-rpc-url=<OP_RPC_URL>      URL of the Optimism RPC node
  -c, --cache[=<CACHE>]              Use a local directory as a cache for RPC calls. Accepts a custom directory. [default: host/testdata]
  -b, --block-number=<BLOCK_NUMBER>  Block number to begin from
  -n, --block-count=<BLOCK_COUNT>    Number of blocks to provably derive [default: 1]
  -h, --help                         Print help

…il/tidy-host

Wollac

nice!

…il/tidy-host

* feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80)

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * test for warnings * nits * docker in readme * create cache file if not found * update copyright * update copyright --------- Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * initial monolith * refactor prompt style * op-info support * delete old host binaries * nits * fix tests * help nit * short params * bump zkvm, fix CI * clippies * save receipts * refactor * move clap structs to cli module * refactor * refactor main monolith * fixes and changes * vs code change * default cache folder nit * build after derive * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * missing cache files for test * update crates * prove on bonsai * verify bonsai receipts * readme,cache,cuda,no-dev-mode,receipt uuid output, etc.. * logging nit * rlp decoding transactions * provably bad blocks * build on derive * derivation test fix * support erroneous preflights * host: derive with receipts * fix op head * verification nit * repeat bonsai requests until success * wider bonsai loop * switch to async bonsai * update readme * patch blocking code * patch blocking code * rpc_cache -> cache_rpc * det build, receipt cache, info->debug * fix nondet receipt label * logging and TransactionKind rlp decode fix * update risc0 to release v0.20 * cargo locks * nit * move more info output to trace * update GH action to 0.20 * use old actions * stark2snark * changes * refactor rlp & comments * update to v20.1 * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * fmt * SNARK Verification via RPC call * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * test for warnings * nits * docker in readme * remove env unset code because #91 --------- Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Wolfgang Welz <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * deterministic cache * test for warnings * nits * docker in readme * create cache file if not found * fmt * update copyright * update copyright --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Rami <[email protected]>

* transactions module * TxEssence trait * Generic Transaction struct * tx essence as a generic parameter * OptimismTxEssence * op chain spec * OpTxExecStrategy * redundant block builder type params * strategy bundles * host binary parameters * profiling flag * optimism * new derivation binary skeleton, copy over libs * providers and conversion utils * epoch transitioning * basic derive flow * host-side derivation * read metadata from op head * op-derive guest * disable guest memory leaks * heapless batch derivation * Add missing import * Remove heapless BinaryHeap * Remove heapless * Introduce op-derive tool * Remove ethers types from BatcherDb trait * Verify new op block has correct transaction list * Move derive logic into library * Fix bug in transaction trie reconstruction * Introduce get_op_header() to BatcherDb * Clippy warning * Default Serde value for FileProvider::receipts * Disable bloom filter checks * Clippy * Clippy * Fix parsing of from and to fields for deposits * Reintroduce filtering by log bloom * fmt * Add support for local exec to op-derive * Enforce block_number is correct in MemDb * Split derive() into multiple functions * Remove redundant check for batch parent hash * Remove redundant copy of system config * Remove redundant block number check * Cleanup * Remove redundant vector of eth blocks * Add base_fee_per_gas to Epoch * Store deposits in Epoch * Add Eth tail to DeriveOutput * Cleanup * Move deque_next_epoch_if_none to State * Move eth block processing to Batches * update zkvm, basic guest * more compose guest code, comment bonsai code * untested composition draft guest code * Reorg and cleanup * More cleanup * run cargo fmt --all * fix clippy warnings * remove unused imports * basic prep/left/fin in-memory flow * in memory aggregation workflow * format guest code * composition with receipts * add op-derive cmd test * add cmd tests * cleanup optimism/mod * Rename command line args * Fix test arguments * Rename config field to max_channel_bank_size * Enforce decompression limit of MAX_RLP_BYTES_PER_CHANNEL * Use constant OPTIMISM_DEPOSITED_TX_TYPE when checking batch validity * Import from std instead of alloc/core * Re-enable core::mem::forget() optimization * Replace asserts with ensures; enforce absence of receipts for Op blocks * Simplfy iteration through derived transactions * Add Bonsai support to op-derive. Also add Bonsai session status to output when polling * More println * Rework Batcher initialization * More logging * More log output if Bonsai workflow fails * refactor rpc db * variable derive step support * re-enable profiling * ignore rpc_cache dir * add -profile support to compose binary * upgrade zkvm, modify code comments * bump zkvm, fix CI * clippies * fixes and changes * vs code change * disambiguate merkle ranges and proofs Co-authored-by: Wolfgang Welz <[email protected]> * update risc0 to release v0.20 * update GH action to 0.20 * use old actions * changes * refactor: Unified zeth utility (#72) * feat: Provably build derived op blocks (#78) * feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80) * cleanup toml files * remove unused imports * cli cleanup * change block count to u32 * test composition * fix arguments * update risc0 to 0.20.1 * compact json files * update lock files * improve execution * test for warnings * nits * docker in readme * create cache file if not found * update copyright * update copyright --------- Co-authored-by: Rami Khalil <[email protected]> Co-authored-by: Timothy Carstens <[email protected]> Co-authored-by: Rami <[email protected]>

hashcashier added 4 commits January 8, 2024 19:02

initial monolith

61e5847

refactor prompt style

fa9c279

op-info support

5425737

delete old host binaries

c8dad6b

hashcashier self-assigned this Jan 10, 2024

hashcashier added 6 commits January 10, 2024 15:13

nits

be6f732

fix tests

9af3aa8

help nit

451de67

short params

786be8a

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

74e4c64

…il/tidy-host

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

451de61

…il/tidy-host

hashcashier changed the title ~~Unify zeth host binary~~ Unified zeth utility Jan 12, 2024

hashcashier added 8 commits January 12, 2024 14:47

save receipts

08eadac

refactor

abbc3c5

move clap structs to cli module

a926380

refactor

3a72eae

refactor main monolith

bdf9521

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

df5b43b

…il/tidy-host

default cache folder nit

0b6a455

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

b1a1436

…il/tidy-host

hashcashier marked this pull request as ready for review January 22, 2024 08:48

hashcashier requested review from intoverflow and Wollac January 22, 2024 08:48

hashcashier changed the title ~~Unified zeth utility~~ refactor: Unified zeth utility Jan 22, 2024

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

d5b4b1e

…il/tidy-host

Wollac approved these changes Jan 25, 2024

View reviewed changes

hashcashier and others added 3 commits January 30, 2024 16:16

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

707be50

…il/tidy-host

Merge branch 'rkhalil/op-compose' of github.com:risc0/zeth into rkhal…

07dc903

…il/tidy-host

feat: Provably build derived op blocks (#78)

cf4be86

* feat: Composition via Bonsai (#79) * feat: Deterministic builds with receipt cache (+misc) (#80)

hashcashier merged commit d2ab2f6 into rkhalil/op-compose Feb 5, 2024
3 checks passed

hashcashier deleted the rkhalil/tidy-host branch February 5, 2024 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Unified zeth utility #72

refactor: Unified zeth utility #72

hashcashier commented Jan 10, 2024 •

edited

Loading

Wollac left a comment

refactor: Unified zeth utility #72

refactor: Unified zeth utility #72

Conversation

hashcashier commented Jan 10, 2024 • edited Loading

Wollac left a comment

Choose a reason for hiding this comment

hashcashier commented Jan 10, 2024 •

edited

Loading