-
Notifications
You must be signed in to change notification settings - Fork 857
Conversation
…nk (#1641) ### Description [_PR description_] ### Issue Link #1601 ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update ### Highlights - separate rw_table in evm_circuit/state_circuit and each have its own permutationchip. fingerprints checking will be deferred to root circuit. - disable rwtable first row check (by disable selector offset=0 in state_circuit, other offset toggle on as usual), since it will be copy constraints via public input. - keep `Rw::Start` and introduce `Rw::Padding` to support post-padding in address-sorted rwtable in last_chunk. - instance (new public input) column are design into two, one for prev_chunk context, another for current_chunk context to be used for next chunk. So in root circuit commitment of instance column can be used in snark-verifier without need to unwrap it. > Update on 18 Oct 2023: beginchunk/endchunk virtual step, inner_rw_counter, chunkctx - add Begin/EndChunk virtual step. BeginChunk is not mandatory in first chunk first step. while EndChunk is not mandatory in last chunk last step. - add `inner_rw_counter` which is local rw_counter within each chunk. This is used to count valid rw_row and assure Rw::Padding are append consecutively in `end_block.rs` logic => EndChunk should apply similar check later on - introduce chunkctx->{chunk_index, total_chunks} to tell first chunk (chunk_index==0) and last chunk (chunk_index + 1 == total_chunks) during witness generation/constraints assignment - add chunkctx_table to able to lookup chunk context (chunk_index, next_chunk_index, total_chunk, initial_rwc, end_rwc..etc) in exec step to allow various conditional check based on current chunk context ### How Has This Been Tested? [_explanation_] ### More context on Rw::Padding (Not cover in current PR, will be handled in later multiple chunk PR to make scope smaller) In new logic, `Rw::Start` will be insert in first chunk offset 0, while other holes are filled by `Rw::Padding` in last chunk(s). The address-sorted rwtable layout will be ``` address-sorted rwtable ## first chunk [ Rw::start, // offset 0, Rw::Start is only in first chunk, and only in offset 0, constrainted by public input ....(normal Rw), ...(Rw::Padding), // padding if there are only one chunk ] ## end chunk (if there are > 1 chunk) [ ....(normal Rw), // offset 0 are carry over from previous chunk, other are address sorted ...(Rw::Padding) // padding in end chunk ] ``` For chronologically rwtable, since there is no in-row constraints to check each Rw operation, so theoretically Rw::Padding rows can be filled in any offset. However, we also need to assure there is no malicious insertion other than Rw::Padding. To do that, we will rewrite this logic in later PR https://github.com/privacy-scaling-explorations/zkevm-circuits/blob/main/zkevm-circuits/src/evm_circuit/execution/end_block.rs#L86-L90 to lookup consecutive `Rw::Padding` at **chronologically** sorted table, at the END of EACH chunk. > A tricks: first Rw::Padding rw_counter need to be set as last (globally) valid row rw_counter + 1. This is to make sure in both chronologically rw_table or address-sorted rw_table it's always append in the end of rw_table. ``` chronologically rwtable, ascending sorted by `rw_counter`. ## first chunk [ Rw::start, // offset 0, Rw::Start is only in first chunk, constrainted by public input ...(normal Rw), ...(Rw::Padding), // first Rw::Padding rw_counter need to be set as last (globally) valid row rw_counter + 1, last means from last !!chunk!!. It assures Rw::Padding always append at the end of each chunk ] ## end chunk (if there are > 1 chunk) [ ....(normal Rw), // offset 0 are carry over from previous chunk, other are rw_counter sorted ...(Rw::Padding) // padding, also rw_counter sorted ] ```
### Description Depends on #1641 with extra commit: adding fingerprint equality check on chronological/by address rw_table Fingerprint check gate will be enable in last chunk last row ### instance columns, top down order match instance array order chunk ctx - [current chunk index, total chunk, initial rwc] // equal with chunk_{i-1} - [next chunk index, total chunk, next rwc] equal with chunk_{i+1} pi circuit - [pi digest lo, pi digest hi] // same across all chunks state circuit - [prev permutation fingerprint] // equal with chunk_{i-1} - [next permutation fingerprint] // equal with chunk_{i+1} - [alpha, gamma] // same across all chunks evm circuit - [prev permutation fingerprint] // equal with chunk_{i-1} - [next permutation fingerprint] // equal with chunk_{i+1} - [alpha, gamma] // same across all chunks ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update
### Description covered item - [x] supercircuit simplify instance columns related to chunkctx to just one column - [x] first chunk instance: chunk continuity related fields should be constraints as constant. - [x] aggregate multiple super circuit proof chunk with chunk consistency check . - [x] compute permutation fingerprint challenges `alpha,gamma` from rw_table advices commitments and assert its equality - [x] generalized `challenge([column_indexs], challenge_column_index)` to a new protocol structure so more challenges in the future can be added. TODO - [ ] verify multiple chunk from bus_mapping => rely on #1690 ### Issue Link #1603 ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update ### Tests Light tests pass. integration test failed due to challenge computation in witness not implemented yet, which will be done in bus mapping chunk PR.
### Description Instantiate circuits with `Block` and specified `Chunk` from bus-mapping. Refactor `CircuitInputBuilder` to generate given number of `Chunk`s with fixed or dynamic params. Note that one instance of the circuit corresponds to one chunk instead of one block. <img width="606" alt="Screen Shot 2023-12-08 at 11 37 30 PM" src="https://github.com/privacy-scaling-explorations/zkevm-circuits/assets/45245961/84eac662-5adf-49d6-99bf-eefc295cf5aa"> ### Issue Link #1696 ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update ### Contents - Initialize the `CircuitInputBuilder` with either `FixedCParam` or just specify the total number of chunk and dynamically derive parameter after a dry run. - In `handle_tx` split the chunk whenever the local `rwc` exceeds the target amount, specifically by setting the `EndChunk` and proceed to the next `BeginChunk` if needed. - Commit the `ChunkCtx` for each chunk at the end, which tracks the global `Rw` range & local `rwc` later used in the State circuits. - After the block is handled, generate witness for a specific chunk with `builder.chunk_convert(idx)`, analogous to `builder.block_convert()`. - Config the circuit with both block and chunk, which leads to API changes in `SubCircuit<F: Field>` trait including `new_from_block(&block, &chunk)` and `min_num_rows_block(&block, &chunk)`. ### Workflow For `CircuitInputBuilder` chunk is needed to build **all** circuits: ``` let builder = BlockData::new_from_geth_data(geth_data.clone()) .new_circuit_input_builder() .handle_block(ð_block, &geth_traces) .expect("handle_block"); let block = block_convert(&builder); let chunk = chunk_convert(&builder, 0); let circuit = SuperCircuit::new_from_block(&block, &chunk); ``` For `CircuitTestBuilder` **modifier** is applied to both block and chunk, and you can run with fixed or dynamic params: ``` CircuitTestBuilder::new_from_test_ctx( TestContext::<2, 1>::simple_ctx_with_bytecode(bytecode).unwrap(), ) .modifier(Box::new(move |block, chunk| { // do something.. })) .run_dynamic_chunk(4, 2); ``` ``` CircuitTestBuilder::new_from_block(block) .params(FixedCParams { total_chunks: 2, max_rws: 60, ..Default::default() }) .run_chunk(1); ``` Default `run()` still works since it runs one chunk internally. ``` CircuitTestBuilder::new_from_block(block).run() ``` --------- Co-authored-by: sm.wu <[email protected]>
…test (#1773) ### Content Reported issues found on multi-chunk testing - [x] refactor 1: simply chunking boundary judgement logic in bus-mapping to ready for finish other incompleted features + avoid tech dept in the future - [x] add uncompleted logic in bus-mapping: chronological and by-address rwtable not propagate pre-chunk last rw correctly. - [x] edge case: deal with dummy chunk for real chunk less than desired chunk in circuit params - [x] allow zero limb diff in state_circuit lexicoordering => we allow duplicated `rw_counter` in `padding`, and rely on permutation constraints on by-address/chronological rw_table to avoid malicious padding insert. - [x] super_circuit/root_circuit tests adopt multiple chunk ### Related Issue To close #1778
Docs & materials
|
Permutation FingerprintsTo constrain 2 rw_table
are equals across multiple chunks, we constrain In the end, equality are constraints by gate in super_circuit zkevm-circuits/zkevm-circuits/src/super_circuit.rs Lines 250 to 270 in 0ab41c9
|
Continuity constraint on evm_circuit execution stateWe introduce new virtual steps
to carry on the execution state begin chunk. For An exception: begin_chunk/end_chunkwe can NOT read/write exec state
to rw_table because we rely rwc to assure rw lookup correctness. Therefore zkevm-circuits/zkevm-circuits/src/evm_circuit.rs Lines 390 to 391 in 2fb87b0
and in execution.rs circuit first/last row we lookup rwc from chunk context table There is one interesting property of zkevm-circuits/bus-mapping/src/circuit_input_builder.rs Lines 440 to 446 in 0ab41c9
|
Bus-mapping witness chunkingOverall it follows the original design of With zkevm-circuits/bus-mapping/src/circuit_input_builder.rs Lines 371 to 378 in 0ab41c9
we finalize current chunk and bump chunk_ctx to the next, until we reach the end of desired number of total_chunks. To support dynamic max_rws/max_evms calculation in unittest, there will be 2 pass going over whole block. First pass is for estimated max_rws/max_evms needed per chunk, and then second pass go over block again and finalized circuit_input_builder<FixedParams> .
New |
permutation fingerprint challengesPermutation challenge is specially designed as an application-level challenge out of native halo2. The reason is because permutation challenge are cross different ConstrainSystem which halo2 yet support. The challenge are computed in bus-mapping via get_rwtable_cols_commitment Giving rw_table column vector from multiple chunk, this function will derive respective KZG commitments for each column and then poseidon hash all of them for challenge. challenge will be feed to super circuit as public input. In aggregation circuit, user challenge value will be verified via poseidon circuit. |
chunk consistency check on aggregation circuitEach chunk will derive respective super circuit proof and in final aggregation circuit we layout all of them to prove
Above verifier logic are arithmetization into circuit constrain in root_circuit With those magic to hide all the detail from final decider, final snark public_input is just lo-hi hash without leak any chunking context and complexity. |
rw_table padding logicrw table will be padding to We will lookup consecutive padding in
|
What can be done after nextWith first move of proof chunk we achieve the milestone to be able to deal with proof parallism on block data level by concurrent provers, each deal with smaller k to get final chunk, and then aggregate all proof. An known limitation on root circuit, with current complex linearization(gate) functions, under k=26 in root circuit we can only verify 2 super circuit prove. To fully relax the power, we can design root circuit as recursive proof (issue #1631) so this part can be slightly eliminated.
|
relevant tests
|
circuit stats (before & after)chunkctx_tablenew added table
state circuitnew
evm circuitconstraints increase due to
super circuit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some initial questions and nitpicks
instance_i.sc_rwtable_row_next_fingerprint.assigned(), | ||
instance_i_plus_one | ||
.sc_rwtable_row_prev_fingerprint | ||
.assigned(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am wondering whether it's meant instance_i.sc_rwtable_row_next_fingerprint = instance_i_plus_one.sc_rwtable_row_curr_fingerprint.assigned
, so curr
instead of prev
in i_plus_one
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catch up this!
instance_i
latest accumuated value will propagate to instance_i+1
Fix by renamed it to i.curr
= {i+1}.prev
semantically and erase next
term to avoid confused
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First round
tx_ctx: TransactionContext, | ||
next_geth_step: Option<(usize, &GethExecStep)>, | ||
last_call: Option<Call>, | ||
) -> Result<bool, Error> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should comment on what this bool
mean.
It seems to me true
means we have the next chunk and false means we are at the last chunk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good idea, explanation was added in latest commit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM modulo Miha's feedback.
Epic work on completing the proof chunking!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tremendous work! I added just some nitpicks.
Description
This PR makes the first move toward single block chunking into multiple proof & aggregation
Issue Link
#1498
Type of change
Rationale
design rationale are list below as separated section