Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark feature #31

Merged
merged 20 commits into from
Jul 16, 2024
Merged

Benchmark feature #31

merged 20 commits into from
Jul 16, 2024

Conversation

JulianGCalderon
Copy link
Contributor

@JulianGCalderon JulianGCalderon commented Jul 10, 2024

Adds BenchBlockRange Command to Replay crate under the Benchmark features.

The command is used to measure the performance of Cairo Native without the overhead of rpc requests. Caching native compilation is not yet possible in the current blockifier version

Supersedes #12

Overview

The bench block range command has roughly three distinct steps:

  1. Fetch all block range data, this includes all transactions and block context information needed to execute them
  2. Executes the whole block range once, to make sure all internal data required by blockifier is cached
  3. Executes (and measures) the whole block range n times. This third step won't make any rpc call as all data has already been cached.

The benchmark command ended up being quite lengthy, so I left the high level command implementation in main.rs and put the inner details in benchmark.rs

execute_tx_with_blockifier

Internally, both execute_tx_configurable and execute_tx_configurable_with_state depended on our rpc state reader, bypassing the cached state. When executing transaction for benchmarking, I needed it to be indepent of the inner state reader: CachedState<impl StateReader>. To solve this I made a new function execute_tx_with_blockifier, that does not depend on our rpc state and is just a wrapper around blockifier execution. It receives all contextual information to execute the transaction.

execute_tx_configurable_with_state could be replaced by this new function. I didn't do it in this PR to avoid having to many changes.

OptionalStateReader

To ensure that no requests are made when benchmarking, an OptionalStateReader was used instead of our StateReader. This reader can be disabled so that it panics if it is attempted to read from when benchmarking

Benchmark Result in the Server

I benchmarked a single run of block 606173 in `starknet-benchmarks'. It contains 28 transactions.

  • total run time: 858.34s
image

@JulianGCalderon JulianGCalderon force-pushed the benchmark-feature branch 3 times, most recently from 03ac734 to 719b717 Compare July 11, 2024 17:23
@JulianGCalderon JulianGCalderon linked an issue Jul 11, 2024 that may be closed by this pull request
@JulianGCalderon JulianGCalderon marked this pull request as ready for review July 11, 2024 18:03
@edg-l edg-l merged commit 9034fba into main Jul 16, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add benchmark feature in replay crate
3 participants