feat: New prover documentation (#2466)

New prover documentation.

[Rendered
form](https://github.com/matter-labs/zksync-era/tree/popzxc-new-prover-docs/prover/docs/README.md)
is recommended, as it's easier to view.

Additionally, does the following:

- Introduces ordering for advanced guides (it was a mix, not it's more
or less sorted and suggests an order for reading).
- Fixes a few things in the `setup-dev.md`.
- Creates a `prover-local` config profile, so that no manual
interactions with configs are needed.
- Gets rid of `prover/setup.sh` as it's no longer needed that much.

.gitignore

@@ -46,6 +46,7 @@ Cargo.lock
 !/etc/env/configs/stage-proofs.toml
 !/etc/env/configs/testnet.toml
 !/etc/env/configs/mainnet.toml
+!/etc/env/configs/prover-local.toml
 /etc/env/l1-inits
 !/etc/env/l1-inits/stage.env
 !/etc/env/l1-inits/stage_proofs.env

docs/guides/advanced/0_alternative_vm_intro.md → docs/guides/advanced/12_alternative_vm_intro.md

@@ -2,8 +2,6 @@
 
 ## zkEVM clarifier
 
-[Back to ToC](../../specs/README.md)
-
 The ZKsync zkEVM plays a fundamentally different role in the zkStack than the EVM does in Ethereum. The EVM is used to
 execute code in Ethereum's state transition function. This STF needs a client to implement and run it. Ethereum has a
 multi-client philosophy, there are multiple clients, and they are written in Go, Rust, and other traditional programming

docs/guides/advanced/deeper_overview.md → docs/guides/advanced/14_zk_deeper_overview.md

@@ -1,6 +1,4 @@
-# Deeper Overview
-
-[Back to ToC](../../../README.md)
+# Proof System Deeper Overview
 
 The purpose of this section is to explain our new proof system from an engineering standpoint. We will examine the code
 examples and how the libraries communicate.

docs/guides/advanced/README.md

@@ -0,0 +1,28 @@
+# ZKsync advanced guides
+
+This section contains more advanced guides that aim to explain complex internals of ZKsync ecosystem in an easy to grasp
+way.
+
+## Table of Contents
+
+- [Local initialization](./01_initialization.md)
+- [Deposits](./02_deposits.md)
+- [Withdrawals](./03_withdrawals.md)
+- [Contracts](./04_contracts.md)
+- [Calls](./05_how_call_works.md)
+- [Transactions](./06_how_transaction_works.md)
+- [Fee model](./07_fee_model.md)
+- [L2 messaging](./08_how_l2_messaging_works.md)
+- [Pubdata](./09_pubdata.md)
+- [Pubdata with blobs](./10_pubdata_with_blobs.md)
+- [Bytecode compression](./11_compression.md)
+- [EraVM intro](./12_alternative_vm_intro.md)
+- [ZK intuition](./13_zk_intuition.md)
+- [ZK deeper overview](./14_zk_deeper_overview.md)
+- [Prover keys](./15_prover_keys.md)
+
+Additionally, there are a few articles that cover specific topics that may be useful for developers actively working on
+`zksync-era` repo.
+
+- [Advanced debugging](./90_advanced_debugging.md)
+- [Docker and CI](./91_docker_and_ci.md)

docs/guides/setup-dev.md

@@ -2,42 +2,67 @@
 
 ## TL;DR
 
-If you run on 'clean' Debian on GCP:
+This is a shorter version of setup guide to make it easier subsequent initializations. If it's the first time you're
+initializing the workspace, it's recommended that you read the whole guide below, as it provides more context and tips.
+
+If you run on 'clean' Ubuntu on GCP:
 
 ```bash
+# For VMs only! They don't have SSH keys, so we override SSH with HTTPS
+git config --global url."https://github.com/".insteadOf [email protected]:
+git config --global url."https://".insteadOf git://
+
 # Rust
 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 # NVM
 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash
 # All necessary stuff
-sudo apt-get install build-essential pkg-config cmake clang lldb lld libssl-dev postgresql
-# Docker
-sudo usermod -aG docker YOUR_USER
+sudo apt-get update
+sudo apt-get install build-essential pkg-config cmake clang lldb lld libssl-dev postgresql apt-transport-https ca-certificates curl software-properties-common
+# Install docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
+sudo apt install docker-ce
+sudo usermod -aG docker ${USER}
+
+# Stop default postgres (as we'll use the docker one)
+sudo systemctl stop postgresql
+sudo systemctl disable postgresql
+# Start docker.
+sudo systemctl start docker
 
 ## You might need to re-connect (due to usermod change).
 
 # Node & yarn
 nvm install 20
+# Important: there will be a note in the output to load
+# new paths in your local session, either run it or reload the terminal.
 npm install -g yarn
 yarn set version 1.22.19
 
+# For running unit tests
+cargo install cargo-nextest
 # SQL tools
 cargo install sqlx-cli --version 0.7.4
-# Stop default postgres (as we'll use the docker one)
-sudo systemctl stop postgresql
-# Start docker.
-sudo systemctl start docker
 
 # Foundry
 curl -L https://foundry.paradigm.xyz | bash
 foundryup --branch master
+# You will need to reload your `*rc` file here
+
+# Clone the repo to the desired location
+git clone [email protected]:matter-labs/zksync-era.git
+cd zksync-era
+git submodule update --init --recursive
 ```
 
+Don't forget to [add env variables](#Environment) and look at [tips](#tips).
+
 ## Supported operating systems
 
 ZKsync currently can be launched on any \*nix operating system (e.g. any linux distribution or MacOS).
 
-If you're using Windows, then make sure to use WSL 2, since WSL 1 is known to cause troubles.
+If you're using Windows, then make sure to use WSL 2.
 
 Additionally, if you are going to use WSL 2, make sure that your project is located in the _linux filesystem_, since
 accessing NTFS partitions from within WSL is very slow.
@@ -90,38 +115,9 @@ If logging out does not resolve the issue, restarting the computer should.
 
 ## `Node` & `Yarn`
 
-1. Install `Node` (requires version `v18.18.0`). Since our team attempts to always use the latest LTS version of
-   `Node.js`, we suggest you to install [nvm](https://github.com/nvm-sh/nvm). It will allow you to update `Node.js`
-   version easily in the future (by running `nvm use v18.18.0` in the root of the repository)
-2. Install `yarn` (make sure to get version 1.22.19 - you can change the version by running `yarn set version 1.22.19`).
-   Instructions can be found on the [official site](https://classic.yarnpkg.com/en/docs/install/). Check if `yarn` is
-   installed by running `yarn -v`. If you face any problems when installing `yarn`, it might be the case that your
-   package manager installed the wrong package.Make sure to thoroughly follow the instructions above on the official
-   website. It contains a lot of troubleshooting guides in it.
-
-## `Axel`
-
-Install `axel` for downloading keys:
-
-On mac:
-
-```bash
-brew install axel
-```
-
-On debian-based linux:
-
-```bash
-sudo apt-get install axel
-```
-
-Check the version of `axel` with the following command:
-
-```
-axel --version
-```
-
-Make sure the version is higher than `2.17.10`.
+1. Install `Node` (requires version `v20`). The recommended way is via [nvm](https://github.com/nvm-sh/nvm).
+2. Install `yarn`. Can be done via `npm install -g yarn`. Make sure to get version 1.22.19 - you can change the version
+   by running `yarn set version 1.22.19`.
 
 ## `clang`
 
@@ -221,33 +217,9 @@ SQLx is a Rust library we use to interact with Postgres, and its CLI is used to
 features of the library.
 
 ```bash
-cargo install --locked sqlx-cli --version 0.7.3
+cargo install --locked sqlx-cli --version 0.7.4
 ```
 
-## Solidity compiler `solc`
-
-Install the latest solidity compiler.
-
-On mac:
-
-```bash
-brew install solidity
-```
-
-On debian-based linux:
-
-```bash
-sudo add-apt-repository ppa:ethereum/ethereum
-sudo apt-get update
-sudo apt-get install solc
-```
-
-Alternatively, download a [precompiled version](https://github.com/ethereum/solc-bin) and add it to your PATH.
-
-## Python
-
-Most environments will have this preinstalled but if not, install Python.
-
 ## Easier method using `nix`
 
 Nix is a tool that can fetch _exactly_ the right dependencies specified via hashes. The current config is Linux-only but
@@ -274,11 +246,10 @@ Edit the lines below and add them to your shell profile file (e.g. `~/.bash_prof
 export ZKSYNC_HOME=/path/to/zksync
 
 export PATH=$ZKSYNC_HOME/bin:$PATH
-
-# If you're like me, uncomment:
-# cd $ZKSYNC_HOME
 ```
 
+## Tips
+
 ### Tip: `mold`
 
 Optionally, you may want to optimize the build time with the modern linker, [`mold`](https://github.com/rui314/mold).

etc/env/base/fri_prover.toml

@@ -1,5 +1,5 @@
 [fri_prover]
-setup_data_path = "vk_setup_data_generator_server_fri/data"
+setup_data_path = "crates/bin/vk_setup_data_generator_server_fri/data"
 prometheus_port = 3315
 max_attempts = 10
 generation_timeout_in_secs = 600

etc/env/configs/prover-local.toml

@@ -0,0 +1,8 @@
+# Config for running prover locally
+__imports__ = [ "base", "l1-inits/.init.env", "l2-inits/dev.init.env" ]
+
+[eth_sender.sender]
+proof_sending_mode = "OnlyRealProofs"
+
+[fri_prover]
+cloud_type = "Local"

prover/README.md

@@ -2,36 +2,11 @@
 
 This directory contains all the libraries and binaries related to proving of the blocks.
 
-Directories with 'fri' suffix, are mostly used with the new proof system (Boojum).
-
-## Components
-
-### vk_setup_data_generator_server_fri
-
-Set of tools to create setup keys, verification keys and verification key hashes for the circuits.
-
-Usually run once, and then we use their outputs in multiple places in the system.
-
-### prover_fri_gateway
-
-Communication module between the 'main' server running the state keeper, and the proving subsystem.
-
-### witness_generator
-
-Creating prover jobs and saving necessary artifacts.
-
-### prover_fri
-
-This directory contains the main 'prover'. It can be run in two modes: either as CPU or as GPU. (controlled via 'gpu'
-feature flag).
-
-### witness_vector_generator
-
-Only used in GPU proving mode. Prepares all the witness data using CPU, and then streams it to the prover_fri.
-
-This is mostly used for resource efficiency (as machines with GPUs are more expensive, it allows us to run many
-witness_vector_generators, that can 'share' as single gpu based prover_fri).
-
-### proof_fri_compressor
-
-Used as a 'last step' to compress/wrap the final FRI proof into a SNARK (to make L1 verification cheaper).
+## Documentation
+
+- [Intro](00_intro.md)
+- [Setting up a GCP VM](01_gcp_vm.md)
+- [Workspace setup](02_setup.md)
+- [Running prover subsystem](03_launch.md)
+- [Proof generation flow](04_flow.md)
+- [Further reading](99_further_reading.md)