Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to Sync from Block 0 Without Using Snapshots? #2869

Open
0x1un opened this issue Sep 13, 2024 · 1 comment
Open

How to Sync from Block 0 Without Using Snapshots? #2869

0x1un opened this issue Sep 13, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@0x1un
Copy link

0x1un commented Sep 13, 2024

I need to obtain trace data for blocks, and I understand that using snapshot mode for fast synchronization means that historical trace data is no longer available.

However, I have encountered the following issues, which have been troubling me for several days:

2024-09-13T08:26:17.393259Z  INFO zksync_external_node: Main node URL is: "https://zksync2-mainnet.zksync.io/"
2024-09-13T08:26:35.030247Z  INFO zksync_external_node::config: Diamond proxy address is not specified in config; will use address returned by main node: 0x32400084c286cf3e17e7b677ea9583e60a000324
2024-09-13T08:26:35.030432Z  INFO zksync_external_node::node_builder: Pruning is disabled
2024-09-13T08:26:35.031205Z  INFO zksync_utils::env: locate_workspace() failed. You are using an already compiled version
2024-09-13T08:26:35.137405Z  INFO zksync_node_framework::service::context: Layer sigint_handler_layer has added a new task: sigint_handler
2024-09-13T08:26:35.137467Z  INFO zksync_node_framework::service::context: Layer healthcheck_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource, but it is not available
2024-09-13T08:26:35.137525Z  INFO zksync_node_framework::service::context: Layer healthcheck_layer has created a new resource common/app_health_check
2024-09-13T08:26:35.137541Z  INFO zksync_node_framework::service::context: Layer healthcheck_layer has added a new task: healthcheck_server
2024-09-13T08:26:35.137553Z  INFO zksync_node_framework::service::context: Layer prometheus_exporter has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.137586Z  INFO zksync_node_framework::service::context: Layer prometheus_exporter has added a new task: prometheus_exporter
2024-09-13T08:26:35.137629Z  INFO zksync_node_framework::service::context: Layer pools_layer has provided a new resource common/master_pool
2024-09-13T08:26:35.137655Z  INFO zksync_node_framework::service::context: Layer pools_layer has provided a new resource common/replica_pool
2024-09-13T08:26:35.137669Z  INFO zksync_node_framework::service::context: Layer main_node_client_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.137857Z  INFO zksync_node_framework::service::context: Layer main_node_client_layer has provided a new resource external_node/main_node_client
2024-09-13T08:26:35.137898Z  INFO zksync_node_framework::service::context: Layer query_eth_client_layer has provided a new resource common/eth_interface
2024-09-13T08:26:35.137920Z  INFO zksync_node_framework::service::context: Layer reorg_detector_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.137930Z  INFO zksync_node_framework::service::context: Layer reorg_detector_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.137939Z  INFO zksync_node_framework::service::context: Layer reorg_detector_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.148299Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 10, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.148364Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 10
2024-09-13T08:26:35.148406Z  INFO zksync_node_framework::service::context: Layer reorg_detector_layer has added a new task: reorg_detector
2024-09-13T08:26:35.148424Z  INFO zksync_node_framework::service::context: Layer postgres_metrics_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.153098Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.153126Z  INFO zksync_node_framework::implementations::resources::pools: Created a new replica pool. Total connections count: 1
2024-09-13T08:26:35.153146Z  INFO zksync_node_framework::service::context: Layer postgres_metrics_layer has added a new task: postgres_metrics_scraping
2024-09-13T08:26:35.153165Z  INFO zksync_node_framework::service::context: Layer external_node_metrics has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.153205Z  INFO zksync_shared_metrics::rustc: Metadata for rustc that this binary was compiled with: RustcMetadata { version: "1.82.0-nightly", commit_hash: Some("28a58f2fa7f0c46b8fab8237c02471a915924fe5"), commit_date: Some("2024-07-31"), channel: "nightly", host: "x86_64-unknown-linux-gnu", llvm: Some("19.1") }
2024-09-13T08:26:35.153228Z  INFO zksync_external_node::metrics: Setting general node information: ExternalNodeInfo { server_version: "24.23.0", l1_chain_id: 1, sl_chain_id: 1, l2_chain_id: 324, postgres_pool_size: 10 }
2024-09-13T08:26:35.157756Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.157807Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 11
2024-09-13T08:26:35.157824Z  INFO zksync_node_framework::service::context: Layer external_node_metrics has added a new task: en_protocol_version_metrics
2024-09-13T08:26:35.157840Z  INFO zksync_node_framework::service::context: Layer block_reverter_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.157853Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.157892Z  INFO zksync_node_framework::service::context: Layer block_reverter_layer has provided a new resource common/block_reverter
2024-09-13T08:26:35.157904Z  INFO zksync_node_framework::service::context: Layer external_node_role_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.157911Z  INFO zksync_node_framework::service::context: Layer external_node_role_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.157924Z  INFO zksync_node_framework::service::context: Layer external_node_role_layer has requested resource common/block_reverter of type zksync_node_framework::implementations::resources::reverter::BlockReverterResource
2024-09-13T08:26:35.157937Z  INFO zksync_node_framework::service::context: Layer external_node_role_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.157949Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.157960Z  INFO zksync_node_framework::resource::unique: Resource zksync_node_framework::resource::unique::Unique<zksync_block_reverter::BlockReverter> has been taken
2024-09-13T08:26:35.157986Z  INFO zksync_node_framework::service::context: Layer external_node_role_layer has provided a new resource node_initialization_strategy
2024-09-13T08:26:35.158001Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.158009Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_layer has requested resource node_initialization_strategy of type zksync_node_framework::implementations::layers::node_storage_init::NodeInitializationStrategyResource
2024-09-13T08:26:35.158035Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.158051Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_layer has added a new task: node_storage_initializer
2024-09-13T08:26:35.158060Z  INFO zksync_node_framework::service::context: Layer l1_batch_commitment_mode_validation_layer has requested resource common/eth_interface of type zksync_node_framework::implementations::resources::eth_interface::EthInterfaceResource
2024-09-13T08:26:35.158084Z  INFO zksync_node_framework::service::context: Layer l1_batch_commitment_mode_validation_layer has added a new task: l1_batch_commitment_mode_validation
2024-09-13T08:26:35.158096Z  INFO zksync_node_framework::service::context: Layer validate_chain_ids_layer has requested resource common/eth_interface of type zksync_node_framework::implementations::resources::eth_interface::EthInterfaceResource
2024-09-13T08:26:35.158104Z  INFO zksync_node_framework::service::context: Layer validate_chain_ids_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.158123Z  INFO zksync_node_framework::service::context: Layer validate_chain_ids_layer has added a new task: validate_chain_ids
2024-09-13T08:26:35.158131Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_precondition_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.158137Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_precondition_layer has requested resource node_initialization_strategy of type zksync_node_framework::implementations::layers::node_storage_init::NodeInitializationStrategyResource
2024-09-13T08:26:35.158148Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.158155Z  INFO zksync_node_framework::service::context: Layer node_storage_initializer_precondition_layer has added a new task: node_storage_initializer_precondition
2024-09-13T08:26:35.158166Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.158173Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.158181Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has requested resource common/object_store of type zksync_node_framework::implementations::resources::object_store::ObjectStoreResource, but it is not available
2024-09-13T08:26:35.158191Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.158203Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.162498Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 10, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.162522Z  INFO zksync_node_framework::implementations::resources::pools: Created a new replica pool. Total connections count: 11
2024-09-13T08:26:35.162672Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has added a new task: metadata_calculator
2024-09-13T08:26:35.162688Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has provided a new resource api/tree_api_client
2024-09-13T08:26:35.162699Z  INFO zksync_node_framework::service::context: Layer metadata_calculator_layer has added a new shutdown hook: rocksdb_terminaton
2024-09-13T08:26:35.162708Z  INFO zksync_node_framework::service::context: Layer external_io_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.162716Z  INFO zksync_node_framework::service::context: Layer external_io_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.162748Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.162782Z  INFO zksync_node_framework::service::context: Layer external_io_layer has provided a new resource common/sync_state
2024-09-13T08:26:35.162793Z  INFO zksync_node_framework::service::context: Layer external_io_layer has provided a new resource external_node/action_queue_sender
2024-09-13T08:26:35.162803Z  INFO zksync_node_framework::service::context: Layer external_io_layer has provided a new resource state_keeper/io
2024-09-13T08:26:35.162816Z  INFO zksync_node_framework::service::context: Layer external_io_layer has provided a new resource state_keeper/conditional_sealer
2024-09-13T08:26:35.162825Z  INFO zksync_node_framework::service::context: Layer state_keeper_output_handler_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.162838Z  INFO zksync_node_framework::service::context: Layer state_keeper_output_handler_layer has requested resource common/sync_state of type zksync_node_framework::implementations::resources::sync_state::SyncStateResource
2024-09-13T08:26:35.167083Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 6, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.167113Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 17
2024-09-13T08:26:35.167173Z  INFO zksync_node_framework::service::context: Layer state_keeper_output_handler_layer has provided a new resource state_keeper/output_handler
2024-09-13T08:26:35.167186Z  INFO zksync_node_framework::service::context: Layer state_keeper_output_handler_layer has added a new task: state_keeper/l2_block_sealer
2024-09-13T08:26:35.167222Z  INFO zksync_node_framework::service::context: Layer main_batch_executor_layer has provided a new resource state_keeper/batch_executor
2024-09-13T08:26:35.167234Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has requested resource state_keeper/io of type zksync_node_framework::implementations::resources::state_keeper::StateKeeperIOResource
2024-09-13T08:26:35.167243Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has requested resource state_keeper/batch_executor of type zksync_node_framework::implementations::resources::state_keeper::BatchExecutorResource
2024-09-13T08:26:35.167256Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has requested resource state_keeper/output_handler of type zksync_node_framework::implementations::resources::state_keeper::OutputHandlerResource
2024-09-13T08:26:35.167265Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has requested resource state_keeper/conditional_sealer of type zksync_node_framework::implementations::resources::state_keeper::ConditionalSealerResource
2024-09-13T08:26:35.167279Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.167291Z  INFO zksync_node_framework::resource::unique: Resource zksync_node_framework::resource::unique::Unique<alloc::boxed::Box<dyn zksync_state_keeper::io::StateKeeperIO>> has been taken
2024-09-13T08:26:35.167304Z  INFO zksync_node_framework::resource::unique: Resource zksync_node_framework::resource::unique::Unique<alloc::boxed::Box<dyn zksync_state_keeper::batch_executor::BatchExecutor<zksync_state::storage_factory::PgOrRocksdbStorage>>> has been taken
2024-09-13T08:26:35.167311Z  INFO zksync_node_framework::resource::unique: Resource zksync_node_framework::resource::unique::Unique<zksync_state_keeper::io::output_handler::OutputHandler> has been taken
2024-09-13T08:26:35.171699Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 2, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.171728Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 19
2024-09-13T08:26:35.171755Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has added a new task: state_keeper
2024-09-13T08:26:35.171792Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has added a new task: state_keeper/rocksdb_catchup_task
2024-09-13T08:26:35.171804Z  INFO zksync_node_framework::service::context: Layer state_keeper_layer has added a new shutdown hook: rocksdb_terminaton
2024-09-13T08:26:35.171822Z  INFO zksync_node_framework::service::context: Layer external_node_consensus_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.171832Z  INFO zksync_node_framework::service::context: Layer external_node_consensus_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.171844Z  INFO zksync_node_framework::service::context: Layer external_node_consensus_layer has requested resource common/sync_state of type zksync_node_framework::implementations::resources::sync_state::SyncStateResource
2024-09-13T08:26:35.171853Z  INFO zksync_node_framework::service::context: Layer external_node_consensus_layer has requested resource external_node/action_queue_sender of type zksync_node_framework::implementations::resources::action_queue::ActionQueueSenderResource
2024-09-13T08:26:35.171875Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.171885Z  INFO zksync_node_framework::resource::unique: Resource zksync_node_framework::resource::unique::Unique<zksync_node_sync::sync_action::ActionQueueSender> has been taken
2024-09-13T08:26:35.171896Z  INFO zksync_node_framework::service::context: Layer external_node_consensus_layer has added a new task: consensus_fetcher
2024-09-13T08:26:35.171906Z  INFO zksync_node_framework::service::context: Layer consistency_checker_layer has requested resource common/eth_interface of type zksync_node_framework::implementations::resources::eth_interface::EthInterfaceResource
2024-09-13T08:26:35.171918Z  INFO zksync_node_framework::service::context: Layer consistency_checker_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.171927Z  INFO zksync_node_framework::service::context: Layer consistency_checker_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.176449Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.176485Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 20
2024-09-13T08:26:35.179043Z  INFO zksync_node_framework::service::context: Layer consistency_checker_layer has added a new task: consistency_checker
2024-09-13T08:26:35.179082Z  INFO zksync_node_framework::service::context: Layer commitment_generator_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.179096Z  INFO zksync_node_framework::service::context: Layer commitment_generator_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.185508Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 16, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.185564Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 36
2024-09-13T08:26:35.185621Z  INFO zksync_node_framework::service::context: Layer commitment_generator_layer has added a new task: commitment_generator
2024-09-13T08:26:35.185648Z  INFO zksync_node_framework::service::context: Layer batch_status_updater_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.185665Z  INFO zksync_node_framework::service::context: Layer batch_status_updater_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.185687Z  INFO zksync_node_framework::service::context: Layer batch_status_updater_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.185709Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing master unbound pool
2024-09-13T08:26:35.185795Z  INFO zksync_node_framework::service::context: Layer batch_status_updater_layer has added a new task: batch_status_updater
2024-09-13T08:26:35.185814Z  INFO zksync_node_framework::service::context: Layer logs_bloom_backfill_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.190922Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.190973Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 37
2024-09-13T08:26:35.190993Z  INFO zksync_node_framework::service::context: Layer logs_bloom_backfill_layer has added a new task: logs_bloom_backfill
2024-09-13T08:26:35.191013Z  INFO zksync_node_framework::service::context: Layer sync_state_updater_layer has requested resource common/sync_state of type zksync_node_framework::implementations::resources::sync_state::SyncStateResource
2024-09-13T08:26:35.191031Z  INFO zksync_node_framework::service::context: Layer sync_state_updater_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.191046Z  INFO zksync_node_framework::service::context: Layer sync_state_updater_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.191090Z  INFO zksync_node_framework::implementations::layers::sync_state_updater: SyncState was provided by another layer, skipping SyncStateUpdaterLayer
2024-09-13T08:26:35.191175Z  INFO zksync_node_framework::service::context: Layer mempool_cache_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.196096Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 10, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.196154Z  INFO zksync_node_framework::implementations::resources::pools: Created a new replica pool. Total connections count: 21
2024-09-13T08:26:35.196193Z  INFO zksync_state::cache::sequential_cache: Configured sequential cache `mempool` with capacity 10000 items
2024-09-13T08:26:35.196385Z  INFO zksync_node_framework::service::context: Layer mempool_cache_layer has provided a new resource api/mempool_cache
2024-09-13T08:26:35.196413Z  INFO zksync_node_framework::service::context: Layer mempool_cache_layer has added a new task: mempool_cache_update_task
2024-09-13T08:26:35.196442Z  INFO zksync_node_framework::service::context: Layer tree_api_client_layer has requested resource api/tree_api_client of type zksync_node_framework::implementations::resources::web3_api::TreeApiClientResource
2024-09-13T08:26:35.196467Z  INFO zksync_node_framework::service::context: Layer tree_api_client_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.196494Z  INFO zksync_node_framework::implementations::layers::web3_api::tree_api_client: Tree API client is already provided
2024-09-13T08:26:35.196526Z  INFO zksync_node_framework::service::context: Layer main_node_fee_params_fetcher_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.196571Z  INFO zksync_node_framework::service::context: Layer main_node_fee_params_fetcher_layer has provided a new resource common/sequencer_fee_input
2024-09-13T08:26:35.196587Z  INFO zksync_node_framework::service::context: Layer main_node_fee_params_fetcher_layer has provided a new resource common/api_fee_input
2024-09-13T08:26:35.196601Z  INFO zksync_node_framework::service::context: Layer main_node_fee_params_fetcher_layer has added a new task: main_node_fee_params_fetcher
2024-09-13T08:26:35.196617Z  INFO zksync_node_framework::service::context: Layer proxy_sink_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.196637Z  INFO zksync_node_framework::service::context: Layer proxy_sink_layer has requested resource common/master_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::MasterPool>
2024-09-13T08:26:35.201550Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 1, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.201599Z  INFO zksync_node_framework::implementations::resources::pools: Created a new master pool. Total connections count: 38
2024-09-13T08:26:35.201633Z  INFO zksync_node_framework::service::context: Layer proxy_sink_layer has provided a new resource api/tx_sink
2024-09-13T08:26:35.201649Z  INFO zksync_node_framework::service::context: Layer proxy_sink_layer has added a new task: account_nonce_sweeper_task
2024-09-13T08:26:35.201676Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has requested resource api/tx_sink of type zksync_node_framework::implementations::resources::web3_api::TxSinkResource
2024-09-13T08:26:35.201694Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.201719Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has requested resource common/api_fee_input of type zksync_node_framework::implementations::resources::fee_input::ApiFeeInputResource
2024-09-13T08:26:35.201735Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has requested resource external_node/main_node_client of type zksync_node_framework::implementations::resources::main_node_client::MainNodeClientResource
2024-09-13T08:26:35.201755Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has requested resource state_keeper/conditional_sealer of type zksync_node_framework::implementations::resources::state_keeper::ConditionalSealerResource
2024-09-13T08:26:35.201870Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing replica unbound pool
2024-09-13T08:26:35.201959Z DEBUG zksync_state::postgres: Initialized VM execution cache with 134217728B capacity for factory deps, 33554432B capacity for initial writes
2024-09-13T08:26:35.202013Z  INFO zksync_state::cache::lru_cache: Configured LRU cache `factory_deps_cache` with capacity 134217728B
2024-09-13T08:26:35.203077Z  INFO zksync_state::cache::lru_cache: Configured LRU cache `initial_writes_cache` with capacity 16777216B
2024-09-13T08:26:35.203247Z  INFO zksync_state::cache::lru_cache: Configured LRU cache `negative_initial_writes_cache` with capacity 16777216B
2024-09-13T08:26:35.203360Z DEBUG zksync_state::postgres: Initializing VM storage values cache with 134217728B capacity
2024-09-13T08:26:35.203445Z  INFO zksync_state::cache::lru_cache: Configured LRU cache `values_cache` with capacity 134217728B
2024-09-13T08:26:35.203587Z  INFO zksync_node_api_server::execution_sandbox: Initializing the VM concurrency limiter with max concurrency 2048
2024-09-13T08:26:35.203661Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has provided a new resource api/tx_sender
2024-09-13T08:26:35.203684Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has added a new task: vm_concurrency_barrier_task
2024-09-13T08:26:35.203700Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has added a new task: postgres_storage_caches
2024-09-13T08:26:35.203714Z  INFO zksync_node_framework::service::context: Layer tx_sender_layer has added a new task: whitelisted_tokens_for_aa_update_task
2024-09-13T08:26:35.203741Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.203795Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource api/tx_sender of type zksync_node_framework::implementations::resources::web3_api::TxSenderResource
2024-09-13T08:26:35.203812Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource common/sync_state of type zksync_node_framework::implementations::resources::sync_state::SyncStateResource
2024-09-13T08:26:35.203832Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource api/tree_api_client of type zksync_node_framework::implementations::resources::web3_api::TreeApiClientResource
2024-09-13T08:26:35.203857Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource api/mempool_cache of type zksync_node_framework::implementations::resources::web3_api::MempoolCacheResource
2024-09-13T08:26:35.203888Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource common/circuit_breakers of type zksync_node_framework::implementations::resources::circuit_breakers::CircuitBreakersResource, but it is not available
2024-09-13T08:26:35.203913Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has created a new resource common/circuit_breakers
2024-09-13T08:26:35.203928Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.208924Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 2, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.208974Z  INFO zksync_node_framework::implementations::resources::pools: Created a new replica pool. Total connections count: 23
2024-09-13T08:26:35.208992Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing replica unbound pool
2024-09-13T08:26:35.209064Z  INFO zksync_node_api_server::web3: Using tree API client: LazyAsyncTreeReader(Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<core::option::Option<zksync_metadata_calculator::helpers::AsyncTreeReader>>>, RwLock { data: None }), version: Version(0), is_closed: false, ref_count_rx: 2 }, version: Version(0) })
2024-09-13T08:26:35.209234Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has added a new task: web3_http_server
2024-09-13T08:26:35.209251Z  INFO zksync_node_framework::service::context: Layer web3_http_server_layer has added a new task: api_task_garbage_collector
2024-09-13T08:26:35.209269Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource common/replica_pool of type zksync_node_framework::implementations::resources::pools::PoolResource<zksync_node_framework::implementations::resources::pools::ReplicaPool>
2024-09-13T08:26:35.209286Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource api/tx_sender of type zksync_node_framework::implementations::resources::web3_api::TxSenderResource
2024-09-13T08:26:35.209301Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource common/sync_state of type zksync_node_framework::implementations::resources::sync_state::SyncStateResource
2024-09-13T08:26:35.209316Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource api/tree_api_client of type zksync_node_framework::implementations::resources::web3_api::TreeApiClientResource
2024-09-13T08:26:35.209332Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource api/mempool_cache of type zksync_node_framework::implementations::resources::web3_api::MempoolCacheResource
2024-09-13T08:26:35.209347Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource common/circuit_breakers of type zksync_node_framework::implementations::resources::circuit_breakers::CircuitBreakersResource
2024-09-13T08:26:35.209363Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has requested resource common/app_health_check of type zksync_node_framework::implementations::resources::healthcheck::AppHealthCheckResource
2024-09-13T08:26:35.214429Z  INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { database_url: "postgres://***:***@192.168.224.138:30967/zksync", max_size: 2, acquire_timeout: 30s, statement_timeout: None, db: "zksync_dal::Core" }
2024-09-13T08:26:35.214477Z  INFO zksync_node_framework::implementations::resources::pools: Created a new replica pool. Total connections count: 25
2024-09-13T08:26:35.214494Z  INFO zksync_node_framework::implementations::resources::pools: Provided a new copy of an existing replica unbound pool
2024-09-13T08:26:35.214514Z  INFO zksync_node_api_server::web3: Using tree API client: LazyAsyncTreeReader(Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<core::option::Option<zksync_metadata_calculator::helpers::AsyncTreeReader>>>, RwLock { data: None }), version: Version(0), is_closed: false, ref_count_rx: 2 }, version: Version(0) })
2024-09-13T08:26:35.214581Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has added a new task: web3_ws_server
2024-09-13T08:26:35.214597Z  INFO zksync_node_framework::service::context: Layer web3_ws_server_layer has added a new task: api_task_garbage_collector
2024-09-13T08:26:35.214758Z  INFO zksync_node_framework::service: Wiring complete
2024-09-13T08:26:35.215861Z  INFO NamedFuture{name=node_storage_initializer_precondition}: zksync_node_framework::implementations::layers::node_storage_init: Waiting for node storage to be initialized
2024-09-13T08:26:35.215888Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_framework::implementations::layers::node_storage_init: Starting the node storage initialization task
2024-09-13T08:26:35.224709Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_storage_init: Node has neither genesis L1 batch, nor snapshot recovery info
2024-09-13T08:26:35.224796Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_storage_init: Performing genesis initialization
2024-09-13T08:26:36.305526Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that L1 chain ID 1 is returned by main node client
2024-09-13T08:26:36.305579Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that L2 chain ID L2ChainId(324) is returned by main node client
2024-09-13T08:26:36.432286Z  INFO NamedFuture{name=l1_batch_commitment_mode_validation}: zksync_commitment_generator::validation_task: Checked that the configured L1 batch commitment mode (Rollup) matches the commitment mode used on L1 contract 0x32400084c286cf3e17e7b677ea9583e60a000324
2024-09-13T08:26:36.943460Z  INFO NamedFuture{name=validate_chain_ids}: zksync_node_sync::validate_chain_ids_task: Checked that L1 chain ID 1 is returned by Ethereum client
2024-09-13T08:26:54.774285Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_genesis: running regenesis
2024-09-13T08:26:54.824628Z  INFO NamedFuture{name=node_storage_initializer}: zksync_node_genesis: chain_schema_genesis is complete
2024-09-13T08:26:54.828201Z  INFO NamedFuture{name=node_storage_initializer}: zksync_merkle_tree::domain: Creating Merkle tree for genesis batch with 35  writes
2024-09-13T08:26:54.840290Z  INFO NamedFuture{name=node_storage_initializer}: zksync_merkle_tree::domain: Processed genesis batch; root hash is 0x79b3…ffe3, 35 leaves in total
2024-09-13T08:26:54.847944Z  INFO NamedFuture{name=node_storage_initializer}: zksync_db_connection::connection: Long-living connection for `en` created at core/node/node_storage_init/src/external_node/genesis.rs:24: 19.612134016s
2024-09-13T08:26:54.848392Z ERROR zksync_node_framework::service: Task oneshot_runner failed: Oneshot task node_storage_initializer failed
2024-09-13T08:26:54.848476Z  INFO zksync_node_framework::service: One of the task has exited, shutting down the node
2024-09-13T08:26:54.848744Z  INFO NamedFuture{name=node_storage_initializer_precondition}: zksync_node_framework::implementations::layers::node_storage_init: Node storage initialization precondition completed
2024-09-13T08:26:54.849053Z  INFO zksync_node_api_server::healthcheck: Stop signal received, healthcheck server is shutting down
2024-09-13T08:26:54.849014Z  INFO zksync_node_framework::implementations::layers::postgres_metrics: Stop signal received, Postgres metrics scraping is shutting down
2024-09-13T08:26:54.849366Z  INFO zksync_node_api_server::healthcheck: Healthcheck server shut down
2024-09-13T08:26:54.850476Z  INFO zksync_node_framework::service: Task sigint_handler finished
2024-09-13T08:26:54.850519Z  INFO zksync_node_framework::service: Task healthcheck_server finished
2024-09-13T08:26:54.850535Z  INFO zksync_node_framework::service: Task prometheus_exporter finished
2024-09-13T08:26:54.850548Z  INFO zksync_node_framework::service: Task reorg_detector finished
2024-09-13T08:26:54.850561Z  INFO zksync_node_framework::service: Task postgres_metrics_scraping finished
2024-09-13T08:26:54.850574Z  INFO zksync_node_framework::service: Task en_protocol_version_metrics finished
2024-09-13T08:26:54.850587Z  INFO zksync_node_framework::service: Task metadata_calculator finished
2024-09-13T08:26:54.850600Z  INFO zksync_node_framework::service: Task state_keeper/l2_block_sealer finished
2024-09-13T08:26:54.850613Z  INFO zksync_node_framework::service: Task state_keeper finished
2024-09-13T08:26:54.850626Z  INFO zksync_node_framework::service: Task consensus_fetcher finished
2024-09-13T08:26:54.850639Z  INFO zksync_node_framework::service: Task consistency_checker finished
2024-09-13T08:26:54.850652Z  INFO zksync_node_framework::service: Task commitment_generator finished
2024-09-13T08:26:54.850665Z  INFO zksync_node_framework::service: Task batch_status_updater finished
2024-09-13T08:26:54.850678Z  INFO zksync_node_framework::service: Task mempool_cache_update_task finished
2024-09-13T08:26:54.850691Z  INFO zksync_node_framework::service: Task main_node_fee_params_fetcher finished
2024-09-13T08:26:54.850704Z  INFO zksync_node_framework::service: Task account_nonce_sweeper_task finished
2024-09-13T08:26:54.850717Z  INFO zksync_node_framework::service: Task vm_concurrency_barrier_task finished
2024-09-13T08:26:54.850730Z  INFO zksync_node_framework::service: Task postgres_storage_caches finished
2024-09-13T08:26:54.850743Z  INFO zksync_node_framework::service: Task whitelisted_tokens_for_aa_update_task finished
2024-09-13T08:26:54.850756Z  INFO zksync_node_framework::service: Task web3_http_server finished
2024-09-13T08:26:54.850808Z  INFO zksync_node_framework::service: Task api_task_garbage_collector finished
2024-09-13T08:26:54.850826Z  INFO zksync_node_framework::service: Task web3_ws_server finished
2024-09-13T08:26:54.850840Z  INFO zksync_node_framework::service: Task api_task_garbage_collector finished
2024-09-13T08:26:54.851218Z  INFO zksync_storage::db: All the RocksDB instances are dropped
2024-09-13T08:26:54.851352Z  INFO zksync_node_framework::service: Shutdown hook rocksdb_terminaton completed
2024-09-13T08:26:54.851491Z  INFO zksync_storage::db: All the RocksDB instances are dropped
2024-09-13T08:26:54.851629Z  INFO zksync_node_framework::service: Shutdown hook rocksdb_terminaton completed
2024-09-13T08:26:54.851655Z  INFO zksync_node_framework::service: Exiting the service
Error: One or more tasks failed: [TaskFailed(TaskId("oneshot_runner"), Oneshot task node_storage_initializer failed
Caused by:
    0: performing genesis failed
    1: ensure_genesis_state
    2: Root hash mismatched: From config: 0xbc59c242d551e3939b9b2939b8b686efa77ba3833183045d548aa5f53357ba95, Calculated 0x79b383138554ae78daa0cd096ca63d7e69dc7e1868daf3c9a86d01918df1ffe3)]

Here are the parameters I’m using for the synchronization:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zksync-en-138
  namespace: fullnode
  labels:
    app: zksync-en-138
    component: core
spec:
  serviceName: zksync-en-138
  replicas: 1
  selector:
    matchLabels:
      app: zksync-en-138
      component: core
  template:
    metadata:
      labels:
        app: zksync-en-138
        component: core
    spec:
      terminationGracePeriodSeconds: 3000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - worker-17-192.168.224.138
              topologyKey: kubernetes.io/hostname
      priorityClassName: system-cluster-critical
      hostNetwork: false
      dnsPolicy: ClusterFirstWithHostNet
      imagePullSecrets:
        - name: harbor-auth

      containers:
        - name: en
          image: 'harbor.local/da/external-node:2.0-v24.23.0'
          env:
            - name: DATABASE_URL
              value: "postgres://postgres:[email protected]:30967/zksync"
            - name: DATABASE_POOL_SIZE
              value: "10"
            - name: EN_HTTP_PORT
              value: "3060"
            - name: EN_WS_PORT
              value: "3061"
            - name: EN_HEALTHCHECK_PORT
              value: "3081"
            - name: EN_PROMETHEUS_PORT
              value: "3322"
            - name: EN_ETH_CLIENT_URL
              value: "https://ethereum-rpc.publicnode.com"
            - name: EN_MAIN_NODE_URL
              value: "https://zksync2-mainnet.zksync.io"
            - name: EN_L1_CHAIN_ID
              value: "1"
            - name: EN_L2_CHAIN_ID
              value: "324"

            - name: EN_STATE_CACHE_PATH
              value: "./db/ext-node/state_keeper"
            - name: EN_MERKLE_TREE_PATH
              value: "./db/ext-node/lightweight"

            - name: EN_SNAPSHOTS_RECOVERY_ENABLED
              value: "false"
            - name: EN_SNAPSHOTS_OBJECT_STORE_BUCKET_BASE_URL
              value: "zksync-era-mainnet-external-node-snapshots"
            - name: EN_SNAPSHOTS_OBJECT_STORE_MODE
              value: "GCSAnonymousReadOnly"

            - name: EN_L1_BATCH_COMMIT_DATA_GENERATOR_MODE
              value: "Rollup"
            - name: RUST_LOG
              value: "warn,zksync=info,zksync_core::metadata_calculator=debug,zksync_state=debug,zksync_utils=debug,zksync_web3_decl::client=error"
            
            - name: EN_API_NAMESPACES
              value: "eth,net,web3,debug,zks,en,pubsub,snapshots,unstable"

            - name: EN_PRUNING_ENABLED
              value: "false"
            # - name: EN_PRUNING_DATA_RETENTION_SEC
            #   value: "5184000"

          ports:
            - containerPort: 3060
              name: http
            - containerPort: 3061
              name: ws
            - containerPort: 3081
              name: healthcheck
            - containerPort: 3322
              name: prometheus

          volumeMounts:
            - name: zksync-en-storage
              mountPath: /db

  volumeClaimTemplates:
    - metadata:
        name: zksync-en-storage
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: zksync-en-data-138
        resources:
          requests:
            storage: 4000Gi
@0x1un 0x1un added the bug Something isn't working label Sep 13, 2024
@0x1un
Copy link
Author

0x1un commented Sep 20, 2024

Nobody can answer this question?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@0x1un