diff --git a/SUMMARY.md b/SUMMARY.md index e9e2361a2b..a5ba61d2ce 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -96,8 +96,9 @@ - [Mine Testnet Stacks Tokens](guides-and-tutorials/run-a-miner/mine-testnet-stacks-tokens.md) - [Mine Mainnet Stacks Tokens](guides-and-tutorials/run-a-miner/mine-mainnet-stacks-tokens.md) - [Verify Miner](guides-and-tutorials/run-a-miner/verify-miner.md) -- [Run a Nakamoto Signer](guides-and-tutorials/running-a-signer/README.md) +- [Run a Signer](guides-and-tutorials/running-a-signer/README.md) - [How to Read Signer Logs](guides-and-tutorials/running-a-signer/how-to-read-signer-logs.md) + - [How to Monitor a Signer](guides-and-tutorials/running-a-signer/how-to-monitor-signer.md) - [Stack STX](guides-and-tutorials/stack-stx/README.md) - [Solo Stack](guides-and-tutorials/stack-stx/stacking-flow.md) - [Operate a Pool](guides-and-tutorials/stack-stx/operate-a-pool.md) diff --git a/guides-and-tutorials/running-a-signer/README.md b/guides-and-tutorials/running-a-signer/README.md index 1c48340715..f5c1a1a93c 100644 --- a/guides-and-tutorials/running-a-signer/README.md +++ b/guides-and-tutorials/running-a-signer/README.md @@ -1,9 +1,9 @@ # Run a Signer {% hint style="info" %} -The block for Nakamoto activation has been chosen as Bitcoin block 864,864, which is currently expected on October 9th. This block is subject to change should core developers need additional time for testing or unexpected issues. +The block for Nakamoto activation has been chosen as Bitcoin block 867,867, which is currently expected on October 29th. This block is subject to change should core developers need additional time for testing or unexpected issues. -Binaries will be provided roughly a week in advance and your normal upgrade procedure should apply here, you’ll want to be running the latest node and Signer software. Note that if you do not upgrade ahead of the hard fork, your nodes will be dropped from the network. Ideally, you will have Stacked for Cycle #94 ahead of time, any time now is good. +Binaries will be provided roughly a week in advance and your normal upgrade procedure should apply here, you’ll want to be running the latest node and Signer software. Note that if you do not upgrade ahead of the hard fork, your nodes will be dropped from the network. {% endhint %} {% hint style="info" %} diff --git a/guides-and-tutorials/running-a-signer/how-to-monitor-signer.md b/guides-and-tutorials/running-a-signer/how-to-monitor-signer.md new file mode 100644 index 0000000000..73cbde051f --- /dev/null +++ b/guides-and-tutorials/running-a-signer/how-to-monitor-signer.md @@ -0,0 +1,227 @@ +# How to Monitor a Signer + +We will use [Grafana Cloud](https://grafana.com) to observe and monitor both the +Signer and its corresponding Stacks node. + +## Requirements + +Grafana's application observability docs have a [great +quick-start](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/). We will use: + +- Grafana Cloud to collect metrics and visualize them. +- Grafana Alloy, on the Signer host, to push the metrics. + +### Creating a Grafana Cloud account + +Before we begin, create a [Grafana +Cloud](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/grafana-cloud/) account (they offer a free tier that you can use). + +Once done, access your dashboard and: + +1. Click on "Connections", then +2. "Add new connection", and +3. select "Hosted Prometheus metrics". +4. Now select "Via Grafana Alloy", then +5. on step 2, select "Run Grafana Alloy" to generate an API token. + +Note the token `GCLOUD_RW_API_KEY` and the parameters `GCLOUD_HOSTED_METRICS_URL` +and `GCLOUD_HOSTED_METRICS_ID`, we will use them later. + +### Configuring the Signer and the Stacks node + +Ensure both your Signer configuration and your node configuration include the +following lines: + +```toml +# signer-config.toml +# ... +metrics_endpoint = "127.0.0.1:30001" +``` + +```toml +# node-config.toml +# ... +prometheus_bind = "127.0.0.1:9153" +``` + +If you compile the application binaries yourself, remember to enable the Cargo +feature `monitoring_prom` while building them, for example: + +```bash +cargo build --features monitoring_prom,slog_json --release +``` + +Once both binaries are running with the updated configuration, you can peek +at the metrics being exposed: + +```bash +curl 127.0.0.1:30001/metrics + +# HELP stacks_signer_current_reward_cycle The current reward cycle +# TYPE stacks_signer_current_reward_cycle gauge +stacks_signer_current_reward_cycle 95 +# HELP stacks_signer_node_rpc_call_latencies_histogram Time (seconds) measuring round-trip RPC call latency to the Stacks node +# TYPE stacks_signer_node_rpc_call_latencies_histogram histogram +... +stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.005"} 0 +stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.01"} 0 +stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.025"} 0 +stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.05"} 985 +stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.1"} 1194 +... +``` + +Also, you'll have a `/info` endpoint on the same port: + +```bash +curl 127.0.0.1:30001/info +``` + +### Install Alloy + +Follow these instructions to install [Grafana +Alloy](https://grafana.com/docs/alloy/latest/set-up/install/linux/). + +On Debian-based distributions: + +```bash +sudo apt install gpg +sudo mkdir -p /etc/apt/keyrings/ +wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null +echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list +sudo apt-get update +sudo apt-get install alloy +``` + +### Configure Alloy + +Edit the file `/etc/alloy/config.alloy` as follows, by taking care of replacing +the placeholders related to the `prometheus` endpoint with the parameters +obtained when creating a [Grafana Cloud account before](#creating-a-grafana-cloud-account): + +- `GCLOUD_HOSTED_METRICS_URL` +- `GCLOUD_HOSTED_METRICS_ID` +- `GCLOUD_RW_API_KEY` + +```conf +// For a full configuration reference, see https://grafana.com/docs/alloy +logging { + level = "warn" +} + +prometheus.exporter.unix "default" { + include_exporter_metrics = true + disable_collectors = ["mdadm"] +} + +prometheus.scrape "default" { + targets = array.concat( + prometheus.exporter.unix.default.targets, + [ + { + // Self-collect metrics + job = "alloy", + __address__ = "127.0.0.1:12345", + }, + { + // stacks-signer + job = "stacks-signer", + __address__ = "127.0.0.1:30001", + }, + { + // stacks-node + job = "stacks-node", + __address__ = "127.0.0.1:9153", + }, + ], + ) + + forward_to = [prometheus.remote_write.default.receiver] +} + +prometheus.remote_write "default" { + external_labels = {"instance" = constants.hostname} + endpoint { + # TODO: Edit the URL below with your Grafana production URL. + # should end with /api/prom/push + url = "" + + # TODO: Edit with your Grafana Cloud ID and Token + basic_auth { + username = "" + password = "" + } + } +} +``` + +```bash +sudo systemctl daemon-reload +sudo systemctl enable alloy.service +sudo systemctl start alloy.service +``` + +Metrics from your Signer and node will now start being pushed to Grafana Cloud. + +## Visualizing the metrics + +You can now start building a dashboard to visualize the metrics. + +1. Log-in to Grafana Cloud and create a new Dashboard. +2. Pick the Prometheus instance you created before as the data source. +3. Create a new panel and pick `stacks_signer_current_reward_cycle` from the + metrics. + +You should now be able to see Stacks' current reward cycle, as measured by the +Signer, into the dashboard. + +Grafana comes with powerful data visualization tools. You can read about how to +query and transform data +[here](https://grafana.com/docs/grafana-cloud/visualizations/panels-visualizations/query-transform-data/), +while here you will find examples on how to build +[Prometheus queries](https://prometheus.io/docs/prometheus/latest/querying/basics/). + +## Bonus: monitoring the host + +Since we are here, we can also monitor the host itself. Debian-based +distributions make it very easy for us by using +`[node-exporter](https://github.com/prometheus/node_exporter/tree/master)`. + +```bash +sudo apt install prometheus-node-exporter +sudo systemctl enable prometheus-node-exporter +sudo systemctl start prometheus-node-exporter +``` + +This will expose metrics on port `9100` of `localhost`. + +We can now configure `alloy` to push them to Grafana. Edit your +`/etc/alloy/config.alloy` file and add the following: + +```txt +prometheus.scrape "default" { + targets = array.concat([ + + ... + + { + job = "node_exporter", + __address__ = "127.0.0.1:9100", + } + + ... +])} +``` + +Now reload `alloy` and check its status: + +```bash +sudo systemctl reload alloy +sudo systemctl status alloy +``` + +`node-exporter` provides a _lot_ of metrics. Poke at them through the Grafana +Explorer or use one of the many prepared dashboard (e.g., [this +one](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)) to see +comprehensive information. Once you have a dashboard ready, you can also +use it to configure a few alerts (e.g., on disk space, etc). diff --git a/nakamoto-upgrade/nakamoto-activation-guide-for-signers/README.md b/nakamoto-upgrade/nakamoto-activation-guide-for-signers/README.md index 34cdde81c3..555f92e0e2 100644 --- a/nakamoto-upgrade/nakamoto-activation-guide-for-signers/README.md +++ b/nakamoto-upgrade/nakamoto-activation-guide-for-signers/README.md @@ -1,9 +1,9 @@ # Nakamoto Activation Guide for Signers {% hint style="info" %} -The block for Nakamoto activation has been chosen as Bitcoin block 864,864, which is currently expected on October 9th. This block is subject to change should core developers need additional time for testing or unexpected issues. +The block for Nakamoto activation has been chosen as Bitcoin block 867,867, which is currently expected on October 29th. This block is subject to change should core developers need additional time for testing or unexpected issues. -Binaries will be provided roughly a week in advance and your normal upgrade procedure should apply here, you’ll want to be running the latest node and Signer software. Note that if you do not upgrade ahead of the hard fork, your nodes will be dropped from the network. Ideally, you will have Stacked for Cycle #94 ahead of time, any time now is good. +Binaries will be provided roughly a week in advance and your normal upgrade procedure should apply here, you’ll want to be running the latest node and Signer software. Note that if you do not upgrade ahead of the hard fork, your nodes will be dropped from the network. {% endhint %} #### Testnet Activation Window (August 19) diff --git a/nakamoto-upgrade/nakamoto-rollout-plan/README.md b/nakamoto-upgrade/nakamoto-rollout-plan/README.md index c47378fa1b..101eb1a47a 100644 --- a/nakamoto-upgrade/nakamoto-rollout-plan/README.md +++ b/nakamoto-upgrade/nakamoto-rollout-plan/README.md @@ -16,7 +16,7 @@ Nakamoto has completed step 1 of the rollout (Instantiation). Next, a hard fork The rollout will follow this two-step process, each of which is implemented via hard fork. 1. **Step 1 - Instantiation:** The pox-4 contract and the majority of the Nakamoto code are shipped, but Nakamoto rules are inactive. This is so other aspects of the contract can be tested before layering on the complexity that comes with the testnet (and later mainnet) being dependent on it. Importantly, this phase also allows time for Signers to register without the network being dependent on them to sign blocks. -2. **Step 2 - Activation:** A process to make Nakamoto rules live begins on August 28. Once completely rolled out, the full set of Nakamoto features including Signer-based functions, fast blocks, and Bitcoin finality. In other words, ‘the switch is flipped’! +2. **Step 2 - Activation:** Once completely rolled out, the full set of Nakamoto features including Signer-based functions, fast blocks, and Bitcoin finality. In other words, ‘the switch is flipped’! This switch is scheduled to occur at Bitcoin Block #867867 (\~October 29th). It’s important to note the heaviest lift of any hard fork is historically the sync from genesis. With the two Nakamoto forks, one goal is not to require this, making the upgrade more akin to a push-button software update and much simpler for all node operators. @@ -30,7 +30,7 @@ Nakamoto rules are the logic that makes Nakamoto different than the version befo ### Nakamoto Activation Sequence -
Step Overview Date/Period
✅ A, BActivation Window Opens & Binaries DeliveredPending no new bugs, final binaries are delivered - this is all the code Signers, Miners, and Node Operators need to run the network.Aug 28th
✅ CCycle Handoff - SignersAt the end of Cycle 92, core developers will watch for a successful ‘handoff’, meaning a successful change of the Signer sets between Stacking cycles.Cycle 92 to Cycle 93
✅ DFirst Testnet Hard ForkCore developers performed a successful testnet hardfork (on Nakamoto testnet).Sept 27
EDetermine Hard Fork BlockCore developers will select a mainnet hard fork block after performing a final testnet hardfork.~October 10
FEpoch 3.0 - Nakamoto Rules StartFast blocks, full Bitcoin finality! Nakamoto rules go live on mainnet at hard fork block.Hard Fork Block
+
Step Overview Date/Period
✅ A, BActivation Window Opens & Binaries DeliveredPending no new bugs, final binaries are delivered - this is all the code Signers, Miners, and Node Operators need to run the network.Aug 28th
✅ CCycle Handoff - SignersAt the end of Cycle 92, core developers will watch for a successful ‘handoff’, meaning a successful change of the Signer sets between Stacking cycles.Cycle 92 to Cycle 93
✅ DFirst Testnet Hard ForkCore developers performed a successful testnet hardfork (on Nakamoto testnet).Sept 27
✅ EDetermine Hard Fork BlockCore developers have selected Bitcoin block #867867.October 17
FEpoch 3.0 - Nakamoto Rules StartFast blocks, full Bitcoin finality! Nakamoto rules go live on mainnet at hard fork block.~October 29
### Factors that could affect timelines: diff --git a/nakamoto-upgrade/nakamoto-rollout-plan/nakamoto-for-exchanges.md b/nakamoto-upgrade/nakamoto-rollout-plan/nakamoto-for-exchanges.md index aa38981c71..d28e7b1cf0 100644 --- a/nakamoto-upgrade/nakamoto-rollout-plan/nakamoto-for-exchanges.md +++ b/nakamoto-upgrade/nakamoto-rollout-plan/nakamoto-for-exchanges.md @@ -44,7 +44,7 @@ Exchanges that support native Bitcoin yield via Stacking may have extra upgrade Core developers expect to release binaries on or around August 28th. Exchanges can expect their usual ecosystem point of contact to provide them with detailed information and updates during the process. 1. **Instantiation block:** The first hard fork occurred at Bitcoin block 840,360. If you haven't already, you will need to upgrade to the latest node software or they will not process valid blocks and users will not be able to move funds on the network. -2. **Activation block:** The second hard fork occurs around August 28. At this time, the Nakamoto consensus rules will activate, meaning the new Nakamoto features will be made live. Exchanges will need to repeat the upgrade process at this time. Again, no genesis sync, just a download and restart. +2. **Activation block:** The second hard fork will occur at Bitcoin Block #867867 (\~October 29th). At this time, the Nakamoto consensus rules will activate, meaning the new Nakamoto features will be made live. Exchanges will need to repeat the upgrade process at this time. Again, no genesis sync, just a download and restart. The hard fork will occur **after** a series of activation steps. You can learn more about this upcoming Nakamoto Activation sequence here: [#nakamoto-activation-sequence](./#nakamoto-activation-sequence "mention"). diff --git a/nakamoto-upgrade/nakamoto-upgrade-start-here.md b/nakamoto-upgrade/nakamoto-upgrade-start-here.md index 29885c2d37..047bba09c7 100644 --- a/nakamoto-upgrade/nakamoto-upgrade-start-here.md +++ b/nakamoto-upgrade/nakamoto-upgrade-start-here.md @@ -32,7 +32,7 @@ If you are operating as a signer, you'll want to make sure you familiarize yours ### Application Developers -The instantiation phase (current phase) focuses on activation the new stacking rules in pox-4. Fast blocks won't be available until after Activation, projected \~August 28th. This means that most developers won't need to do anything different, although there are some changes to various Hiro products and tools you should be aware of. Details on this can be found in the [Nakamoto for App Developers](nakamoto-rollout-plan/nakamoto.md) guide. +The instantiation phase (current phase) focuses on activation the new stacking rules in pox-4. Fast blocks won't be available until after Activation, projected \~October 29th. This means that most developers won't need to do anything different, although there are some changes to various Hiro products and tools you should be aware of. Details on this can be found in the [Nakamoto for App Developers](nakamoto-rollout-plan/nakamoto.md) guide. ### Exchanges