-
Notifications
You must be signed in to change notification settings - Fork 238
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into update-sbtc-docs
- Loading branch information
Showing
7 changed files
with
237 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
227 changes: 227 additions & 0 deletions
227
guides-and-tutorials/running-a-signer/how-to-monitor-signer.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| # How to Monitor a Signer | ||
|
|
||
| We will use [Grafana Cloud](https://grafana.com) to observe and monitor both the | ||
| Signer and its corresponding Stacks node. | ||
|
|
||
| ## Requirements | ||
|
|
||
| Grafana's application observability docs have a [great | ||
| quick-start](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/). We will use: | ||
|
|
||
| - Grafana Cloud to collect metrics and visualize them. | ||
| - Grafana Alloy, on the Signer host, to push the metrics. | ||
|
|
||
| ### Creating a Grafana Cloud account | ||
|
|
||
| Before we begin, create a [Grafana | ||
| Cloud](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/grafana-cloud/) account (they offer a free tier that you can use). | ||
|
|
||
| Once done, access your dashboard and: | ||
|
|
||
| 1. Click on "Connections", then | ||
| 2. "Add new connection", and | ||
| 3. select "Hosted Prometheus metrics". | ||
| 4. Now select "Via Grafana Alloy", then | ||
| 5. on step 2, select "Run Grafana Alloy" to generate an API token. | ||
|
|
||
| Note the token `GCLOUD_RW_API_KEY` and the parameters `GCLOUD_HOSTED_METRICS_URL` | ||
| and `GCLOUD_HOSTED_METRICS_ID`, we will use them later. | ||
|
|
||
| ### Configuring the Signer and the Stacks node | ||
|
|
||
| Ensure both your Signer configuration and your node configuration include the | ||
| following lines: | ||
|
|
||
| ```toml | ||
| # signer-config.toml | ||
| # ... | ||
| metrics_endpoint = "127.0.0.1:30001" | ||
| ``` | ||
|
|
||
| ```toml | ||
| # node-config.toml | ||
| # ... | ||
| prometheus_bind = "127.0.0.1:9153" | ||
| ``` | ||
|
|
||
| If you compile the application binaries yourself, remember to enable the Cargo | ||
| feature `monitoring_prom` while building them, for example: | ||
|
|
||
| ```bash | ||
| cargo build --features monitoring_prom,slog_json --release | ||
| ``` | ||
|
|
||
| Once both binaries are running with the updated configuration, you can peek | ||
| at the metrics being exposed: | ||
|
|
||
| ```bash | ||
| curl 127.0.0.1:30001/metrics | ||
|
|
||
| # HELP stacks_signer_current_reward_cycle The current reward cycle | ||
| # TYPE stacks_signer_current_reward_cycle gauge | ||
| stacks_signer_current_reward_cycle 95 | ||
| # HELP stacks_signer_node_rpc_call_latencies_histogram Time (seconds) measuring round-trip RPC call latency to the Stacks node | ||
| # TYPE stacks_signer_node_rpc_call_latencies_histogram histogram | ||
| ... | ||
| stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.005"} 0 | ||
| stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.01"} 0 | ||
| stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.025"} 0 | ||
| stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.05"} 985 | ||
| stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.1"} 1194 | ||
| ... | ||
| ``` | ||
|
|
||
| Also, you'll have a `/info` endpoint on the same port: | ||
|
|
||
| ```bash | ||
| curl 127.0.0.1:30001/info | ||
| ``` | ||
|
|
||
| ### Install Alloy | ||
|
|
||
| Follow these instructions to install [Grafana | ||
| Alloy](https://grafana.com/docs/alloy/latest/set-up/install/linux/). | ||
|
|
||
| On Debian-based distributions: | ||
|
|
||
| ```bash | ||
| sudo apt install gpg | ||
| sudo mkdir -p /etc/apt/keyrings/ | ||
| wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null | ||
| echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list | ||
| sudo apt-get update | ||
| sudo apt-get install alloy | ||
| ``` | ||
|
|
||
| ### Configure Alloy | ||
|
|
||
| Edit the file `/etc/alloy/config.alloy` as follows, by taking care of replacing | ||
| the placeholders related to the `prometheus` endpoint with the parameters | ||
| obtained when creating a [Grafana Cloud account before](#creating-a-grafana-cloud-account): | ||
|
|
||
| - `GCLOUD_HOSTED_METRICS_URL` | ||
| - `GCLOUD_HOSTED_METRICS_ID` | ||
| - `GCLOUD_RW_API_KEY` | ||
|
|
||
| ```conf | ||
| // For a full configuration reference, see https://grafana.com/docs/alloy | ||
| logging { | ||
| level = "warn" | ||
| } | ||
| prometheus.exporter.unix "default" { | ||
| include_exporter_metrics = true | ||
| disable_collectors = ["mdadm"] | ||
| } | ||
| prometheus.scrape "default" { | ||
| targets = array.concat( | ||
| prometheus.exporter.unix.default.targets, | ||
| [ | ||
| { | ||
| // Self-collect metrics | ||
| job = "alloy", | ||
| __address__ = "127.0.0.1:12345", | ||
| }, | ||
| { | ||
| // stacks-signer | ||
| job = "stacks-signer", | ||
| __address__ = "127.0.0.1:30001", | ||
| }, | ||
| { | ||
| // stacks-node | ||
| job = "stacks-node", | ||
| __address__ = "127.0.0.1:9153", | ||
| }, | ||
| ], | ||
| ) | ||
| forward_to = [prometheus.remote_write.default.receiver] | ||
| } | ||
| prometheus.remote_write "default" { | ||
| external_labels = {"instance" = constants.hostname} | ||
| endpoint { | ||
| # TODO: Edit the URL below with your Grafana production URL. | ||
| # should end with /api/prom/push | ||
| url = "<your GCLOUD_HOSTED_METRICS_URL>" | ||
| # TODO: Edit with your Grafana Cloud ID and Token | ||
| basic_auth { | ||
| username = "<your GCLOUD_HOSTED_METRICS_ID>" | ||
| password = "<your GCLOUD_RW_API_KEY>" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ```bash | ||
| sudo systemctl daemon-reload | ||
| sudo systemctl enable alloy.service | ||
| sudo systemctl start alloy.service | ||
| ``` | ||
|
|
||
| Metrics from your Signer and node will now start being pushed to Grafana Cloud. | ||
|
|
||
| ## Visualizing the metrics | ||
|
|
||
| You can now start building a dashboard to visualize the metrics. | ||
|
|
||
| 1. Log-in to Grafana Cloud and create a new Dashboard. | ||
| 2. Pick the Prometheus instance you created before as the data source. | ||
| 3. Create a new panel and pick `stacks_signer_current_reward_cycle` from the | ||
| metrics. | ||
|
|
||
| You should now be able to see Stacks' current reward cycle, as measured by the | ||
| Signer, into the dashboard. | ||
|
|
||
| Grafana comes with powerful data visualization tools. You can read about how to | ||
| query and transform data | ||
| [here](https://grafana.com/docs/grafana-cloud/visualizations/panels-visualizations/query-transform-data/), | ||
| while here you will find examples on how to build | ||
| [Prometheus queries](https://prometheus.io/docs/prometheus/latest/querying/basics/). | ||
|
|
||
| ## Bonus: monitoring the host | ||
|
|
||
| Since we are here, we can also monitor the host itself. Debian-based | ||
| distributions make it very easy for us by using | ||
| `[node-exporter](https://github.com/prometheus/node_exporter/tree/master)`. | ||
|
|
||
| ```bash | ||
| sudo apt install prometheus-node-exporter | ||
| sudo systemctl enable prometheus-node-exporter | ||
| sudo systemctl start prometheus-node-exporter | ||
| ``` | ||
|
|
||
| This will expose metrics on port `9100` of `localhost`. | ||
|
|
||
| We can now configure `alloy` to push them to Grafana. Edit your | ||
| `/etc/alloy/config.alloy` file and add the following: | ||
|
|
||
| ```txt | ||
| prometheus.scrape "default" { | ||
| targets = array.concat([ | ||
| ... | ||
| { | ||
| job = "node_exporter", | ||
| __address__ = "127.0.0.1:9100", | ||
| } | ||
| ... | ||
| ])} | ||
| ``` | ||
|
|
||
| Now reload `alloy` and check its status: | ||
|
|
||
| ```bash | ||
| sudo systemctl reload alloy | ||
| sudo systemctl status alloy | ||
| ``` | ||
|
|
||
| `node-exporter` provides a _lot_ of metrics. Poke at them through the Grafana | ||
| Explorer or use one of the many prepared dashboard (e.g., [this | ||
| one](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)) to see | ||
| comprehensive information. Once you have a dashboard ready, you can also | ||
| use it to configure a few alerts (e.g., on disk space, etc). |
4 changes: 2 additions & 2 deletions
4
nakamoto-upgrade/nakamoto-activation-guide-for-signers/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters