Skip to content

Commit

Permalink
fix: metrics dev-env for linux
Browse files Browse the repository at this point in the history
Fixes a hardcoded path in the compose setup for local metrics. Updates
the docs, as well, to clarify that the first-run experience via docker
compose is only relevant for Linux hosts, due to use of host-networking.
Will work on a macos-compatible dev env later, but for now focusing on
making sure what's in the repo is clear and runnable for actual testnet
node operators, which means Linux boxes.

Closes #4565.
  • Loading branch information
conorsch committed Jun 20, 2024
1 parent 3045645 commit 757d224
Show file tree
Hide file tree
Showing 6 changed files with 22 additions and 11 deletions.
3 changes: 0 additions & 3 deletions deployments/compose/justfile

This file was deleted.

7 changes: 7 additions & 0 deletions deployments/compose/metrics.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,17 @@ services:
dockerfile: deployments/containerfiles/Dockerfile-grafana
network_mode: host
user: "${UID:-1000}"
# Don't use ports, since ports conflicts with network_mode=host.
# We use network_mode=host so that metrics can talk to services bound
# to localhost.
# ports:
# - "3000:3000"

# The Prometheus service, for scraping metrics from Penumbra's pd metrics port.
prom:
image: "docker.io/prom/prometheus"
network_mode: host
# ports:
# - "9090:9090"
volumes:
- "${PWD:?}/../config/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
2 changes: 1 addition & 1 deletion deployments/config/prometheus.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
scrape_configs:
- job_name: 'Tendermint Node'
- job_name: 'CometBFT Node'
scrape_interval: 10s
scheme: http
metrics_path: metrics
Expand Down
4 changes: 2 additions & 2 deletions deployments/containerfiles/Dockerfile-grafana
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ARG GRAFANA_VERSION="10.1.5"
FROM docker.io/grafana/grafana:${GRAFANA_VERSION}
COPY deployments/config/grafana/provisioning /etc/grafana/provisioning
COPY deployments/config/grafana/provisioning/ /etc/grafana/provisioning/
COPY deployments/config/grafana/grafana.ini /etc/grafana/grafana.ini
COPY deployments/config/grafana/dashboards /var/lib/grafana/dashboards
COPY deployments/config/grafana/dashboards/ /var/lib/grafana/dashboards/
12 changes: 7 additions & 5 deletions docs/guide/src/dev/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,20 +62,22 @@ After being changed, Grafana dashboards should be backed up to the repository fo
Grafana has an [import/export](https://grafana.com/docs/grafana/latest/dashboards/export-import/) feature that
we use for maintaining our dashboards.

1. Export the dashboard as JSON with the default settings
2. Rename the JSON file and copy into the repo (`config/grafana/dashboards/`)
3. PR the changes into main, and confirm on preview post-deploy that it works as expected.

1. View the dashboard you want to export, and click the share icon in the top bar.
2. Choose **Export**, and enable **Export for sharing externally**, which will generalized the datasource.
3. Download the JSON file, renaming it as necessary, and copy into the repo (`config/grafana/dashboards/`)
4. PR the changes into main, and confirm on preview post-deploy that it works as expected.

## Editing metrics locally

To facilitate working with metrics locally, first run a `pd` node on your machine with the metrics endpoint
exposed. Then, you can spin up a metrics sidecar deployment:

```bash
cd deployments/compose
just metrics
```

Note that this setup only works on Linux hosts, due to the use of host networking, so the metrics
containers can reach network ports on the host machine.

To add new Grafana visualizations, open http://localhost:3000 and edit the existing dashboards.
When you're happy with what you've got, follow the "Backing up Grafana" instructions above to save your work.
5 changes: 5 additions & 0 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ fmt:
proto:
./deployments/scripts/protobuf-codegen

# Run a local prometheus/grafana setup, in containers, to scrape a local node. Linux only.
metrics:
cd ./deployments/compose/ \
&& docker-compose -f metrics.yml up --build --abort-on-container-exit --force-recreate --remove-orphans

# Configures and runs a relayer instance between "preview" (latest main) and local devnet on current HEAD
relayer-local-devnet:
./deployments/scripts/relayer-local-devnet
Expand Down

0 comments on commit 757d224

Please sign in to comment.