From 6218d9282112a98c2bac15453ad637775fd5714b Mon Sep 17 00:00:00 2001 From: Jordan Schalm Date: Fri, 6 Sep 2024 16:23:08 -0700 Subject: [PATCH 1/4] Add docs about machine account metrics --- .../node-operation/monitoring-nodes.md | 31 +++++++++++++++++++ docs/networks/staking/11-machine-account.md | 2 +- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/networks/node-ops/node-operation/monitoring-nodes.md b/docs/networks/node-ops/node-operation/monitoring-nodes.md index e9e30da26a..90b65330de 100644 --- a/docs/networks/node-ops/node-operation/monitoring-nodes.md +++ b/docs/networks/node-ops/node-operation/monitoring-nodes.md @@ -60,6 +60,37 @@ The following are some important metrics produced by the node. | consensus_hotstuff_cur_view | Current view of the HotStuff consensus algorith; Consensus/Collection only; should increase at a constant rate. | | consensus_hotstuff_timeout_seconds | How long it takes to timeout failed rounds; Consensus/Collection only; values consistently larger than 5s are abnormal. | +### Machine Account + +Collection and consensus nodes use a machine account that must be kept funded. See [here](../../staking/11-machine-account.md) for details. + +Nodes check their machine account's configuration and funding and produce metrics. + +| Metric Name | Description | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| machine_account_balance | The current balance (FLOW) | +| machine_account_recommended_min_balance | The recommended minimum balance (FLOW) | +| machine_account_is_misconfigured | 0 if the node is configured correctly; 1 if the node is misconfigured | + +To be notified when your node's machine account needs to be refilled or has a configuration error, you can set up alerts. + +When the machine account balance needs to be refilled: +``` +machine_account_balance < machine_account_recommended_min_balance +``` + +When the machine account has a configuration error: +``` +machine_account_is_misconfigured > 0 +``` + +The metrics include the account address of the machine account for convenience: +``` +# HELP machine_account_balance the last observed balance of this node's machine account, in units of FLOW +# TYPE machine_account_balance gauge +machine_account_balance{acct_address="7b16b57ae0a3c6aa"} 9.99464935 +``` + ## Monitoring a Flow node using Metrika Monitoring Metrika has developed the Flow node monitoring service and is the recommended way of monitoring a Flow node. diff --git a/docs/networks/staking/11-machine-account.md b/docs/networks/staking/11-machine-account.md index 08a9b3f54e..499906ea38 100644 --- a/docs/networks/staking/11-machine-account.md +++ b/docs/networks/staking/11-machine-account.md @@ -34,7 +34,7 @@ however more may be required under certain circumstances and network conditions. Because some transactions sent by the Machine Account are system critical, we recommend maintaining -a balance sufficient to accommodate worst-case transaction submission numbers at all times. +a balance sufficient to accommodate worst-case transaction submission numbers at all times. See [] From 34dcfb2cd4645386e54312a30088effc33a9c9b9 Mon Sep 17 00:00:00 2001 From: Jordan Schalm Date: Fri, 6 Sep 2024 16:24:42 -0700 Subject: [PATCH 2/4] fix link --- docs/networks/staking/11-machine-account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networks/staking/11-machine-account.md b/docs/networks/staking/11-machine-account.md index 499906ea38..0e40c8d1f2 100644 --- a/docs/networks/staking/11-machine-account.md +++ b/docs/networks/staking/11-machine-account.md @@ -34,7 +34,7 @@ however more may be required under certain circumstances and network conditions. Because some transactions sent by the Machine Account are system critical, we recommend maintaining -a balance sufficient to accommodate worst-case transaction submission numbers at all times. See [] +a balance sufficient to accommodate worst-case transaction submission numbers at all times. See [here](./../node-ops/node-operation/monitoring-nodes.md) for how to monitor. From 2e72550b6c40f7c52fa2f8c3ef97e60357833d1a Mon Sep 17 00:00:00 2001 From: Jordan Schalm Date: Mon, 9 Sep 2024 08:56:09 -0700 Subject: [PATCH 3/4] Update docs/networks/node-ops/node-operation/monitoring-nodes.md --- docs/networks/node-ops/node-operation/monitoring-nodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networks/node-ops/node-operation/monitoring-nodes.md b/docs/networks/node-ops/node-operation/monitoring-nodes.md index 90b65330de..903db9210e 100644 --- a/docs/networks/node-ops/node-operation/monitoring-nodes.md +++ b/docs/networks/node-ops/node-operation/monitoring-nodes.md @@ -84,7 +84,7 @@ When the machine account has a configuration error: machine_account_is_misconfigured > 0 ``` -The metrics include the account address of the machine account for convenience: +The metrics include the account address of the machine account (`acct_address` label) for convenience: ``` # HELP machine_account_balance the last observed balance of this node's machine account, in units of FLOW # TYPE machine_account_balance gauge From 5503b49ac4af0b9a06226db9f0a13be9516ebb10 Mon Sep 17 00:00:00 2001 From: Jordan Schalm Date: Mon, 9 Sep 2024 08:59:24 -0700 Subject: [PATCH 4/4] Update docs/networks/staking/11-machine-account.md --- docs/networks/staking/11-machine-account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networks/staking/11-machine-account.md b/docs/networks/staking/11-machine-account.md index 0e40c8d1f2..fd5b6e7ad6 100644 --- a/docs/networks/staking/11-machine-account.md +++ b/docs/networks/staking/11-machine-account.md @@ -34,7 +34,7 @@ however more may be required under certain circumstances and network conditions. Because some transactions sent by the Machine Account are system critical, we recommend maintaining -a balance sufficient to accommodate worst-case transaction submission numbers at all times. See [here](./../node-ops/node-operation/monitoring-nodes.md) for how to monitor. +a balance sufficient to accommodate worst-case transaction submission numbers at all times. **See [here](./../node-ops/node-operation/monitoring-nodes.md#machine-account) for how to monitor.**