Monitor Type: etcd (Source)
Accepts Endpoints: Yes
Multiple Instances Allowed: Yes
This monitor reports etcd server metrics under the /metrics path on its
client port and optionally on locations given by --listen-metrics-urls.
Note that this monitor collects metrics solely from the prometheus endpoint,
unlike the collectd/etcd monitor which collects metrics from the /stats
endpoint.
An example configuration for this monitor:
monitors:
- type: etcd
discoveryRule: kubernetes_pod_name =~ "etcd" && target == "pod"
port: 2379
useHTTPS: true
skipVerify: true
sendAllMetrics: true
clientCertPath: /var/lib/minikube/certs/etcd/server.crt
clientKeyPath: /var/lib/minikube/certs/etcd/server.key
extraDimensions:
metric_source: etcdNote that the above config assumes that the client certificate and key are accessible by the SignalFx Agent in the specified path.
To activate this monitor in the Smart Agent, add the following to your agent config:
monitors: # All monitor config goes under this key
- type: etcd
... # Additional config
For a list of monitor options that are common to all monitors, see Common Configuration.
| Config option | Required | Type | Description |
|---|---|---|---|
httpTimeout |
no | int64 |
HTTP timeout duration for both read and writes. This should be a duration string that is accepted by https://golang.org/pkg/time/#ParseDuration (default: 10s) |
username |
no | string |
Basic Auth username to use on each request, if any. |
password |
no | string |
Basic Auth password to use on each request, if any. |
useHTTPS |
no | bool |
If true, the agent will connect to the server using HTTPS instead of plain HTTP. (default: false) |
httpHeaders |
no | map of strings |
A map of HTTP header names to values. Comma separated multiple values for the same message-header is supported. |
skipVerify |
no | bool |
If useHTTPS is true and this option is also true, the exporter's TLS cert will not be verified. (default: false) |
caCertPath |
no | string |
Path to the CA cert that has signed the TLS cert, unnecessary if skipVerify is set to false. |
clientCertPath |
no | string |
Path to the client TLS cert to use for TLS required connections |
clientKeyPath |
no | string |
Path to the client TLS key to use for TLS required connections |
host |
yes | string |
Host of the exporter |
port |
yes | integer |
Port of the exporter |
useServiceAccount |
no | bool |
Use pod service account to authenticate. (default: false) |
metricPath |
no | string |
Path to the metrics endpoint on the exporter server, usually /metrics (the default). (default: /metrics) |
sendAllMetrics |
no | bool |
Send all the metrics that come out of the Prometheus exporter without any filtering. This option has no effect when using the prometheus exporter monitor directly since there is no built-in filtering, only when embedding it in other monitors. (default: false) |
These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.
etcd_cluster_version(gauge)
Which version is running. 1 for 'cluster_version' label with current cluster versionetcd_debugging_disk_backend_commit_rebalance_duration_seconds(cumulative)
The latency distributions of commit.rebalance called by bboltdb backend. (sum)etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket(cumulative)
The latency distributions of commit.rebalance called by bboltdb backend. (bucket)etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count(cumulative)
The latency distributions of commit.rebalance called by bboltdb backend. (count)etcd_debugging_disk_backend_commit_spill_duration_seconds(cumulative)
The latency distributions of commit.spill called by bboltdb backend. (sum)etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket(cumulative)
The latency distributions of commit.spill called by bboltdb backend. (bucket)etcd_debugging_disk_backend_commit_spill_duration_seconds_count(cumulative)
The latency distributions of commit.spill called by bboltdb backend. (count)etcd_debugging_disk_backend_commit_write_duration_seconds(cumulative)
The latency distributions of commit.write called by bboltdb backend. (sum)etcd_debugging_disk_backend_commit_write_duration_seconds_bucket(cumulative)
The latency distributions of commit.write called by bboltdb backend. (bucket)etcd_debugging_disk_backend_commit_write_duration_seconds_count(cumulative)
The latency distributions of commit.write called by bboltdb backend. (count)etcd_debugging_lease_granted_total(cumulative)
The total number of granted leases.etcd_debugging_lease_renewed_total(cumulative)
The number of renewed leases seen by the leader.etcd_debugging_lease_revoked_total(cumulative)
The total number of revoked leases.etcd_debugging_lease_ttl_total(cumulative)
Bucketed histogram of lease TTLs. (sum)etcd_debugging_lease_ttl_total_bucket(cumulative)
Bucketed histogram of lease TTLs. (bucket)etcd_debugging_lease_ttl_total_count(cumulative)
Bucketed histogram of lease TTLs. (count)etcd_debugging_mvcc_compact_revision(gauge)
The revision of the last compaction in store.etcd_debugging_mvcc_current_revision(gauge)
The current revision of store.etcd_debugging_mvcc_db_compaction_keys_total(cumulative)
Total number of db keys compacted.etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds(cumulative)
Bucketed histogram of db compaction pause duration. (sum)etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket(cumulative)
Bucketed histogram of db compaction pause duration. (bucket)etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_count(cumulative)
Bucketed histogram of db compaction pause duration. (count)etcd_debugging_mvcc_db_compaction_total_duration_milliseconds(cumulative)
Bucketed histogram of db compaction total duration. (sum)etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket(cumulative)
Bucketed histogram of db compaction total duration. (bucket)etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_count(cumulative)
Bucketed histogram of db compaction total duration. (count)etcd_debugging_mvcc_db_total_size_in_bytes(gauge)
Total size of the underlying database physically allocated in bytes.etcd_debugging_mvcc_delete_total(cumulative)
Total number of deletes seen by this member.etcd_debugging_mvcc_events_total(cumulative)
Total number of events sent by this member.etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds(cumulative)
Bucketed histogram of index compaction pause duration. (sum)etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket(cumulative)
Bucketed histogram of index compaction pause duration. (bucket)etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_count(cumulative)
Bucketed histogram of index compaction pause duration. (count)etcd_debugging_mvcc_keys_total(gauge)
Total number of keys.etcd_debugging_mvcc_pending_events_total(gauge)
Total number of pending events to be sent.etcd_debugging_mvcc_put_total(cumulative)
Total number of puts seen by this member.etcd_debugging_mvcc_range_total(cumulative)
Total number of ranges seen by this member.etcd_debugging_mvcc_slow_watcher_total(gauge)
Total number of unsynced slow watchers.etcd_debugging_mvcc_txn_total(cumulative)
Total number of txns seen by this member.etcd_debugging_mvcc_watch_stream_total(gauge)
Total number of watch streams.etcd_debugging_mvcc_watcher_total(gauge)
Total number of watchers.etcd_debugging_server_lease_expired_total(cumulative)
The total number of expired leases.etcd_debugging_snap_save_marshalling_duration_seconds(cumulative)
The marshalling cost distributions of save called by snapshot. (sum)etcd_debugging_snap_save_marshalling_duration_seconds_bucket(cumulative)
The marshalling cost distributions of save called by snapshot. (bucket)etcd_debugging_snap_save_marshalling_duration_seconds_count(cumulative)
The marshalling cost distributions of save called by snapshot. (count)etcd_debugging_snap_save_total_duration_seconds(cumulative)
The total latency distributions of save called by snapshot. (sum)etcd_debugging_snap_save_total_duration_seconds_bucket(cumulative)
The total latency distributions of save called by snapshot. (bucket)etcd_debugging_snap_save_total_duration_seconds_count(cumulative)
The total latency distributions of save called by snapshot. (count)etcd_debugging_store_expires_total(cumulative)
Total number of expired keys.etcd_debugging_store_reads_total(cumulative)
Total number of reads action by (get/getRecursive), local to this member.etcd_debugging_store_watch_requests_total(cumulative)
Total number of incoming watch requests (new or reestablished).etcd_debugging_store_watchers(gauge)
Count of currently active watchers.etcd_debugging_store_writes_total(cumulative)
Total number of writes (e.g. set/compareAndDelete) seen by this member.etcd_disk_backend_commit_duration_seconds(cumulative)
The latency distributions of commit called by backend. (sum)etcd_disk_backend_commit_duration_seconds_bucket(cumulative)
The latency distributions of commit called by backend. (bucket)etcd_disk_backend_commit_duration_seconds_count(cumulative)
The latency distributions of commit called by backend. (count)etcd_disk_backend_defrag_duration_seconds(cumulative)
The latency distribution of backend defragmentation. (sum)etcd_disk_backend_defrag_duration_seconds_bucket(cumulative)
The latency distribution of backend defragmentation. (bucket)etcd_disk_backend_defrag_duration_seconds_count(cumulative)
The latency distribution of backend defragmentation. (count)etcd_disk_backend_snapshot_duration_seconds(cumulative)
The latency distribution of backend snapshots. (sum)etcd_disk_backend_snapshot_duration_seconds_bucket(cumulative)
The latency distribution of backend snapshots. (bucket)etcd_disk_backend_snapshot_duration_seconds_count(cumulative)
The latency distribution of backend snapshots. (count)etcd_disk_wal_fsync_duration_seconds(cumulative)
The latency distributions of fsync called by WAL. (sum)etcd_disk_wal_fsync_duration_seconds_bucket(cumulative)
The latency distributions of fsync called by WAL. (bucket)etcd_disk_wal_fsync_duration_seconds_count(cumulative)
The latency distributions of fsync called by WAL. (count)etcd_grpc_proxy_cache_hits_total(gauge)
Total number of cache hitsetcd_grpc_proxy_cache_keys_total(gauge)
Total number of keys/ranges cachedetcd_grpc_proxy_cache_misses_total(gauge)
Total number of cache missesetcd_grpc_proxy_events_coalescing_total(cumulative)
Total number of events coalescingetcd_grpc_proxy_watchers_coalescing_total(gauge)
Total number of current watchers coalescingetcd_mvcc_db_open_read_transactions(gauge)
The number of currently open read transactionsetcd_mvcc_db_total_size_in_bytes(gauge)
Total size of the underlying database physically allocated in bytes.etcd_mvcc_db_total_size_in_use_in_bytes(gauge)
Total size of the underlying database logically in use in bytes.etcd_mvcc_delete_total(cumulative)
Total number of deletes seen by this member.etcd_mvcc_hash_duration_seconds(cumulative)
The latency distribution of storage hash operation. (sum)etcd_mvcc_hash_duration_seconds_bucket(cumulative)
The latency distribution of storage hash operation. (bucket)etcd_mvcc_hash_duration_seconds_count(cumulative)
The latency distribution of storage hash operation. (count)etcd_mvcc_hash_rev_duration_seconds(cumulative)
The latency distribution of storage hash by revision operation. (sum)etcd_mvcc_hash_rev_duration_seconds_bucket(cumulative)
The latency distribution of storage hash by revision operation. (bucket)etcd_mvcc_hash_rev_duration_seconds_count(cumulative)
The latency distribution of storage hash by revision operation. (count)etcd_mvcc_put_total(cumulative)
Total number of puts seen by this member.etcd_mvcc_range_total(cumulative)
Total number of ranges seen by this member.etcd_mvcc_txn_total(cumulative)
Total number of txns seen by this member.etcd_network_client_grpc_received_bytes_total(cumulative)
The total number of bytes received from grpc clients.etcd_network_client_grpc_sent_bytes_total(cumulative)
The total number of bytes sent to grpc clients.etcd_server_file_descriptors_used_total(gauge)
The total number of file descriptors used.etcd_server_go_version(gauge)
Which Go version server is running with. 1 for 'server_go_version' label with current version.etcd_server_has_leader(gauge)
Whether or not a leader exists. 1 is existence, 0 is not.etcd_server_health_failures_total(cumulative)
The total number of failed health checksetcd_server_health_success_total(cumulative)
The total number of successful health checksetcd_server_heartbeat_send_failures_total(cumulative)
The total number of leader heartbeat send failures (likely overloaded from slow disk).etcd_server_id(gauge)
Server or member ID in hexadecimal format. 1 for 'server_id' label with current ID.etcd_server_is_leader(gauge)
Whether or not this member is a leader. 1 if is, 0 otherwise.etcd_server_is_learner(gauge)
Whether or not this member is a learner. 1 if is, 0 otherwise.etcd_server_leader_changes_seen_total(cumulative)
The number of leader changes seen.etcd_server_learner_promote_successes_total(cumulative)
The total number of successful learner promotions while this member is leader.etcd_server_pending_proposal_total(gauge)
The total number of pending proposals.etcd_server_proposal_durations_seconds(cumulative)
The latency distributions of committing proposal. (sum)etcd_server_proposal_durations_seconds_bucket(cumulative)
The latency distributions of committing proposal. (bucket)etcd_server_proposal_durations_seconds_count(cumulative)
The latency distributions of committing proposal. (count)etcd_server_proposal_failed_total(cumulative)
The total number of failed proposals.etcd_server_proposals_applied_total(gauge)
The total number of consensus proposals applied.etcd_server_proposals_committed_total(gauge)
The total number of consensus proposals committed.etcd_server_proposals_failed_total(cumulative)
The total number of failed proposals seen.etcd_server_proposals_pending(gauge)
The current number of pending proposals to commit.etcd_server_quota_backend_bytes(gauge)
Current backend storage quota size in bytes.etcd_server_read_indexes_failed_total(cumulative)
The total number of failed read indexes seen.etcd_server_slow_apply_total(cumulative)
The total number of slow apply requests (likely overloaded from slow disk).etcd_server_slow_read_indexes_total(cumulative)
The total number of pending read indexes not in sync with leader's or timed out read index requests.etcd_server_snapshot_apply_in_progress_total(gauge)
1 if the server is applying the incoming snapshot. 0 if none.etcd_server_version(gauge)
Which version is running. 1 for 'server_version' label with current version.etcd_snap_db_fsync_duration_seconds(cumulative)
The latency distributions of fsyncing .snap.db file (sum)etcd_snap_db_fsync_duration_seconds_bucket(cumulative)
The latency distributions of fsyncing .snap.db file (bucket)etcd_snap_db_fsync_duration_seconds_count(cumulative)
The latency distributions of fsyncing .snap.db file (count)etcd_snap_db_save_total_duration_seconds(cumulative)
The total latency distributions of v3 snapshot save (sum)etcd_snap_db_save_total_duration_seconds_bucket(cumulative)
The total latency distributions of v3 snapshot save (bucket)etcd_snap_db_save_total_duration_seconds_count(cumulative)
The total latency distributions of v3 snapshot save (count)etcd_snap_fsync_duration_seconds(cumulative)
The latency distributions of fsync called by snap. (sum)etcd_snap_fsync_duration_seconds_bucket(cumulative)
The latency distributions of fsync called by snap. (bucket)etcd_snap_fsync_duration_seconds_count(cumulative)
The latency distributions of fsync called by snap. (count)etcd_snapshot_save_marshalling_durations_seconds(cumulative)
The marshalling cost distributions of save called by snapshot. (sum)etcd_snapshot_save_marshalling_durations_seconds_bucket(cumulative)
The marshalling cost distributions of save called by snapshot. (bucket)etcd_snapshot_save_marshalling_durations_seconds_count(cumulative)
The marshalling cost distributions of save called by snapshot. (count)etcd_snapshot_save_total_durations_seconds(cumulative)
The total latency distributions of save called by snapshot. (sum)etcd_snapshot_save_total_durations_seconds_bucket(cumulative)
The total latency distributions of save called by snapshot. (bucket)etcd_snapshot_save_total_durations_seconds_count(cumulative)
The total latency distributions of save called by snapshot. (count)etcd_storage_db_compaction_pause_duration_milliseconds(cumulative)
Bucketed histogram of db compaction pause duration. (sum)etcd_storage_db_compaction_pause_duration_milliseconds_bucket(cumulative)
Bucketed histogram of db compaction pause duration. (bucket)etcd_storage_db_compaction_pause_duration_milliseconds_count(cumulative)
Bucketed histogram of db compaction pause duration. (count)etcd_storage_db_compaction_total_duration_milliseconds(cumulative)
Bucketed histogram of db compaction total duration. (sum)etcd_storage_db_compaction_total_duration_milliseconds_bucket(cumulative)
Bucketed histogram of db compaction total duration. (bucket)etcd_storage_db_compaction_total_duration_milliseconds_count(cumulative)
Bucketed histogram of db compaction total duration. (count)etcd_storage_db_total_size_in_bytes(gauge)
Total size of the underlying database in bytes.etcd_storage_delete_total(cumulative)
Total number of deletes seen by this member.etcd_storage_events_total(cumulative)
Total number of events sent by this member.etcd_storage_index_compaction_pause_duration_milliseconds(cumulative)
Bucketed histogram of index compaction pause duration. (sum)etcd_storage_index_compaction_pause_duration_milliseconds_bucket(cumulative)
Bucketed histogram of index compaction pause duration. (bucket)etcd_storage_index_compaction_pause_duration_milliseconds_count(cumulative)
Bucketed histogram of index compaction pause duration. (count)etcd_storage_keys_total(gauge)
Total number of keys.etcd_storage_pending_events_total(gauge)
Total number of pending events to be sent.etcd_storage_put_total(cumulative)
Total number of puts seen by this member.etcd_storage_range_total(cumulative)
Total number of ranges seen by this member.etcd_storage_slow_watcher_total(gauge)
Total number of unsynced slow watchers.etcd_storage_txn_total(cumulative)
Total number of txns seen by this member.etcd_storage_watch_stream_total(gauge)
Total number of watch streams.etcd_storage_watcher_total(gauge)
Total number of watchers.etcd_store_expires_total(cumulative)
Total number of expired keys.etcd_store_reads_total(cumulative)
Total number of reads action by (get/getRecursive), local to this member.etcd_store_watch_requests_total(cumulative)
Total number of incoming watch requests (new or reestablished).etcd_store_watchers(gauge)
Count of currently active watchers.etcd_store_writes_total(cumulative)
Total number of writes (e.g. set/compareAndDelete) seen by this member.etcd_wal_fsync_durations_seconds(cumulative)
The latency distributions of fsync called by wal. (sum)etcd_wal_fsync_durations_seconds_bucket(cumulative)
The latency distributions of fsync called by wal. (bucket)etcd_wal_fsync_durations_seconds_count(cumulative)
The latency distributions of fsync called by wal. (count)etcd_wal_last_index_saved(gauge)
The index of the last entry saved by wal.go_gc_duration_seconds(cumulative)
A summary of the GC invocation durations. (sum)go_gc_duration_seconds_count(cumulative)
A summary of the GC invocation durations. (count)go_gc_duration_seconds_quantile(gauge)
A summary of the GC invocation durations. (quantized)go_goroutines(gauge)
Number of goroutines that currently exist.go_info(gauge)
Information about the Go environment.go_memstats_alloc_bytes(gauge)
Number of bytes allocated and still in use.go_memstats_alloc_bytes_total(cumulative)
Total number of bytes allocated, even if freed.go_memstats_buck_hash_sys_bytes(gauge)
Number of bytes used by the profiling bucket hash table.go_memstats_frees_total(cumulative)
Total number of frees.go_memstats_gc_cpu_fraction(gauge)
The fraction of this program's available CPU time used by the GC since the program started.go_memstats_gc_sys_bytes(gauge)
Number of bytes used for garbage collection system metadata.go_memstats_heap_alloc_bytes(gauge)
Number of heap bytes allocated and still in use.go_memstats_heap_idle_bytes(gauge)
Number of heap bytes waiting to be used.go_memstats_heap_inuse_bytes(gauge)
Number of heap bytes that are in use.go_memstats_heap_objects(gauge)
Number of allocated objects.go_memstats_heap_released_bytes(gauge)
Number of heap bytes released to OS.go_memstats_heap_released_bytes_total(cumulative)
Total number of heap bytes released to OS.go_memstats_heap_sys_bytes(gauge)
Number of heap bytes obtained from system.go_memstats_last_gc_time_seconds(gauge)
Number of seconds since 1970 of last garbage collection.go_memstats_lookups_total(cumulative)
Total number of pointer lookups.go_memstats_mallocs_total(cumulative)
Total number of mallocs.go_memstats_mcache_inuse_bytes(gauge)
Number of bytes in use by mcache structures.go_memstats_mcache_sys_bytes(gauge)
Number of bytes used for mcache structures obtained from system.go_memstats_mspan_inuse_bytes(gauge)
Number of bytes in use by mspan structures.go_memstats_mspan_sys_bytes(gauge)
Number of bytes used for mspan structures obtained from system.go_memstats_next_gc_bytes(gauge)
Number of heap bytes when next garbage collection will take place.go_memstats_other_sys_bytes(gauge)
Number of bytes used for other system allocations.go_memstats_stack_inuse_bytes(gauge)
Number of bytes in use by the stack allocator.go_memstats_stack_sys_bytes(gauge)
Number of bytes obtained from system for stack allocator.go_memstats_sys_bytes(gauge)
Number of bytes obtained from system.go_threads(gauge)
Number of OS threads created.grpc_server_handled_total(cumulative)
Total number of RPCs completed on the server, regardless of success or failure.grpc_server_msg_received_total(cumulative)
Total number of RPC stream messages received on the server.grpc_server_msg_sent_total(cumulative)
Total number of gRPC stream messages sent by the server.grpc_server_started_total(cumulative)
Total number of RPCs started on the server.http_request_duration_microseconds(cumulative)
The HTTP request latencies in microseconds. (sum)http_request_duration_microseconds_count(cumulative)
The HTTP request latencies in microseconds. (count)http_request_duration_microseconds_quantile(gauge)
The HTTP request latencies in microseconds. (quantized)http_request_size_bytes(cumulative)
The HTTP request sizes in bytes. (sum)http_request_size_bytes_count(cumulative)
The HTTP request sizes in bytes. (count)http_request_size_bytes_quantile(gauge)
The HTTP request sizes in bytes. (quantized)http_requests_total(cumulative)
Total number of HTTP requests made.http_response_size_bytes(cumulative)
The HTTP response sizes in bytes. (sum)http_response_size_bytes_count(cumulative)
The HTTP response sizes in bytes. (count)http_response_size_bytes_quantile(gauge)
The HTTP response sizes in bytes. (quantized)process_cpu_seconds_total(cumulative)
Total user and system CPU time spent in seconds.process_max_fds(gauge)
Maximum number of open file descriptors.process_open_fds(gauge)
Number of open file descriptors.process_resident_memory_bytes(gauge)
Resident memory size in bytes.process_start_time_seconds(gauge)
Start time of the process since unix epoch in seconds.process_virtual_memory_bytes(gauge)
Virtual memory size in bytes.process_virtual_memory_max_bytes(gauge)
Maximum amount of virtual memory available in bytes.promhttp_metric_handler_requests_in_flight(gauge)
Current number of scrapes being served.promhttp_metric_handler_requests_total(cumulative)
Total number of scrapes by HTTP status code.
To emit metrics that are not default, you can add those metrics in the
generic monitor-level extraMetrics config option. Metrics that are derived
from specific configuration options that do not appear in the above list of
metrics do not need to be added to extraMetrics.
To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.