-
Notifications
You must be signed in to change notification settings - Fork 218
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into hermes_console_vue
- Loading branch information
Showing
159 changed files
with
3,905 additions
and
1,575 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,44 +1,20 @@ | ||
| # Metrics | ||
|
|
||
| Hermes Frontend and Consumers use [Dropwizard Metrics](https://dropwizard.github.io/metrics/3.1.0/) library to gather | ||
| and publish metrics to Metric Store. | ||
| Hermes Frontend, Consumers and Management use [Micrometer Metrics](https://github.com/micrometer-metrics/micrometer) library to gather | ||
| and expose metrics. | ||
|
|
||
| If you would like to preview or debug metrics, set `{modulePrefix}.metrics.consoleReporterEnabled` to `true`, so they will be printed | ||
| to stdout. | ||
| ## Prometheus | ||
| By default, Hermes includes Prometheus reporter. It exposes metrics on `/status/prometheus` endpoint. | ||
| Reporter configuration can be configured using following options: | ||
|
|
||
| ## Graphite | ||
| Option | Description | Default value | ||
| ---------------------------------------------- |-------------------------------------------------------| ------------- | ||
| {modulePrefix}.metrics.prometheus.step | The step size to use in computing windowed statistics | 60s | ||
| {modulePrefix}.metrics.prometheus.descriptions | If meter descriptions should be sent to Prometheus | true | ||
|
|
||
| By default, Hermes includes Graphite reporter, which can be configured using following options: | ||
| In order to be able to access basic metrics via Management API, it needs to be configured to reach VictoriaMetrics API: | ||
|
|
||
| Option | Description | Default value | ||
| ---------------------------------------------- | -------------------------------------- | ------------- | ||
| {modulePrefix}.metrics.graphiteReporterEnabled | enable Graphite reporter | false | ||
| {modulePrefix}.graphite.host | Graphite host | localhost | ||
| {modulePrefix}.graphite.port | Graphite port | 2003 | ||
| {modulePrefix}.graphite.prefix | prefix for all metrics | stats.tech.hermes | ||
| {modulePrefix}.metrics.reportPeriod | how often to report metrics | 20s | ||
|
|
||
| In order to be able to access basic metrics via Management API, it needs to be configured to reach Graphite API: | ||
|
|
||
| Option | Description | Default value | ||
| ----------------------- | ------------------------ | ------------- | ||
| metrics.graphiteHttpUri | URI to Graphite HTTP API | http://localhost:80 | ||
| metrics.prefix | prefix for all metrics | stats.tech.hermes | ||
|
|
||
| ## Custom | ||
|
|
||
| You can register any custom reporter that is compatible with Dropwizard `MetricRegistry`. | ||
|
|
||
| For the Consumers and Frontend modules register the reporter as a bean, for example: | ||
|
|
||
| ```java | ||
| @Configuration | ||
| public class CustomHermesConsumersConfiguration { | ||
|
|
||
| @Bean | ||
| @Primary | ||
| public MetricRegistry myMetricRegistry(MetricRegistry metricRegistry) { | ||
| return new MyMetricsReporter(metricRegistry); | ||
| } | ||
| } | ||
| ``` | ||
| Option | Description | Default value | ||
| ------------------------------------------|-----------------------------------------------| ------------- | ||
| prometheus.client.enabled | Should fetch external metrics from Prometheus | true | ||
| prometheus.client.externalMonitoringUrl | URI to VictoriaMetrics HTTP API | http://localhost:18090 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
...rc/main/java/pl/allegro/tech/hermes/common/di/factories/MicrometerRegistryParameters.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,12 @@ | ||
| package pl.allegro.tech.hermes.common.di.factories; | ||
|
|
||
| import java.time.Duration; | ||
| import java.util.List; | ||
|
|
||
| public interface MicrometerRegistryParameters { | ||
| List<Double> getPercentiles(); | ||
|
|
||
| boolean zookeeperReporterEnabled(); | ||
|
|
||
| Duration zookeeperReportPeriod(); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
76 changes: 73 additions & 3 deletions
76
hermes-common/src/main/java/pl/allegro/tech/hermes/common/metric/ConsumerMetrics.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +1,91 @@ | ||
| package pl.allegro.tech.hermes.common.metric; | ||
|
|
||
| import io.micrometer.core.instrument.MeterRegistry; | ||
| import io.micrometer.core.instrument.Tags; | ||
| import pl.allegro.tech.hermes.api.Subscription; | ||
| import pl.allegro.tech.hermes.metrics.HermesCounter; | ||
| import pl.allegro.tech.hermes.metrics.HermesTimer; | ||
| import pl.allegro.tech.hermes.metrics.counters.HermesCounters; | ||
|
|
||
| import java.util.function.ToDoubleFunction; | ||
|
|
||
| import static pl.allegro.tech.hermes.common.metric.Gauges.BATCH_BUFFER_AVAILABLE_BYTES; | ||
| import static pl.allegro.tech.hermes.common.metric.Gauges.BATCH_BUFFER_TOTAL_BYTES; | ||
| import static pl.allegro.tech.hermes.common.metric.Gauges.THREADS; | ||
| import static pl.allegro.tech.hermes.common.metric.SubscriptionTagsFactory.subscriptionTags; | ||
|
|
||
| public class ConsumerMetrics { | ||
| private final HermesMetrics hermesMetrics; | ||
| private final MeterRegistry meterRegistry; | ||
| private final GaugeRegistrar gaugeRegistrar; | ||
|
|
||
| public ConsumerMetrics(HermesMetrics hermesMetrics, MeterRegistry meterRegistry) { | ||
| this.hermesMetrics = hermesMetrics; | ||
| this.meterRegistry = meterRegistry; | ||
| this.gaugeRegistrar = new GaugeRegistrar(meterRegistry, hermesMetrics); | ||
| } | ||
|
|
||
| public <T> void registerQueueUtilizationGauge(T obj, String queueName, ToDoubleFunction<T> f) { | ||
| gaugeRegistrar.registerGauge("queue." + queueName + ".utilization", obj, f); | ||
| } | ||
|
|
||
| public HermesCounter queueFailuresCounter(String name) { | ||
| return HermesCounters.from( | ||
| meterRegistry.counter("queue." + name + ".failures"), | ||
| hermesMetrics.counter("queue." + name + ".failures") | ||
| ); | ||
| } | ||
|
|
||
| public <T> void registerConsumerProcessesThreadsGauge(T obj, ToDoubleFunction<T> f) { | ||
| gaugeRegistrar.registerGauge(THREADS, "consumer-processes.threads", obj, f); | ||
| } | ||
|
|
||
| public <T> void registerRunningConsumerProcessesGauge(T obj, ToDoubleFunction<T> f) { | ||
| hermesMetrics.registerRunningConsumerProcessesCountGauge(() -> (int) f.applyAsDouble(obj)); | ||
| meterRegistry.gauge("consumer-processes.running", obj, f); | ||
| } | ||
|
|
||
| public <T> void registerDyingConsumerProcessesGauge(T obj, ToDoubleFunction<T> f) { | ||
| hermesMetrics.registerDyingConsumerProcessesCountGauge(() -> (int) f.applyAsDouble(obj)); | ||
| meterRegistry.gauge("consumer-processes.dying", obj, f); | ||
| } | ||
|
|
||
| public HermesTimer offsetCommitterDuration() { | ||
| public <T> void registerBatchBufferTotalBytesGauge(T obj, ToDoubleFunction<T> f) { | ||
| gaugeRegistrar.registerGauge(BATCH_BUFFER_TOTAL_BYTES, "batch-buffer.total-bytes", obj, f); | ||
| } | ||
|
|
||
| public <T> void registerBatchBufferAvailableBytesGauge(T obj, ToDoubleFunction<T> f) { | ||
| gaugeRegistrar.registerGauge(BATCH_BUFFER_AVAILABLE_BYTES, "batch-buffer.available-bytes", obj, f); | ||
| } | ||
|
|
||
| public HermesCounter oAuthSubscriptionTokenRequestCounter(Subscription subscription, String providerName) { | ||
| return HermesCounters.from( | ||
| meterRegistry.counter("oauth.token-requests", Tags.concat( | ||
| subscriptionTags(subscription.getQualifiedName()), | ||
| "provider", providerName | ||
| )), | ||
| hermesMetrics.oAuthSubscriptionTokenRequestMeter(subscription, providerName) | ||
| ); | ||
| } | ||
|
|
||
| public HermesTimer oAuthProviderLatencyTimer(String providerName) { | ||
| return HermesTimer.from( | ||
| meterRegistry.timer("offset-committer.duration"), | ||
| hermesMetrics.timer("offset-committer.duration") | ||
| meterRegistry.timer("oauth.token-request-latency", Tags.of("provider", providerName)), | ||
| hermesMetrics.oAuthProviderLatencyTimer(providerName) | ||
| ); | ||
| } | ||
|
|
||
| public HermesCounter processedSignalsCounter(String name) { | ||
| return HermesCounters.from( | ||
| meterRegistry.counter("signals.processed", Tags.of("signal", name)), | ||
| hermesMetrics.counter("supervisor.signal." + name) | ||
| ); | ||
| } | ||
|
|
||
| public HermesCounter droppedSignalsCounter(String name) { | ||
| return HermesCounters.from( | ||
| meterRegistry.counter("signals.dropped", Tags.of("signal", name)), | ||
| hermesMetrics.counter("supervisor.signal.dropped." + name) | ||
| ); | ||
| } | ||
| } |
Oops, something went wrong.