- [BUGFIX] Fixed a regression that caused the ThanosRuler statefulsets to be stuck after upgrading the operator to v0.61.0. #5183
- [CHANGE] Updated
RuleGroup
description and add validation for the CRD. #5072 - [CHANGE] Removed validations in the operator that are already covered at the CRD level. #5108
- [CHANGE] jsonnet: Enforced existence of the TLS secret for the admission webhook deployment. #5112
- [CHANGE] jsonnet: Changed default port of the admission webhook service from 8443 to 443. #5112
- [CHANGE] Added a filter for non-running pods in the
ServiceMonitor
CRD. #5149 - [FEATURE] Added
spec.attachMetadata.node
in theServiceMonitor
CRD. #5147 - [ENHANCEMENT] Updated
ProbeTLSConfig
andSafeTLSConfig
description. #5081 - [ENHANCEMENT] Updated admission webhook deployment's jsonnet to avoid down-time on updates. #5099
- [ENHANCEMENT] Added the
filterExternalLabels
field to the remote read configuration of thePrometheus
CRD. #5142 - [ENHANCEMENT] Added
enableHttp2
field toAlertingEndpoints
#5152 - [ENHANCEMENT] Updated
ThanosRuler
arguments (QueryConfig
,AlertManagerConfig
,ObjectStorageConfig
andTracingConfig
) to be directly read from secrets instead of using ENV vars. #5122 - [ENHANCEMENT] Add
alertmanagerConfigMatcherStrategy
toAlertmanager
CRD in order to disable auto-generated namespace matchers. #5084 - [BUGFIX] Ignore
PartialResponseStrategy
in thePrometheus
CRD. This field is only applicable for the Thanos Ruler. #5125
- [BUGFIX] Fixed configuration when
spec.tsdb.outOfOrderTimeWindow
is set in the Prometheus CRD. #5078
- [CHANGE] Added
filterRunning
field to the PodMonitor CRD. By default, non-running pods are dropped by the Prometheus service discovery. To preserve the old behavior and keep pods which aren't running, setfilterRunning: false
. #5049 - [FEATURE] Added
grpcListenLocal
andhttpListenLocal
fields to the Thanos sidecar configuration of the Prometheus CRD. #5045 - [FEATURE] Added
hostNetwork
field to the Prometheus CRD. #5010 - [FEATURE] Added
spec.tsdb.outOfOrderTimeWindow
field to the Prometheus CRD to allow out-of-order samples in TSDB. #5071 - [ENHANCEMENT] Added columns showing the Prometheus conditions to the output of
kubectl get prometheus
. #5055 - [ENHANCEMENT] Added
observedGeneration
field to the Prometheus status conditions. #5005
- [CHANGE/BUGFIX] Removed
FOWNER
capability from the Thanos sidecar. #5030
- [BUGFIX] Fixed secret and configmap volume names that need to be mounted in additional containers. #5000
- [BUGFIX] Removed
CAP_FOWNER
capability for the Thanos sidecar when not required. #5004 - [BUGFIX] Removed the
CAP_
prefix of theFOWNER
capability on Thanos sidecar. #5014
- [FEATURE] Added validations for timeout and time settings of alertmanager at CRD level. #4898
- [FEATURE] Added support for global
resolveTimeout
andhttpConfig
in Alertmanager CRD. #4622 - [FEATURE] Added support for
additionalArgs
field to the Prometheus CRD for Prometheus, Alertmanager and Thanos sidecar. #4863 - [ENHANCEMENT] Added
tracingConfigFile
option to ThanosRuler CRD. #4962 - [BUGFIX] Fixed compress alertmanager secret to circumvent maximum size limit of 1048576 bytes. #4906
- [BUGFIX] Fixed namespace enforcement exclusion on newly created Prometheus objects. #4915
- [BUGFIX] Fixed
CAP_FOWNER
capability to Thanos sidecar container. #4931 - [BUGFIX] Fixed
spec.query.maxSamples
andspec.query.maxConcurrency
fields of Prometheus CRD. #4951 - [BUGFIX] Fixed Thanos sidecar connectivity issue when Prometheus TLS is enabled. #4954
- [BUGFIX] Fixed Prometheus and Alertmanager Pods not created when Secret name exceeds 63 characters. #4988
- [FEATURE] Add validations for timeout and time settings of alertmanager at CRD level. #4827, #4881
- [FEATURE] Extend the PrometheusSpec to allow to configure the
max_exemplars
. #4834 - [FEATURE] Add support for web TLS configuration for Alertmanager CRD. #4868
- [ENHANCEMENT] Add support for
uppercase
,lowercase
, andCamelCase
relabel actions. #4840, #4873 - [ENHANCEMENT] Added support for
enable_http2
in endpoint scrape configuration. #4836 - [BUGFIX] Fixed missing conversion of the
followRedirects
field in HTTP configuration for AlertmanagerConfig v1beta1. #4854 - [BUGFIX] fix AlertmanagerConfig.Spec.Route nil panic. #4853
- [BUGFIX] Optimise warning log message during sanitization of OpsGenie configuration. #4833
The main change introduced by this release is a new v1beta1 API version for the AlertmanagerConfig CRD.
Changes compared to the v1alpha1 API:
- Renamed
spec.muteTimeIntervals
field toto spec.timeIntervals
. - Removed
regex
field from theMatcher
type. - Replaced all
v1.SecretKeySelector
types by theSecretKeySelector
type- Removed
optional
field. name
andkey
fields are required.
- Removed
As a pre-requisite, you need to deploy the admission webhook and configure the
conversion webhook in the AlertmanagerConfig CRD object so that users can use
both v1alpha1 and v1beta1 versions at the same time. There are more details in
Documentation/user-guides/webhook.md
about the webhook configuration.
Because of the conversion webhook requirement, the new version is an opt-in
feature: the bundle.yaml
file and the manifests from
example/prometheus-operator-crd
don't deploy the new API version (the
manifests to enable the v1beta1 version are under the
example/prometheus-operator-crd-full
directory). We will wait until v0.59.0
(at least) before enabling the new API version by default.
- [CHANGE] Added validations at the API level for the time-based fields of the ThanosRuler CRD. #4815
- [CHANGE] Added validations at the API level for the OpsGenie's
responders
field of the AlertmanagerConfig CRD. #4725 - [FEATURE] Added v1beta1 version for AlertmanagerConfig CRD. #4709
- [FEATURE] Added support for Telegram receiver in the AlertmanagerConfig CRD. #4726
- [FEATURE] Added
updateAlerts
field to the OpsGenie configuration of the AlertmanagerConfig CRD. #4726 - [FEATURE] Added
hostAliases
field to the the Alertmanager, Prometheus and ThanosRuler CRDs. #4787 - [ENHANCEMENT] Added configuration option in the jsonnet mixins to specify the aggregation labels. #4814
- [ENHANCEMENT] Added
attachMetadata
field to the PodMonitor CRD. #4792 - [BUGFIX] Fixed the curl command for exec probes when
listenLocal
is set to true in the Prometheus object. It avoids temporary service outage due to long WAL replays. #4804
- [BUGFIX] Fixed errors for Alertmanager configurations using the new
entity
,actions
andopsgenie_api_key_file
fields. #4797 - [BUGFIX] Fixed high CPU usage by reducing the number of number of reconciliations on Prometheus objects. #4798 #4806
- [BUGFIX] Fix StatefulSet spec's generation to be determistic when
spec.containers
is not empty. #4772
- [BUGFIX] Avoid unnecessary updates of the Prometheus StatefulSet object. #4762
- [CHANGE] Added validation at the API level for size-based fields of the Prometheus CRD. #4661
- [CHANGE] Added validation at the API level for log level and format fields of the Alertmanager, Prometheus and ThanosRuler CRDs. #4638
- [CHANGE] Added validation at the API level for duration and time-based fields of the Prometheus CRD. #4684
- [CHANGE] Added shortnames for custom resources (
amcfg
for AlertmanagerConfig,am
for Alertmanager,pmon
for PodMonitor,prb
for Probe,prom
for Prometheus,smon
for ServiceMonitor,ruler
for Thanos Ruler). #4680 - [FEATURE] Added
status
subresource to the Prometheus CRD. #4580 - [ENHANCEMENT] Added
excludedFromEnforce
field to the Prometheus CRD. It allows to define PodMonitor, ServiceMonitor, Probe or PrometheusRule objects for which the namespace label enforcement (if enabled) should not be applied. This deprecatesprometheusRulesExcludedFromEnforce
which is still supported but users are encouraged to migrate to the new field. #4397 - [ENHANCEMENT] Added
enableRemoteWriteReceiver
field to the Prometheus CRD. #4633 - [ENHANCEMENT] Added
entity
andactions
fields for the OpsGenie receiver to the AlertmanagerConfig CRD. #4697 - [ENHANCEMENT] Added
prometheus_operator_reconcile_duration_seconds
histogram metric. #4706 - [BUGFIX] Added support for
opsgenie_api_key_file
andapi_key_file
in the generated Alertmanager configuration. #4666 #4738
- [BUGFIX] Fixed Prometheus configuration when
spec.queryLogFile
has no path ("query.log" for instance). #4683
- [CHANGE] Enabled read-only root filesystem for containers generated from the Prometheus, Alertmanager and ThanosRuler objects. #4552
- [CHANGE] Disabled privilege escalation for the containers generated from Prometheus, Alertmanager and ThanosRuler objects. #4552
- [CHANGE] Dropped all capabilities for the containers generated from Prometheus, Alertmanager and ThanosRuler objects. #4552
- [CHANGE] Added
emptyDir
volume to the Prometheus statefulset whenspec.queryLogFile
is only a base filename (e.g.query.log
as opposed to/tmp/query.log
). When the path contains a full path, a volume + volume mount should be explicitly given in the Prometheus spec since the root file system is now read-only. #4566 - [CHANGE/BUGFIX] Added skip TLS verify for the config-reloader HTTP client when informing Prometheus/Alertmanager on a config reload (partial fix for #4273). #4592
- [CHANGE] Switched using the
endpointslice
role for Prometheus by default if it is supported by the Kubernetes API. #4535 - [FEATURE] Added standalone admission webhook. #4494
- [FEATURE] Support the definition of Alertmanager configuration via AlertManagerConfig instead of Kubernetes secret (EXPERIMENTAL). #4220
- [FEATURE] Support sharding for Probe objects. #4587
- [ENHANCEMENT] Restore Prometheus StatefulSet liveness probe so that deadlocks are detected and recovered from. #4387, #4534
- [ENHANCEMENT] Added
-alertmanager-config-namespaces
CLI flag to the operator. #4455, #4619 - [ENHANCEMENT]
remoteWrite
andremoteRead
fields of the Prometheus CRD not considered experimental anymore. #4555 - [ENHANCEMENT] Added support for follow_redirects in endpoint scrape configuration. #4563
- [ENHANCEMENT] Added support for Oauth2 in AlertmanagerConfig CRD. #4501
- [ENHANCEMENT] Improved logging when the given Prometheus version doesn't support some CR fields. #4571
- [ENHANCEMENT] Added
__tmp_ingress_address
label to preserve the initial host address of the ingress object. #4603 - [ENHANCEMENT] Fixed potential name collisions in Alertmanager configuration when merging AlertmanagerConfig objects. #4626
- [BUGFIX] Fixed panic when validating
Probe
. #4541 - [BUGFIX] Added validation for sourceLabels in relabel configuration. #4568
- [BUGFIX] Allow retention to be set only by size. #4590
- [BUGFIX] Updated relabelConfig validation to accept Prometheus default config on labeldrop relabelConfig. #4579
- [BUGFIX] Fixed relabelConfigs for labelmap action. #4574
- [FEATURE] Support SNS Receiver in AlertmanagerConfig CR. #4468
- [ENHANCEMENT] Specify SA token automounting on pod-level for operator and prometheus operand. #4514
- [ENHANCEMENT] Support following redirects and Oauth2 in HTTP Client config in raw alertmanager config secret. #4499
- [ENHANCEMENT] Add Replicas column for Thanos Ruler. #4496
- [ENHANCEMENT] Set User-Agent for the kubernetes client. #4506
- [BUGFIX] Avoid race during recreation of StatefulSet(s). #4504
- [BUGFIX] Add validation for proberSpec
url
field inProbeSpec
. #4483 - [BUGFIX] Add validation for relabel configs. #4429
- [BUGFIX] Add validation for scrapeTimeout validation. #4491
- [BUGFIX] Fixed the validation pattern for the
february
month in the AlertManagerConfig CRD. #4458
- [CHANGE] Added startup probe to Prometheus. #4433 #4369
- [FEATURE] Added support for mute time intervals in the AlertManagerConfig CRD. #4388
- [FEATURE] Added support for new matching syntax in the routes configuration of the AlertmanagerConfig CRD. #4332
- [FEATURE] Added support for new matching syntax in the inhibit rules configuration of the AlertmanagerConfig CRD. #4329
- [FEATURE] Added
headers
in the Remote Read configuration of the Prometheus CRD. #4323 - [FEATURE] Added
retryOnRateLimit
in the Remote Write configuration of the Prometheus CRD. #4420 - [FEATURE] Added support for PagerDuty links and images in the AlertManagerConfig CR. #4425
- [ENHANCEMENT] Optimized the generated Prometheus configuration to make it compatible with the new Prometheus agent mode. #4417
- [BUGFIX] Improved validation for the Alertmanager CRD to match with upstream Alertmanager. #4434
- [BUGFIX] Fixed propagation of annotations from spec.podMetadata to the StatefulSet's pods. #4422
- [BUGFIX] Ensured that
group_by
values are unique in the generated Alertmanager configuration. #4413 - [BUGFIX] Fixed generated secrets being larger than the maximum size limit of 1048576 bytes. #4427 #4449
- [BUGFIX] Fixed regex in relabel_configs. #4395
- [CHANGE] Extend sharding capabilities to
additionalScrapeConfigs
. #4324 - [CHANGE] Remove
app
label from Prometheus, Alertmanager and Thanos Ruler statefulsets/pods. #4350 - [FEATURE] Add
alertRelabelConfigs
field to the Thanos Ruler CRD for configuring Prometheus alert relabeling features. #4303 - [FEATURE] Add support for updated matching syntax in Alertmanager's raw config for
inhibit_rules
androute
. #4307, #4309 - [FEATURE] Add validating webhook for AlertManagerConfig. #4338
- [FEATURE] Adds support for Sigv4 when configuring RemoteWrite. #3994
- [ENHANCEMENT] Add "generic ephemeral storage" as a data storage option for Alertmanager, Prometheus and Thanos Ruler. #4326
- [ENHANCEMENT] Improve docs and error message for "smarthost" field. #4299
- [ENHANCEMENT] Add alerts for config reloader sidecars. #4294
- [ENHANCEMENT] Add validations for duration and size fields for Prometheus, Alertmanager, and Thanos Ruler resources #4308, #4352
- [ENHANCEMENT] Add s390x support to docker images. #4351
- [ENHANCEMENT] Only load alertmanager configuration when writing configuration. #4333
- [BUGFIX] Fix
matchLabels
selector to have empty label values in ServiceMonitor, PodMonitor, and Probe. #4327 - [BUGFIX] Prevent rule file name collision. #4347
- [BUGFIX] Update native kubernetes fields used in prometheus-operator CRDs. #4221
- [BUGFIX] Validated the value of the
EnforcedBodySizeLimit
field to avoid Prometheus crash. #4285
No change since v0.51.0.
The CI automation failed to build the v0.51.0 images so we had to create a new patch release.
- [FEATURE] Added
metricRelabelings
field to the Probe CRD for configuring the metric relabel configs. #4226 - [FEATURE] Added
volumeMounts
field to the Prometheus CRD for configuring the volume mounts of the thanos-sidecar container. #4238 - [FEATURE] Added
enforcedBodySizeLimit
field to the Prometheus CRD. #4275 - [FEATURE] Added
authorization
field to all HTTP configurations in the AlertmanagerConfig CRD. #4110 - [FEATURE] Added
minReadySeconds
field to AlertManager, Prometheus and ThanosRuler CRDs. #4246 - [FEATURE] Added support for Slack webhook URL via file path in the Alertmanager configuration secret. #4234
- [FEATURE] Added support the
authorization
field for all HTTP configurations in the Alertmanager configuration secret. #4234 - [ENHANCEMENT] Improved detection and rollback of manual changes to Alertmanager statefulsets. #4228
- [BUGFIX] Invalid probes are discarded instead of stopping after the first error when reconciling probes. #4248
- [BUGFIX] Empty basic auth username is allowed in the AlertmanagerConfig CRD. #4260
- [BUGFIX] Update conflicts for secrets are handled properly which avoids overwriting user-defined metadata. #4235
- [BUGFIX] The namespace label is always enforced with metricRelabelConfigs. #4272
- [CHANGE] Remove deprecated flags
--config-reloader-memory
and--config-reloader-cpu
in favor of--config-reloader-memory-limit
,--config-reloader-memory-request
,--config-reloader-cpu-limit
, and--config-reloader-cpu-request
. #3884 - [CHANGE] Remove use of Kubernetes API versions being removed in v1.22. #4171
- [FEATURE] Added support for OAuth2 authentication in remote read and remote write configuration. #4113
- [FEATURE] Added OAuth2 configuration for ServiceMonitor, PodMonitor and Probe. #4170
- [FEATURE] Added
prometheus_operator_spec_shards
metric for exposing the number of shards set on prometheus operator spec. #4173 - [FEATURE] Support for
Authorization
section in various prometheus sections. #4180 - [FEATURE] Validate prometheus rules when generating rule file content. #4184
- [FEATURE] Support
label_limit
,label_name_length_limit
andlabel_value_length_limit
configuration fields at the Prometheus CRD level as well as support individual limits per ServiceMonitor, PodMonitor and Probe resources. #4195 - [FEATURE] Added sample and target limits to Probe. #4207
- [FEATURE] Added
send_exemplars
field to theremote_write
configuration in Prometheus. #4215 #4160 - [ENHANCEMENT] Support loading ClusterConfig from concatenated KUBECONFIG env. #4154
- [ENHANCEMENT] Include PrometheusRule in prometheus-operator CRD category. #4213
- [ENHANCEMENT] Preserve annotations set by kubectl. #4185
- [BUGFIX] Thanos: listen to all available IP addresses instead of
POD_IP
, simplifies istio management. #4038 - [BUGFIX] Add port name mapping to ConfigReloader to avoid reloader-web probe failure. #4187
- [BUGFIX] Handle Thanos rules
partial_response_strategy
field in validating admission webhook. #4217
- [CHANGE] Flag "storage.tsdb.no-lockfile" will now default to false. #4066
- [CHANGE] Remove
app.kubernetes.io/version
label selector from Prometheus and Alertmanager statefulsets. #4093 - [CHANGE] Exit if the informers cache synchronization doesn't complete after 10 minutes. #4143, #4149
- [FEATURE] Added web TLS configuration support for Prometheus. #4025
- [FEATURE] Add proxy_url support for Probes. #4043
- [ENHANCEMENT] Set proper build context in version package. #4019
- [ENHANCEMENT] Publish images on GitHub Container Registry. #4060
- [ENHANCEMENT] Automatically generate document for operator executable. #4112
- [ENHANCEMENT] Adds configuration to set the Prometheus ready timeout to Thanos sidecar #4118
- [BUGFIX] Fixed bug in Alertmanager config where URLS that are taken from Kubernetes secrets might contain whitespace or newline characters. #4068
- [BUGFIX] Generate correct scraping configuration for Probes with empty or unset
module
parameter. #4074 - [BUGFIX] Operator does not generate
max_retries
option inremote_write
for Prometheus version 2.11.0 and higher. #4103 - [BUGFIX] Preserve the dual-stack immutable fields on service sync. #4119
- [BUGFIX] Added an
app
label on Prometheus pods. #4055
Deprecation notice: app labels will be removed in v0.50.
- [CHANGE] Replace app label names with app.kubernetes.io/name. #3939
- [CHANGE] Drop ksonnet as a dependency in jsonnetfile.json. #4002
- [ENHANCEMENT] Add default container annotation to Alertmanager pod. #3978
- [ENHANCEMENT] Add default container annotation to Thanos ruler pod. #3981
- [ENHANCEMENT] Optimize asset secret update logic. #3986
- [ENHANCEMENT] jsonnet: set default container in prometheus-operator pod. #3979
- [BUGFIX] Watch configmaps from the Prometheus allowed namespaces only. #3992
- [BUGFIX] Reconcile resources on namespace updates when using privileged lister/watcher. #3879
- [BUGFIX] Don't generate broken Alertmanager configuration when
http_config
is defined in the global parameters. #4041
- [BUGFIX] Avoid reconciliations for Alertmanager statefulset on resource version changes. #3948
The --config-reloader-cpu
and --config-reloader-memory
flags are deprecated
and will be removed in v0.49.0. They are replaced respectively by the
--config-reloader-cpu-request
/--config-reloader-cpu-limit
and
config-reloader-memory-request
/config-reloader-memory-limit
flags.
- [FEATURE] Add
enableFeatures
field to the Prometheus CRD for enabling feature flags. #3878 - [FEATURE] Add
metadataConfig
field to the Prometheus CRD for configuring how remote-write sends metadata information. #3915 - [FEATURE] Add support for TLS and authentication configuration to the Probe CRD. #3876
- [ENHANCEMENT] Allow CPU requests/limits and memory requests/limits of the config reloader to be set independently. #3826
- [ENHANCEMENT] Add rules validation to
po-lint
. #3894 - [ENHANCEMENT] Add common Kubernetes labels to statefulset objects managed by the Prometheus operator. #3841
- [ENHANCEMENT] Avoid unneeded synchronizations on Alertmanager updates. #3943
- [ENHANCEMENT] Retain the original job's name
__tmp_prometheus_job_name label
in the generated scrape configurations. #3828 - [BUGFIX] Fix
app.kubernetes.io/managed-by
label on kubelet endpoint object. #3902 - [BUGFIX] Avoid name collisions in the generated Prometheus configuration. #3913
- [BUGFIX] Restore
prometheus_operator_spec_replicas
metrics for Alertmanager and Thanos Ruler controllers. #3924 - [BUGFIX] Allow
smtp_require_tls
to be false in Alertmanager configuration. #3960
- [CHANGE] Drop support for Prometheus 1.x #2822
- [FEATURE] Add relabelingConfigs to ProbeTargetStaticConfig #3817
- [ENHANCEMENT] Add custom HTTP headers in remoteWrite-config #3851
- [ENHANCEMENT] CRDs are now part of prometheus-operator group allowing
kubectl get prometheus-operator
operation #3843 - [ENHANCEMENT] Support app.kubernetes.io/managed-by label to kubelet service and endpoint objects. #3834
- [BUGFIX] Fix the loss of the
headers
key in AlertmanagerConfig. #3856 - [BUGFIX] Preserve user-added labels and annotations on Service, Endpoint and StatefulSet resources managed by the operator. #3810
- [BUGFIX] Do not require child routes in AlertmanagerConfig to have a receiver. #3749
- [CHANGE] Add schema validations to AlertmanagerConfig CRD. #3742
- [CHANGE] Refactored jsonnet library to remove ksonnet and align with kube-prometheus #3781
- [ENHANCEMENT] Add
app.kubernetes.io/name
label to Kubelet Service/Endpoints object. #3768 - [ENHANCEMENT] Improve HTTP server's logging #3772
- [ENHANCEMENT] Add namespace label to static probe metrics #3752
- [ENHANCEMENT] Add
TracingConfigFile
field into thanos configuration. #3762 - [BUGFIX] Fix log messages when syncing node endpoints. #3758
- [BUGFIX] fix discovery of
AlertmanagerConfig
resources when--alertmanager-instance-namespaces
is defined. #3759
- [BUGFIX] Fix Alertmanager configuration for OpsGenie receiver. #3728
- [CHANGE] Fix child routes support in AlertmanagerConfig. #3703
- [FEATURE] Add Slack receiver type to AlertmanagerConfig CRD. #3618
- [FEATURE] Add WeChat receiver type to AlertmanagerConfig CRD. #3619
- [FEATURE] Add Email receiver type to AlertmanagerConfig CRD. #3692
- [FEATURE] Add Pushover receiver type to AlertmanagerConfig CRD. #3697
- [FEATURE] Add VictorOps receiver type to AlertmanagerConfig CRD. #3701
- [FEATURE] Add sharding support for prometheus cluster. #3241
- [ENHANCEMENT] Add option to allow configuring object storage for Thanos. #3668
- [ENHANCEMENT] Add TLS support for remote read. #3714
- [ENHANCEMENT] Include EnforcedSampleLimit as a metric. #3617
- [ENHANCEMENT] Adjust config reloader memory requests and limits. #3660
- [ENHANCEMENT] Add
clusterGossipInterval
,clusterPushpullInterval
andclusterPeerTimeout
fields to Alertmanager CRD. #3663 - [BUGFIX] Handle all possible receiver types in AlertmanagerConfig. #3689
- [BUGFIX] Fix operator crashing on empty Probe targets. #3637
- [BUGFIX] Fix usage of
--prometheus-default-base-image
,--alertmanager-default-base-image
, and--thanos-default-base-image
flags. #3642 - [BUGFIX] Fix matching labels with empty values when using Exists/NotExists operators. #3686
- [BUGFIX] Fix issue with additional data from the Alertmanager config's secret not being kept. #3647
- [BUGFIX] Fix Alertmanager controller to wait for all informers to be synced before reconciling. #3641
This release introduces a new AlertmanagerConfig
CRD that allows to split the
Alertmanager configuration in different objects. For now the CRD only supports
the PagerDuty, OpsGenie and webhook receivers, other
integrations
will follow in future releases of the operator. The current version of the CRD
is v1alpha1
meaning that testing/feedback is encouraged and welcome but the
feature is not yet considered stable and the API is subject to change in the
future.
- [CHANGE] Use a single reloader sidecar (instead of 2) for Prometheus. The
--config-reloader-image
flag is deprecated and will be removed in a future release (not before v0.45.0). Make sure to start the operator with a version of--prometheus-config-reloader
that is at leastv0.43.0
otherwise the Prometheus pods will fail to start. #3457 - [FEATURE] Add
targetLimit
andenforcedTargetLimit
to the Prometheus CRD. #3571 - [FEATURE] Add initial support for
AlertmanagerConfig
CRD. #3451 - [FEATURE] Add support for Pod Topology Spread Constraints to Prometheus, Alertmanager, and ThanosRuler CRDs. #3598
- [ENHANCEMENT] Allow customization of the Prometheus web page title. #3525
- [ENHANCEMENT] Add metrics for selected/rejected resources and synchronization status. #3421
- [ENHANCEMENT] Configure Thanos sidecar for uploads only when needed. #3485
- [ENHANCEMENT] Add
--version
flag to all binaries +prometheus_operator_build_info
metric. #359 - [ENHANCEMENT] Add
prometheus_operator_prometheus_enforced_sample_limit
metric. #3617 - [BUGFIX] Remove liveness probes to avoid killing Prometheus during the replay of the WAL. #3502
- [BUGFIX] Fix
spec.ipFamily: Invalid value: "null": field is immutable
error when updating governing services. #3526 - [BUGFIX] Generate more unique job names for Probes. #3481
- [BUGFIX] Don't block when the operator is configured to watch namespaces that don't exist yet. #3545
- [BUGFIX] Use
exec
in readiness probes to reduce the chance of leaking zombie processes. #3567 - [BUGFIX] Fix broken AdmissionReview. #3574
- [BUGFIX] Fix reconciliation when 2 resources share the same secret. #3590
- [BUGFIX] Discard invalid TLS configurations. #3578
- [BUGFIX] Bump client-go to fix watch bug
The Prometheus Operator now lives in its own independent GitHub organization.
We have also added a governance (#3398).
- [FEATURE] Move API types out into their own module (#3395)
- [FEATURE] Create a monitoring mixin for prometheus-operator (#3333)
- [ENHANCEMENT] Remove multilistwatcher and denylistfilter (#3440)
- [ENHANCEMENT] Instrument client-go requests (#3465)
- [ENHANCEMENT] pkg/prometheus: skip invalid service monitors (#3445)
- [ENHANCEMENT] pkg/alertmanager: Use lower value for --cluster.reconnect-timeout (#3436)
- [ENHANCEMENT] pkg/alertmanager: cleanup resources via OwnerReferences (#3423)
- [ENHANCEMENT] Add prometheus_operator_reconcile_operations_total metric (#3415)
- [ENHANCEMENT] pkg/operator/image.go: Adjust image path building (#3392)
- [ENHANCEMENT] Specify timeouts per Alertmanager target when sending alerts. (#3385)
- [ENHANCEMENT] Push container images to Quay into coreos and prometheus-operator orgs (#3390)
- [ENHANCEMENT] Run single replica Alertmanager in HA cluster mode (#3382)
- [BUGFIX] Fix validation logic for SecretOrConfigMap (#3413)
- [BUGFIX] Don't overwrite __param_target (#3377)
- [BUGFIX] Fix image url logic (#3402)
- [CHANGE] Configmap-reload: Update to v0.4.0 (#3334)
- [CHANGE] Update prometheus compatibility matrix to v2.19.2 (#3316)
- [FEATURE] Add Synthetic Probes support. This includes support for job names. (#2832, #3318, #3312, #3306)
- [FEATURE] Support Prometheus vertical compaction (#3281)
- [ENHANCEMENT] pkg: Instrument resources being tracked by the operator (#3360)
- [ENHANCEMENT] Add SecretListWatchSelector to reduce memory and CPU footprint (#3355)
- [ENHANCEMENT] Added support for configuring CA, cert, and key via secret or configmap. (#3249)
- [ENHANCEMENT] Consolidate image url logic, deprecating
baseImage
,sha
, andtag
in favor ofimage
field in CRDs. (#3103, #3358) - [ENHANCEMENT] Decouple alertmanager pod labels from selector labels (#3317)
- [ENHANCEMENT] pkg/prometheus: Ensure relabeling of container label in ServiceMonitors (#3315)
- [ENHANCEMENT] misc: Remove v1beta1 crd remainings (#3311)
- [ENHANCEMENT] Normalize default durations (#3308)
- [ENHANCEMENT] pkg/prometheus: Allow enforcing namespace label in Probe configs (#3304)
- [BUGFIX] Revert "Normalize default durations" (#3364)
- [BUGFIX] Reload alertmanager on configmap/secret change (#3319)
- [BUGFIX] listwatch: Do not duplicate resource versions (#3373)
- [CHANGE] Update dependencies to prometheus 2.18 (#3231)
- [CHANGE] Add support for new prometheus versions (v2.18 & v2.19) (#3284)
- [CHANGE] bump Alertmanager default version to v0.21.0 (#3286)
- [FEATURE] Automatically disable high availability mode for 1 replica alertmanager (#3233)
- [FEATURE] thanos-sidecar: Add minTime arg (#3253)
- [FEATURE] Add scrapeTimeout as global configurable parameter (#3250)
- [FEATURE] Add EnforcedSampleLimit which enforces a global sample limit (#3276)
- [FEATURE] add ability to exclude rules from namespace label enforcement (#3207)
- [BUGFIX] thanos sidecar: log flags double definition (#3242)
- [BUGFIX] Mutate rule labels, annotations to strings (#3230)
- [CHANGE] Introduce release schedule (#3135)
- [CHANGE] Remove options configuring CRD management (manage-crds, crd-kinds, with-validation) (#3155)
- [CHANGE] Add CRD definitions to bundle.yaml (#3171)
- [CHANGE] Switch to apiextensions.k8s.io/v1 CRD and require kubernetes v1.16 or newer (#3175, #3187)
- [FEATURE] Add support prometheus query log file (#3116)
- [FEATURE] Add support for watching specified rules directory by config-relader (#3128)
- [FEATURE] Add TLS support for operator web server (#3134, #3157)
- [FEATURE] Allow to set address for operator http endpoint (#3098)
- [FEATURE] Allow setting the alertmanagers cluster.advertiseAddress (#3160)
- [FEATURE] Build operator images for ARM and ARM64 platforms (#3177)
- [ENHANCEMENT] Allow setting log level and format for thanos sidecar (#3112)
- [ENHANCEMENT] Support naming of remote write queues (#3144)
- [ENHANCEMENT] Allow disabling mount subPath for volumes (#3143)
- [ENHANCEMENT] Update k8s libraries to v1.18 (#3154)
- [ENHANCEMENT] Create separate namespace informers when needed (#3182)
- [BUGFIX] Tolerate version strings which aren't following semver (#3101)
- [BUGFIX] Retain metadata for embedded PVC objects (#3115)
- [BUGFIX] Fix definition of thanos-ruler-operated service (#3126)
- [BUGFIX] Allow setting the cluster domain (#3138)
- [BUGFIX] Allow matching only PodMonitors (#3173)
- [BUGFIX] Fix typo in statefulset informer (#3179)
- [BUGFIX] Fix definition of web service port for Alertmanager (#3125)
- [BUGFIX] Support external alert query URL for THanos Ruler (#3129)
- [BUGFIX] Do not modify the PrometheusRule cache object (#3105)
- [CHANGE] Changed ThanosRuler custom resource field alertmanagersURL type from string to []string (#3067)
- [CHANGE] Deprecate PodMonitor targetPort field (#3071, #3078)
- [FEATURE] Add queryConfig field to ThanosRuler spec (#3068)
- [FEATURE] GRPC TLS config for Thanos Ruler and Sidecar (#3059)
- [FEATURE] MergePatch Alertmanager containers (#3080)
- [FEATURE] Add VolumeMounts to Prometheus custom resources (#2871)
- [ENHANCEMENT] Update Thanos to v0.11.0 (#3066)
- [ENHANCEMENT] Clarify that Endpoint.targetPort is pod port (#3064)
- [BUGFIX] ThanosRuler restarts instead of reloading with new PrometheusRules (#3056)
- [BUGFIX] Omit QueryEndpoints if empty (#3091)
- [FEATURE] Add routePrefix to ThanosRuler spec (#3023)
- [FEATURE] Add externalPrefix to ThanosRuler spec (#3058)
- [FEATURE] Add pod template fields to ThanosRuler custom resource (#3034)
- [ENHANCEMENT] Make ports on kubelet service and endpoints match (#3039)
- [ENHANCEMENT] Update kubernetes API dependencies to v0.17.3/1.17.3 (#3042)
- [ENHANCEMENT] Simplify multi-arch building (#3035)
- [ENHANCEMENT] Default to Prometheus v2.16.0 (#3050, #3051)
- [BUGFIX] Fix stateful set being pruned by kubectl (#3029, #3030)
- [BUGFIX] Fix prometheus rule validator admitting rules with invalid label types (#2727,#2962)
- [BUGFIX] Fix flaky test in Thanos ruler (#3038)
- [BUGFIX] Fix ThanosRuler status reporting (#3045)
- [BUGFIX] Preserve pod labels and annotations in custom resources (#3041, #3043)
- [BUGFIX] Prevent stateful set update loop for alertmanager and thonos types (#3048, #3049)
- [CHANGE] Rename binary
lint
topo-lint
(#2964) - [CHANGE] Restrict api extension RBAC rules (#2974)
- [FEATURE] Add operator for Thanos Ruler resources (#2943)
- [FEATURE] Thanos Ruler Improvements (#2986, #2991, #2993, #2994, #3018, #3019)
- [FEATURE] Add additional printer columns for custom resources (#2922)
- [ENHANCEMENT] Set config-reloader containers resources (#2958)
- [ENHANCEMENT] Fix broken links and remove spec.version in examples (#2961)
- [ENHANCEMENT] Reduce deprecation warning verbosity (#2978)
- [ENHANCEMENT] Build tooling improvements (#2979, #2982, #2983)
- [ENHANCEMENT] Update Prometheus compatible version list (#2998)
- [ENHANCEMENT] Fix broken links in documentation (#3005)
- [ENHANCEMENT] Update default container image versions (#3007)
- [BUGFIX] Fix statefulset crash loop in Kube 1.17 (#2987)
- [CHANGE] Deprecate baseImage, tag, and sha fields in custom resources (#2914)
- [FEATURE] Add APIVersion field to Prometheus.Spec.Alerting (#2884)
- [FEATURE] Add an option to disable compaction (#2886)
- [FEATURE] Allow configuring PVC access mode (#978)
- [ENHANCEMENT] Do not disable compaction on sidecar without object storage configuration (#2845)
- [ENHANCEMENT] Fix StatefulSet being needlessly re-created (#2857)
- [ENHANCEMENT] Add metric for statefulset re-create (#2859)
- [ENHANCEMENT] Rename
mesh
ports to be prefixed with protocol (#2863) - [ENHANCEMENT] Use kubebuilder controller-gen for creating CRDs (#2855)
- [ENHANCEMENT] Add metrics about node endpoints synchronization (#2887)
- [ENHANCEMENT] Turn off preserveUnknownFields to enable kubectl explain (#2903)
- [ENHANCEMENT] Instrument operator's list and watch operations (#2893)
- [BUGFIX] Modified prometheus wget probe when listenLocal=true (#2929)
- [BUGFIX] Fix generated statefulset being pruned by kubectl (#2944)
- [CHANGE] Make arbitraryFSAccessThroughSMs optional (#2797)
- [FEATURE] Add [prometheus,alertmanager]-instance-namespaces cmdline parameter (#2783)
- [FEATURE] Add configSecret element to the AlertmanagerSpec (#2827)
- [FEATURE] Add enforcedNamespaceLabel to Prometheus CRD (#2820)
- [FEATURE] Add exec probes against localhost:9090 to Prometheus if listenLocal is set to true (#2763)
- [FEATURE] Add honorTimestamps field to Prometheus, Podmonitor, and ServiceMonitor CRD (#2800)
- [FEATURE] Add ignoreNamespaceSelectors field to Prometheus CRD (#2816)
- [FEATURE] Add local configuration options in jsonnet/prometheus-operator (#2794)
- [FEATURE] Add overrideHonorLabels to Prometheus CRD (#2806)
- [FEATURE] Reference secrets instead of local files (#2716)
- [ENHANCEMENT] Add missing json struct tag for ArbitraryFSAccessThroughSMsConfig (#2808)
- [ENHANCEMENT] Ensure containers have "FallbackToLogsOnError" termination policy (#2819)
- [ENHANCEMENT] Improve detection of StatefulSet changes (#2801)
- [ENHANCEMENT] Only append relabelings if EnforcedNamespaceLabel value is set (#2830)
- [ENHANCEMENT] Remove unneeded Ownership change in prometheus-config-reloader (#2761)
- [ENHANCEMENT] Update k8s client to 1.16 (#2778)
- [BUGFIX] Update prometheus dependency to v2.12.0 to fix validation failure for .externalLabels admission webhook (#2779)
- [FEATURE] Add Thanos service port to governing service (#2754)
- [FEATURE] Add VolumeMounts to Alertmanager (#2755)
- [ENHANCEMENT] Bump default thanos image and version (#2746)
- [CHANGE] Change PodManagement policy to parallel in Alertmanager (#2676)
- [FEATURE] Adding label selector for Alertmanager objects discovery filtering (#2662)
- [FEATURE] Provide option to turn on WAL compression (#2683)
- [FEATURE] Support namespace denylist for listwatcher (#2710)
- [FEATURE] Add support for Volumes to Prometheus Custom Resource (#2734)
- [FEATURE] Add support for Volumes to Alertmanager Custom Resource (#2737)
- [FEATURE] Add support for InitContainers to Prometheus Custom Resource (#2522)
- [BUGFIX] Increase terminationGracePeriod for alertmanager statefulSet as it cannot be 0. (#2657)
- [CHANGE] Remove gossip configuration from Thanos sidecar. This means only non-gossip configurations can be used going forward. (#2623, #2629)
- [FEATURE] Add PodMonitor, allowing monitoring pods directly without the necessity to go through a Endpoints of a Service, this is an experimental feature, it may break at any time without notice. (#2566)
- [FEATURE] Add admission webhook to validate
PrometheusRule
objects with Prometheus' promtool linting. (#2551) - [FEATURE] Add ability to select subset of Prometheus objects to reconcile against, configurable with
--prometheus-instance-selector
flag. (#2615) - [FEATURE] Add ability to configure size based retention on Prometheus. (#2608)
- [FEATURE] Add ability to use StatefulSet ordinal in external labels. (#2591)
- [ENHANCEMENT] Use /-/healthy and /-/ready for probes in Alertmanager. (#2600)
Note: Both kube-prometheus (#2554) and the Helm Chart (#2416) have been removed from this repository. kube-prometheus is not hosted as github.com/coroes/kube-prometheus and the helm chart is available at https://github.com/helm/charts/tree/master/stable/prometheus-operator
- [CHANGE] Drop support for Alertmanager < v0.15.0 (#2568)
- [FEATURE] Add Prometheus Config Reloader CPU and Memory flags (#2466)
- [FEATURE] Support
--max-samples
flag in QuerySpec (#2505) - [FEATURE] Adding kustomization files for remote bases (#2497)
- [FEATURE] Allow disabling limits on sidecars (#2560)
- [FEATURE] Modify arbitrary parts of the operator generated containers (#2445)
- [ENHANCEMENT] Add proper Operator labels as recommended by SIG-Apps (#2427)
- [ENHANCEMENT] Watch ConfigMaps having the prometheus-name selector (#2454)
- [ENHANCEMENT] Add prometheusExternalLabelName field to Prometheus object (#2430)
- [ENHANCEMENT] Optional secret in scrapeconfig (#2511)
- [ENHANCEMENT] Update PodSecurityContext docs (#2569)
- [ENHANCEMENT] Update Kubernetes client libraries to 1.14.0 (#2570)
- [ENHANCEMENT] Use Go modules with Kubernetes 1.14 (#2571)
- [ENHANCEMENT] Update to Alertmanager v0.17.0 (#2587)
- [ENHANCEMENT] Add support for setting Log Format for Alertmanager (#2577)
- [ENHANCEMENT] Switch Deployments and StatefulSets from apps/v1beta to apps/v1 (#2593)
- [ENHANCEMENT] Add Service and Servicemonitor to bundle.yaml (#2595)
- [BUGFIX] Fix startup nodeSyncEndpoints (#2475)
- [BUGIFX] Update Thanos vendoring to include config reloader fixes (#2504)
- [FEATURE] Thanos sidecar supports external Thanos clusters (#2412)
- [FEATURE] Make replicas external label name configurable (#2411)
- [FEATURE] Flags for config reloader memory and cpu limits (#2403)
- [ENHANCEMENT] Update to Prometheus v2.7.1 as default (#2374)
- [ENHANCEMENT] Update to Alertmanager v0.16.1 as default (#2362)
- [FEATURE] CLI tool to lint YAML against CRD definitions (#2269)
- [FEATURE] Support Thanos v0.2 arbitrary object storage configuration (#2264)
- [ENHANCEMENT] Update Alertmanager to v0.16.0 (#2145)
- [ENHANCEMENT] Added AlertResendDelay to Prometheus resource (#2265)
- [ENHANCEMENT] Support min_shards configuration of the queueConfig (#2284)
- [ENHANCEMENT] Write compressed Prometheus config into Kubernetes Secret (#2243)
- [ENHANCEMENT] Add flag to enable Prometheus web admin API (#2300)
- [ENHANCEMENT] Add logFormat support for Prometheus (#2307)
- [ENHANCEMENT] Configure Thanos sidecar with route prefix (#2345)
- [BUGFIX] Fix omitting source_labels where they are unnecessary (#2292)
- [BUGFIX] Guard against nil targetPort (#2318)
- [FEATURE] Add
image
field to specify full Prometheus, Alertmanager and Thanos images. - [FEATURE] Add prometheus query options (lookback-delta, max-concurrency, timeout).
- [CHANGE] Remove attempting to set "secure" security context (#2109).
- [CHANGE] Remove deprecated StorageSpec fields (#2132).
- [ENHANCEMENT] Better handling for pod/node labels from ServiceMonitors (#2089).
- [ENHANCEMENT] Update to Prometheus v2.5.0 as default (#2101).
- [ENHANCEMENT] Update to Alertmanager v0.15.3 as default (#2128).
- [ENHANCEMENT] Increase CPU limits for small containers to not being throttled as much (#2144).
- [BUGFIX] Sanitize thanos secret volume mount name (#2159).
- [BUGFIX] Fix racy Kubernetes multi watch (#2177).
- [FEATURE] Allow passing additional alert relabel configs in Prometheus custom resource (#2022)
- [FEATURE] Add ability to mount custom ConfigMaps into Alertmanager and Prometheus (#2028)
This release has a breaking changes for prometheus_operator_.*
metrics.
prometheus_operator_alertmanager_reconcile_errors_total
and prometheus_operator_prometheus_reconcile_errors_total
are now combined and called prometheus_operator_reconcile_errors_total
.
Instead the metric has a "controller" label which indicates the errors from the Prometheus or Alertmanager controller.
The same happened with prometheus_operator_alertmanager_spec_replicas
and prometheus_operator_prometheus_spec_replicas
which is now called prometheus_operator_spec_replicas
and also has the "controller" label.
The prometheus_operator_triggered_total
metric now has a "controller" label as well and finally instruments the
Alertmanager controller.
For a full description see: prometheus-operator#1984 (comment)
In order to support multiple namespaces, the --namespace
flag changed to --namespaces
and accepts and comma-separated list of namespaces as a string.
- [CHANGE] Default to Node Exporter v0.16.0 (#1812)
- [CHANGE] Update to Go 1.11 (#1855)
- [CHANGE] Default to Prometheus v2.4.3 (#1929) (#1983)
- [CHANGE] Default to Thanos v0.1.0 (#1954)
- [CHANGE] Overhaul metrics while adding triggerBy metric for Alertmanager (#1984)
- [CHANGE] Add multi namespace support (#1813)
- [FEATURE] Add SHA field to Prometheus, Alertmanager and Thanos for images (#1847) (#1854)
- [FEATURE] Add configuration for priority class to be assigned to Pods (#1875)
- [FEATURE] Configure sampleLimit per ServiceMonitor (#1895)
- [FEATURE] Add additionalPeers to Alertmanager (#1878)
- [FEATURE] Add podTargetLabels to ServiceMonitors (#1880)
- [FEATURE] Relabel target name for Pods (#1896)
- [FEATURE] Allow configuration of relabel_configs per ServiceMonitor (#1879)
- [FEATURE] Add illegal update reconciliation by deleting StatefulSet (#1931)
- [ENHANCEMENT] Set Thanos cluster and grpc ip from pod.ip (#1836)
- [BUGFIX] Add square brackets around pod IPs for IPv6 support (#1881)
- [BUGFIX] Allow periods in secret name (#1907)
- [BUGFIX] Add BearerToken in generateRemoteReadConfig (#1956)
- [BUGFIX] Do not abort kubelet endpoints update due to nodes without IP addresses defined (#1816)
- [BUGFIX] Fix high CPU usage of Prometheus Operator when annotating Prometheus resource (#1785)
- [CHANGE] Deprecate specification of Prometheus rules via ConfigMaps in favor of
PrometheusRule
CRDs - [FEATURE] Introduce new flag to control logging format (#1475)
- [FEATURE] Ensure Prometheus Operator container runs as
nobody
user by default (#1393) - [BUGFIX] Fix reconciliation of Prometheus StatefulSets due to ServiceMonitors and PrometheusRules changes when a single namespace is being watched (#1749)
[BUGFIX] Do not migrate rule config map for Prometheus statefulset on rule config map to PrometheusRule migration (#1679)
- [ENHANCEMENT] Enable operation when CRDs are created externally (#1640)
- [BUGFIX] Do not watch for new namespaces if a specific namespace has been selected (#1640)
- [FEATURE] Allow setting volume name via volumetemplateclaimtemplate in prom and alertmanager (#1538)
- [FEATURE] Allow setting custom tags of container images (#1584)
- [ENHANCEMENT] Update default Thanos to v0.1.0-rc.2 (#1585)
- [ENHANCEMENT] Split rule config map mounted into Prometheus if it exceeds Kubernetes config map limit (#1562)
- [BUGFIX] Mount Prometheus data volume into Thanos sidecar & pass correct path to Thanos sidecar (#1583)
- [CHANGE] Default to Prometheus v2.3.1.
- [CHANGE] Default to Alertmanager v0.15.0.
- [FEATURE] Make remote write queue configurations configurable.
- [FEATURE] Add Thanos integration (experimental).
- [BUGFIX] Fix usage of console templates and libraries.
With this release we introduce a new Custom Resource Definition - the
PrometheusRule
CRD. It addresses the need for rule syntax validation and rule
selection across namespaces. PrometheusRule
replaces the configuration of
Prometheus rules via K8s ConfigMaps. There are two migration paths:
-
Automated live migration: If the Prometheus Operator finds Kubernetes ConfigMaps that match the
RuleSelector
in aPrometheus
specification, it will convert them to matchingPrometheusRule
resources. -
Manual migration: We provide a basic CLI tool to convert Kubernetes ConfigMaps to
PrometheusRule
resources.
go get -u github.com/coreos/prometheus-operator/cmd/po-rule-migration
po-rule-migration \
--rule-config-map=<path-to-config-map> \
--rule-crds-destination=<path-to-rule-crd-destination>
- [FEATURE] Add leveled logging to Prometheus Operator (#1277)
- [FEATURE] Allow additional Alertmanager configuration in Prometheus CRD (#1338)
- [FEATURE] Introduce
PrometheusRule
Custom Resource Definition (#1333) - [ENHANCEMENT] Allow Prometheus to consider all namespaces to find ServiceMonitors (#1278)
- [BUGFIX] Do not attempt to set default memory request for Prometheus 2.0 (#1275)
- [FEATURE] Allow specifying additional Prometheus scrape configs via secret (#1246)
- [FEATURE] Enable Thanos sidecar (#1219)
- [FEATURE] Make AM log level configurable (#1192)
- [ENHANCEMENT] Enable Prometheus to select Service Monitors outside own namespace (#1227)
- [ENHANCEMENT] Enrich Prometheus operator CRD registration error handling (#1208)
- [BUGFIX] Allow up to 10m for Prometheus pod on startup for data recovery (#1232)
- [BUGFIX] Fix alertmanager >=0.15.0 cluster gossip communication (#1193)
From this release onwards only Kubernetes versions v1.8 and higher are supported. If you have an older version of Kubernetes and the Prometheus Operator running, we recommend upgrading Kubernetes first and then the Prometheus Operator.
While multiple validation issues have been fixed, it will remain a beta feature in this release. If you want to update validations, you need to either apply the CustomResourceDefinitions located in example/prometheus-operator-crd
or delete all CRDs and restart the Prometheus Operator.
Some changes cause Prometheus and Alertmanager clusters to be redeployed. If you do not have persistent storage backing your data, this means you will loose the amount of data equal to your retention time.
- [CHANGE] Use canonical
/prometheus
and/alertmanager
as data dirs in containers. - [FEATURE] Allow configuring Prometheus and Alertmanager servers to listen on loopback interface, allowing proxies to be the ingress point of those Pods.
- [FEATURE] Allow configuring additional containers in Prometheus and Alertmanager Pods.
- [FEATURE] Add ability to whitelist Kubernetes labels to become Prometheus labels.
- [FEATURE] Allow specifying additional secrets for Alertmanager Pods to mount.
- [FEATURE] Allow specifying
bearer_token_file
for Alertmanger configurations of Prometheus objects in order to authenticate with Alertmanager. - [FEATURE] Allow specifying TLS configuration for Alertmanger configurations of Prometheus objects.
- [FEATURE] Add metrics for reconciliation errors:
prometheus_operator_alertmanager_reconcile_errors_total
andprometheus_operator_prometheus_reconcile_errors_total
. - [FEATURE] Support
read_recent
andrequired_matchers
fields for remote read configurations. - [FEATURE] Allow disabling any defaults of
SecurityContext
fields of Pods. - [BUGFIX] Handle Alertmanager >=v0.15.0 breaking changes correctly.
- [BUGFIX] Fix invalid validations for metric relabeling fields.
- [BUGFIX] Fix validations for
AlertingSpec
. - [BUGFIX] Fix validations for deprecated storage fields.
- [BUGFIX] Fix remote read and write basic auth support.
- [BUGFIX] Fix properly propagate errors of Prometheus config reloader.
This release adds validations as a beta feature. It will only be installed on new clusters, existing CRD definitions will not be updated, this will be done in a future release. Please try out this feature and give us feedback!
- [CHANGE] Default Prometheus version v2.2.0-rc.0.
- [CHANGE] Default Alertmanager version v0.14.0.
- [FEATURE] Generate and add CRD validations.
- [FEATURE] Add ability to set
serviceAccountName
for Alertmanager Pods. - [FEATURE] Add ability to specify custom
securityContext
for Alertmanager Pods. - [ENHANCEMENT] Default to non-root security context for Alertmanager Pods.
- [CHANGE] Default to Alertmanager v0.13.0.
- [BUGFIX] Alertmanager flags must be double dashed starting v0.13.0.
- [FEATURE] Add support for specifying remote storage configurations.
- [FEATURE] Add ability to specify log level.
- [FEATURE] Add support for dropping metrics at scrape time.
- [ENHANCEMENT] Ensure that resource limit can't make Pods unschedulable.
- [ENHANCEMENT] Allow configuring emptyDir volumes
- [BUGFIX] Use
--storage.tsdb.no-lockfile
for Prometheus 2.0. - [BUGFIX] Fix Alertmanager default storage.path.
- [CHANGE] Default Prometheus version v2.0.0
- [BUGFIX] Generate ExternalLabels deterministically
- [BUGFIX] Fix incorrect mount path of Alertmanager data volume
- [EXPERIMENTAL] Add ability to specify CRD Kind name
- [BUGFIX] Ignore illegal change of PodManagementPolicy to StatefulSet.
- [CHANGE] Default Prometheus version v2.0.0-rc.1.
- [CHANGE] Default Alertmanager version v0.9.1.
- [BUGFIX] Set StatefulSet replicas to 0 if 0 is specified in Alertmanager/Prometheus object.
- [BUGFIX] Glob for all files in a ConfigMap as rule files.
- [FEATURE] Add ability to run Prometheus Operator for a single namespace.
- [FEATURE] Add ability to specify CRD api group.
- [FEATURE] Use readiness and health endpoints of Prometheus 1.8+.
- [ENHANCEMENT] Add OwnerReferences to managed objects.
- [ENHANCEMENT] Use parallel pod creation strategy for Prometheus StatefulSets.
After a long period of not having broken any functionality in the Prometheus Operator, we have decided to promote the status of this project to beta.
Compatibility guarantees and migration strategies continue to be the same as for the v0.12.0
release.
- [CHANGE] Remove analytics collection.
- [BUGFIX] Fix memory leak in kubelet endpoints sync.
- [FEATURE] Allow setting global default
scrape_interval
. - [FEATURE] Allow setting Pod objectmeta to Prometheus and Alertmanger objects.
- [FEATURE] Allow setting tolerations and affinity for Prometheus and Alertmanager objects.
Starting with this release only Kubernetes v1.7.x
and up is supported as CustomResourceDefinitions are a requirement for the Prometheus Operator and are only available from those versions and up.
Additionally all objects have been promoted from v1alpha1
to v1
. On start up of this version of the Prometheus Operator the previously used ThirdPartyResource
s and the associated v1alpha1
objects will be automatically migrated to their v1
equivalent CustomResourceDefinition
.
- [CHANGE] All manifests created and used by the Prometheus Operator have been promoted from
v1alpha1
tov1
. - [CHANGE] Use Kubernetes
CustomResourceDefinition
s instead ofThirdPartyResource
s. - [FEATURE] Add ability to set scrape timeout to
ServiceMonitor
. - [ENHANCEMENT] Use
StatefulSet
rolling deployments. - [ENHANCEMENT] Properly set
SecurityContext
for Prometheus 2.0 deployments. - [ENHANCEMENT] Enable web lifecycle APIs for Prometheus 2.0 deployments.
- [BUGFIX] Fix memory leak in kubelet endpoints sync.
- [ENHANCEMENT] Add profiling endpoints.
- [BUGFIX] Adapt Alertmanager storage usage to not use deprecated storage definition.
Warning: This release deprecates the previously used storage definition in favor of upstream PersistentVolumeClaim templates. While this should not have an immediate effect on a running cluster, Prometheus object definitions that have storage configured need to be adapted. The previously existing fields are still there, but have no effect anymore.
- [FEATURE] Add Prometheus 2.0 alpha3 support.
- [FEATURE] Use PVC templates instead of custom storage definition.
- [FEATURE] Add cAdvisor port to kubelet sync.
- [FEATURE] Allow default base images to be configurable.
- [FEATURE] Configure Prometheus to only use necessary namespaces.
- [ENHANCEMENT] Improve rollout detection for Alertmanager clusters.
- [BUGFIX] Fix targetPort relabeling.
- [BUGFIX] Use computed route prefix instead of directly from manifest.
Attention: if the basic auth feature was previously used, the key
and name
fields need to be switched. This was not intentional, and the bug is not fixed,
but causes this change.
- [CHANGE] Prometheus default version v1.7.1.
- [CHANGE] Alertmanager default version v0.7.1.
- [BUGFIX] Fix basic auth secret key selector
key
andname
switched. - [BUGFIX] Fix route prefix flag not always being set for Prometheus.
- [BUGFIX] Fix nil panic on replica metrics.
- [FEATURE] Add ability to specify Alertmanager path prefix for Prometheus.
- [CHANGE] Prometheus route prefix defaults to root.
- [CHANGE] Default to Prometheus v1.7.0.
- [CHANGE] Default to Alertmanager v0.7.0.
- [FEATURE] Add route prefix support to Alertmanager resource.
- [FEATURE] Add metrics on expected replicas.
- [FEATURE] Support for running Alertmanager v0.7.0.
- [BUGFIX] Fix sensitive rollout triggering.
- [FEATURE] Add experimental Prometheus 2.0 support.
- [FEATURE] Add support for setting Prometheus external labels.
- [BUGFIX] Fix non-deterministic config generation.
- [CHANGE] The
kubelet-object
flag has been renamed tokubelet-service
. - [CHANGE] Remove automatic relabelling of Pod and Service labels onto targets.
- [CHANGE] Remove "non-namespaced" alpha annotation in favor of
honor_labels
. - [FEATURE] Add ability make use of the Prometheus
honor_labels
configuration option. - [FEATURE] Add ability to specify image pull secrets for Prometheus and Alertmanager pods.
- [FEATURE] Add basic auth configuration option through ServiceMonitor.
- [ENHANCEMENT] Add liveness and readiness probes to Prometheus and Alertmanger pods.
- [ENHANCEMENT] Add default resource requests for Alertmanager pods.
- [ENHANCEMENT] Fallback to ExternalIPs when InternalIPs are not available in kubelet sync.
- [ENHANCEMENT] Improved change detection to trigger Prometheus rollout.
- [ENHANCEMENT] Do not delete unmanaged Prometheus configuration Secret.
- [ENHANCEMENT] Use new Prometheus 1.6 storage flags and make it default.
- [ENHANCEMENT] Include kubelet insecure port in kubelet Enpdoints object.
- [FEATURE] Add ability to mount custom secrets into Prometheus Pods. Note that secrets cannot be modified after creation, if the list if modified after creation it will not effect the Prometheus Pods.
- [FEATURE] Attach pod and service name as labels to Pod targets.
This release introduces breaking changes to the generated StatefulSet's PodTemplate, which cannot be modified for StatefulSets. The Prometheus and Alertmanager objects have to be deleted and recreated for the StatefulSets to be created properly.
- [CHANGE] Use Secrets instead of ConfigMaps for configurations.
- [FEATURE] Allow ConfigMaps containing rules to be selected via label selector.
- [FEATURE]
nodeSelector
added to the Alertmanager kind. - [ENHANCEMENT] Use Prometheus v2 chunk encoding by default.
- [BUGFIX] Fix Alertmanager cluster mesh initialization.
- [FEATURE] Allow not tagging targets with the
namespace
label. - [FEATURE] Allow specifying
ServiceAccountName
to be used by Prometheus pods. - [ENHANCEMENT] Label governing services to uniquely identify them.
- [ENHANCEMENT] Reconcile Service and Endpoints objects.
- [ENHANCEMENT] General stability improvements.
- [BUGFIX] Hostname cannot be fqdn when syncing kubelets into Endpoints object.
- [BUGFIX] Use correct governing
Service
for PrometheusStatefulSet
.
- [FEATURE] Allow synchronizing kubelets into an
Endpoints
object. - [FEATURE] Allow specifying custom configmap-reload image
- [CHANGE] Split endpoint and job in separate labels instead of a single concatenated one.
- [BUGFIX] Properly exit on errors communicating with the apiserver.
This release introduces breaking changes to the underlying naming schemes. It is recommended to destroy existing Prometheus and Alertmanager objects and recreate them with new namings.
With this release support for v1.4.x
clusters is dropped. Changes will not be
backported to the 0.1.x
release series anymore.
- [CHANGE] Prefixed StatefulSet namings based on managing resource
- [FEATURE] Pass labels and annotations through to StatefulSets
- [FEATURE] Add tls config to use for Prometheus target scraping
- [FEATURE] Add configurable
routePrefix
for Prometheus - [FEATURE] Add node selector to Prometheus TPR
- [ENHANCEMENT] Stability improvements
- [BUGFIX] Fix config reloading when using external url.
The 0.1.x
releases are backport releases with Kubernetes v1.4.x
compatibility.
- [BUGFIX] Fix config reloading when using external url.
- [FEATURE] Add ability to set the external url the Prometheus/Alertmanager instances will be available under.
The 0.1.x
releases are backport releases with Kubernetes v1.4.x
compatibility.
- [FEATURE] Add ability to set the external url the Prometheus/Alertmanager instances will be available under.
- [BUGFIX] Fix
subPath
behavior when not using storage provisioning
This release requires a Kubernetes cluster >=1.5.0. See the readme for instructions on how to upgrade if you are currently running on a lower version with the operator.
- [CHANGE] Use StatefulSet instead of PetSet
- [BUGFIX] Fix Prometheus config generation for labels containing "-"