prometheus-kubernetes

Base from contrib/kube-prometheus/

This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The content of this project is written in jsonnet. This project could both be described as a package as well as a library.

Components included in this package:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• kube-state-metrics
• Grafana

This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project, similar to this project it provides composable jsonnet as a library for users to customize to their needs.

Quickstart

Although this project is intended to be used as a library, a compiled version of the Kubernetes manifests generated with this library is checked into this repository in order to try the content out quickly.

Simply create the stack:

kubectl apply -f manifests

Usage

The content of this project consists of a set of jsonnet files making up a library to be consumed.

Install this library in your own project with jsonnet-bundler:

$ git clone https://github.com/gavinzhou/prometheus-kubernetes.git
$ jb install

jb can be installed with go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

You may wish to not use ksonnet and simply render the generated manifests to files on disk, this can be done with:

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + {
    _config+:: {
        namespace: 'monitor',

        prometheus+:: {
            replicas: 1,
        },

        alertmanager+:: {
            replicas: 1,
        },

        grafana+:: {
            config: {
                sections: {
                    "auth.anonymous": {enabled: true},
                },
            },
        },
    },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

This renders all manifests in a json structure of {filename: manifest-content}.

Compiling

To compile the above and get each manifest in a separate file on disk use the following script:

#!/usr/bin/env bash
set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir manifests

                                               # optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-kube-prometheus.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- {}

Note you need jsonnet and gojsonyaml (go get github.com/brancz/gojsontoyaml) installed. If you just want json output, not yaml, then you can skip the pipe and everything afterwards.

This script reads each key of the generated json and uses that as the file name, and writes the value of that key to that file.

Install Prometheus Kubernetes

./build.sh ## need some time
kubectl apply -f manifests

Check running Prometheus pod

kubectl get po -n monitor -w

The grafana definition is located in a different project gavinzhou/kubernetes-grafana ,it fixed some bugs.

Show grafana with port-forward

kubectl port-forward $(kubectl get pod -l app=grafana -n monitor -o 'jsonpath={.items[0].metadata.name}') -n monitor 3000

Customization

Get more Readme from prometheus-operator