Skip to content

Setup without Docker

Jump to bottom
ekatchko edited this page Nov 6, 2020 · 2 revisions

Prometheus setup for the federated de.NBI cloud

This wiki entry contains the prometheus monitoring setup for the de.NBI cloud, especially for the cloud site Tübingen on CentOS 7. A general wiki and more in detail how to use Docker container can be found here. If you want to use Docker container follow the instructions there. In the following is described how to set up Prometheus bare metal.

Used components

  • Prometheus
  • OpenStack project usage exporter
  • HAProxy (or something similar)
  • Grafana Dashboard (can be used but is not required)

Installation

Everything described in the following needs to be installed on a node with access to the OpenStack API.

Prometheus installation

  1. Create new user prometheus and change to user prometheus 
useradd -m -s /bin/bash prometheus
su - prometheus
  1. Download and unpack Prometheus sources
wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz
tar -xzvf prometheus-2.6.0.linux-amd64.tar.gz
  1. Rename the directory
mv prometheus-2.6.0.linux-amd64/ prometheus/
  1. Configure Prometheus to run as a systemd service (change back to root user) Create new service file
vim /etc/systemd/system/prometheus.service

Enter the following lines:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Restart=on-failure

#Change this line if you download the 
#Prometheus on different path user
ExecStart=/home/prometheus/prometheus/prometheus \
  --config.file=/home/prometheus/prometheus/prometheus.yml \
  --storage.tsdb.path=/home/prometheus/prometheus/data

[Install]
WantedBy=multi-user.target

Save and exit the file.

Start Prometheus via systemd

systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus

Check if it is running correct (default port is 9090)

systemctl status prometheus
netstat -plntu
  1. Check if the dashboard is available Open a browser and access the URL http://localhost:9090. You should now have access to the Prometheus web interface. Choose a graph to test if everything works.

Install the Prometheus OpenStack usage exporter

The OpenStack usage exporter has been implemented by Tilmann Luettje, is maintained by deNBI and can be found here. All the following steps are performed as root.

  1. Clone git repository.
git clone https://github.com/deNBI/OS_project_usage_exporter
  1. As the usage exporter requires Python 3.7 this has to be installed, if not already there.

Download and unpack Python 3.7

wget https://www.python.org/ftp/python/3.7.6Python-3.7.6.tgz
tar -xf Python-3.7.6.tgz

Install Python 3.7 without breaking the system Python, make sure to have the packages libffi-devel and openssl-devel installed via yum. Otherwise the installation will fail. After the installation make sure that python3.7 is in your path.

cd Python-3.7.6/
./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
  1. Create a virtual environment for the exporter and test the exporter

Change to the exporter git repo

cd OS_project_usage_exporter

Run the following commands to create the environment for this directory with all required dependencies

python3 -m venv .venv
source .venv/bin/activate
pip3.7 install -r requirements.txt

You can test if everything worked fine by displaying the help output

python3 ./project_usage_exporter.py -h

In order to run the exporter in production it is required that the OpenStack credentials have been sourced as environment variables.

source /root/admin

You can now test if the exporter runs without any errors

python3 ./project_usage_exporter.py --start-date-endpoint https://cloud.denbi.de/portal/public/startdate/ -p 8095 --verbose

This will start the exporter using data since the date you get from the portal start-date-endpoint and it is running on port 8095. API calls will be made every 300s. There might come some INFO messages about projects without any data and additional DEBUG messages.

  1. Integrate the exporter into Prometheus In order to make our Prometheus installation aware of the exporter we have to edit the configuration. Change to the user prometheus and open the config file
su - prometheus
vim /home/prometheus/prometheus/prometheus.yml

Add the following part after the line - targets: ['localhost:9090']

- job_name: project_usages
    metrics_path: /

    # do not collect any metrics of the python client
    params:
      'match[]':
        - '{__name__=~"project_.*_usage"}'

    static_configs:
      - targets: ['localhost:8095']

Now Prometheus and the exporter should be connected. You can check if everything is okay through the Prometheus web interface. Open the web interface and go to Status then to Targets. The State should be up and no errors should be displayed.

If everything is working, we will write a systemd script to handle the exporter more conveniently.
You may exit the usage exporter now, as we will add some more parameters.

  1. Create systemd script for prometheus exporter

Before we start with the systemd file we need an additional file for the OpenStack credentials as they have to be sourced in advance to get the OpenStack API access. You can copy the admin file under /root/admin and delete the export commands so that only the environment variables are left. The file can be placed for example under /root/prometheus_credentials.

Create a systemd file

vim /etc/systemd/system/prometheus-usage-exporter.service

Enter the following lines

[Unit]
Description=Prometheus exporter for OpenStack usage statistics
After=multi-user.target

[Service]
Restart=always
Type=simple
EnvironmentFile=/root/prometheus_credentials
WorkingDirectory=/root/OS_project_usage_exporter
ExecStart=/root/OS_project_usage_exporter/.venv/bin/python3 /root/OS_project_usage_exporter/project_usage_exporter.py --start-date-endpoint https://cloud.denbi.de/portal/public/startdate/ ----weight-update-endpoint https://cloud.denbi.de/portal/public/creditsweights/ -p 8095

[Install]
WantedBy=multi-user.target

Please have a look at the possible parameters.
Especially --domain, --domain-id, --simple-vm-id, --weight-update-frequency and -i should be set correctly if you need to set them. Please make sure that the EnvironmentFile path is set correctly. The EnvironmentFile is required to load the OpenStack credentials.

Afterwards stop the running exporter on the commandline if not stopped already and start the systemd service.

systemctl daemon-reload
systemctl enable prometheus-usage-exporter
systemctl start prometheus-usage-exporter

Check if everything is running fine

systemctl status prometheus-usage-exporter

Check also the webinterface if everything is up and data is collected.

Now Prometheus and the exporter are fully installed.

Establish connection to global Prometheus instance in Bielefeld

In this setup an HAProxy is used to establish the connection.

  1. Enable URL access and add Bearer Token. Open the /etc/haproxy/haproxy.cfg file and enter the following lines:
listen prometheus
    bind 193.196.20.3:9091 ssl crt /etc/pki/tls/private/combinded.pem
    balance source
    http-response set-header Strict-Transport-Security "max-age=15552000; includeSubDomains; preload;"
    mode http
    option tcpka
    server localhost localhost:9090 check
capture request header Authorization len 50
capture request header User-Agent len 64

# Authorization header, called 'bearer token' inside prometheus config
# must be the same as in the global_prometheus config
acl auth_header req.fhdr(Authorization) "Bearer $BEARER_TOKEN"
http-request deny if ! auth_header

# %5B%5D equals [] escaped
# match={job="project_usages",__name__=~"project_.*_usage"}
acl valid_query urlp(match%5B%5D) %7Bjob%3D%22project_usages%22%2C__name__%3D~%22project_.%2A_usage%22%7D
http-request deny if ! valid_query</pre>
  • In older versions of HAProxy (e.g. 1.5.18) the syntax is slightly different and you have to adjust the following two lines in your /etc/haproxy/haproxy.cfg file to:
http-response set-header Strict-Transport-Security max-age=15552000;\ includeSubDomains;\ preload;

acl auth_header req.fhdr(Authorization) Bearer\ $BEARER_TOKEN
  1. Update iptables config As not the whole world should have access to the Prometheus Interface iptables are used to limit the connections to including a single IP address from Bielefeld but further security actions are also necessary.
iptables -A INPUT -s IPRANGE -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - accept" -j ACCEPT
iptables -A INPUT -s IP_BIELEFELD/32 -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - accept" -j ACCEPT
iptables -A INPUT -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - drop" -j DROP
  1. Query the desired information If everything is set up correctly you should now be able to access the Prometheus API and run the following curl query to get the results.
curl 'YOUR_URL:9091/api/v1/series?match%5B%5D=%7Bjob%3D%22project_usages%22%2C__name__%3D~%22project_.%2A_usage%22%7D' -H 'Authorization:Bearer $BEARER_TOKEN'
Clone this wiki locally