-
Notifications
You must be signed in to change notification settings - Fork 0
Setup without Docker
This wiki entry contains the prometheus monitoring setup for the de.NBI cloud, especially for the cloud site Tübingen on CentOS 7. A general wiki and more in detail how to use Docker container can be found here. If you want to use Docker container follow the instructions there. In the following is described how to set up Prometheus bare metal.
- Prometheus
- OpenStack project usage exporter
- HAProxy (or something similar)
- Grafana Dashboard (can be used but is not required)
Everything described in the following needs to be installed on a node with access to the OpenStack API.
- Create new user
prometheus
and change to userprometheus
useradd -m -s /bin/bash prometheus su - prometheus
- Download and unpack Prometheus sources
wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz tar -xzvf prometheus-2.6.0.linux-amd64.tar.gz
- Rename the directory
mv prometheus-2.6.0.linux-amd64/ prometheus/
- Configure Prometheus to run as a systemd service (change back to root user) Create new service file
vim /etc/systemd/system/prometheus.service
Enter the following lines:
[Unit] Description=Prometheus Server Documentation=https://prometheus.io/docs/introduction/overview/ After=network-online.target [Service] User=prometheus Restart=on-failure #Change this line if you download the #Prometheus on different path user ExecStart=/home/prometheus/prometheus/prometheus \ --config.file=/home/prometheus/prometheus/prometheus.yml \ --storage.tsdb.path=/home/prometheus/prometheus/data [Install] WantedBy=multi-user.target
Save and exit the file.
Start Prometheus via systemd
systemctl daemon-reload systemctl enable prometheus systemctl start prometheus
Check if it is running correct (default port is 9090
)
systemctl status prometheus netstat -plntu
- Check if the dashboard is available
Open a browser and access the URL
http://localhost:9090
. You should now have access to the Prometheus web interface. Choose a graph to test if everything works.
The OpenStack usage exporter has been implemented by Tilmann Luettje, is maintained by deNBI and
can be found here.
All the following steps are performed as root
.
- Clone git repository.
git clone https://github.com/deNBI/OS_project_usage_exporter
- As the usage exporter requires
Python 3.7
this has to be installed, if not already there.
Download and unpack Python 3.7
wget https://www.python.org/ftp/python/3.7.6Python-3.7.6.tgz tar -xf Python-3.7.6.tgz
Install Python 3.7 without breaking the system Python, make sure to have the packages libffi-devel
and
openssl-devel installed via yum.
Otherwise the installation will fail. After the installation make sure that python3.7
is in your path.
cd Python-3.7.6/ ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib" make && make altinstall
- Create a virtual environment for the exporter and test the exporter
Change to the exporter git repo
cd OS_project_usage_exporter
Run the following commands to create the environment for this directory with all required dependencies
python3 -m venv .venv source .venv/bin/activate pip3.7 install -r requirements.txt
You can test if everything worked fine by displaying the help output
python3 ./project_usage_exporter.py -h
In order to run the exporter in production it is required that the OpenStack credentials have been sourced as environment variables.
source /root/admin
You can now test if the exporter runs without any errors
python3 ./project_usage_exporter.py --start-date-endpoint https://cloud.denbi.de/portal/public/startdate/ -p 8095 --verbose
This will start the exporter using data since the date you get from the portal start-date-endpoint and it is running on port 8095. API calls will be made every 300s. There might come some INFO messages about projects without any data and additional DEBUG messages.
- Integrate the exporter into Prometheus
In order to make our Prometheus installation aware of the exporter we have to edit the configuration.
Change to the user
prometheus
and open the config file
su - prometheus vim /home/prometheus/prometheus/prometheus.yml
Add the following part after the line - targets: ['localhost:9090']
- job_name: project_usages metrics_path: / # do not collect any metrics of the python client params: 'match[]': - '{__name__=~"project_.*_usage"}' static_configs: - targets: ['localhost:8095']
Now Prometheus and the exporter should be connected. You can check if everything is okay through the
Prometheus web interface. Open the web interface and go to Status
then to Targets
. The State
should be up and
no errors should be displayed.
If everything is working, we will write a systemd script to handle the exporter more conveniently.
You may exit the usage exporter now, as we will add some more parameters.
- Create systemd script for prometheus exporter
Before we start with the systemd file we need an additional file for the OpenStack credentials as they have to
be sourced in advance to get the OpenStack API access. You can copy the admin file under /root/admin
and delete
the export commands so that only the environment variables are left. The file can be placed
for example under /root/prometheus_credentials
.
Create a systemd file
vim /etc/systemd/system/prometheus-usage-exporter.service
Enter the following lines
[Unit] Description=Prometheus exporter for OpenStack usage statistics After=multi-user.target [Service] Restart=always Type=simple EnvironmentFile=/root/prometheus_credentials WorkingDirectory=/root/OS_project_usage_exporter ExecStart=/root/OS_project_usage_exporter/.venv/bin/python3 /root/OS_project_usage_exporter/project_usage_exporter.py --start-date-endpoint https://cloud.denbi.de/portal/public/startdate/ ----weight-update-endpoint https://cloud.denbi.de/portal/public/creditsweights/ -p 8095 [Install] WantedBy=multi-user.target
Please have a look at the possible parameters.
Especially --domain, --domain-id, --simple-vm-id, --weight-update-frequency and -i should be set correctly if you need
to set them.
Please make sure that the EnvironmentFile
path is set correctly.
The EnvironmentFile
is required to load the OpenStack credentials.
Afterwards stop the running exporter on the commandline if not stopped already and start the systemd service.
systemctl daemon-reload systemctl enable prometheus-usage-exporter systemctl start prometheus-usage-exporter
Check if everything is running fine
systemctl status prometheus-usage-exporter
Check also the webinterface if everything is up and data is collected.
Now Prometheus and the exporter are fully installed.
In this setup an HAProxy is used to establish the connection.
- Enable URL access and add Bearer Token.
Open the
/etc/haproxy/haproxy.cfg
file and enter the following lines:
listen prometheus bind 193.196.20.3:9091 ssl crt /etc/pki/tls/private/combinded.pem balance source http-response set-header Strict-Transport-Security "max-age=15552000; includeSubDomains; preload;" mode http option tcpka server localhost localhost:9090 check
capture request header Authorization len 50
capture request header User-Agent len 64
# Authorization header, called 'bearer token' inside prometheus config
# must be the same as in the global_prometheus config
acl auth_header req.fhdr(Authorization) "Bearer $BEARER_TOKEN"
http-request deny if ! auth_header
# %5B%5D equals [] escaped
# match={job="project_usages",__name__=~"project_.*_usage"}
acl valid_query urlp(match%5B%5D) %7Bjob%3D%22project_usages%22%2C__name__%3D~%22project_.%2A_usage%22%7D
http-request deny if ! valid_query</pre>
- In older versions of HAProxy (e.g. 1.5.18) the syntax is slightly different and you have to adjust the following two lines in your
/etc/haproxy/haproxy.cfg
file to:
http-response set-header Strict-Transport-Security max-age=15552000;\ includeSubDomains;\ preload; acl auth_header req.fhdr(Authorization) Bearer\ $BEARER_TOKEN
- Update
iptables
config As not the whole world should have access to the Prometheus Interface iptables are used to limit the connections to including a single IP address from Bielefeld but further security actions are also necessary.
iptables -A INPUT -s IPRANGE -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - accept" -j ACCEPT
iptables -A INPUT -s IP_BIELEFELD/32 -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - accept" -j ACCEPT
iptables -A INPUT -d HOST_IP/32 -p tcp -m tcp --dport 9091 -m comment --comment "Prometheus webapi - drop" -j DROP
- Query the desired information If everything is set up correctly you should now be able to access the Prometheus API and run the following curl query to get the results.
curl 'YOUR_URL:9091/api/v1/series?match%5B%5D=%7Bjob%3D%22project_usages%22%2C__name__%3D~%22project_.%2A_usage%22%7D' -H 'Authorization:Bearer $BEARER_TOKEN'