Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

Latest commit

 

History

History
260 lines (206 loc) · 11.7 KB

gitlab.md

File metadata and controls

260 lines (206 loc) · 11.7 KB
Raw

gitlab

Monitor Type: gitlab (Source)

Accepts Endpoints: Yes

Multiple Instances Allowed: Yes

Overview

GitLab is an open-source web-based git repository manager developed by GitLab Inc. GitLab has built-in features for creating wiki pages, issue-tracking and CI/CD pipelines. GitLab is bundled with Prometheus exporters that can be configured to export performance metrics of itself and that of the bundled software that GitLab depends on. These exporters publish Prometheus metrics at endpoints are scraped by this monitor.

REQUIREMENTS AND DEPENDENCIES

Software Version
GitLab 9.3+

INSTALLATION

CONFIGURATION

GitLab Configuration

Follow the instructions here to configure the GitLab's Prometheus exporters to expose metric endpoint targets. For GitLab Runner monitoring configuration go here.

Note that configuring GitLab by editing /etc/gitlab/gitlab.rb should be accompanied by running the command gitlab-ctl reconfigure in order for the changes to take effect.

Also, configuring Nginx by editing the file /var/opt/gitlab/nginx/conf/nginx-status.conf, for instance, should be accompanied by running command gitlab-ctl restart. Note that changes to the configuration file /var/opt/gitlab/nginx/conf/nginx-status.conf in particular are erased by subsequent runs of command gitlab-ctl reconfigure because gitlab-ctl reconfigure restores the original configuration file.

Below is a list of some of the Prometheus endpoint targets with links to their respective configuration pages. Note that target gitlab_monitor metrics are just targets gitlab_monitor_database, gitlab_monitor_process and gitlab_monitor_sidekiq metrics combined.

Agent Monitor Type Gitlab Doc Standard Port Standard Path
gitlab Gitlab doc 9168 /metrics
gitlab-gitaly Gitlab doc 9236 /metrics
gitlab-sidekiq Gitlab doc 8082 /metrics
gitlab-unicorn Gitlab doc 8080 /-/metrics
gitlab-workhorse Gitlab doc 9229 /metrics
prometheus/nginx-vts Gitlab doc 8060 /metrics
prometheus/node Gitlab doc 9100 /metrics
promteheus/postgres Gitlab doc 9187 /metrics
prometheus/prometheus Gitlab doc 9090 /metrics
prometheus/redis Gitlab doc 9121 /metrics
gitlab-runner Gitlab doc 9252 /metrics

GitLab Prometheus exporters, Nginx and GitLab Runner must be configured to listen to IP address(es) that include the IP address of the host or docker container of the SignalFx Smart Agent. For example, the configuration below in /etc/gitlab/gitlab.rb configures the GitLab Postgres Prometheus exporter to allow network connections on port 9187 from any IP address.

postgres_exporter['listen_address'] = '0.0.0.0:9187'

The above configuration can also be written as:

postgres_exporter['listen_address'] = ':9187'

Below is part of file /var/opt/gitlab/nginx/conf/nginx-status.conf showing the location /metrics block for metric related configuration. This file configures Nginx. The statement allow 172.17.0.0/16; allows network connection in the 172.17.0.0/16 IP range. The assumption is that the IP address associated with the SignalFx Smart Agent is in that IP range.

server {
    ...
    location /metrics {
    ...
    allow 172.17.0.0/16;
    deny all;
    }
}

Below is part of the global section of /etc/gitlab-runner/config.toml. This file configures GitLab Runner. The statement below configures GitLab Runner's Prometheus metrics HTTP server to allows network connection on port 9252 from any IP address.

listen_address = "0.0.0.0:9252"
...

Sample Config

To monitor everything we support in Gitlab, use the following configuration in the Smart Agent config:

monitors:
 - type: gitlab-unicorn
   host: localhost
   port: 8080

 - type: gitlab
   host: localhost
   port: 9168

 - type: gitlab-runner
   host: localhost
   port: 9252

 - type: gitlab-workhorse
   host: localhost
   port: 9229

 - type: gitlab-sidekiq
   host: localhost
   port: 8082

 - type: gitlab-gitaly
   host: localhost
   port: 9236

 - type: prometheus/postgres
   host: localhost
   port: 9187

 - type: prometheus/nginx-vts
   host: localhost
   port: 8060

You can, of course, use auto-discovery by specifying a discoveryRule instead of host and port.

Configuration

To activate this monitor in the Smart Agent, add the following to your agent config:

monitors:  # All monitor config goes under this key
 - type: gitlab
   ...  # Additional config

For a list of monitor options that are common to all monitors, see Common Configuration.

Config option Required Type Description
httpTimeout no int64 HTTP timeout duration for both read and writes. This should be a duration string that is accepted by https://golang.org/pkg/time/#ParseDuration (default: 10s)
username no string Basic Auth username to use on each request, if any.
password no string Basic Auth password to use on each request, if any.
useHTTPS no bool If true, the agent will connect to the server using HTTPS instead of plain HTTP. (default: false)
httpHeaders no map of strings A map of HTTP header names to values. Comma separated multiple values for the same message-header is supported.
skipVerify no bool If useHTTPS is true and this option is also true, the exporter's TLS cert will not be verified. (default: false)
sniServerName no string If useHTTPS is true and skipVerify is true, the sniServerName is used to verify the hostname on the returned certificates. It is also included in the client's handshake to support virtual hosting unless it is an IP address.
caCertPath no string Path to the CA cert that has signed the TLS cert, unnecessary if skipVerify is set to false.
clientCertPath no string Path to the client TLS cert to use for TLS required connections
clientKeyPath no string Path to the client TLS key to use for TLS required connections
host yes string Host of the exporter
port yes integer Port of the exporter
useServiceAccount no bool Use pod service account to authenticate. (default: false)
metricPath no string Path to the metrics endpoint on the exporter server, usually /metrics (the default). (default: /metrics)
sendAllMetrics no bool Send all the metrics that come out of the Prometheus exporter without any filtering. This option has no effect when using the prometheus exporter monitor directly since there is no built-in filtering, only when embedding it in other monitors. (default: false)

Metrics

These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.

  • ci_stale_builds (gauge)
  • gitlab_cache_misses_total (cumulative)
  • gitlab_database_rows (gauge)
  • gitlab_projects_with_jid (gauge)
  • gitlab_projects_without_jid (gauge)
  • gitlab_stuck_import_jobs_worker_runs_total (cumulative)
  • http_request_duration_seconds (cumulative)
  • http_request_duration_seconds_bucket (cumulative)
  • http_request_duration_seconds_count (cumulative)
  • pg_stat_table_idx_tup_fetch (gauge)
  • pg_stat_table_n_dead_tup (gauge)
  • pg_stat_table_n_tup_del (gauge)
  • pg_stat_table_n_tup_hot_upd (gauge)
  • pg_stat_table_n_tup_ins (gauge)
  • pg_stat_table_n_tup_upd (gauge)
  • pg_stat_table_seq_scan (gauge)
  • pg_stat_table_seq_tup_read (gauge)
  • process_age_seconds (gauge)
  • process_count (gauge)
  • process_memory_bytes (gauge)
  • ruby_file_descriptors (gauge)
  • ruby_gc_duration_seconds_total (cumulative)
  • ruby_gc_stat_count (gauge)
  • ruby_gc_stat_heap_allocatable_pages (gauge)
  • ruby_gc_stat_heap_allocated_pages (gauge)
  • ruby_gc_stat_heap_available_slots (gauge)
  • ruby_gc_stat_heap_eden_pages (gauge)
  • ruby_gc_stat_heap_final_slots (gauge)
  • ruby_gc_stat_heap_free_slots (gauge)
  • ruby_gc_stat_heap_live_slots (gauge)
  • ruby_gc_stat_heap_marked_slots (gauge)
  • ruby_gc_stat_heap_sorted_length (gauge)
  • ruby_gc_stat_heap_tomb_pages (gauge)
  • ruby_gc_stat_major_gc_count (gauge)
  • ruby_gc_stat_malloc_increase_bytes (gauge)
  • ruby_gc_stat_malloc_increase_bytes_limit (gauge)
  • ruby_gc_stat_minor_gc_count (gauge)
  • ruby_gc_stat_old_objects (gauge)
  • ruby_gc_stat_old_objects_limit (gauge)
  • ruby_gc_stat_oldmalloc_increase_bytes (gauge)
  • ruby_gc_stat_oldmalloc_increase_bytes_limit (gauge)
  • ruby_gc_stat_remembered_wb_unprotected_objects (gauge)
  • ruby_gc_stat_remembered_wb_unprotected_objects_limit (gauge)
  • ruby_gc_stat_total_allocated_objects (gauge)
  • ruby_gc_stat_total_allocated_pages (gauge)
  • ruby_gc_stat_total_freed_objects (gauge)
  • ruby_gc_stat_total_freed_pages (gauge)
  • ruby_memory_bytes (gauge)
  • ruby_sampler_duration_seconds_total (cumulative)
  • sidekiq_dead_jobs_total (gauge)
  • sidekiq_queue_latency (gauge)
  • sidekiq_queue_size (gauge)

Non-default metrics (version 4.7.0+)

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.