Skip to content

Latest commit

 

History

History
128 lines (92 loc) · 6.88 KB

README.md

File metadata and controls

128 lines (92 loc) · 6.88 KB
Raw

Observability

alt version alt production alt docker alt ARM

Why

This repository contain an observability stack using Observability tools. These one is deploying with Docker Compose to test all the stack's tools.

It is only a testing stack with no security, no production-ready configuration, ... This for testing purpose.

Architecture

The observability stack is composed by Mimir, Loki and Tempo which are deployed in Monolithic mode with 3 instances of each.
It's also include a Grafana vizualisation tool.
The application stack is composed by a FastAPI application with some routes and PostgreSQL database with 1 table.
Logs are written in a log file, scrape by Grafana Promtail and send to Grafana Agent.
Metrics are exposed via Prometheus Client and scraped by Grafana Agent.
Traces are sent via Opentelemetry SDK to Grafana Agent.
Unit tests are performed by k6.io.

Mimir, Loki and Tempo metrics are also exposed througth Grafana Agent.
The metrics of all containers are exposed with cAdvisor.

alt schema

Prerequisite

This repository is run on Mac M1 (ARM architecture). If you want to use another architecture, be sure to modify the Dockerfiles which import arm binary (ex. promtail on the app Dockerfile).

  1. Install Docker.
  2. Create Docker network : docker network create observability-network

How deploy it

  1. Clone this repository : git clone https://github.com/thomasdesplaces/observability_docker.git

  2. Build & deploy observability stack : docker-compose --profile grafana up --build

  3. Access to Grafana for visualization : http://localhost:3000

  4. Build & deploy application stack : docker-compose --profile application up --build

How to send logs/metrics/traces

Deploy an application or database or other on the same docker network with these parameters :

  1. Traces :
  • Protocol : http
  • Host : nginx
  • Port : 3600
  • Path : /v1/traces
  1. Logs :
  • Protocol : http
  • Host : nginx
  • Port : 3500
  • Path : /loki/api/v1/push
  1. Metrics :
  • Expose with Prometheus (or based on Prometheus) on example port : 8000 and modify the targets value in agent.yaml (line 24).

Ports configuration

Observability stack

  1. Get Logs : Promtail --> Nginx : 3500 --> Agent : 3501 --> Nginx : 3502 --> Loki : 3503

  2. Get Traces : oTel SDK --> Nginx : 3600 --> Agent : 3601 --> Nginx : 3602 --> Tempo : 3603

  3. Get Metrics : Prometheus Exporter : 5050 (on path /metrics) <-- Agent --> Nginx : 3702 --> Mimir : 3703

  4. Consult Logs on Grafana : Grafana --> Nginx : 4502 --> Loki : 3503

  5. Consult Traces on Grafana : Grafana --> Nginx : 4602 --> Tempo : 4603

  6. Consult Metrics on Grafana : Grafana --> Nginx : 4702 --> Mimir : 3703

Application stack

  1. Front port : 5055

  2. Backend/API port : 5050

  3. Database port : 5432

Sources

Observability stack

Logo/Link Version Usage
Grafana Agent main-6f9d397 Used to scrape data from applications (front, back, databases, ...) and send to each specific storage (like OpenTelemetry Collector). Based on https://github.com/grafana/agent/blob/main/example/docker-compose/docker-compose.yaml
Grafana Tempo v1.5.0 Used to store Traces (like Jaeger or Zipkin). Based on https://github.com/grafana/tempo/tree/main/example/docker-compose/scalable-single-binary
Grafana Mimir v2.4.0 Used to store Metrics (like Prometheus or Cortex). Based on https://github.com/grafana/mimir/blob/main/docs/sources/mimir/tutorials/play-with-grafana-mimir/docker-compose.yml
Grafana Loki v2.7.0 Used to store Logs (like Elasticsearch). Based on https://github.com/grafana/loki/tree/main/examples/getting-started
Grafana v9.3.1 Used to visualize data (like Kibana). Based on https://github.com/grafana/grafana
Minio Latest Used to load balance traffic between each instance (on Cloud, use S3, Google Cloud Storage or similar). Based on https://github.com/minio/minio
NGinx Latest Used to load balance traffic between each instance. Based on https://github.com/nginx/nginx
cAdvisor Latest Used to expose all containers metrics

Application stack

Thanks to @Blueswen for the FastAPI observability configuration.

Logo/Link Version Usage
Promtail Latest Used to scrape application logs from file.
FastAPI 0.88.0 Used to expose API. Based on https://github.com/Blueswen/fastapi-observability
PostgreSQL 15-Bullseye Used to store application data.
Latest Used to export Postgres metrics. Based on https://hub.docker.com/r/wrouesnel/postgres_exporter
k6.io Latest Used to performed unit tests.
OpenTelemetry 1.15.0 Used to generate traces.