Logging With Docker, Promtail and Grafana Loki

Ruan Bekker
3 min readNov 18, 2022

--

In this post we will use Grafana Promtail to collect all our logs and ship it to Grafana Loki.

About

We will be using Docker Compose and mount the docker socket to Grafana Promtail so that it is aware of all the docker events and configure it that only containers with docker labels logging=promtail needs to be enabled for logging, which will then scrape those logs and send it to Grafana Loki where we will visualize it in Grafana.

Note

For more Grafana tutorials, have a look at my website:

Promtail

In our promtail configuration config/promtail.yaml:

# https://grafana.com/docs/loki/latest/clients/promtail/configuration/
# https://docs.docker.com/engine/api/v1.41/#operation/ContainerList
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:
- url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: flog_scrape
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
filters:
- name: label
values: ["logging=promtail"]
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
- source_labels: ['__meta_docker_container_log_stream']
target_label: 'logstream'
- source_labels: ['__meta_docker_container_label_logging_jobname']
target_label: 'job'

You can see we are using the docker_sd_configs provider and filter only docker containers with the docker labels logging=promtail and once we have those logs we relabel our labels to have the container name and we also use docker labels like log_stream and logging_jobname to add labels to our logs.

Grafana Config

We would like to auto configure our datasources for Grafana and in config/grafana-datasources.yml we have:

apiVersion: 1

datasources:
- name: Loki
type: loki
access: proxy
url: http://loki:3100
version: 1
editable: false
isDefault: true

Docker Compose

Then lastly we have our docker-compose.yml that wire up all our containers:

version: '3.8'

services:
nginx-app:
container_name: nginx-app
image: nginx
labels:
logging: "promtail"
logging_jobname: "containerlogs"
ports:
- 8080:80
networks:
- app

grafana:
image: grafana/grafana:latest
ports:
- 3000:3000
volumes:
- ./config/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
networks:
- app

loki:
image: grafana/loki:latest
ports:
- 3100:3100
command: -config.file=/etc/loki/local-config.yaml
networks:
- app

promtail:
image: grafana/promtail:latest
container_name: promtail
volumes:
- ./config/promtail.yaml:/etc/promtail/docker-config.yaml
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
command: -config.file=/etc/promtail/docker-config.yaml
depends_on:
- loki
networks:
- app

networks:
app:
name: app

As you can see with our nginx container we define our labels:

  nginx-app:
container_name: nginx-app
image: nginx
labels:
logging: "promtail"
logging_jobname: "containerlogs"

Which uses logging: "promtail" to let promtail know this log container’s log to be scraped and logging_jobname: "containerlogs" which will assign containerlogs to the job label.

Start the stack

If you are following along all this configuration is available in my github repository https://github.com/ruanbekker/docker-promtail-loki .

Once you have everything in place you can start it with:

docker-compose up -d

Access nginx on http://localhost:8080

Nginx Home Page

Then navigate to grafana on http://localhost:3000 and select explore on the left and select the container:

Grafana Loki Label Filters

And you will see the logs:

Grafana Loki Logs

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.

--

--