Guide - Collect logs using Grafana Agent (Loki)

in   3.685 minutes read

This is a beginners guide on how to collect log files on your linux machine using Grafana Agent and publish them to Grafana Cloud. Disclaimer: I conside myself a beginner in both linux, docker, grafana and yaml - this guide is written for beginners like me.

There are multiple ways of installing Grafana Agent. I prefer to run Grafana Agent inside an docker container.

Pre: Install docker

First, install docker on your linux machine. There are multiple ways of doing this, depeneding on your OS. I’m using AMI2 (Amazon EC2 default OS):

yum install docker
sudo service docker start
sudo usermod -a -G docker ec2-user

This script installs docker, make sure docker starts on reboot and enables the ec2-user to run docker-commands without sudo.

Create your grafana-agent config.

The config instructs the grafana agent what files to collect and how to upload them to Grafana Cloud. You can create a boilerplate config in Grafana Cloud by doing the onboarding and picking Loki.

It will setup an API key automatically. However, creating an API key can be done separately.

Your grafana-config.yaml should look something like this. You can save this file anywhere, I keep it in ~/grafana-config.yaml.

logs:
  configs:
  - name: "Anders Grafana Cloud"
  
            
    positions:
      filename: /var/log/anders/positions.yaml #This file needs to persisted to the host using a `-v volume` or logs might be re-read. 
            
    clients:
    - url: https://89231:[email protected]/loki/api/v1/push
            
    scrape_configs:
    - job_name: anderslogs
      static_configs:
      - targets:
          - localhost
        labels:
          job: anderslogs
          host: "anders_vm1"
          __path__: /var/log/anders/**/*.log

There are many more options available, read about them in the Grafana Agent documentation.

Start the grafana agent in docker

We will now launch the grafana agent as an docker container. However, since the agent is running inside a contianer it only has access to read files that exists inside that container. It won’t have access the files on your machine that it needs:

  • /home/ec2-user/grafana-config.yaml - read config
  • /var/log/anders/positions.yaml - keep track of what has already been read from files
  • /var/log/anders/*.log - the log-files that will be read

Let’s fix that by using volumes in docker. When you run a docker container, you can map files and folders on the host system to be part of the docker containers filesysem. This is done using docker volumes and the -v or --read-only arguments to docker run.

For each directory we want to make available to docker, we add mount. E.g. -v /home/ec2-user/grafana-config.yaml:/etc/agent/agent.yaml.

Note that the grafana-agent needs to save some temporary and operational data on the host, so we make they are mounted:

  • -v /tmp/agent:/etc/agent/data - store temporary data
  • -v /home/ec2-user/grafana-config.yaml:/etc/agent/agent.yaml - Config
  • -v /home/ec2-user/positions.yaml:/var/log/anders/positions.yaml - Positions file

With mappings, our start command looks like this:

docker run \
-v /tmp/agent:/etc/agent/data \
-v /home/ec2-user/grafana-config.yaml:/etc/agent/agent.yaml \
-v /var/log/anders:/var/log/anders \
grafana/agent:v0.22.0

Tip: Since our Grafana Agent config collects all log-files (including sub directories) in /var/log/anders/*, we can simply add mounts to subfolders and have new log files automatically picked up by grafana: -v /home/ec2-user/my_logs/:/var/log/anders/my_logs.

Write your logs to the right place
I either write logs from my apps to /var/log/anders/something.log to have it automatically collected by Grafana Agent. If the logs are written somewhere else, I add a mount to a subfolder, e.g. /var/log/anders/something/ and Grafana Agent will automatically collect it.

Your logs should now be available in your Grafana Cloud.
Open Grafana Cloud > Explore. Click on Log Browser to see what labels/files have been collected.

Always run Grafana Agent (automatically starts on reboots, background)

I put my start command in a script and extended it with a name and dockers built in way to keep a container alive:

#!/bin/sh
# Removes existing container with this name so that we can start a new
docker container stop grafana-agent &> /dev/null
docker container rm grafana-agent &> /dev/null
# Use -v to map a file/folder to /var/log/anders to have it automatically picked up by Grafana Agent
# -d, start the container in the background 
# --name is optional, but helps to make sure only one agent is running at once.
# --restart unless-stopped will automatically start your container on crashes/reboot etc.
docker run \
-d \
--name grafana-agent \
--restart unless-stopped \
-v /tmp/agent:/etc/agent/data \
-v /home/ec2-user/grafana-config.yaml:/etc/agent/agent.yaml \
-v /var/log/anders:/var/log/anders \
-v /home/ec2-user/logs:/var/log/anders/user \
grafana/agent:v0.22.0

Run the script everytime you change volume mounts or the config to restart the Grafana Agent.

Tip: If you have a problem beeing duplicated upon restarts, make sure the position-path in your grafana-config.yaml is persisted to the host machine using a volume mount.


 Tagged with  •  •  •  •