Alloy Role¶

Overview¶

Grafana Alloy (formerly Grafana Agent) is a flexible telemetry collector for logs, metrics, and traces. In solti-monitoring, Alloy is primarily used for log collection and shipping to Loki.

Purpose¶

Collect logs from multiple sources (files, journald, syslog)
Parse and label logs with metadata
Ship logs to Loki for storage and querying
Support for complex log processing pipelines

Installation¶

The alloy role installs and configures Alloy on target hosts:

- role: jackaltx.solti_monitoring.alloy
  vars:
    alloy_loki_endpoint: "http://monitor.example.com:3100"
    alloy_config_sources:
      - fail2ban
      - apache
      - syslog

Key Configuration Options¶

Loki Output¶

alloy_loki_endpoint: "http://10.10.0.11:3100"  # Loki API endpoint
alloy_loki_auth: false                          # Enable if using authentication

Log Sources¶

Available sources: - fail2ban - Security ban/unban events - apache - Apache access and error logs - bind9 - DNS query logs - mail - Postfix/mail system logs - wireguard - VPN connection logs - gitea - Git service logs - syslog - System logs via journald

Configuration example:

alloy_config_sources:
  - name: fail2ban
    type: journald
    matches:
      - "_SYSTEMD_UNIT=fail2ban.service"
    labels:
      service_type: "fail2ban"

  - name: apache
    type: file
    path: "/var/log/apache2/access.log"
    labels:
      service_type: "web"
      log_type: "access"

Service Arguments¶

alloy_args:
  - "--disable-reporting"                      # Disable telemetry
  - "--server.http.listen-addr=127.0.0.1:12345"  # Local HTTP API only

Alloy Configuration Language¶

Alloy uses its own configuration language (River format):

// Log source from journald
loki.source.journal "fail2ban" {
  matches {
    _SYSTEMD_UNIT = "fail2ban.service"
  }

  labels = {
    service_type = "fail2ban",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

// Loki endpoint
loki.write "default" {
  endpoint {
    url = "http://10.10.0.11:3100/loki/api/v1/push"
  }
}

Configuration Validation¶

IMPORTANT: Always validate configuration before deploying!

Test Workflow¶

# Validates config without restarting service
alloy fmt /tmp/config.alloy
alloy validate /tmp/config.alloy

Deploy Workflow¶

Only deploy after successful validation:

# 1. Test (safe, no service restart)
ansible-playbook test-alloy-config.yml

# 2. Deploy (writes config and restarts)
ansible-playbook deploy-alloy.yml

Service Management¶

Systemd Service¶

# Check status
systemctl status alloy

# View logs
journalctl -u alloy -f

# Restart service
systemctl restart alloy

Configuration Reload¶

Alloy supports live config reload (experimental):

# Send SIGHUP to reload config
systemctl reload alloy

Log Parsing and Labeling¶

Journald Sources¶

Read from systemd journal with filtering:

loki.source.journal "service_logs" {
  matches {
    _SYSTEMD_UNIT = "myservice.service"
  }

  labels = {
    service_type = "myservice",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

File Sources¶

Read from log files:

loki.source.file "apache_access" {
  targets = [
    {
      __path__ = "/var/log/apache2/access.log",
      service_type = "web",
      log_type = "access",
    },
  ]

  forward_to = [loki.write.default.receiver]
}

Log Processing¶

Apply transformations and parsing:

// Parse fail2ban logs
loki.process "fail2ban" {
  forward_to = [loki.write.default.receiver]

  stage.regex {
    expression = "\[(?P<jail>[^\]]+)\]\s+(?P<action>Ban|Unban)\s+(?P<ip>\d+\.\d+\.\d+\.\d+)"
  }

  stage.labels {
    values = {
      jail = "",
      action = "",
      banned_ip = "ip",
    }
  }
}

Multi-Destination Setup¶

Ship logs to multiple Loki instances:

loki.write "primary" {
  endpoint {
    url = "http://primary.example.com:3100/loki/api/v1/push"
  }
}

loki.write "backup" {
  endpoint {
    url = "http://backup.example.com:3100/loki/api/v1/push"
  }
}

// Forward to both
loki.source.journal "logs" {
  // ... config ...
  forward_to = [
    loki.write.primary.receiver,
    loki.write.backup.receiver,
  ]
}

Resource Requirements¶

Typical footprint: - CPU: 1-2% average - Memory: 256MB - Disk: 100MB for binary and configs - Network: Depends on log volume

Troubleshooting¶

Check Service Status¶

systemctl status alloy
journalctl -u alloy -n 50

Validate Configuration¶

alloy fmt /etc/alloy/config.alloy
alloy validate /etc/alloy/config.alloy

Test Loki Connection¶

curl http://monitor.example.com:3100/ready

View Alloy Metrics¶

Alloy exposes metrics about itself:

curl http://127.0.0.1:12345/metrics

Common Issues¶

Config syntax errors: Run alloy validate before deploying
Connection to Loki failed: Check network and Loki status
No logs appearing: Verify log sources are active and matches are correct
High memory usage: Reduce log volume or processing complexity

Fail2ban Journald Migration¶

Important note: As of 2026-01-01, fail2ban logs moved from file-based to journald.

OLD configuration (deprecated):

loki.source.file "fail2ban" {
  targets = [{
    __path__ = "/var/log/fail2ban.log",
  }]
}

NEW configuration (current):

loki.source.journal "fail2ban" {
  matches {
    _SYSTEMD_UNIT = "fail2ban.service"
  }
  labels = {
    service_type = "fail2ban",
  }
}

Log format changed from pre-parsed to raw, requiring regex parsing in queries.

Reference Deployment¶

See Reference Deployments chapter for real-world examples: - ispconfig3.example.com - Full Alloy deployment monitoring multiple services