Configuration Examples¶

Overview¶

Real-world configuration examples from reference deployments showing actual production usage.

monitor11 InfluxDB Configuration¶

InfluxDB Service¶

Installation: Podman Quadlet Storage: S3-compatible (storage.example.com:8010)

# From inventory.yml - monitor11
influxdb_org: "example-org"
influxdb_bucket: "telegraf"
influxdb_admin_user: "admin"
influxdb_retention: "720h"  # 30 days

# S3 backend
influxdb_s3_enabled: true
influxdb_s3_endpoint: "http://storage.example.com:8010"
influxdb_s3_bucket: "influx11"
influxdb_s3_access_key: "{{ lookup('file', project_root + '/mylab/data/tokens/influx-s3-access.key') }}"
influxdb_s3_secret_key: "{{ lookup('file', project_root + '/mylab/data/tokens/influx-s3-secret.key') }}"

Telegraf on monitor11¶

Output: Localhost InfluxDB

# From inventory.yml - monitor11
telegraf_outputs: ['local']
telgraf2influxdb_configs:
  local:
    url: "http://localhost:8086"
    token: "{{ lookup('file', project_root + '/mylab/data/tokens/influx-monitor11-token.txt') }}"
    bucket: "telegraf"
    org: "example-org"

monitor11 Loki Configuration¶

Loki Service¶

Installation: Podman Quadlet Storage: S3-compatible (storage.example.com:8010)

# From inventory.yml - monitor11
loki_local_storage: false
loki_endpoint: "storage.example.com:8010"
loki_s3_bucket: "loki11"
loki_key_id: "{{ lookup('file', project_root + '/mylab/data/tokens/loki-s3-access.key') }}"
loki_access_key: "{{ lookup('file', project_root + '/mylab/data/tokens/loki-s3-secret.key') }}"

production.example.com Configuration¶

Alloy Log Collection¶

Monitors: Apache, ISPConfig, Fail2ban, Gitea, Mail, Bind9, WireGuard

# From playbooks/production/22-deploy-alloy.yml
vars:
  # File-based log sources
  alloy_monitor_apache: true
  alloy_monitor_ispconfig: true
  alloy_monitor_fail2ban: true
  alloy_monitor_gitea: true

  # Additional groups for file access
  alloy_additional_groups:
    - git

  # Journald-based log sources
  alloy_monitor_mail: true
  alloy_monitor_bind9: true
  alloy_monitor_wg: true

  # Custom startup arguments
  alloy_custom_args: "--disable-reporting --server.http.listen-addr=127.0.0.1:12345"

  # Output destination
  alloy_loki_endpoints:
    - label: monitor11wg
      endpoint: "10.10.0.11"  # Via WireGuard

Telegraf Metrics Collection¶

Outputs: monitor11 via WireGuard

# From inventory.yml - production
telegraf_outputs: ['monitor11wg']
telgraf2influxdb_configs:
  monitor11wg:
    url: "http://10.10.0.11:8086"
    token: "{{ lookup('file', project_root + '/mylab/data/tokens/influx-monitor11-token.txt') }}"
    bucket: "telegraf"
    org: "example-org"

Alloy Config Template Snippets¶

Apache Log Collection¶

// Apache access logs
local.file_match "apache_access" {
  path_targets = [{
    __address__ = "localhost",
    __path__    = "/var/log/apache2/access.log",
  }]
}

loki.source.file "apache_access" {
  targets    = local.file_match.apache_access.targets
  forward_to = [loki.process.apache_access.receiver]
}

loki.process "apache_access" {
  forward_to = [loki.write.loki.receiver]

  stage.labels {
    values = {
      service_type = "web",
      web_service  = "apache",
      log_type     = "access",
      hostname     = "{{ ansible_hostname }}",
    }
  }
}

Fail2ban Journald Collection¶

// Fail2ban from journald
loki.source.journal "fail2ban" {
  forward_to    = [loki.process.fail2ban.receiver]
  relabel_rules = discovery.relabel.journal.rules
  matches       = "_SYSTEMD_UNIT=fail2ban.service"
}

loki.process "fail2ban" {
  forward_to = [loki.write.loki.receiver]

  stage.labels {
    values = {
      service_type = "fail2ban",
      hostname     = "{{ ansible_hostname }}",
    }
  }

  // Extract ban/unban actions
  stage.regex {
    expression = "\[(?P<jail>[^\]]+)\]\s+(?P<action>Ban|Unban)\s+(?P<banned_ip>\d+\.\d+\.\d+\.\d+)"
  }
}

Bind9 DNS Query Logging¶

// Bind9 DNS queries from journald
loki.source.journal "bind9" {
  forward_to    = [loki.process.bind9.receiver]
  relabel_rules = discovery.relabel.journal.rules
  matches       = "_SYSTEMD_UNIT=named.service"
}

loki.process "bind9" {
  forward_to = [loki.write.loki.receiver]

  stage.labels {
    values = {
      service_type = "dns",
      dns_service  = "bind9",
      hostname     = "{{ ansible_hostname }}",
    }
  }

  // Extract query details
  stage.regex {
    expression = "client @(?P<client_ip>[^ ]+).* query: (?P<query_domain>[^ ]+) (?P<query_class>[^ ]+) (?P<query_type>[^ ]+)"
  }
}

Loki Write Endpoint¶

loki.write "loki" {
  endpoint {
    url = "http://{{ endpoint }}:3100/loki/api/v1/push"

    // Optional basic auth
    // basic_auth {
    //   username = "loki"
    //   password = "secret"
    // }
  }
}

Telegraf Config Snippets¶

System Metrics¶

# From roles/telegraf/templates/telegraf.conf.j2

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.mem]]

[[inputs.net]]
  interfaces = ["eth*", "en*"]

[[inputs.system]]

Apache Metrics¶

[[inputs.apache]]
  urls = ["http://localhost/server-status?auto"]

Output to InfluxDB¶

[[outputs.influxdb_v2]]
  urls = ["{{ url }}"]
  token = "{{ token }}"
  organization = "{{ org }}"
  bucket = "{{ bucket }}"

  ## Optional timeout
  timeout = "5s"

  ## Optional user agent
  user_agent = "telegraf"

Playbook Examples¶

Deploy InfluxDB + Telegraf (monitor11)¶

---
- name: "Deploy Metrics Stack on monitor11"
  hosts: monitor11
  become: true

  vars:
    influxdb_reload: true
    influxdb_configure: true

  pre_tasks:
    - name: Ensure ansible temp directory exists
      ansible.builtin.file:
        path: /tmp/ansible-tmp
        state: directory
        owner: root
        group: root
        mode: "0777"

  roles:
    - jackaltx.solti_monitoring.influxdb
    - jackaltx.solti_monitoring.telegraf

Deploy Alloy (Production)¶

---
- name: "Deploy Alloy Log Collector on Production Server"
  hosts: **production**
  become: true

  vars:
    alloy_monitor_apache: true
    alloy_monitor_fail2ban: true
    alloy_monitor_mail: true
    alloy_monitor_bind9: true
    alloy_monitor_wg: true

    alloy_custom_args: "--disable-reporting --server.http.listen-addr=127.0.0.1:12345"

    alloy_loki_endpoints:
      - label: monitor11wg
        endpoint: "10.10.0.11"

  roles:
    - jackaltx.solti_monitoring.alloy

Verification Commands¶

Verify InfluxDB¶

# Check service
systemctl status influxdb

# Test API
curl http://localhost:8086/health

# Query buckets
influx bucket list --org example-org

Verify Loki¶

# Check service
systemctl status loki

# Test API
curl http://localhost:3100/ready

# Test query
curl -G http://localhost:3100/loki/api/v1/query   --data-urlencode 'query={hostname="production.example.com"}'   --data-urlencode 'limit=5'

Verify Telegraf¶

# Check service
systemctl status telegraf

# Test config
telegraf --test --config /etc/telegraf/telegraf.conf

# View logs
journalctl -u telegraf -f

Verify Alloy¶

# Check service
systemctl status alloy

# Validate config
alloy validate /etc/alloy/config.alloy

# View logs
journalctl -u alloy -f

Token Management¶

Tokens stored in mylab/data/tokens/:

mylab/data/tokens/
├── influx-monitor11-token.txt    # InfluxDB API token
├── influx-s3-access.key          # S3 access key ID
├── influx-s3-secret.key          # S3 secret key
├── loki-s3-access.key            # Loki S3 access key
└── loki-s3-secret.key            # Loki S3 secret key

Security: These files are gitignored and stored securely.

References¶

Full playbooks: mylab/playbooks/
Role templates: roles/*/templates/
Inventory: mylab/inventory.yml
CLAUDE.md: Complete configuration examples