Skip to content

prod.example.com

Overview

Full production deployment on Linode VPS demonstrating complete monitoring setup.

Type: Linode VPS (public cloud) Purpose: Full production deployment (complete solti-monitoring reference) Stack: Alloy, Telegraf, ISPConfig, Gitea, Fail2ban, WireGuard client

Deployment Details

Stack Components

  • Alloy - Log collection and forwarding
  • Telegraf - Metrics collection and forwarding
  • ISPConfig - Web hosting control panel
  • Gitea - Git repository hosting
  • Fail2ban - Intrusion prevention
  • Apache - Web server
  • Bind9 - DNS server
  • Postfix - Mail server
  • WireGuard - VPN client

Playbooks

  • 91-prod-alloy-test.yml - Alloy testing (writes to /tmp)
  • 22-proc-alloy.yml - Alloy deployment (writes to /etc)

Located in: mylab/playbooks/prod/

Configuration

Monitored Services

Via Alloy (Logs): - Apache web server logs - ISPConfig control panel logs - Fail2ban security events - Gitea repository activity - Journald services: - Mail (Postfix, Dovecot) - Bind9 DNS queries - WireGuard VPN connections

Via Telegraf (Metrics): - System metrics (CPU, memory, disk, network) - Apache metrics - Service health checks

Alloy Configuration

Log Destinations: - Primary: monitor11wg (10.10.0.11:3100) via WireGuard - Protocol: HTTP to Loki

Custom Arguments: 

--disable-reporting --server.http.listen-addr=127.0.0.1:12345

Log Sources: - File-based: /var/log/apache2/*.log, /var/log/fail2ban.log - Journald: Mail, Bind9, WireGuard, Gitea

Telegraf Configuration

Output: - monitor11wg (10.10.0.11:8086) via WireGuard

Inputs: - System (cpu, mem, disk, net) - Apache - Process monitoring

Network Configuration

Internet
   │
   ├── ispconfig3-server.example.com (Linode VPS)
   │    ├── Public services (HTTP/HTTPS/DNS/Mail)
   │    └── WireGuard client (10.10.0.1)
   │         │
   │         └── Secure tunnel to monitor11
   │
   └── WireGuard tunnel
        │
        └── monitor11.example.com (10.10.0.11)
             ├── Receives logs from ispconfig3
             └── Receives metrics from ispconfig3

Alloy Test/Deploy Workflow

IMPORTANT: Always test Alloy config changes before deploying!

Step 1: Test Configuration

cd mylab
ansible-playbook --become-password-file ~/.secrets/lavender.pass   ./playbooks/ispconfig3/91-ispconfig3-alloy-test.yml

Test playbook behavior: - Renders config to /tmp/alloy-test-config-YYYYMMDDTHHMMSS.alloy on ispconfig3 - Runs alloy fmt and alloy validate to check syntax - Does NOT restart alloy service - Safe to run multiple times

Step 2: Deploy Configuration

cd mylab
ansible-playbook --become-password-file ~/.secrets/lavender.pass   ./playbooks/ispconfig3/22-ispconfig3-alloy.yml

Deploy playbook behavior: - Writes to /etc/alloy/config.alloy - Restarts alloy service - Activates new configuration

Service Management

Check Services

ssh root@ispconfig3-server.example.com
systemctl status alloy
systemctl status telegraf
systemctl status apache2
systemctl status bind9
systemctl status fail2ban
systemctl status wg-quick@wg0

View Alloy Logs

journalctl -u alloy -f

Test Alloy Config

alloy fmt /etc/alloy/config.alloy
alloy validate /etc/alloy/config.alloy

Deployment Commands

Deploy Monitoring Agents

cd mylab
./manage-svc.sh -h ispconfig3 telegraf deploy
./manage-svc.sh -h ispconfig3 alloy deploy

Verify Deployment

cd mylab
./svc-exec.sh -h ispconfig3 telegraf verify
./svc-exec.sh -h ispconfig3 alloy verify

Troubleshooting

Alloy Not Forwarding Logs

Check Alloy status: 

systemctl status alloy
journalctl -u alloy -n 50

Test WireGuard connectivity: 

ping 10.10.0.11
curl http://10.10.0.11:3100/ready

WireGuard Tunnel Down

Check WireGuard status: 

wg show
systemctl status wg-quick@wg0

Restart WireGuard: 

systemctl restart wg-quick@wg0

Logs Not Appearing in Loki

Test Loki query from ispconfig3: 

curl -G http://10.10.0.11:3100/loki/api/v1/query   --data-urlencode 'query={hostname="ispconfig3-server.example.com"}'   --data-urlencode 'limit=5'

Check Alloy config: 

cat /etc/alloy/config.alloy

Migration Notes

Fail2ban Journald Migration (2026-01-01)

Fail2ban logs migrated from direct file monitoring to journald.

OLD source (deprecated): - Labels: {job="fail2ban", action_type="Ban", jail="sshd"} - Last data: 2026-01-01 04:18 UTC

NEW source (current): - Labels: {service_type="fail2ban", hostname="ispconfig3-server.example.com"} - Started: 2026-01-01 04:41 UTC - Requires regex parsing in Loki queries

Example query migration: 

# OLD (don't use)
{job="fail2ban", action_type="Ban", jail="sshd"}

# NEW (current)
{service_type="fail2ban"}
| regexp `\[(?P<jail>[^\]]+)\]\s+(?P<action>Ban|Unban)\s+(?P<banned_ip>\d+\.\d+\.\d+\.\d+)`
| action="Ban"
| jail="sshd"

References

  • Playbooks: mylab/playbooks/ispconfig3/*.yml
  • Alloy config template: roles/alloy/templates/config.alloy.j2
  • Inventory: mylab/inventory.yml (ispconfig3 host definition)
  • CLAUDE.md: Reference Machines, Alloy Test/Deploy Workflow