Skip to content

Deployment Workflows

General Deployment Workflow

Step 1: Prepare Environment

Install base requirements on target system:

# Update system
sudo apt update && sudo apt upgrade -y  # Debian/Ubuntu
sudo dnf update -y                       # Rocky/RHEL

# Install Ansible
sudo apt install ansible -y              # Debian/Ubuntu
sudo dnf install ansible -y              # Rocky/RHEL

# Install collection
ansible-galaxy collection install jackaltx.solti_monitoring

Step 2: Configure Inventory

Create inventory file with target hosts:

# inventory.yml
all:
  children:
    monitoring_servers:
      hosts:
        monitor.example.com:
          ansible_user: admin
          ansible_become: true

Step 3: Create Playbook

Create deployment playbook:

# deploy-monitoring.yml
---
- name: Deploy monitoring server
  hosts: monitoring_servers
  become: true

  roles:
    - role: jackaltx.solti_monitoring.influxdb
      vars:
        influxdb_admin_token: "{{ vault_influxdb_token }}"
        influxdb_org: "myorg"
        influxdb_bucket: "telegraf"

Step 4: Run Deployment

Execute the playbook:

ansible-playbook -i inventory.yml deploy-monitoring.yml

Step 5: Verify Deployment

Run verification tasks:

# Check service status
ansible monitoring_servers -i inventory.yml -m shell \
  -a "systemctl status influxdb"

# Test API endpoint
curl http://monitor.example.com:8086/health

Alloy Deployment Workflow

IMPORTANT: Always test Alloy config changes before deploying!

Test Workflow (Safe)

# 1. TEST - Validates config, does NOT restart service
ansible-playbook --become-password-file ~/.secrets/admin.pass \
  ./playbooks/test-alloy-config.yml

Test playbook behavior: - Renders config to /tmp/alloy-test-config-YYYYMMDDTHHMMSS.alloy - Runs alloy fmt and alloy validate to check syntax - Does NOT restart alloy service - Safe to run multiple times

Deploy Workflow (Production)

# 2. DEPLOY - Writes to /etc/alloy/config.alloy and restarts service
ansible-playbook --become-password-file ~/.secrets/admin.pass \
  ./playbooks/deploy-alloy.yml

Deploy playbook behavior: - Writes config to /etc/alloy/config.alloy - Restarts alloy service - Verifies service started successfully

Multi-Component Deployment

Deploy complete monitoring stack (server + client):

# deploy-full-stack.yml
---
- name: Deploy monitoring server
  hosts: monitoring_servers
  become: true
  roles:
    - jackaltx.solti_monitoring.influxdb
    - jackaltx.solti_monitoring.loki

- name: Deploy monitoring clients
  hosts: monitoring_clients
  become: true
  roles:
    - jackaltx.solti_monitoring.telegraf
    - jackaltx.solti_monitoring.alloy

Execute: 

ansible-playbook -i inventory.yml deploy-full-stack.yml

Incremental Updates

Update configuration without full redeployment:

# Update only Telegraf configuration
ansible-playbook -i inventory.yml deploy-monitoring.yml \
  --tags telegraf_config

# Restart only Alloy service
ansible-playbook -i inventory.yml deploy-monitoring.yml \
  --tags alloy_restart

Rollback Procedure

If deployment fails:

  1. Check logs: Review Ansible output for errors
  2. Verify services: Check systemd status
  3. Restore config: Revert to previous working configuration
  4. Redeploy: Run playbook with corrected variables

Configuration Validation

Before deploying to production:

  1. Syntax check: Use --syntax-check flag
  2. Dry run: Use --check flag (check mode)
  3. Test environment: Deploy to test host first
  4. Verify services: Ensure all services start successfully
# Syntax check
ansible-playbook --syntax-check deploy-monitoring.yml

# Dry run (no changes)
ansible-playbook -i inventory.yml deploy-monitoring.yml --check

# Deploy to test environment first
ansible-playbook -i inventory-test.yml deploy-monitoring.yml

WireGuard-Based Deployment

For remote collectors using WireGuard:

  1. Deploy WireGuard server on monitoring host
  2. Configure WireGuard clients on collector hosts
  3. Update inventory with WireGuard IPs (10.10.0.x)
  4. Deploy monitoring stack using WireGuard endpoints

Example: 

# Client connects to server via WireGuard
telegraf_output_url: "http://10.10.0.11:8086"
alloy_loki_endpoint: "http://10.10.0.11:3100"

Troubleshooting Deployments

Common Issues

  1. Port conflicts: Check if ports 8086, 3100 already in use
  2. Permission errors: Ensure become: true is set
  3. Network connectivity: Verify hosts are reachable
  4. Missing dependencies: Install required packages first

Debug Mode

Run playbook with verbose output:

ansible-playbook -i inventory.yml deploy-monitoring.yml -vvv

Service Verification

Check service status after deployment:

# InfluxDB
systemctl status influxdb
curl http://localhost:8086/health

# Loki
systemctl status loki
curl http://localhost:3100/ready

# Telegraf
systemctl status telegraf

# Alloy
systemctl status alloy