Skip to content

Testing Overview

Overview

Testing in solti-monitoring has evolved as roles matured and automation patterns emerged. What started with unit tests shifted toward integration tests as they proved more informative for validating real deployments.

Current State: - Integration tests are primary validation - run-podman-tests.sh and run-proxmox-tests.sh orchestrate testing - Some unit tests exist at role level but are minimally maintained - GitHub Actions CI required custom containers to work properly

Testing Philosophy: - Test real deployments, not isolated functions - Integration tests catch more issues than unit tests - Molecule provides test infrastructure, not the testing strategy - Testing evolves with the roles - what works now might change later

Test Execution Scripts

run-podman-tests.sh

Orchestrates integration tests using Podman containers:

#!/bin/bash
# Fast local integration testing

cd solti-monitoring

# Test individual roles
molecule test -s default -- influxdb
molecule test -s default -- loki
molecule test -s default -- telegraf
molecule test -s default -- alloy

# Test full stack
molecule test -s combined

Why Podman: - Fast (containers spin up in seconds) - Isolated (clean environment per test) - Local (no external dependencies) - Repeatable (consistent results)

run-proxmox-tests.sh

Orchestrates integration tests on Proxmox VMs:

#!/bin/bash
# Slower, production-like testing

cd solti-monitoring

# Test on real VMs
molecule test -s proxmox -- influxdb
molecule test -s proxmox -- loki

# Test across distributions
molecule test -s proxmox-rocky9
molecule test -s proxmox-debian12
molecule test -s proxmox-ubuntu24

Why Proxmox: - Production-like (real VMs, real systemd, real networking) - Comprehensive (catches platform-specific issues) - Multi-distro (validates across Rocky, Debian, Ubuntu) - Pre-release validation (run before merging to main)

Test Platform Matrix

Platform Speed When Used Distros Tested
Podman Seconds Development iteration Debian 12, Rocky 9, Ubuntu 24
Proxmox Minutes Pre-release validation Rocky 9, Debian 12, Ubuntu 24
GitHub Actions Medium CI on PR Debian 12 (custom containers)

Why three platforms: - Podman: Rapid feedback during development - Proxmox: Catch production issues before release - GitHub Actions: Automated regression checks

Integration Test Focus

What Integration Tests Validate:

  1. Deployment succeeds
  2. Service/container starts
  3. Systemd unit running

  4. Ports listening

  5. Configuration correct

  6. Config files generated
  7. Secrets/tokens loaded

  8. Service-specific settings applied

  9. Functionality works

  10. API endpoints respond
  11. Data flows (Telegraf → InfluxDB, Alloy → Loki)

  12. Queries return expected results

  13. Integration points work

  14. Components can talk to each other
  15. End-to-end data flow validated
  16. Real service behavior confirmed

Example (InfluxDB integration test):

# verify.yml - validates real behavior
- name: InfluxDB health check
  uri:
    url: "http://localhost:8086/health"
    status_code: 200

- name: Write test metric
  uri:
    url: "http://localhost:8086/api/v2/write?bucket=telegraf&org=example-org"
    method: POST
    headers:
      Authorization: "Token {{ influxdb_admin_token }}"
    body: "test_metric,host=test value=42"

- name: Query metric back
  uri:
    url: "http://localhost:8086/api/v2/query"
    method: POST
    headers:
      Authorization: "Token {{ influxdb_admin_token }}"
    body: 'from(bucket: "telegraf") |> range(start: -1m)'
  register: result

- name: Verify query succeeded
  assert:
    that: result.json.results is defined

Unit Tests (Minimally Used)

Reality: Unit tests exist in some roles but are rarely maintained.

Why unit tests didn't stick: - Too granular (testing individual tasks misses integration issues) - High maintenance (more tests to update when code changes) - Low return (integration tests catch the same issues) - Mock complexity (mocking systemd, containers, networking is hard)

Where unit tests still exist: - Template rendering (verify Jinja2 output) - Variable validation (ensure required vars set) - Edge case testing (unusual configurations)

But: If integration tests provide better coverage, maintaining unit tests becomes low priority.

Evolution of Testing Strategy

Early Approach (Unit-focused)

Role development → Write unit tests → Write integration tests

Problems: - Unit tests gave false confidence - Integration tests caught issues unit tests missed - Maintaining both was time-consuming

Current Approach (Integration-focused)

Role development → Write integration tests → Skip/minimize unit tests

Benefits: - Integration tests validate real deployments - Faster development (fewer tests to maintain) - Better coverage (tests actual behavior)

Future Evolution

Testing strategy will continue evolving: - May add more unit tests if integration tests become slow - May consolidate test scenarios if coverage overlaps - May add new platforms (cloud VMs, bare metal)

Key point: Testing adapts to what works, not what documentation says it should be.

GitHub Actions CI Challenges

Problem: GitHub Actions runners don't support systemd containers properly.

Initial attempts: - Standard container images (failed - no systemd) - Nested Podman (failed - no privilege support) - Docker-in-Docker (failed - networking issues)

Solution: Created jackaltx/testing-containers

Custom containers provide: - Systemd-enabled base images - Pre-configured Podman - Test fixtures and dependencies - Consistent behavior (local and CI)

GitHub Actions workflow:

name: Test Monitoring Roles

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/jackaltx/testing-containers/debian12-systemd:latest
      options: --privileged

    steps:
      - uses: actions/checkout@v2

      - name: Run integration tests
        run: |
          cd solti-monitoring
          molecule test -s default

Result: CI tests now work, using same approach as local testing.

Molecule Framework (Tool, Not Strategy)

Molecule is the test runner, not the testing strategy:

What Molecule provides: - Test lifecycle management (create, converge, verify, destroy) - Platform abstraction (Podman, Proxmox, delegated) - Scenario isolation (default, proxmox, combined) - Ansible integration (uses standard playbooks)

What Molecule doesn't provide: - Testing strategy (integration vs. unit) - Test content (you write verify.yml) - Platform-specific behavior (you configure molecule.yml)

Molecule scenarios:

roles/influxdb/molecule/
├── default/          # Podman container test (fast)
├── proxmox/         # Proxmox VM test (slow, comprehensive)
└── combined/        # Full stack test (InfluxDB + Telegraf + Loki + Alloy)

See: "Molecule Framework" page for detailed configuration.

Running Tests

Local Development

# Quick test during development
./run-podman-tests.sh

# Test specific role
cd solti-monitoring
molecule test -s default -- influxdb

# Debug failing test
molecule --debug converge -s default -- influxdb
molecule verify -s default -- influxdb

Pre-Release Validation

# Comprehensive validation
./run-proxmox-tests.sh

# Test specific platform
cd solti-monitoring
molecule test -s proxmox-debian12 -- influxdb

CI (Automated)

# Runs automatically on PR
# Uses custom testing containers
# See .github/workflows/*.yml

Test Results

Success:

PLAY RECAP *******************************************
instance: ok=15 changed=8 unreachable=0 failed=0

INFO     Verifier completed successfully.
INFO     Running default > destroy

Failure:

TASK [Verify InfluxDB health] *************************
fatal: [instance]: FAILED! => {
  "status": 503,
  "msg": "Service Unavailable"
}

ERROR    Command returned 2 code

Debugging failures:

# Leave instance running
molecule converge -s default -- influxdb

# SSH into test instance
molecule login -s default

# Check logs
podman logs influxdb
journalctl -u influxdb

Documentation Evolution

This chapter documents current reality, not idealized testing practices.

What changed: - Unit tests de-emphasized (low value) - Integration tests emphasized (high value) - Test orchestration scripts (run-podman-tests.sh, run-proxmox-tests.sh) documented - GitHub Actions custom containers explained

What might change: - Testing strategy will continue evolving - New platforms may be added - Test scenarios may be consolidated or expanded

Philosophy: Document what works now, update as it evolves.

Chapter Contents

This chapter contains:

  1. Molecule Framework - How Molecule works and configuration
  2. Test Scenarios - Specific test scenario examples
  3. Platform Matrix - Distro coverage and platform behavior
  4. CI/CD Integration - GitHub Actions workflows and custom containers
  5. Verification Tasks - Common verification patterns

Start with: "Molecule Framework" if new to Molecule, "Test Scenarios" if familiar with testing.

References

  • Testing Containers: https://github.com/jackaltx/testing-containers
  • Test Scripts: run-podman-tests.sh, run-proxmox-tests.sh in repo root
  • Molecule Docs: https://molecule.readthedocs.io/
  • GitHub Actions: .github/workflows/ in repo