Cleanup and Management¶

Proper cleanup and management of test containers ensures efficient resource usage and prevents conflicts.

Cleanup Strategies¶

Strategy 1: Soft Cleanup (Keep Data)¶

Removes containers but preserves data directories:

./manage-svc.sh redis remove
./manage-svc.sh elasticsearch remove

When to use: - Between test runs that need same data - When debugging failed tests - When you might need to inspect data later

What's removed: - Containers - Pods - Systemd unit files - Quadlet definitions

What's kept: - Data directories - Configuration files - Logs - Backups

Strategy 2: Hard Cleanup (Delete Everything)¶

Completely removes services including all data:

./manage-svc.sh redis remove -e redis_delete_data=true
./manage-svc.sh elasticsearch remove -e elasticsearch_delete_data=true

When to use: - After test suite completes - When starting fresh tests - When cleaning up CI/CD environment - When freeing disk space

What's removed: - Everything from soft cleanup, PLUS: - Data directories - All persisted data - Configuration files - Logs

Strategy 3: Selective Cleanup¶

Remove specific services while keeping others:

# Keep Redis, remove others
./manage-svc.sh elasticsearch remove -e elasticsearch_delete_data=true
./manage-svc.sh mattermost remove -e mattermost_delete_data=true
# Redis continues running

Strategy 4: Clean Slate¶

Remove all containers and data:

#!/bin/bash
# cleanup-all.sh

SERVICES="redis elasticsearch mattermost vault minio traefik"

for service in $SERVICES; do
    echo "Removing $service..."
    ./manage-svc.sh $service remove -e ${service}_delete_data=true
done

echo "Cleanup complete!"

Lifecycle Management¶

Full Lifecycle Example¶

# 1. Prepare (one-time per host)
./manage-svc.sh redis prepare

# 2. Deploy
./manage-svc.sh redis deploy

# 3. Verify
./svc-exec.sh redis verify

# 4. Use in tests
pytest tests/

# 5. Soft cleanup (keep data)
./manage-svc.sh redis remove

# 6. Redeploy with same data
./manage-svc.sh redis deploy

# 7. More tests
pytest tests/

# 8. Hard cleanup (delete all)
./manage-svc.sh redis remove -e redis_delete_data=true

Automated Lifecycle¶

# test_lifecycle.py
import subprocess
import pytest

class TestWithRedis:
    @classmethod
    def setup_class(cls):
        """Deploy Redis before all tests in class."""
        subprocess.run(["./manage-svc.sh", "redis", "deploy"], check=True)
        subprocess.run(["./svc-exec.sh", "redis", "verify"], check=True)

    @classmethod
    def teardown_class(cls):
        """Remove Redis after all tests in class."""
        subprocess.run([
            "./manage-svc.sh", "redis", "remove",
            "-e", "redis_delete_data=true"
        ])

    def test_something(self):
        # Your test here
        pass

Service Management¶

Check Service Status¶

# Single service
systemctl --user status redis-pod

# All services
systemctl --user status | grep -E "(redis|elasticsearch|mattermost)"

# Detailed status
systemctl --user status redis-pod --no-pager -l

View Logs¶

# Recent logs
podman logs redis-svc

# Follow logs
podman logs -f redis-svc

# Last 100 lines
podman logs --tail 100 redis-svc

# With timestamps
podman logs --timestamps redis-svc

# All containers in pod
podman pod logs redis-pod

Restart Services¶

# Restart service
systemctl --user restart redis-pod

# Reload systemd after config changes
systemctl --user daemon-reload

# Restart specific container
podman restart redis-svc

Stop/Start Services¶

# Stop (keep data)
systemctl --user stop redis-pod

# Start
systemctl --user start redis-pod

# Enable auto-start
systemctl --user enable redis-pod

# Disable auto-start
systemctl --user disable redis-pod

Resource Management¶

Check Resource Usage¶

# CPU and memory usage
podman stats

# Specific container
podman stats redis-svc

# One-time snapshot
podman stats --no-stream

# With formatting
podman stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

Disk Usage¶

# Check data directory sizes
du -sh ~/redis-data
du -sh ~/elasticsearch-data
du -sh ~/mattermost-data

# Total container usage
du -sh ~/.local/share/containers/storage

# Detailed breakdown
podman system df

Clean Up Resources¶

# Remove unused images
podman image prune -a

# Remove unused volumes
podman volume prune

# Remove unused networks
podman network prune

# Clean everything unused
podman system prune -a --volumes

CI/CD Cleanup Patterns¶

GitHub Actions¶

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      # ... setup and deploy services ...

      - name: Run tests
        run: pytest tests/

      - name: Cleanup containers
        if: always()  # Run even if tests fail
        run: |
          ./manage-svc.sh redis remove -e redis_delete_data=true
          ./manage-svc.sh elasticsearch remove -e elasticsearch_delete_data=true

      - name: Cleanup images
        if: always()
        run: podman system prune -a -f

GitLab CI¶

test:
  script:
    - ./manage-svc.sh redis deploy
    - pytest tests/

  after_script:
    - ./manage-svc.sh redis remove -e redis_delete_data=true || true
    - podman system prune -a -f || true

Jenkins Pipeline¶

pipeline {
    agent any

    stages {
        stage('Deploy Services') {
            steps {
                sh './manage-svc.sh redis deploy'
                sh './manage-svc.sh elasticsearch deploy'
            }
        }

        stage('Test') {
            steps {
                sh 'pytest tests/'
            }
        }
    }

    post {
        always {
            sh './manage-svc.sh redis remove -e redis_delete_data=true || true'
            sh './manage-svc.sh elasticsearch remove -e elasticsearch_delete_data=true || true'
            sh 'podman system prune -a -f || true'
        }
    }
}

Troubleshooting Cleanup Issues¶

Container Won't Stop¶

# Check what's happening
podman logs redis-svc

# Force stop
podman stop -t 0 redis-svc

# Force kill
podman kill redis-svc

# Remove forcefully
podman rm -f redis-svc

Pod Won't Delete¶

# List pods
podman pod ps

# Stop pod
podman pod stop redis-pod

# Force remove pod and all containers
podman pod rm -f redis-pod

Data Directory Won't Delete¶

# Check permissions
ls -la ~/redis-data

# Check for open files
lsof ~/redis-data

# Force delete
sudo rm -rf ~/redis-data

Orphaned Resources¶

# Find orphaned containers
podman ps -a --filter "status=exited"

# Find orphaned pods
podman pod ps --filter "status=exited"

# Clean up orphans
podman container prune
podman pod prune

Best Practices¶

1. Always Use Cleanup Handlers¶

In scripts and tests, always include cleanup:

#!/bin/bash
trap cleanup EXIT

cleanup() {
    echo "Cleaning up..."
    ./manage-svc.sh redis remove -e redis_delete_data=true
}

# Your test code here

2. Verify Cleanup Completed¶

# After cleanup, verify no containers remain
if podman ps -a | grep -q redis; then
    echo "ERROR: Redis containers still exist!"
    exit 1
fi

3. Clean Between Test Runs¶

# In CI, start with clean slate
./cleanup-all.sh
./deploy-all.sh
pytest tests/
./cleanup-all.sh

4. Monitor Disk Usage¶

# Set up monitoring
df -h /home
du -sh ~/.local/share/containers/storage

# Alert if usage too high
USAGE=$(df /home | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $USAGE -gt 80 ]; then
    echo "WARNING: Disk usage above 80%"
    podman system prune -a -f
fi

5. Keep Cleanup Scripts¶

Maintain cleanup utilities:

# cleanup-all.sh - Remove all test services
# cleanup-soft.sh - Remove containers, keep data
# cleanup-images.sh - Remove unused images
# cleanup-aged.sh - Remove data older than N days

6. Document Cleanup Process¶

In your project README:

## Cleanup

After testing:

bash
# Quick cleanup (keep data)
make clean

# Full cleanup (delete everything)
make clean-all

# Nuclear option (reset to pristine state)
make reset

Automation Scripts¶

Cleanup by Age¶

#!/bin/bash
# cleanup-aged.sh - Remove data directories older than N days

DAYS=7
SERVICES="redis elasticsearch mattermost"

for service in $SERVICES; do
    DATA_DIR=~/${service}-data
    if [ -d "$DATA_DIR" ]; then
        AGE=$(find "$DATA_DIR" -maxdepth 0 -mtime +${DAYS} | wc -l)
        if [ $AGE -gt 0 ]; then
            echo "Removing old $service data (>$DAYS days)"
            rm -rf "$DATA_DIR"
        fi
    fi
done

Cleanup by Usage¶

#!/bin/bash
# cleanup-unused.sh - Remove stopped containers

# Find stopped containers older than 1 hour
podman ps -a --filter "status=exited" \
  --format "{{.ID}} {{.Status}}" | \
  grep "hours ago\|days ago\|weeks ago" | \
  awk '{print $1}' | \
  xargs -r podman rm

# Clean up orphaned resources
podman system prune -f

Scheduled Cleanup¶

# Add to crontab for daily cleanup
# crontab -e

# Clean up daily at 2 AM
0 2 * * * /path/to/cleanup-aged.sh

# Prune weekly on Sunday
0 3 * * 0 podman system prune -a -f

Monitoring¶

Log Cleanup Status¶

#!/bin/bash
# Log cleanup operations

LOG_FILE=~/cleanup.log

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a $LOG_FILE
}

log "Starting cleanup..."
./manage-svc.sh redis remove -e redis_delete_data=true
log "Redis removed"

./manage-svc.sh elasticsearch remove -e elasticsearch_delete_data=true
log "Elasticsearch removed"

log "Cleanup complete"

Cleanup Metrics¶

# Track cleanup metrics
echo "Cleanup metrics:"
echo "- Containers removed: $(podman ps -a | wc -l)"
echo "- Disk space freed: $(df -h /home | tail -1 | awk '{print $4}')"
echo "- Images removed: $(podman images | wc -l)"

Summary¶

Effective cleanup management:

Choose appropriate strategy (soft vs hard cleanup)
Always cleanup in CI/CD (use if: always())
Monitor resource usage (disk, memory, containers)
Automate cleanup (scripts, cron jobs)
Log cleanup operations (for debugging)
Verify cleanup completed (check for orphans)
Document cleanup process (in project docs)