Skip to content

Testing Philosophy

SOLTI Framework

The Solti-Containers collection follows the SOLTI framework principles:

  • Systems: Managing system-of-systems
  • Oriented: Structured and purposeful
  • Laboratory: Controlled testing environment
  • Testing: Verification and validation
  • Integration: Component interconnection

Why Each Service

Mattermost

I want a private collector of notices that can do phone and screen with the ability to send out messages. I can use Gist in the wild, but some things need to be private. Jabber and MQTT appeal to me for the "notice", but it's good to have a collector that one can look at for debugging.

There are many integrations with Mattermost, so I want to evaluate collecting test result/reports. To be honest, the signal to noise on these things leave me cold. I have seen so many different ways of grouping information for technical and business reporting.

Future projects: Tie Mattermost back to Claude with MCP for test analysis.

HashiCorp's Vault

Seems every time I need to find a better way to manage secrets, this one comes up. Seems very versatile. I have set these up a few times and they work ok. This will be my first container version.

The testing and verification of this role is better than the others.

The configuration seems complete. Have Claude document it.

Redis

I want a fast Key-Value store to collect my testing reports as they run. File systems can add too much time in my testing cycle.

Their license changes may cause me to move on to Valkey in the future.

Elasticsearch

I have only used it once before and it was impressive as a backend. Currently I am using Alloy-Loki for logs. I would like to have an alternative as well as learn how to better use no-SQL databases. My gut says I should look at Mongo, but their license worries me.

Traefik

This is my first experience. I have learned much and know so little about container networking. I don't like it yet, but this may be a love-hate thing. Another high-maintenance relationship. It reminds me of a project long ago and far away. This one has layers.

However, container networking still is fuzzy in my mind.

MinIO

S3-compatible storage for testing purposes without AWS costs. Useful for testing backup workflows and object storage integrations.

Pattern Development

Between us (human and AI) we have created a decent pattern for creating small testing services using Podman. The pattern includes:

  1. Consistent directory structure for all roles
  2. Common variable naming conventions
  3. Standardized task organization
  4. Reusable management scripts
  5. Predictable lifecycle operations (prepare, deploy, verify, remove)

The AI Collaboration Challenge

Ask an AI to duplicate the pattern and watch for the variances. It will deviate. This is why having a documented standard pattern is crucial - it provides the "box" that constrains AI-generated solutions to remain consistent with existing infrastructure.

Where Next

So far the only services I'm missing are:

  • A good fuzzer (Jepson)
  • Vulnerability assessment tools (Grype, Trivy)
  • S3 storage alternative (MinIO) - now included

Testing Cycles

These containers are designed for fast testing cycles:

  • Prepare: One-time system setup (directories, SELinux, network)
  • Deploy: Quick container startup (seconds, not minutes)
  • Test: Run your integration tests
  • Remove: Clean removal when done
  • Iterate: Repeat as needed

Documentation Order

I am not sure how I want to document this yet. I feel there is a proper order, but I hate to impose until I get a better feel.