Test Services Lifecycle, Fixture Management & Ephemeral Environment Subsystem Operational Architecture Guide
1. Architectural Overview
The Test Services Subsystem (pkg/testservices) provides robust lifecycle management, fixture orchestration, and ephemeral container coordination for integration and system tests across the ZQK Knowledge Kernel execution plane (POL-DEFAULT-b17f54c251eb3c4d, CRI-TEST-SERVICES).
In complex, multi-agent distributed systems and graph-centric environments, integration tests frequently depend on auxiliary infrastructure services (such as the MemGraph knowledge graph engine, TSDB backends, or caching layers). Naively spinning up unbounded, uncontrolled external processes introduces severe non-determinism, port collisions, resource leakage, and slow test execution.
pkg/testservices delivers a lightweight, reliable, self-healing lifecycle harness that:
- Verifies Container Engine Availability: Safely probes host Docker availability before attempting ephemeral service deployment, failing closed with clean error wrapping when prerequisites are absent.
- Eliminates Port Conflicts via Listener Reuse: Detects existing long-lived service instances (such as a local background MemGraph instance listening on port 7687) and seamlessly reuses established listeners rather than failing docker bindings with
port is already allocated. - Serializes Parallel Container Spawns: Employs fine-grained, per-container mutex synchronization (
lockStartService) to prevent race conditions during concurrent test executions across package boundaries. - Enforces Multi-Protocol Health Polling: Supports active health verification via raw TCP socket dialing, HTTP endpoint status checks, and in-container command execution before declaring an ephemeral service ready.
- Guarantees Hermetic Teardown: Provides automated cleanup mechanisms with graceful stop timeouts and forced container removal fallback, ensuring zero residual zombie containers across test suites.
2. Component Topology
The subsystem is structured into five core functional components:
flowchart TD
subgraph TestExecutionPlane["Test Execution Plane"]
Runner["Test Runner / Suite"]
Setup["SetupTestServices(ctx)"]
TestServices["TestServices Container Handle"]
end
subgraph TestServicesCore["pkg/testservices Subsystem"]
SM["ServiceManager"]
LockRegistry[("Container Lock Registry (startServiceLocks)")]
Config["ServiceConfig & HealthCheck Specs"]
HealthEngine["HealthCheck Engine (TCP / HTTP / Exec)"]
PortSentinel["Port Publisher Sentinel"]
DockerGovernor["Container Lifecycle Governor"]
end
subgraph HostPlane["Host & Container Runtime"]
DockerCLI["Docker Engine CLI / Socket"]
DaemonListener["Existing Host Listener (e.g. :7687)"]
EphemeralBox["Ephemeral Container Instance"]
end
Runner -->|Initialize| Setup
Setup -->|Instantiate| SM
Setup -->|Config Specs| Config
SM -->|Per-Container Mutex| LockRegistry
SM -->|Host Port Inspection| PortSentinel
PortSentinel -->|Check Port Allocation| DockerCLI
PortSentinel -.->|Listener Detected| DaemonListener
SM -->|Command Dispatch| DockerGovernor
DockerGovernor -->|docker run / rm / stop| DockerCLI
DockerCLI -->|Spawn| EphemeralBox
SM -->|Poll Readiness| HealthEngine
HealthEngine -->|nc / curl / exec| EphemeralBox
Setup -->|Yield Handle| TestServices
TestServices -->|Teardown / Cleanup| SM
Component Roles & Responsibilities
| Component | Responsibility | Key Invariants |
|---|---|---|
ServiceManager |
Coordinates test service initialization, health verification, and teardown. | Immutable configuration access; structured logging through pkg/logging when verbose mode is enabled. |
ServiceConfig |
Defines container name, base image, port bindings, environment variables, timeouts, and health check criteria. | Passed by value to ensure immutability across concurrent test workers. |
Container Lock Registry |
Global synchronization map (sync.Map) managing per-container mutexes (lockStartService). |
Prevents duplicate docker run or docker rm race conditions when parallel tests launch simultaneously. |
Port Publisher Sentinel |
Probes active host container publish rules via docker ps --filter publish=<port>. |
Reuses pre-existing running service listeners instead of failing duplicate port bindings. |
HealthCheck Engine |
Polls service endpoints across TCP (nc -z), HTTP (curl -f), or in-container command execution. |
Bounded deadline loops; retries transient failures; automatically cascades container teardown on failure. |
Container Lifecycle Governor |
Standardized command argument builder (BuildDockerRunArgs) and process execution wrapper. |
Uses pkg/execwrap for contextual process termination and clean signal propagation. |
3. Test Services Lifecycle Sequence
The lifecycle of an ephemeral test service spans configuration resolution, environment preflight, port allocation probing, container execution, readiness verification, and guaranteed teardown:
sequenceDiagram
autonumber
actor Test as Test Runner
participant SM as ServiceManager
participant Locks as Lock Registry
participant Docker as Docker CLI / Daemon
participant Probe as HealthCheck Engine
participant Target as Container / Service
Test->>SM: StartService(ctx, config)
SM->>Docker: docker info (preflight check)
alt Docker Engine Unavailable
Docker-->>SM: Error (daemon unreachable)
SM-->>Test: Error: docker is not available
else Docker Engine Ready
SM->>Locks: lockStartService(containerName)
Locks-->>SM: Mutex Acquired
SM->>Docker: docker ps --filter name=containerName
alt Container Already Running
Docker-->>SM: Container Name Match
SM-->>Test: Success (Reused Running Container)
else Container Not Running
SM->>Docker: docker ps --filter publish=hostPort
alt Host Port Already Published
Docker-->>SM: Port Listener ID Found
SM-->>Test: Success (Reused Host Listener)
else Host Port Free
SM->>Docker: docker rm -f containerName (purge stale)
SM->>Docker: docker run -d --name containerName ...
Docker-->>SM: Container ID
loop Health Check Polling until Deadline
SM->>Probe: checkHealth(ctx, config)
Probe->>Target: Probe (TCP / HTTP / Command)
alt Probe Succeeded
Target-->>Probe: Healthy (Exit 0 / HTTP 200 / Open Port)
Probe-->>SM: Healthy (true)
else Probe Failed
Target-->>Probe: Unhealthy / Connection Refused
Probe-->>SM: Retry (false)
end
end
alt Health Verified
SM-->>Test: Service Started & Healthy (nil)
else Health Check Timed Out
SM->>Docker: docker stop containerName
SM->>Docker: docker rm containerName
SM-->>Test: Error: service failed health check
end
end
end
end
4. Health Check Protocols & Probing Mechanisms
pkg/testservices implements three dedicated health check protocols to ensure services are fully operational before tests proceed:
1. TCP Port Dialing (Type: "port")
- Mechanism: Executes
nc -z localhost <port>under the request context. - Use Case: Validates database listener availability (e.g. MemGraph Bolt port
7687, Redis port6379, Postgres port5432). - Characteristics: Fast detection of TCP socket acceptance; minimal resource overhead.
func (sm *ServiceManager) checkPortHealth(ctx context.Context, port string) (bool, error) {
cmd := exec.CommandContext(ctx, "nc", "-z", "localhost", port)
err := cmd.Run()
return err == nil, nil
}
2. HTTP Endpoint Verification (Type: "http")
- Mechanism: Dispatches
curl -f -s http://localhost:<port><path>to query service status endpoints. - Use Case: Verifies HTTP API services, metrics endpoints, or health probes (
/health,/readyz). - Characteristics: Requires a successful HTTP 2xx response status code; handles path routing.
func (sm *ServiceManager) checkHTTPHealth(ctx context.Context, port, path string) (bool, error) {
url := fmt.Sprintf("http://localhost:%s%s", port, path)
cmd := exec.CommandContext(ctx, "curl", "-f", "-s", url)
err := cmd.Run()
return err == nil, nil
}
3. In-Container Exec Command Probing (Type: "command")
- Mechanism: Executes configured diagnostic commands (e.g.
redis-cli ping,pg_isready,memgraph-client) usingexec.CommandContext. - Use Case: Validates internal engine state when raw port listening does not guarantee readiness.
- Characteristics: Returns error on empty command definitions; requires zero exit code for health.
func (sm *ServiceManager) checkCommandHealth(ctx context.Context, command []string) (bool, error) {
if len(command) == 0 {
return false, errfmt.Errorf("health check command is empty")
}
cmd := exec.CommandContext(ctx, command[0], command[1:]...)
err := cmd.Run()
return err == nil, nil
}
5. Port Conflict Resolution & Listener Reuse
In local developer environments and long-running daemon setups, services such as MemGraph are often already running in background containers or native system daemons (e.g. zqk-memgraph listening on 7687 and 7444).
Attempting to run docker run -p 7687:7687 ... against an occupied port results in immediate Docker daemon failure:
Bind for 0.0.0.0:7687 failed: port is already allocated
pkg/testservices resolves this friction through its PortPublisherSentinel:
-
Before invoking
docker run,hostPortsAlreadyPublishedqueries Docker:bash docker ps --filter publish=<hostPort> --format "{{.ID}}" -
If every host port required by the
ServiceConfigis already published by an active container, the manager logs listener reuse and returns success immediately:text [SERVICE] Service memgraph host ports already published; reusing existing listener -
If only a partial subset of required ports is active or conflicting, the manager does not guess; it proceeds to spawn the configured container, allowing standard error escalation if allocation is invalid.
6. Concurrency & Thread-Safety
When parallel test packages execute in Go (go test -parallel), multiple test suites may independently invoke SetupTestServices or sm.StartService with identical container names.
Serialization via lockStartService
To guarantee thread-safety without cross-test deadlock, pkg/testservices maintains a synchronized lock registry:
var startServiceLocks sync.Map // containerName -> *sync.Mutex
func lockStartService(containerName string) *sync.Mutex {
v, _ := startServiceLocks.LoadOrStore(containerName, &sync.Mutex{})
return v.(*sync.Mutex)
}
Managed Concurrency with goroutinelabels
All background routines and concurrent test workers adhere to the ZQK concurrency mandate:
- Goroutines are managed via
goroutinelabels.NewGoroutine("<name>", "<purpose>").StartSimple(...). - Naked
go ...statements are strictly prohibited to maintain trace attribution and prevent goroutine leakage. - Clean shutdown cascades ensure that context cancellation unblocks pending health loops without hanging test executors.
7. Operational Recipes & Usage Patterns
Recipe 1: Initializing Standard Test Services
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
// SetupTestServices respects config/zqk.yaml (storage.graph_enabled)
services, err := testservices.SetupTestServices(ctx)
if err != nil {
t.Fatalf("failed to initialize test services: %v", err)
}
defer func() {
if cleanupErr := services.Cleanup(); cleanupErr != nil {
t.Errorf("cleanup error: %v", cleanupErr)
}
}()
Recipe 2: Starting a Custom Ephemeral Container
sm := testservices.NewServiceManager()
cfg := testservices.ServiceConfig{
Name: "redis-cache",
ContainerName: "zqk-test-redis",
Image: "redis:7-alpine",
Ports: map[string]string{
"6379": "6379",
},
Environment: map[string]string{
"ALLOW_EMPTY_PASSWORD": "yes",
},
HealthCheck: testservices.HealthCheck{
Type: "port",
Port: "6379",
Interval: 500 * time.Millisecond,
Timeout: 10 * time.Second,
},
StartTimeout: 15 * time.Second,
StopTimeout: 5 * time.Second,
}
if err := sm.StartService(ctx, cfg); err != nil {
log.Fatalf("failed to start custom ephemeral service: %v", err)
}
defer sm.StopService(context.Background(), cfg.ContainerName)
8. Verification & Coverage Invariant
The pkg/testservices package enforces a strict statement test coverage floor of >= 85.0% (achieving 100.0% statement coverage):
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/testservices
Verification Matrix
lockStartService: Concurrent access, pointer uniqueness, distinct mapping safety.NewServiceManager&log: Verbose enabled logging and silent disabled logging.StartService: Docker unavailable branch, already-running container reuse, published port reuse, run failure, health check timeout cascade stop, and healthy start.StopService: Non-running container early return, stop command execution failure, clean removal.waitForHealthy: Default timeouts, context cancellation unblocking, transient error retries.BuildDockerRunArgs: Argument order, port flag formatting, environment flag sorting, extra arguments.SetupTestServices&Cleanup: Storage graph enabled/disabled pathways, start failure wrapping, and cleanup error aggregation.