Kernel Test Scaffolding, Deterministic Waiting & Scenario Test Harness Subsystem Operational Architecture Guide
1. Executive Architectural Summary & Purpose
The Kernel Test Scaffolding, Deterministic Waiting & Scenario Test Harness Subsystem (pkg/testing) provides the authoritative foundation for hermetic, deterministic, and isolated testing across all layers of the ZQK Knowledge Kernel.
Testing high-concurrency event-driven kernel architectures introduces critical failure modes when relying on standard test paradigms:
- Flaky Sleep Invariants: Arbitrary
time.Sleep()calls lead to non-deterministic race conditions, test flakes, CI timeouts, or unnecessarily inflated test suite execution times. - Filesystem Cross-Contamination: Tests executing directly against repository trees risk corrupting
.zqk/process, polluting git worktrees, or causing state collisions across parallel test workers. - Unmanaged Background Goroutines: Orphaned goroutines that outlive individual test runs leak file descriptors, mutate shared memory, and trigger race detectors.
- Fragile Snapshot Assertions: Snapshot tests that fail to normalize dynamic timestamps, absolute paths, or machine-specific environments produce false positives in CI environments.
Governed by core platform mandates (including POL-DEFAULT-142277d3 for deterministic test verification and test data isolation), pkg/testing establishes an integrated suite of architectural primitives:
- Hermetic Environment Isolation: Automatically provisions per-test temporary working directories via
t.TempDir(), redirectingZQK_TEST_ROOTand suppressing external state leaks. - Deterministic Waiting & Polling: Replaces arbitrary sleeps with condition polling (
WaitForCondition,CallbackWaiter,WorkerStateTracker) bounded by explicit context timeouts and small tick intervals. - Scenario Sandboxing & Virtualization: Virtualizes checked-in scenario environments (
ScenarioTestEnvironment), creating executable isolated copies while preserving file permissions and directory hierarchies. - Snapshot Agnostic Verification: Provides structured snapshot restoration and validation patterns (
SnapshotTestHelper), enabling tests to verify system state against canonical baseline snapshots. - Thread-Safe Concurrency & Labeling: Guarantees parallel test safety across goroutines using explicit read-write synchronization and named execution labels via
goroutinelabels.NewGoroutine. - Strict Statement Coverage Floor: Enforces a static floor invariant of statement test coverage strictly \(\ge 85.0\%\) across the subsystem.
2. Component Topology & Subsystem Architecture
The testing subsystem is structured into modular components that coordinate to provide complete test lifecycle isolation and verification:
flowchart TD
subgraph TestExecution["Test Runner & Harness Client"]
TestRunner["Go Test Runner\n(go test ./pkg/...)"]
TestFn["Test Function (t *testing.T)"]
end
subgraph ScaffoldingCore["Kernel Test Scaffolding Core (pkg/testing)"]
direction TB
TestHarness["TestEnvironment\n(pkg/testing/test_setup.go)"]
ConfigManager["TestConfig\n(pkg/testing/config.go)"]
DeterministicWaiter["Deterministic Waiter\n(WaitForCondition / WaitForInvocation)"]
ScenarioEnv["ScenarioEnvironment\n(pkg/testing/scenario_environment.go)"]
CallbackRegistry["CallbackWaiter & AtomicCounter\n(pkg/testing/callback_helpers.go)"]
StateTracker["WorkerStateTracker\n(pkg/testing/callback_helpers.go)"]
SnapshotHelper["SnapshotTestHelper\n(pkg/testing/snapshot_test_helpers.go)"]
TimingTracker["TestTiming & ResultParser\n(pkg/testing/timing.go, result_parser.go)"]
end
subgraph IsolationSeams["Hermetic Sandboxing & Seams"]
TempDir["t.TempDir()\n(Isolated Filesystem Sandbox)"]
StorageSeam["storagetesting.IsolationFactory\n(File Storage & Audit Buffer Hook)"]
EnvGuard["zqkenv.ApplyIsolatedStorageEnv\n(ZQK_TEST_ROOT Redirect)"]
end
TestRunner --> TestFn
TestFn -->|"Initializes"| TestHarness
TestFn -->|"Polls Deterministically"| DeterministicWaiter
TestFn -->|"Loads Scenarios"| ScenarioEnv
TestFn -->|"Tracks Callbacks"| CallbackRegistry
TestFn -->|"Monitors Workers"| StateTracker
TestFn -->|"Validates Snapshots"| SnapshotHelper
TestFn -->|"Records Durations"| TimingTracker
TestHarness -->|"Binds"| ConfigManager
TestHarness -->|"Provisions"| TempDir
TestHarness -->|"Configures"| EnvGuard
TestHarness -->|"Injects Factory"| StorageSeam
Component Responsibilities
| Component | Architecture Role | Key Responsibilities | Source Reference |
|---|---|---|---|
TestEnvironment |
Complete Test Harness | Manages test temporary root creation, spec copying and generation, isolated storage factory instantiation, security context creation, and clean teardown. | test_setup.go#L29-L49 |
TestConfig |
Configuration Membrane | Evaluates ZQK_TEST_ROOT and ZQK_TEST_DATA_DIR, overriding project root paths and ensuring commands resolve to isolated test sandboxes. |
config.go#L20-L31 |
WaitForCondition |
Deterministic Poller | Executes polling over condition predicates with short interval ticks and context cancellation awareness, eliminating time.Sleep. |
deterministic_wait.go#L11-L29 |
CallbackWaiter |
Event Synchronization | Provides thread-safe callback notification tracking with fast-path checks and channel waiting for asynchronous callbacks. | callback_helpers.go#L11-L22 |
WorkerStateTracker |
Lifecycle State Monitor | Tracks worker startup and shutdown states deterministically, with non-blocking event dispatch and timeout-bounded condition waiting. | callback_helpers.go#L178-L190 |
ScenarioTestEnvironment |
Scenario Virtualizer | Copies checked-in test scenario trees from test-scenarios/ into ephemeral test roots, preserving file modes and binaries. |
scenario_environment.go#L22-L35 |
SnapshotTestHelper |
Snapshot Verifier | Handles test environment preparation and baseline snapshot evaluation, decoupling tests from static entity IDs. | snapshot_test_helpers.go#L17-L23 |
ParseGoTestOutput |
Test Output Parser | Parses go test stdout into structured TestRunSummary structs with status breakdowns and data race detection. |
result_parser.go#L12-L14 |
3. Test Lifecycle & Deterministic Wait Sequence
The following sequence illustrates the complete lifecycle of a hermetic test run: initializing the sandbox, applying environment isolation, waiting deterministically for asynchronous operations, and executing teardown.
sequenceDiagram
autonumber
actor Test as Unit / Integration Test
participant Harness as SetupCompleteTestEnvironment
participant Sandbox as t.TempDir() & Paths
participant Env as zqkenv / OS Environment
participant Seam as storagetesting.IsolationFactory
participant Waiter as Deterministic Waiter
participant Teardown as t.Cleanup Hook
Test->>Harness: SetupCompleteTestEnvironment(t, opts, factory)
Harness->>Sandbox: Create temp directory (testRoot)
Harness->>Env: Set ZQK_TEST_ROOT = testRoot
Harness->>Env: ApplyIsolatedStorageEnv()
Harness->>Sandbox: SetupTestEnvironment(testRoot) directories
Harness->>Sandbox: Write config/zqk-test.yaml settings
opt factory provided
Harness->>Seam: ConfigureTestGlobals(t, opts)
Harness->>Seam: CreateFileStorage(testRoot)
Seam-->>Harness: storageInstance
end
Harness->>Teardown: Register LIFO cleanup closure
Harness-->>Test: TestEnvironment (Root, Storage, Loaders)
Note over Test,Waiter: Async Kernel Work Execution
Test->>Waiter: WaitForCondition(ctx, conditionFn, pollInterval)
loop Every pollInterval until true or ctx.Done()
Waiter->>Test: Evaluate conditionFn()
alt conditionFn() == true
Waiter-->>Test: Return true (Success)
else Context Timeout / Cancel
Waiter-->>Test: Return false (Timeout)
end
end
Note over Test,Teardown: Test Completion & Hermetic Cleanup
Test-->>Teardown: Test function finishes
Teardown->>Seam: storageInstance.GetTestCleanup()()
Teardown->>Sandbox: Reclaim temp directories
4. Snapshot Testing & State Verification State Machine
Snapshot verification ensures that storage states, graph models, and process data structures match expected baseline representations without coupling tests to non-deterministic factors (like absolute machine paths or wall-clock timestamps).
stateDiagram-v2
[*] --> Initialized: SetupSnapshotTestEnvironment(snapshotPath)
state Initialized {
[*] --> TempDirAllocated
TempDirAllocated --> TestRootBound: Set ZQK_TEST_ROOT
TestRootBound --> OriginalRootResolved: findOriginalProjectRoot()
}
Initialized --> SnapshotRestoring: RestoreSnapshot(t)
state SnapshotRestoring {
[*] --> ExecuteInitWipe: zqk system init --from-snapshot --wipe
ExecuteInitWipe --> MarkRestored: Set IsRestored = true
}
SnapshotRestoring --> ReadyForVerification: State restored to isolated root
state ReadyForVerification {
[*] --> EvaluateStructure: Query Objects via Traversal
EvaluateStructure --> DiffDetection: Compare with Canonical Baseline
DiffDetection --> AssertionPassed: Zero Structural Divergence
DiffDetection --> AssertionFailed: Schema or Graph Divergence
}
ReadyForVerification --> [*]: Hermetic Teardown
Snapshot Agnostic Principles
- Dynamic ID Decoupling: Tests never assert literal object IDs minted at runtime. Comparison relies on structural properties, relationship types, and semantic payloads.
- Environment Variable Redirection: All snapshot restorations are pinned to
ZQK_TEST_ROOT, guaranteeing that developer checkouts or production databases are never touched. - Idempotent Restoration: Restorations can be replayed repeatedly within the isolated temporary folder without residual side-effects.
5. Scenario Environment Isolation & Filesystem Sandboxing
When verifying multi-command CLI behaviors or complex workflows, checked-in scenarios stored under test-scenarios/ provide rich starting states. The pkg/testing subsystem virtualizes these directories cleanly:
flowchart LR
subgraph RepoTree["Checked-In Repository State (Read-Only)"]
CheckedInScenario["test-scenarios/<scenario-name>\n⢠.zqk/process\n⢠test scripts\n⢠fixtures"]
end
subgraph SandboxedRoot["Ephemeral Test Sandbox (Read-Write)"]
IsolatedScenario["<t.TempDir()>/<scenario-name>\n⢠Isolated copy of fixtures\n⢠Runnable binaries\n⢠Ephemeral process data"]
end
subgraph TestContext["Test Process Context"]
ZqkTestRoot["ZQK_TEST_ROOT = IsolatedScenario"]
ExecTarget["cmd.Dir = IsolatedScenario"]
end
CheckedInScenario -->|"CopyDir(src, dst)\nPreserve File Modes"| SandboxedRoot
SandboxedRoot -->|"Points To"| ZqkTestRoot
SandboxedRoot -->|"Executes In"| ExecTarget
Sandboxing Mechanics
- Recursive Directory Copying:
CopyDirwalks the scenario directory tree usingfilepath.WalkDir, ensuring subdirectories are created withfileutil.EnsureDirand files are cloned withio.Copy. - Mode Preservation: Source file permissions (including POSIX executable bits
0o755) are preserved so that scenario initialization binaries remain executable in sandbox copies. - Fail-Safe Missing Directory Handling: If a scenario directory is not present,
SetupScenarioTestEnvironmentcallst.Skipfrather than failing, preventing local configuration variations from breaking test suites.
6. Concurrency Safety & Thread-Safe Test Harnesses
To support parallel test execution (t.Parallel()) and multi-agent swarm testing, all synchronization primitives in pkg/testing enforce strict concurrency guarantees:
1. CallbackWaiter Mutex Isolation
The CallbackWaiter coordinates asynchronous callbacks across arbitrary goroutines using an atomic counter (atomic.Int64) combined with a sync.Mutex guarding the waiter channel slice:
// Atomic fast-path avoids lock acquisition when already invoked:
if cw.invocations.Load() > 0 {
return true
}
2. WorkerStateTracker Read-Write Locking
WorkerStateTracker tracks worker status maps using sync.RWMutex, allowing concurrent read operations (IsRunning) while serializing state updates (SetState) and broadcasting state change notifications through non-blocking channel sends.
3. Named Goroutine Labels
When tests launch asynchronous support routines, they must not invoke raw unmonitored go func(). Tests utilize goroutinelabels.NewGoroutine:
goroutinelabels.NewGoroutine("testing", "concurrent worker").StartSimple(func() {
// Isolated worker logic
})
7. Verification & Coverage Invariant
In conformance with POL-DEFAULT-142277d3, the pkg/testing package enforces a strict statement test coverage invariant:
- Static Floor Invariant: Total package statement coverage must remain strictly \(\ge 85.0\%\).
- Target Threshold: Maintenance target of \(\ge 90.0\% - 100.0\%\).
- Verification Command:
bash ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/testing
AST Quality & Clean Code Invariants
- Zero Swallowed Errors: All error returns must be explicitly asserted or handled.
_ = ...pattern assignments on error returns are strictly prohibited. - Function Line Boundaries: Functions must not exceed 100 lines; complex test setups are factored into modular subtests and test helpers.
- Hermetic Storage Isolation: Test cases must never touch
.zqk/processin the actual repository; all filesystem mutations occur withint.TempDir(). - Open-Core Payload Integrity: Documentation guides and test files must not contain literal nanosecond kernel IDs.