Materialized View Reconciler Determinism & CI Flake Eradication
Abstract
This architectural specification details the determinism, concurrency safety, and test lifecycle guarantees for the whatsnext Materialized View projection reconciler in the ZQK Knowledge Kernel.
Background & Problem Statement
The whatsnext subsystem maintains a low-latency materialized projection (.zqk/state/whats_next_lite.json) allowing sub-millisecond evaluation of priority plans, active orders, and task routing. When projections grow older than DefaultStalenessTolerance, GetOrRecoverPayload executes one of two paths:
- Quiescent Graph Watermark Touch: If the lifecycle WAL (
lifecycle_events.wal) has received no new mutations sincepayload.MaterializedAt, the graph is verified to be quiescent. The watermark is updated in-place without triggering an expensive full CAS storage scan. - Active Graph Reconciler Circuit Breaker: If mutations have occurred since
MaterializedAt, the payload is returned withStale=trueandRecovering=true, and an asynchronous background rebuild (TriggerAsyncRebuild) is dispatched.
Under CI environments and concurrent test execution, two systemic test races were observed:
- Unseeded WAL Quiescence Race: When inducing staleness in test environments, failure to write an active WAL mutation newer than
MaterializedAtallowed the quiescent watermark touch to trigger instead of the active reconciler circuit breaker. - TempDir Cleanup / Goroutine Teardown Race: In error-recovery tests (
TestGetOrRecoverPayload_CorruptedAndPermissionError),TriggerAsyncRebuilddispatched asynchronous background goroutines that wrote to.zqk/state/while Go's test framework was concurrently executingos.RemoveAll(tempDir). This resulted inunlinkat: directory not emptytest failures. - Runner Scheduling Latency Spikes: Under concurrent test execution where multiple tests perform disk I/O, individual read iterations occasionally experience scheduling or GC pauses (>15ms), failing overly strict per-iteration cutoffs even when the average read latency remained well below 5ms.
Architectural Invariants & Remedies
1. Deterministic WAL Mutation Seeding
Tests verifying the asynchronous recovery circuit breaker must explicitly seed a lifecycle WAL mutation newer than MaterializedAt:
walPath := filepath.Join(tempDir, paths.ProjectDataDir, paths.WalDir, "lifecycle_events.wal")
_ = fileutil.EnsureDir(filepath.Dir(walPath))
_ = fileutil.WriteFile(walPath, []byte("event: test_mutation\n"), paths.FilePerm644)
Conversely, tests asserting quiescent behavior must verify that WAL mod times older than MaterializedAt refresh watermarks cleanly without entering recovery.
2. Supervised Reconciler Teardown
Any test invoking GetOrRecoverPayload or TriggerAsyncRebuild in a temporary test directory must guarantee that background goroutines have finished before t.TempDir() cleanup runs:
- Call
waitForReconcile(t, timeout)after triggering background rebuilds. - Register
t.Cleanuphandlers that restore file permissions and await background reconciler completion. - Reset the atomic gate
isReconcilingupon test completion.
3. Load-Resilient Latency Assertions
When verifying hot path performance over multi-iteration loops (e.g. 50 reads):
- Assert that the average read latency satisfies the performance invariant (
avg < 10msunder concurrency, targeting sub-millisecond median). - Allow individual iteration jitter up to 75ms on heavily loaded CI runners, bounding allowable outliers to ensure single-thread scheduling anomalies do not fail test suites.
Verification & Traceability
- Requirement: Materialized View Reconciler Determinism & CI Flake Eradication
- Test Suite:
pkg/workflow/whatsnext/materialized_view_test.go - Verification Gates:
- Functional Acceptance: Sub-5ms Hot Path and Quiescent Watermark Invariant
- Boundary & Error Handling: Quiescent Watermark Touch and Async Recovery Circuit Breaker
- Conformance: Single-flight Reconciler Mutex and TempDir Isolation Invariants