Scheduler & Ambient TempDir Teardown Lifecycle Determinism Guide
1. Executive Summary & Problem Statement
In high-concurrency, race-detected test suites across pkg/scheduler and pkg/ambient, test suites that manipulate .zqk/ or initialize subsystem storage inside temporary project roots (t.TempDir()) historically exhibited intermittent flakes:
testing.go:1234: TempDir RemoveAll cleanup: unlinkat /var/folders/.../001/.zqk/scheduler: directory not empty
These errors stem from two systemic lifecycle hazards:
- Asynchronous Storage & Buffer Persistence: Subsystems instantiate or dynamically resolve storage providers (e.g. via
storage.GetGlobalStorageProviderCache()andstorage.GetGlobalBufferRegistry()) without registering an explicit teardown pipeline. As a consequence, background WAL flushers, listing index queues, and audit event buffers continue dispatching filesystem writes after the test function returns. - Unjoined Ambient Daemons & Stream Writers: Background stream writers, event mesh cancel-forwarders, and ambient ingest holons remain active when tests complete without deterministic joins, racing with Go's built-in
testing.TB.TempDir()cleanup (os.RemoveAll).
This guide documents the core invariants, registration patterns, and lifecycle contracts required to ensure 100% deterministic test cleanup across all packages.
2. Core Teardown Architecture & Pipeline
Deterministic teardown is achieved by integrating test temporary directories into the StandardProjectTestTeardownPipeline via pkg/testkit.
flowchart TD
subgraph TestExecution["Test Execution Phase"]
TDir["tmpDir := t.TempDir()"]
RegClean["registerProjectDirTestTeardown(t, tmpDir)\nOR testkit.RegisterStorageTestCleanup(t, tmpDir, sp)"]
TestWork["Execute Unit / Integration Logic\n(Scheduler / Ambient Ingest / Storage Writes)"]
end
subgraph LIFOCleanup["LIFO Teardown Phase (t.Cleanup)"]
direction TB
Stage1["1. Flush & Drain Listing Index Queues"]
Stage2["2. Wait for WAL & FileStorage.Shutdown()"]
Stage3["3. Provider Cache Eviction (GetGlobalStorageProviderCache().Delete)"]
Stage4["4. Buffer Registry Unregister (GetGlobalBufferRegistry().Unregister)"]
Stage5["5. Drain Async Storage & Orphan Cleanup"]
Stage6["6. Strip Process Artifacts & Scrub Project Root"]
Stage7["7. Force RemoveAll(tmpDir) Safety Floor"]
Stage8["8. t.TempDir() os.RemoveAll (Guaranteed Empty)"]
end
TDir --> RegClean --> TestWork
TestWork --> Stage1
Stage1 --> Stage2 --> Stage3 --> Stage4 --> Stage5 --> Stage6 --> Stage7 --> Stage8
2.1 The Standard Teardown Pipeline Stages
When testkit.RegisterStorageTestCleanup is registered:
- Global Listing Index & Write Queue Flush: Flushes any queued listing updates to disk and shuts down background worker channels.
- WAL Draining & FileStorage Shutdown: Halts the WAL write loop within a bounded timeout (45s max, fast exit when empty) and invokes
storage.FileObjectStorage.Shutdown(). - Global Storage Provider Cache Eviction: Calls
Shutdown()on the cached provider registered underprojectRootand removes it fromGetGlobalStorageProviderCache(). - Audit Buffer Unregistration: Unregisters and flushes any active in-memory audit buffers from
GetGlobalBufferRegistry(). - Process Artifact Stripping & Root Scrubbing: Strips temporary files and scrubs the project tree before directory deletion.
3. Mandatory Implementation Patterns
Pattern A: Tests Creating Files in .zqk/ Without a Storage Instance
For tests exercising helpers that touch .zqk/ or .zqk/scheduler/ (such as ReportIssue, SaveJobTypeViewCacheAtomic, WriteNoAutoRestartFile, EmitHumanInTheLoomInterrupt, WriteLockMetadata, writeKeepAlive):
func TestSchedulerIssues_Example(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
registerProjectDirTestTeardown(t, tmpDir)
ReportIssue(tmpDir, "SCH-TEST-ERR", "validation", "simulated job failure")
// ... assertions ...
}
Where registerProjectDirTestTeardown is defined as:
func registerProjectDirTestTeardown(t *testing.T, projectRoot string) {
t.Helper()
t.Cleanup(func() {
if err := fileutil.RemoveAll(projectRoot); err != nil {
t.Logf("cleanup: %v", err)
}
})
testkit.RegisterStorageTestCleanup(t, projectRoot, nil)
}
Because t.Cleanup functions execute in Last-In First-Out (LIFO) order:
testkit.RegisterStorageTestCleanupruns first, ensuring all caches, buffers, and async components are stopped and flushed.- The
RemoveAllclosure runs second, wiping the directory tree cleanly. - The underlying
t.TempDir()cleanup runs last and encounters an already-deleted or empty directory, preventingunlinkat ENOTEMPTY.
Pattern B: Tests Instantiating FileObjectStorage
func TestSchedulerWithStorage_Example(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
sp, err := storagepkg.NewFileObjectStorageForTest(tmpDir)
require.NoError(t, err)
registerFileStorageTestTeardown(t, tmpDir, sp)
// ... test logic ...
}
Pattern C: Ambient Ingest & Background Event Streams
For ambient event services and daemons:
- Join Forwarder Goroutines: Ensure all cancel-forwarders and stream consumers are registered in the service's
sync.WaitGroup(loopWg.Add(1)). - Track In-Flight Ingests: Ensure all ingest entry points (
Ingest,IngestAmbientEvent) increment and decrementinFlightatomic counters. - Explicit Stop in Defer / Cleanup: Always call
defer svc.Stop()ort.Cleanup(func() { svc.Stop() }), followed byhub.WaitForAudits().
t.Run("BindToHub event delivery", func(t *testing.T) {
tempDir := t.TempDir()
testkit.RegisterStorageTestCleanup(t, tempDir, nil)
defer storage.ScrubProjectRootForTempCleanup(tempDir, 10, 10*time.Millisecond)
secCtx := pkgctx.NewSystemSecurityContext()
svc := NewAmbientIngestService(tempDir, secCtx)
defer svc.Stop()
hub := NewEventHub()
defer hub.WaitForAudits()
svc.BindToHub(hub)
// ... test logic ...
})
4. Verification & Race Detection
All test suites touching filesystem storage or background goroutines must pass with -race and repeated executions:
go test -race -count=2 ./pkg/scheduler/...
go test -race -count=2 ./pkg/ambient/...
Adherence to these contracts guarantees zero resource leaks, zero unjoined goroutines, and deterministic TempDir removal across all environments.