Scheduler Lifecycle Finalization and Teardown Synchronization
Executive Summary
The ZQK scheduler coordinates periodic, event-driven, and on-demand job executions across multi-agent runtimes, verification suites, and system maintenance tasks. In distributed or high-concurrency environments, non-deterministic job finalization and test fixture teardown can cause subtle race conditions:
- Premature Completion Signalling: If in-memory flags (
job.Running = false) or concurrency conflict registrations are cleared before storage persistence, audit journaling, and log file flushing finish, external observers or subsequent jobs observe stale states or race with in-flight I/O. - Asynchronous Teardown Resource Contention: If test fixtures shut down storage providers or remove temporary directories (
t.TempDir()) while background scheduler workers, log writers, or CAS write-behind queues are still active, OS-level filesystem unlink operations fail withunlinkat ...: directory not emptyor resource contention errors.
This specification formalizes the strict synchronization invariants governing the scheduler job execution lifecycle and test fixture teardown across the Knowledge Kernel.
1. Job Execution Lifecycle Finalization Invariant
1.1 Problem Analysis
In pkg/scheduler/job_execution.go, executeJobAfterHandlerReturns is responsible for recording execution metrics, publishing coordination events, persisting the job's new state (last_run_at, status, enabled), writing job log entries, and releasing execution locks.
Previously, job.Running = false and unregisterCallback() (which unregisters the job from the conflict manager) were executed immediately after handler return, prior to:
- State registry execution completion (
s.stateRegistry.CompleteExecution) - Coordination channel event publication (
s.coordinationChannel.PublishEvent) - Storage persistence (
s.updateJobInStorageands.disableJobInStorage) - Job log writing and audit event recording (
s.handleJobExecutionOutcome) - Operation callback dispatch (
opCallback.OnComplete/opCallback.OnError) - Job log writer flushing and closure (
CloseJobLogWriter)
Consequently, external callers and test assertions awaiting completion via waitForJobFinalized (which polls job.Running == false) or via opCallback would resume execution and initiate teardown while storage writes and log file flushes were still active in the background.
1.2 Ordered Finalization Sequence
To eliminate this race, executeJobAfterHandlerReturns enforces a strict linear completion pipeline:
flowchart TD
A["Job Handler Returns"] --> B["Record Metrics & Emit Cluster Status"]
B --> C["State Registry CompleteExecution & Coordination PublishEvent"]
C --> D["Invoke Configured Job Callbacks"]
D --> E["Persist Storage Updates (disableJobInStorage / updateJobInStorage)"]
E --> F["Handle Outcome: Audit Event Persist & opCallback Dispatch"]
F --> G["Touch Scheduler Activity"]
G --> H["Flush & Close Buffered JobLogWriter"]
H --> I["Clear Context & Set job.Running = false (under RunningMu)"]
I --> J["Unregister From Conflict Manager (unregisterCallback)"]
1.3 Key Invariants
- Storage-Before-Callback Invariant: Job state in storage (including updated
last_run_at,next_run_at, and disabled state for one-time jobs) is guaranteed to be persisted beforeopCallback.OnCompleteis dispatched. - Log-Flush-Before-Running-Clear Invariant:
CloseJobLogWriter(s.getProjectRoot(), job.ID)runs beforejob.Running = false. Callers inspecting the filesystem immediately uponjob.Running == falseare guaranteed that all buffered job logs are flushed and closed. - Conflict-Deregistration-at-Tail Invariant: The job remains registered in the concurrency conflict manager until all background writes and state updates have completed. Subsequent jobs with conflicting locks will never be admitted while the prior job's teardown is in flight.
2. Test Fixture Teardown Synchronization Invariant
2.1 Problem Analysis
In pkg/scheduler/scheduler_test.go, tests create isolated testing environments using setupTestScheduler(t). The test fixture instantiates an in-memory or filesystem-backed ObjectStorageProvider, sets up project roots, and initializes a Scheduler instance.
Previously, two issues led to flaky directory cleanup in CI:
- Uncalled Teardown on Ignored Return: Callers ignoring the 3rd return value (
sched, _, _ := setupTestScheduler(t)) failed to executeenv.Cleanup(), leaving storage provider write-behind threads and CAS index queues alive duringt.TempDir()removal. - Teardown Inversion on Deferred Cleanup: If a test executed
defer cleanup(),env.Cleanup()ran beforet.Cleanup(func() { sched.Stop() }), shutting down storage while the scheduler's background tickers were still running.
2.2 Unified Idempotent Teardown Pattern
setupTestScheduler provides a centralized, idempotent teardown closure governed by sync.Once, registered directly with t.Cleanup:
var teardownOnce sync.Once
teardown := func() {
teardownOnce.Do(func() {
if sched != nil {
sched.Stop()
}
CloseAllJobLogWriters(env.TestRoot)
_ = caspkg.GetGlobalListingIndexWriteQueue().FlushKind("scheduler_job", 2*time.Second)
_ = caspkg.GetGlobalListingIndexWriteQueue().FlushKind("audit_event", 2*time.Second)
storagepkg.FlushListingIndexForProjectRoot(env.TestRoot, "scheduler_job")
if env.Cleanup != nil {
env.Cleanup()
}
_ = fileutil.RemoveAll(datacell.ProcessPrimaryDir(env.TestRoot))
_ = fileutil.RemoveAll(filepath.Join(env.TestRoot, paths.ProjectDataDir))
})
}
t.Cleanup(teardown)
return sched, env.TestRoot, teardown
2.3 Teardown Execution Order
Regardless of whether a test explicitly invokes defer cleanup(), registers t.Cleanup(cleanup), or omits calling cleanup, the teardown executes exactly once in deterministic sequence:
sched.Stop(): Halts all scheduler cron engines, job dispatcher queues, and worker goroutines.CloseAllJobLogWriters(env.TestRoot)(Scoped byprojectRoot): Closes, flushes, and evicts only active file handles forenv.TestRoot. Writers for concurrent tests sharing the process are preserved, preventing cross-test writer eviction and race conditions during parallel execution.- Queue Draining: Flushes global CAS listing index queues and local storage listing index queues.
- Idempotent Storage Shutdown (
FileObjectStorage.Shutdown): Protected byshutdownOnce sync.Once. Subsequent or duplicate cleanup calls (from test teardown andt.Cleanup) are safely ignored, preventing re-entrant CAS directory re-creation. - Pre-Pipeline Async Draining (
RunProjectTestTeardown): Drains Darwin CAS sync queues (DrainDarwinSyncQueue), stops reverse reference indexing, and shuts down hash registries before running project root scrubbing and process artifact stripping. - LIFO StripRoot Cleanup: Empties
.zqk/processand.zqk/after all storage components have completely terminated, guaranteeing cleanos.RemoveAll(t.TempDir())execution withoutunlinkat ...: directory not emptyflakes.
3. Storage Provider & Writer Invariants
3.1 Scoped Log Writer Eviction Invariant
In pkg/scheduler/job_log_writer.go, CloseAllJobLogWriters(projectRoot string) restricts eviction to entries whose key starts with projectRoot + "\x00". When projectRoot is provided, active writers belonging to other running tests remain untouched in the global registry.
3.2 Idempotent Storage Shutdown Invariant
In pkg/storage/object_storage_file.go, FileObjectStorage.Shutdown wraps execution in f.shutdownOnce.Do(...). Calling Shutdown repeatedly will neither re-enter pending cache dumps nor re-create .zqk directory structures after teardown has completed.
3.3 Pre-Scrub Component Drain Invariant
In pkg/storage/project_test_teardown.go, drainStorageAsyncComponents is invoked prior to executing the standard teardown pipeline. Background Darwin fsync goroutines and hash registry workers complete all disk I/O before any directory scrubbing begins.
4. Zero-Flake Compliance & Verification
- Automated Verification Suite:
pkg/scheduler/scheduler_test.goandpkg/scheduler/job_lock_integration_test.go. - Concurrency Safety: Race detector (
go test -race) and parallel execution suites run cleanly without goroutine leaks,unlinkatfailures, or lock acquisition timeouts. - Fail-Closed Invariants: Any panic occurring during job execution is caught by
recover(), correctly publishing ajob_execution_failedcoordination event and invokingexecuteJobAfterHandlerReturnsto guarantee lock release and clean state teardown.