CLI Scheduler Subsystem Architectural & Operational Guide

Overview

The ZQK Scheduler Subsystem (cmd/zqk/scheduler, pkg/scheduler) provides an autonomous, resilient background orchestration engine, event-driven cron trigger dispatcher, cross-process daemon supervisor, and dead-letter failure escalation pipeline for the Zen Quantum Kernel.

Unlike external cron daemons (such as system cron or Jenkins) that run stateless tasks in detached environments, the ZQK Scheduler operates as a deeply integrated kernel subsystem. It executes scheduled and event-triggered jobs defined within the Content Addressable Storage (CAS) plane, records execution lineage into the Write-Ahead Log (WAL), maintains supervisor heartbeats with automated recovery, and manages lifecycle state machines across autonomous continuous improvement (CAP) loops.

zqk scheduler <subcommand> [flags]

Subsystem Command Taxonomy

The zqk scheduler CLI commands provide comprehensive controls across daemon supervision, ad-hoc job triggers, execution telemetry, and operational recovery:

Command / Invocation Primary Mode Description & Operational Behavior
zqk scheduler start Daemon Control Starts the scheduler background daemon (--background, default) or runs attached in foreground (--foreground) with live logging. Performs singleton PID mutual exclusion.
zqk scheduler stop Daemon Control Issues an orderly termination signal to the running scheduler daemon. Supports graceful drain with configurable timeout (--timeout) and force kill fallback (--force).
zqk scheduler restart Daemon Control Performs a coordinated stop-and-start sequence, ensuring existing background jobs cleanly finish before relaunching the supervisor.
zqk scheduler status Diagnostics Displays current daemon operational state, supervisor PID, uptime, active job worker pools, and keep-alive heartbeat freshness.
zqk scheduler health-check Monitoring / Probe External health check probe designed for monitoring agents and external cron watchdogs. Returns standardized exit codes (0 healthy, 1 unhealthy, 2 operational failure).
zqk scheduler submit <file\|-> Job Ingestion Submits ad-hoc jobs, test bundles, or scheduled job definitions to the scheduler trigger queue with environment overrides and callback specifications.
zqk scheduler trigger <job-id> Ad-Hoc Execution Enqueues an immediate trigger execution event for any registered job, bypassing its configured cron interval.
zqk scheduler list Query / Inspection Lists registered scheduler jobs with status, cron expressions, next execution slots, and last run timestamps (--format table\|json\|yaml).
zqk scheduler history Telemetry / Audit Analyzes historical job execution runs, success/failure ratios, and execution duration trends across rolling time windows (--since, --limit).
zqk scheduler activity Live Telemetry Shows real-time scheduler activity, current busyness, queued triggers, stuck jobs, and missed trigger alerts (--watch, --compact).
zqk scheduler wait <job-id> Synchronization Blocks until a submitted or triggered job finishes execution, streams status changes via WAL replay, and exits with the job's completion code.
zqk scheduler dump Diagnostics Emits complete runtime diagnostics, daemon environment variables, memory state, and active worker goroutine stack traces.
zqk scheduler escalate Alerting / Recovery Triggers dead-letter failure notifications to configured webhooks (Slack/custom) and writes persistent alert issues into .zqk/scheduler/issues.json.
zqk scheduler test-failures Quality Engineering Subsystem for inspecting persistent test failures, managing quarantine lists, debriefing convergence rollups, and auto-scheduling regression reruns.
zqk scheduler service Host Integration Manages host OS supervisor integration (systemd service units on Linux or launchd property lists on macOS) for automated crash-restart persistence.
zqk scheduler skip-window Maintenance Windows Sets temporary execution pause windows (set, clear, list) suppressing scheduled job triggers during maintenance.
zqk scheduler config Configuration Manages runtime scheduler configuration flags, worker concurrency limits, and log level verbosity.

Command Flags & Common Options

The scheduler subsystem provides precise flags for background execution, structured data serialization, and duration filtering:

Command Flag Short Default Description
start --background true Runs the scheduler daemon detached in the background and returns immediately.
start --foreground false Runs the scheduler daemon attached to standard output for live debugging.
start --minimal false Launches in minimal footprint mode, omitting secondary background maintenance runners.
stop --force -f false Sends SIGKILL if graceful shutdown does not complete within the timeout period.
stop --timeout -t 30s Maximum grace period allowed for running workers to finish before forced termination.
status --format -f table Output format: table, json, yaml.
list --enabled-only false Filters job output to only currently active, enabled timer jobs.
list --job-type "" Filters jobs by specific execution type (e.g. test-run, ambient-wave, maintenance).
history --since 24h Filters execution history to runs completed within the specified duration window.
history --job-id "" Restricts history report to a specific job identifier.
activity --watch -w 0s Periodic refresh interval for continuous console monitoring (e.g. 2s, 5s).
activity --stuck-stale-after 30m Elapsed duration after which an executing job without progress is classified as stuck.
escalate --severity -s warning Alert severity level: info, warning, critical.
escalate --dry-run -n false Simulates alert payload dispatch without sending actual network requests.
escalate --inbox-only false Writes escalation directly to local operator inbox without external webhooks.
skip-window --for "" Duration string for temporary execution suppression (e.g. 2h, 45m).
skip-window --until "" RFC3339 timestamp after which scheduled executions automatically resume.

Core Architecture & Execution Topology

The Scheduler Subsystem coordinates background concurrency between user commands, persistent storage, process supervisor loops, and filesystem state.

flowchart TD
    subgraph CLI["CLI Ingestion Layer (cmd/zqk/scheduler)"]
        CmdStart["zqk scheduler start"]
        CmdStop["zqk scheduler stop"]
        CmdStatus["zqk scheduler status"]
        CmdSubmit["zqk scheduler submit / trigger"]
        CmdHealth["zqk scheduler health-check"]
        CmdEscalate["zqk scheduler escalate"]
        CmdTestFail["zqk scheduler test-failures"]
    end

    subgraph Supervisor["Daemon Supervision & Process Control"]
        PIDLock["PID Mutex (.zqk/scheduler/scheduler.pid)"]
        KeepAlive["Heartbeat Writer (.zqk/scheduler/keep-alive)"]
        HostService["Host OS Supervisor (launchd / systemd)"]
        SignalHandler["Graceful OS Signal Interceptor"]
    end

    subgraph Engine["Scheduler Core Execution Engine (pkg/scheduler)"]
        TriggerQueue["Trigger Queue & Cron Parser"]
        WorkerPool["Worker Goroutine Pool"]
        JobState["Job State Machine (active -> running -> complete)"]
        DeadLetter["Dead-Letter & Issue Tracker"]
    end

    subgraph Storage["Knowledge Kernel Storage & State"]
        CAS["CAS Plane (.zqk/process/objects)"]
        WAL["Scheduler Events WAL (.zqk/scheduler/events.jsonl)"]
        UnhealthyFile[".zqk/scheduler/daemon-unhealthy.json"]
        IssuesJSON[".zqk/scheduler/issues.json"]
    end

    CmdStart --> PIDLock
    CmdStart --> KeepAlive
    CmdStop --> SignalHandler
    CmdStatus --> PIDLock
    CmdStatus --> KeepAlive
    CmdHealth --> KeepAlive
    CmdHealth --> UnhealthyFile
    CmdSubmit --> TriggerQueue
    CmdEscalate --> IssuesJSON
    CmdTestFail --> WAL

    PIDLock --> Engine
    KeepAlive --> Engine
    TriggerQueue --> WorkerPool
    WorkerPool --> JobState
    JobState --> CAS
    JobState --> WAL
    WorkerPool --> DeadLetter
    DeadLetter --> IssuesJSON

Detailed Architectural Mechanisms

1. Singleton Daemon Supervision & PID Mutual Exclusion

To guarantee that duplicate scheduler processes do not simultaneously poll cron expressions or double-dispatch jobs:

  • The scheduler writes its current OS process ID into .zqk/scheduler/scheduler.pid upon startup.
  • cmd/zqk/scheduler verifies process liveness via cross-process signal probing (os.FindProcess combined with signal(0) semantics).
  • If a stale PID file exists from an unclean reboot or killed process, the CLI detects that the PID is no longer running and safely reclaims the lock.
  • Cross-process commands (zqk scheduler status, zqk scheduler stop) query this PID file to communicate with the daemon without needing dedicated TCP sockets.

2. Keep-Alive Heartbeat & External Health Checking

The scheduler supervisor maintains continuous proof-of-life:

  • Every execution tick, the daemon writes an updated UTC RFC3339 timestamp into .zqk/scheduler/keep-alive.
  • External monitors execute zqk scheduler health-check periodically.
  • If the heartbeat age exceeds the allowable threshold (DefaultKeepAliveTimeout of 2 minutes) while enabled timer jobs exist in the CAS plane:
  • The health check writes an error report into .zqk/scheduler/daemon-unhealthy.json.
  • It records a scheduler_health_metric object into CAS.
  • It exits with code 1, alerting external monitoring tools or automated restarts.
  • If the daemon is healthy or cleanly stopped with no active timer jobs, it removes any lingering daemon-unhealthy.json and exits with code 0.

3. Trigger Queue & Worker Concurrency

Scheduled tasks follow a decoupled trigger-and-worker pattern:

  • Cron Slot Evaluation: The scheduler continuously checks enabled job objects in CAS whose cron expressions match the current minute slot.
  • Ad-Hoc Enqueuing: Commands like zqk scheduler trigger <job-id> or zqk scheduler submit inject execution triggers directly into the FIFO channel.
  • Worker Concurrency Ceiling: Configurable worker pools consume trigger events up to concurrency limits, isolating task execution from the main scheduling loop.
  • Skip Window Enforcement: Active maintenance windows configured via zqk scheduler skip-window are evaluated prior to dispatch; matching jobs have their triggers skipped and logged rather than queued.

4. Write-Ahead Event Logging & State Machine Transitions

Every execution phase is captured in an append-only event stream:

  • Execution starts, progress milestones, worker completions, and exit failures are logged to .zqk/scheduler/events.jsonl and mirrored to the kernel WAL.
  • Job state machines transition predictably: active ➔ running ➔ completed or failed.
  • Telemetry commands (zqk scheduler history, zqk scheduler activity) stream and parse these event logs without taking write locks on the CAS object store.

5. Dead-Letter Queue & Automated Escalation

When worker jobs encounter unrecoverable errors:

  • Failed jobs record failure metadata, error logs, and stack traces.
  • The failure count is compared against retry policies; exceeding thresholds marks the job failed and registers an entry in .zqk/scheduler/issues.json.
  • zqk scheduler escalate evaluates open issues and dispatches webhook notifications (with secret URL masking for security) or delivers notifications to operator inboxes.

Operational Scenarios & Recipes

Scenario A: Starting the Daemon in Foreground Mode for Debugging

# Start scheduler attached to the current shell with verbose streaming
zqk scheduler start --foreground

Scenario B: Checking Scheduler Health from External Monitoring Scripts

# External health check probe suitable for cron or Kubernetes liveness probes
zqk scheduler health-check
STATUS=$?
if [ $STATUS -ne 0 ]; then
  echo "Scheduler daemon unhealthy (exit code $STATUS)! Initiating restart..."
  zqk scheduler restart
fi

Scenario C: Submitting an Ad-Hoc Test Bundle and Waiting for Completion

# Submit a test bundle execution specification and block until complete
zqk scheduler submit ./specs/test-run-bundle.yaml
zqk scheduler wait SCH-test-run-bundle --timeout 5m

Scenario D: Managing Maintenance Windows

# Suppress all scheduled timer jobs for a 2-hour infrastructure upgrade
zqk scheduler skip-window set --for 2h --match-all

# Inspect active skip windows
zqk scheduler skip-window list

# Resume normal scheduling immediately after maintenance concludes
zqk scheduler skip-window clear

Verification & Statement Coverage Invariant

The Scheduler Subsystem enforces strict test coverage and quality invariants across all files in cmd/zqk/scheduler:

  1. High Statement Coverage Standard: Statement coverage across cmd/zqk/scheduler must achieve >= 85.0%.
  2. Hermetic Test Isolation: Tests creating temporary project workspaces must register cli.UnregisterStorageForProjectRoot(tmpDir) in t.Cleanup to eliminate background timer races, lingering goroutines, and directory cleanup flakes.
  3. AST Quality Invariant: Zero swallowed errors across test fixtures and production code. Error returns must be checked with require.NoError, assert.NoError, or explicit conditional handling rather than blank assignments (_ = ...).
  4. Modular Function Length: Test and implementation functions must remain modular (under 100 lines per function), breaking complex test scenarios into structured sub-helpers.
  5. Fast Test Execution: Targeted test runs against ./cmd/zqk/scheduler must execute hermetically within a strict 60-second timeout ceiling (-timeout 60s), avoiding broad unbounded package scans.

To verify the test suite and coverage locally:

ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 60s -cover ./cmd/zqk/scheduler