Runtime Subsystem Architectural & Operational Guide
Comprehensive architectural manual, goroutine lifecycle governance specification, and operational guide for the ZQK Runtime Subsystem (pkg/runtime).
1. Overview & Architectural Compass
The Runtime Subsystem (pkg/runtime) provides the centralized supervision tree, deterministic goroutine lifecycle governance, leak detection heuristics, and graceful shutdown orchestration for the Zen Quantum Knowledge Kernel.
In an autonomous multi-agent operating environment where concurrent background daemons, scheduled cron jobs, event-driven streaming pumps, and worker pools operate concurrently, raw, unmanaged Go goroutines (go func()) introduce serious operational risks:
- Goroutine Leakage: Orphaned background loops that fail to listen to parent context cancellation stay resident indefinitely, consuming memory and thread handles.
- Uncontrolled Panics: An unhandled panic in an unmanaged goroutine crashes the entire process, circumventing application recovery hooks.
- Resource Abandonment: Goroutines allocating persistent resources (such as active tickers, file descriptors, or network streams) without deterministic cleanup hooks cause descriptor exhaustion.
- Observability Voids: Unlabeled goroutines provide zero telemetry regarding their purpose, execution duration, failure status, or CPU cost during crash analysis.
- Ungraceful Termination: When a process receives an interrupt or shutdown signal, executing routines may be abruptly terminated mid-mutation, causing state corruption in CAS databases or transaction logs.
The pkg/runtime subsystem eradicates these liabilities through a supervised Goroutine Manager, context-aware lifecycle hooks, automated panic recovery membranes, and deterministic drain coordination.
flowchart TD
subgraph Clients["Subsystem Callers"]
CLI["CLI Commands & Interactive TUI"]
Daemon["Ambient Daemons & HTTP Listeners"]
Sched["CAP Loop & Scheduler Services"]
Coord["Coordination & Event Mesh"]
end
subgraph RuntimeCore["pkg/runtime Subsystem"]
subgraph Manager["GoroutineManager Supervision Engine"]
GM["GoroutineManager (Start, Stop, StopAll, Shutdown)"]
Limits["MaxGoroutines & Admission Gatekeeper"]
Drain["Graceful Drain & Shutdown WaitGroup"]
end
subgraph Tracking["Lifecycle & State Tracking"]
TG["TrackedGoroutine (ID, Name, Purpose, Category)"]
SM["State Machine (Starting, Running, Stopping, Stopped, Error, Leaked)"]
ResMgr["Resource Registry (Tickers, Channels, File Handlers)"]
end
subgraph Protection["Fault Tolerance & Observability"]
PH["Panic Handler & AST Recovery Membrane"]
LD["Leak Detection Engine (Heuristics & Notification Channel)"]
Stats["Runtime Telemetry & Stats Aggregation"]
end
subgraph Helpers["Reusable Operational Patterns"]
Periodic["StartPeriodicTask"]
WorkerPool["StartWorkerPool"]
AsyncEv["EmitEventAsync"]
ExecAsync["ExecuteAsync"]
BaseSvc["BaseService (Service Interface)"]
end
end
Clients --> Manager
Clients --> Helpers
Helpers --> Manager
Manager --> Tracking
Manager --> Protection
Core Responsibilities
- Supervised Lifecycle Governance: Every managed background routine is assigned a unique identifier, purpose label, category, parent context, and lifecycle state.
- Automatic Resource Cleanup: Associates tickers, channels, and open streams directly with the tracked goroutine, guaranteeing invocation of cleanup routines upon termination.
- Panic Interception & Containment: Captures any panics occurring within managed goroutines, formats structured diagnostic errors, marks the routine as failed, and notifies telemetry hooks without crashing the host process.
- Leak Detection & Notifications: Implements heuristic duration monitoring and provides non-blocking notification channels to catch stalled goroutines during shutdown.
- Two-Phase Graceful Drain: Sequentially cancels individual goroutine contexts, triggers resource cleanup, cancels the root manager context, and coordinates a timed drain wait before flagging leaked routines.
2. Subsystem Taxonomy & Core Primitives
2.1 GoroutineManager Supervision Engine
The central coordinating type is GoroutineManager (goroutine_manager.go), which tracks and coordinates the lifecycle of all goroutines within a service or command boundary.
type GoroutineManager struct {
goroutines map[string]*TrackedGoroutine
coordinator coordination.EventCoordinator
shutdownCtx context.Context
shutdownCancel context.CancelFunc
maxGoroutines int
shutdownTimeout time.Duration
leakNotificationChan chan string
// internal synchronization and atomic counters
}
Manager Lifecycle Operations
-
Creation (
NewGoroutineManager): Derives a cancellation context from the parent process context. Instantiates internal registries, atomic counters, and default timeout thresholds (default shutdown timeout:30s). -
Admission & Start (
Start): Generates a unique goroutine identifier (goroutine_<name>_<timestamp>). Evaluates the active count againstmaxGoroutines(if configured). Constructs a child context with dedicated cancellation, registers theTrackedGoroutine, emits lifecycle start events, executesOnStartcallbacks, and launches execution within thegoroutinelabelswrapper. -
Targeted Stop (
Stop): Locates the tracked routine by ID, invokes its specific context cancellation function, and triggers registered resource cleanup functions in sequence. -
Collective Stop (
StopAll): Acquires a read lock to snapshot all active routine IDs, then sequentially executesStopacross all registered routines. -
Graceful Shutdown (
Shutdown): Guaranteed idempotency viasync.Once. CallsStopAll(), cancels the root manager context, and launches a monitored wait on the internalsync.WaitGroupwith a configurable deadline.
2.2 TrackedGoroutine & Lifecycle State Machine
Each routine managed by GoroutineManager is represented by a TrackedGoroutine instance containing metadata, context handles, timing records, and operational status.
stateDiagram-v2
[*] --> Starting : Start() invoked
Starting --> Running : Registered & Goroutine Launched
Running --> Stopping : Stop() / Context Cancelled
Stopping --> Stopped : Normal Exit (PostCleanup)
Running --> Error : Unhandled Function Error
Running --> Error : Panic Captured
Stopping --> Leaked : Shutdown Timeout Elapsed
Running --> Leaked : Duration > 1h (DetectLeaks)
Stopped --> [*]
Error --> [*]
Leaked --> [*]
Status Enumerations
StatusStarting("starting"): Initial state during configuration allocation before the go routine scheduler dispatches the worker.StatusRunning("running"): Routine is actively executing its payload function.StatusStopping("stopping"): Cancellation signal has been delivered; cleanup routines are currently executing.StatusStopped("stopped"): Routine completed normally and all associated resources have been released.StatusError("error"): Routine terminated with an error or recovered from an unhandled panic.StatusLeaked("leaked"): Routine failed to exit within the graceful shutdown window or exceeded the long-running leak threshold.
2.3 Resource Governance & Safe Teardown
Background tasks often initialize secondary primitives such as time.Ticker, event channels, temporary files, or network sockets. If the worker encounters an error or context cancellation, orphaned tickers prevent memory reclamation.
pkg/runtime provides the Resource struct to bind these primitives directly to the goroutine:
type Resource struct {
Type string // "ticker", "channel", "file", "connection", etc.
ID string // Resource identifier
Description string // Human-readable description
CleanupFunc func() error // Idempotent cleanup logic
}
When Stop or stopGoroutine executes, the manager iterates over all registered resources and invokes each CleanupFunc. Any cleanup error is recorded and emitted as a structured cleanup_error telemetry event without interrupting subsequent resource deallocations.
2.4 Panic Handlers & Fault Isolation
Standard Go unmanaged goroutines that encounter a nil pointer dereference or slice bounds violation terminate the entire operating process immediately. GoroutineManager integrates deep panic recovery via goroutinelabels.WithPanicHandler:
WithPanicHandler(func(r any) {
err := errfmt.Errorf("panic in goroutine %s: %v", id, r)
gm.stopGoroutine(id, err, config.OnStop)
})
When a panic occurs:
- The panic object is captured and wrapped into an informative error recording the routine ID and panic payload.
- The routine's status is atomically shifted to
StatusError. - The lifetime error counter (
totalErrors) is incremented. - The user-provided
OnStopcallback is invoked with the error. - The associated resources are safely cleaned up.
- The host process and other sibling goroutines continue uninterrupted.
3. Leak Detection, Diagnostics & Drain Coordination
3.1 Leak Detection Heuristics (DetectLeaks)
In long-running daemon environments, some background tasks may hang on deadlocked channels or blocked network I/O without throwing errors. The DetectLeaks function inspects all tracked routines against operational thresholds:
func (gm *GoroutineManager) DetectLeaks() []*TrackedGoroutine {
// ...
for _, tracked := range goroutinesCopy {
// ...
duration := now.Sub(startTime)
if status == StatusRunning && duration > time.Hour {
leaks = append(leaks, tracked)
}
}
return leaks
}
A routine is flagged as a suspected leak if:
- Its current status remains
StatusRunning. - Its active execution duration exceeds the leak threshold (default
1 hour). - The manager has initiated shutdown or diagnostic inspection.
3.2 Two-Phase Graceful Drain Protocol
The Shutdown() method implements a fail-safe, two-phase drain sequence designed to prevent split-brain teardowns and resource leaks:
sequenceDiagram
autonumber
participant Caller as Caller / Service
participant GM as GoroutineManager
participant Workers as Tracked Workers
participant Ticker as Timer / Drain Wait
Caller->>GM: Shutdown()
Note over GM: Phase 1: Stop All & Cancel
GM->>Workers: StopAll() -> tracked.Cancel()
GM->>Workers: Resource CleanupFunc()
GM->>GM: shutdownCancel() (Root Context)
Note over GM: Phase 2: Wait with Timeout
GM->>Ticker: WaitGroup.Wait() with Timeout
alt All Workers Exit Within Deadline
Workers-->>GM: Exit & wg.Done()
GM-->>Caller: return nil
else Timeout Exceeded
Ticker-->>GM: Timeout Triggered
Note over GM: Mark Remaining as Leaked
GM->>Workers: Status = StatusLeaked
GM->>GM: Emit "leaked" event & notify leakChan
GM-->>Caller: return shutdown timeout error
end
Deterministic Test Synchronization
To support rigorous, flake-free unit and integration testing of leak behavior, GoroutineManager supports SetLeakNotificationChannel(ch chan string). When a shutdown timeout occurs, routine IDs marked as leaked are non-blockingly pumped into this channel, allowing tests to verify leak detection without brittle sleep timers.
3.3 Lock Operations & Contention Tracking (lock_op_names.go)
All internal state transitions and registry reads within GoroutineManager utilize named lock operations through concurrency.RunInLockWithLogger and concurrency.RunInRLockWithLogger.
Standard lock operation identifiers defined in lock_op_names.go:
LockNameGoroutineManagerDetectLeaksCopy: Registry copy during leak inspection.LockNameGoroutineManagerGetStatsCopy: Snapshotting active maps for telemetry reporting.LockNameGoroutineManagerLeakCheck: Reading individual routine status and start timestamps.LockNameGoroutineManagerMarkLeaked: Transitioning routine status toStatusLeaked.LockNameGoroutineManagerSetRunning: Marking newly launched routine asStatusRunning.LockNameGoroutineManagerStopAllGetIds: Extracting routine ID list for collective termination.LockNameGoroutineManagerStopCheck: Checking if routine is already stopped before teardown.LockNameGoroutineManagerStopUpdate: Setting stop timestamp, error payload, and status.
If any critical section exceeds the system contention threshold (100ms), the lock logger automatically emits structured warnings containing lock hold times and contention metrics.
4. Reusable Runtime Patterns & Helpers (pkg/runtime/helpers.go)
To avoid duplicating boilerplate goroutine lifecycle code across packages, pkg/runtime/helpers.go provides standard high-level abstractions:
4.1 Periodic Background Tasks (StartPeriodicTask)
Safely runs a recurring function at a specified duration interval. Automatically instantiates a time.Ticker, registers it as a tracked Resource, and stops the ticker when context cancellation occurs.
id, ctx, err := runtime.StartPeriodicTask(
manager,
"cache_prune",
10*time.Minute,
func(ctx context.Context) error {
return pruneExpiredCacheEntries(ctx)
},
)
4.2 Supervised Worker Pools (StartWorkerPool)
Spawns a designated number of worker routines executing identical processing loops. Includes automatic rollback: if any worker fails to start (for example, if maxGoroutines is reached), all previously started workers in that pool are automatically stopped before returning an error.
workerIDs, err := runtime.StartWorkerPool(
manager,
"indexer_worker",
4,
func(ctx context.Context, workerID int) error {
return processIndexQueue(ctx, workerID)
},
)
4.3 Async Event Emission (EmitEventAsync)
Asynchronously emits events into the coordination framework. If a GoroutineManager is supplied, the emission is tracked as a short-lived routine with operational audit metadata. If manager == nil, it falls back safely to an untracked background goroutine for backward compatibility.
err := runtime.EmitEventAsync(manager, coordinator, eventContext)
4.4 One-Off Async Operations (ExecuteAsync)
Executes a single async closure with full lifecycle tracking, returning the routine ID, cancellation context, and a buffered error channel (<-chan error) that receives the execution outcome.
id, ctx, errChan := runtime.ExecuteAsync(manager, "artifact_upload", func(ctx context.Context) error {
return uploadArtifact(ctx, payload)
})
4.5 Standard Service Pattern (Service & BaseService)
Defines the standard contract for background daemons and subsystems:
type Service interface {
Start(ctx context.Context) error
Stop() error
GetGoroutineManager() *GoroutineManager
}
The BaseService struct provides a reference implementation, initializing an internal GoroutineManager with system context and delegating Stop() to manager.Shutdown().
5. Practical Operational Recipes
Recipe 1: Initializing a Supervised Daemon Service
package myservice
import (
"context"
"time"
"github.com/zqk-os/zqk/pkg/coordination"
"github.com/zqk-os/zqk/pkg/runtime"
)
type GatewayService struct {
*runtime.BaseService
}
func NewGatewayService(coord coordination.EventCoordinator) *GatewayService {
return &GatewayService{
BaseService: runtime.NewBaseService(coord),
}
}
func (s *GatewayService) Start(ctx context.Context) error {
gm := s.GetGoroutineManager()
// 1. Start background queue consumer
_, _, err := runtime.ExecuteAsync(gm, "queue_consumer", func(ctx context.Context) error {
for {
select {
case <-ctx.Done():
return nil
default:
// Process queue items
time.Sleep(100 * time.Millisecond)
}
}
})
if err != nil {
return err
}
// 2. Start periodic metrics collector
_, _, err = runtime.StartPeriodicTask(gm, "metrics_heartbeat", 15*time.Second, func(ctx context.Context) error {
return s.emitHeartbeat(ctx)
})
return err
}
func (s *GatewayService) emitHeartbeat(ctx context.Context) error {
// Collect and emit metrics
return nil
}
Recipe 2: Graceful Process Teardown on Signal
func RunSupervisor(ctx context.Context, coord coordination.EventCoordinator) error {
manager := runtime.NewGoroutineManager(ctx, coord)
manager.SetShutdownTimeout(10 * time.Second)
// Start worker pool
_, err := runtime.StartWorkerPool(manager, "data_pipeline", 8, func(ctx context.Context, id int) error {
<-ctx.Done()
return nil
})
if err != nil {
return err
}
// Await interrupt signal
<-ctx.Done()
// Coordinate graceful drain
if err := manager.Shutdown(); err != nil {
// Log leak warning or timeout details
return err
}
return nil
}
6. Verification & Statement Coverage Invariant
The Runtime Subsystem enforces a strict >=85.0% statement test coverage invariant across all production files (goroutine_manager.go, helpers.go, lock_op_names.go).
Test Coverage Execution
To verify statement test coverage locally or within automated continuous integration pipelines, run:
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/runtime
Verification Matrix
- Lifecycle & Callback Hooks: Asserts that
OnStartandOnStopcallbacks are executed deterministically upon routine creation and termination. - Panic Trapping & Recovery: Verifies that panics in worker closures are intercepted, error counters updated, and the calling process remains stable.
- Resource Cleanup Guarantees: Confirms that registered
Resource.CleanupFunchandlers are invoked even during unexpected routine failure. - Leak Detection & Drain Timeouts: Validates that routines exceeding shutdown timeouts are categorized as
StatusLeakedand transmitted via notification channels. - Capacity Limits & Rollback: Verifies that when worker pool creation exceeds
maxGoroutines, previously allocated workers are torn down safely. - Zero Swallowed Errors Invariant: Strict compliance ensures no error returns are discarded (
_ = ...) within test verification suites.
7. Open-Core Compliance Invariant
This guide adheres strictly to the Open-Core Public Release Gateway Standards. In compliance with scripts/open-core/check-public-release-payload.sh:
- All identifiers, types, and operational examples are generic, standard public components.
- No internal ephemeral nanosecond IDs, proprietary swarm addresses, or private cluster endpoints are present.