Event Coordination Subsystem Architectural & Operational Guide (pkg/coordination)
1. Executive Architectural Summary
The Event Coordination Subsystem (pkg/coordination) serves as the central event distribution spineβthe "spinal cord"βof the ZQK Knowledge Kernel. It decouples high-frequency operational mutations, status lifecycle transitions, and diagnostic events from external storage, telemetry sinks, and terminal user interfaces.
By establishing an event coordination plane governed by non-blocking pipelines, pkg/coordination guarantees three core properties:
- Zero Coupling Across Layers: Core execution loops, background schedulers, and storage drivers emit standardized
EventContextpayloads without direct dependencies on CLI renderers, TSDB engines, or audit persistence stores. - Best-Effort Non-Blocking Resilience: Telemetry, metrics collection, audit logging, and subscriber callbacks execute as side-effect trigger stages (
TRIGGER_*). Subsystem errors or latency spikes in peripheral channels never crash or block business execution. - Deterministic Verification & CLI Feedback: Terminal completion states and synchronous CLI subscribers support synchronous fast-paths (
EmitOperationalSync, synchronous audit callbacks), ensuring immediate interactive feedback and reproducible automated test assertions.
flowchart TD
subgraph Producers["Kernel Producers & Execution Engines"]
CLICmd["CLI Commands & Do Loop"]
Scheduler["Scheduler Daemon & Jobs"]
Storage["Storage & CAS Subsystem"]
Workers["Agent Swarm Workers"]
end
subgraph Coordination["Event Coordination Subsystem (pkg/coordination)"]
Coord["Coordinator\n(atomic.Pointer[State], WaitGroup)"]
Pipeline["Emit Pipeline\n(pkg/pipeline Engine)"]
State["Type Index & Subscriber Registry\n(Copy-on-Write)"]
Coord --> Pipeline
Coord --> State
end
subgraph Routers["Channel Routers & Sinks"]
LogRouter["LoggingRouter\n(pkg/logging.EventLogger)"]
AuditRouter["AuditRouter\n(pkg/storage Audit Events)"]
MetricsRouter["MetricsRouter\n(MetricPipeline & TSDB)"]
OpRouter["OperationalRouter\n(Subscribers & CLI Feed)"]
end
Producers -->|Emit EventContext| Coord
Pipeline -->|TRIGGER_logging| LogRouter
Pipeline -->|TRIGGER_audit| AuditRouter
Pipeline -->|TRIGGER_metrics| MetricsRouter
Pipeline -->|TRIGGER_operational| OpRouter
2. Component Topology & Interface Contracts
The coordination subsystem is structured around modular abstractions in coordinator.go:
2.1 The EventCoordinator Interface
The primary contract defines emission and pub-sub registration:
type EventCoordinator interface {
// Emit routes an event to all configured channels based on EventContext flags
Emit(ctx context.Context, eventCtx *EventContext) error
// Subscribe adds a subscriber for operational events and returns a unique ID
Subscribe(subscriber OperationalEventSubscriber) string
// Unsubscribe removes a registered subscriber by ID
Unsubscribe(subscriberID string)
}
2.2 Operational Event Subscribers
Subscribers implement OperationalEventSubscriber to receive filtered or broadcast process coordination signals:
type OperationalEventSubscriber interface {
ID() string
HandleEvent(event *OperationalEvent) error
EventTypes() []string // Empty slice matches all events (wildcard broadcast)
IsActive() bool // Inactive subscribers are bypassed and purged
}
2.3 Coordinator State & Lock Isolation
To achieve ultra-low read latency on the hot emit path, the Coordinator employs an atomic pointer to an immutable state snapshot (atomic.Pointer[coordinatorState]):
- Read Operations (
findRelevantSubscribers): Zero mutex contention. Loads the atomic pointer, inspects the pre-indexedtypeIndexmap and wildcard subscribers in \(O(1)\) time. - Write Operations (
Subscribe,Unsubscribe): Mutex-guarded (c.mu.Lock()). Performs a copy-on-write clone of subscriber maps and rebuilds the type index, atomically swapping the state pointer. - In-Flight Tracking: A
sync.WaitGrouptracks every launched router and subscriber goroutine, allowing clean, bounded drains viaDrain(timeout).
3. Event Dispatch & Stage Pipelines
3.1 EventContext Envelope
Events flow through the coordinator wrapped in an EventContext descriptor:
| Field | Type | Description |
|---|---|---|
OperationID |
string |
Unique identifier of the tracked unit of work (e.g. task ID, run ID). |
OperationType |
string |
Classification of the operation (e.g. system_task, queue_shutdown). |
Status |
string |
Lifecycle state (start, progress, completed, error, cancelled). |
Level |
string |
Explicit log level override (debug, info, warn, error). |
Channels |
bool flags |
Selective routing flags: EmitLogging, EmitAudit, EmitMetrics, EmitOperational. |
EventData |
*EventData |
Structured payload containing LoggingFields, AuditMetadata, and MetricsData. |
Error |
error |
Failure error instance when status is error. |
3.2 Emit Pipeline Composition
When Emit is called, the coordinator dynamically builds an execution pipeline via pkg/pipeline.Builder (coordinator_emit_pipeline.go):
sequenceDiagram
autonumber
participant Caller as Caller / Operation
participant Coord as Coordinator
participant Pipeline as pkg/pipeline
participant Logging as LoggingRouter
participant Audit as AuditRouter
participant Metrics as MetricsRouter
participant Ops as OperationalRouter
Caller->>Coord: Emit(ctx, eventCtx)
Coord->>Pipeline: buildEmitPipeline(eventCtx)
Coord->>Pipeline: Run(pctx, eventCtx)
opt EmitLogging Enabled
Pipeline->>Logging: TRIGGER_logging (Sync if terminal, Async otherwise)
end
opt EmitAudit Enabled
Pipeline->>Audit: TRIGGER_audit (Sync if callback set, Async otherwise)
end
opt EmitMetrics Enabled
Pipeline->>Metrics: TRIGGER_metrics (Async dispatch)
end
opt EmitOperational Enabled
Pipeline->>Ops: TRIGGER_operational (Sync if subscribers exist)
end
Pipeline-->>Coord: Pipeline Run Outcome (nil error)
Coord-->>Caller: Returns success (Best-effort side-effects)
3.3 Stage Naming & Lifecycle Rules
Stage names adhere to the system data pipeline lifecycle:
TRIGGER_logging_complete: Synchronous execution for terminal statuses (completed,error) so diagnostic logs appear before process exit.TRIGGER_logging_async: Asynchronous execution for intermediate progress updates.TRIGGER_audit_callback_sync: Synchronous execution when an audit callback is registered (vital for integration tests).TRIGGER_audit_async: Standard background worker audit persistence.TRIGGER_metrics: Asynchronous metric emission to time-series and aggregate collectors.TRIGGER_operational_subscribers_sync: Direct synchronous delivery when subscribers are registered.
4. Helper Abstractions & Adapters
pkg/coordination provides specialized helper bridges that eliminate import cycles between packages:
flowchart LR
OpCB["CoordinatorOperationCallback\n(pkg/concurrency Bridge)"] -->|Dispatches Events| Coord["Coordinator"]
ProgHelp["ProgressHelper\n(CLI & TUI Reporting)"] -->|Emits Progress| Coord
CLINotif["CLINotifierAdapter\n(storage.OperationNotifier Bridge)"] -->|Delegates Updates| ProgHelp
MetricsAdapt["MetricsCollectorAdapter\n(UnifiedMetricsCollector Bridge)"] -->|Emits Metric Records| Coord
4.1 ProgressHelper
ProgressHelper streamlines multi-step task reporting:
EmitProgress(ctx, progress, total, message, fields, emitAudit): Emits progress with automated percentage calculations.EmitProgressSummary(ctx, summary, progress, total, message, fields): Checkpoint updates for milestones.EmitStatusChange(ctx, oldStatus, newStatus, message, fields): Status transitions with built-in deduplication against redundant duplicate emissions.EmitError(ctx, err, message, fields)andEmitCompletion(ctx, duration, message, fields).
4.2 CoordinatorOperationCallback
CoordinatorOperationCallback maps concurrency hooks (OnStart, OnProgress, OnComplete, OnError, OnCancel) directly into structured audit, log, and operational events.
4.3 CLI Notifier Adapter
CLINotifierAdapter and NewCLINotifierWithCoordinator construct fully-wired, coordinator-backed notifiers for terminal and CLI command execution, falling back cleanly to storage.NoopOperationNotifier when prerequisites are absent.
5. Subscriber Topologies & Routing Semantics
5.1 Topic Routing vs Wildcards
Subscribers register with specific event interests:
- Topic Filtered: Declare specific event types (e.g.
[]string{"operation.start", "operation.complete"}). Only matching events triggerHandleEvent. - Wildcard Broadcast: Declare an empty slice (
[]string{}). Receives all operational events.
5.2 Liveness & Dead-Letter Isolation
Before dispatching an event, findRelevantSubscribers checks subscriber.IsActive(). If a subscriber has ceased activity (e.g., worker cancelled or connection dropped), it is skipped and safely removed during subsequent state maintenance. Errors returned by HandleEvent are isolated without interrupting deliveries to sibling subscribers.
6. Concurrency, Backpressure & Thread-Safety
6.1 Lock Hierarchies & Atomic Guarantees
- State Immutability: The internal
coordinatorStatestruct is never mutated in place. State mutations generate a new state instance swapped atomically viac.state.Store(newState). -
Goroutine Bounding & Labeling: Every background goroutine launched by the coordinator is tagged with
goroutinelabelsfor tracing in diagnostics logs:go goroutinelabels.NewGoroutine("coordinator_logging_router", "...").StartSimple(fn) -
Graceful Draining:
Drain(timeout time.Duration)provides bounded teardown:go if err := coordinator.Drain(5 * time.Second); err != nil { // Handle drain timeout }
7. Verification & Coverage Invariant
Statement test coverage across pkg/coordination is strictly maintained at >= 85.0% (validated at 97.4%):
# Targeted subsystem test execution
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/coordination
Three-Fold Proof Verification
- Static Floor Invariant: Verified statement coverage >= 85.0% across all router implementations, helper bridges, and pipeline stages.
- Operational Dynamic Proof: Validated end-to-end multi-channel routing (simultaneous audit creation in storage, structured JSON logging, metrics collection, and live subscriber callbacks).
- Negative Adversarial Boundary: Verified resilience against nil event contexts, non-existent subscriber IDs, dead-letter subscriber errors, transient storage failures, and concurrent high-throughput emissions.