Logging & Structured Event Dispatch Subsystem Operational Architecture Guide
1. Executive Architectural Summary
The Logging and Event Dispatch Subsystem (pkg/logging) provides high-performance, context-aware, structured logging and asynchronous stream dispatch across the ZQK Knowledge Kernel runtime. In a multi-agent orchestration architecture where concurrent worker swarms, background daemons, MCP protocol servers, and human interactive sessions share process space, naive uncoordinated logging causes severe operational hazards:
- Protocol Corruption: Writing diagnostics or raw debug messages to standard output destroys Model Context Protocol (MCP) JSON-RPC streams, causing immediate disconnection of headless agents.
- I/O Contention & Parked OS Threads: Spawning per-logger background flush tickers or performing blocking synchronous disk writes degrades throughput and inflates OS thread counts.
- Memory Pressure: Rapid generation of structured log fields under high-throughput validation loops introduces heavy heap churn and garbage collection pauses without object pooling.
- Security & Secret Leaks: Unsanitized operational payloads risk exposing sensitive credentials, tokens, and authorization headers in plaintext audit files.
The pkg/logging subsystem eliminates these failure modes through a multi-tier architectural design:
- Context-Aware Dynamic Routing (
LogRouter,DecisionContextLogger): Dynamically filters, formats, and routes events based on execution profile (human,ai-agent,mcp,system,debug) and operational decision context. - Strict MCP Protocol Guarding: Automatically diverts diagnostic and warning outputs to standard error or log files whenever MCP serving mode is detected, guaranteeing standard output remains reserved exclusively for JSON-RPC message frames.
- Zero-Allocation Object Recycling (
sync.Pool): Uses an internal pool of fluent builder entries, recycling field slice capacities while capping oversized allocations at 64 entries. - Buffered & Rolling I/O Pipeline (
BufferedWriter,RollingWriter): Batches disk writes into configurable memory buffers with error-triggered immediate flushes, coupled with size-based log file rotation and oldest-file pruning. - Shared Process-Wide Flush Loop: Unifies all buffered loggers onto a single process-lifetime background ticker managed via goroutine labeling to eliminate goroutine leaks and OS-thread parked thread expansion.
The subsystem maintains full test coverage with statement coverage >= 85.0% across all components.
2. Component Topology & Structural Model
The subsystem is partitioned into modular layers spanning interface definitions, fluent builders, profile context extractors, multi-destination routers, formatters, and buffered/rolling I/O writers.
flowchart TD
subgraph Callers["Subsystem Callers & Execution Contexts"]
AgentSwarm["Autonomous Agent Swarm"]
MCPServer["MCP Server (JSON-RPC)"]
CLITUI["CLI & TUI Interactive Commands"]
AsyncEngine["Validation & Scheduler Daemons"]
end
subgraph EntryLayer["Fluent & Event Entry Plane"]
FluentRoot["Fluent(logger) / FluentEvent(el)"]
Pool[("sync.Pool (fluentEntry)")]
EventLogger["EventLogger (Domain Helpers)"]
ContextLogger["ContextLogger (Profile Detection)"]
end
subgraph DecisionPlane["Routing & Decision Plane"]
RouterInit["InitializeGlobalRouter()"]
GlobalRouter["LogRouter (Multi-Destination)"]
DecisionLogger["DecisionContextLogger"]
MCPGuard{"MCP Mode / Server Serving?"}
end
subgraph FormattingPlane["Formatters"]
JSONFmt["JSONFormatter (Compact JSONL)"]
TextFmt["TextFormatter (Colorized Human)"]
CompactFmt["CompactFormatter (Concise)"]
end
subgraph IOPlane["I/O & Persistence Plane"]
BufWriter["BufferedWriter (Buffered I/O)"]
SharedFlusher["Shared Process Flusher (1s Ticker)"]
RollWriter["RollingWriter (Size/Time Rotation)"]
StdErr["os.Stderr (Safe Diagnostics)"]
StdOut["os.Stdout (CLI User Output)"]
LogFiles[("Rotated Log Files (*.json, *.log)")]
end
Callers --> FluentRoot
Callers --> EventLogger
Callers --> ContextLogger
FluentRoot <--> Pool
FluentRoot --> DecisionLogger
EventLogger --> GlobalRouter
ContextLogger --> GlobalRouter
DecisionLogger --> MCPGuard
GlobalRouter --> MCPGuard
MCPGuard -->|Protocol Safe| FormattingPlane
MCPGuard -->|Divert Diagnostic| StdErr
FormattingPlane --> BufWriter
FormattingPlane --> StdOut
FormattingPlane --> StdErr
BufWriter --> RollWriter
SharedFlusher -.->|Periodic Flush| BufWriter
RollWriter --> LogFiles
Core Types & Interface Contracts
Logger: Foundational interface declaringDebug,Info,Warn,Error,Fatal,WithFields,WithContext, andWithObjectRef.ProgressLogger: Dedicated interface for non-blocking interactive progress bars, phase status transitions, and streaming real-time events.Formatter: Pluggable interface converting structuredLogEntryenvelopes into serialized bytes (JSONFormatter,TextFormatter,CompactFormatter).RollingWriter: Extendedio.WriteCloserprovidingSync()semantics and policy-driven file rotation.
3. Structured Event Dispatch & Buffered Writer Lifecycles
3.1 Event Emission and Processing Sequence
When an application component emits an event, it flows through sanitization, profile evaluation, multi-destination fanout, and buffered persistence:
sequenceDiagram
autonumber
participant App as Subsystem Component
participant Fluent as FluentRoot / fluentEntry
participant Router as DecisionContextLogger / LogRouter
participant Guard as MCP Protocol Guard
participant Fmt as JSON / Text Formatter
participant Buf as BufferedWriter
participant Roll as SizeBasedRollingWriter
participant Disk as Filesystem Storage
App->>Fluent: Fluent(logger).Info("event_message").String("key", "val").Log()
activate Fluent
Fluent->>Fluent: Sanitize & mask credentials (zqkenv)
Fluent->>Router: Emit LogEntry (Level, Message, Fields, Timestamp)
deactivate Fluent
activate Router
Router->>Router: Evaluate destination levels & suppress flags
Router->>Guard: Verify destination safety (IsMCPServerServing?)
alt Is MCP Serving and Destination is Stdout
Guard-->>Router: Suppress or Divert to Stderr / Log File
else Safe Destination
Guard-->>Router: Allow dispatch
end
Router->>Fmt: Format(entry)
activate Fmt
Fmt->>Fmt: Serialize JSON / Text (with fallback recovery)
Fmt-->>Router: Serialized byte payload
deactivate Fmt
Router->>Buf: Write(bytes)
activate Buf
Buf->>Buf: Write to in-memory buffer (bufio.Writer)
alt Buffer Full OR ErrorLevel with FlushOnError
Buf->>Roll: Write(p) & Flush()
activate Roll
Roll->>Roll: Check currentSize + len(p) > maxSize
opt Rotation Threshold Reached
Roll->>Disk: Rotate active file to archive timestamp
Roll->>Disk: Purge oldest archives if count > maxFiles
Roll->>Disk: Open fresh active log file
end
Roll->>Disk: Write bytes to active log file
deactivate Roll
end
deactivate Buf
deactivate Router
3.2 Shared Buffered Flusher Lifecycle
To prevent the explosion of background goroutines observed with individual writer timers, the subsystem manages a single shared flusher:
sequenceDiagram
autonumber
participant Registry as Shared Flusher Registry
participant Loop as goroutinelabels Background Ticker
participant WriterA as BufferedWriter (System Log)
participant WriterB as BufferedWriter (Trace Log)
Note over Registry: Process initializes first BufferedWriter
Registry->>Loop: Spawn shared loop (1s Ticker)
loop Every 1 Second
Loop->>Registry: Snapshot active registered writers
Registry-->>Loop: [WriterA, WriterB]
Loop->>WriterA: Flush()
Loop->>WriterB: Flush()
end
Note over WriterB: Subsystem teardown / Close()
WriterB->>Registry: Unregister(WriterB)
WriterB->>WriterB: Final Flush() & underlying Close()
4. Concurrency, Rolling Invariants & Zero-Allocation Hygiene
4.1 Strict Goroutine Concurrency Invariant
All asynchronous tasks spawned inside the subsystem (including background flush loops and test verification routines) must register descriptive debug metadata via goroutinelabels:
goroutinelabels.NewGoroutine(
"buffered_writer_shared_flush",
"flush all buffered log writers on one ticker",
).StartSimple(f.loop)
Direct invocation of uninstrumented goroutines is strictly prohibited.
4.2 Filesystem & Path Mutation Guarding
Log files and archive directories must strictly be manipulated through fileutil abstractions (fileutil.OpenFile, fileutil.EnsureDir, fileutil.Remove, fileutil.Stat). Direct calls to os.* file modification functions are prevented to safeguard against repository corruption and path traversal.
4.3 Memory Allocation Caps & Recycling Hygiene
fluentEntryPooling: Fluent builder entries are pooled usingsync.Pool. Upon.Log()or.release(), slice headers are reset to zero length (fields[:0]).- Memory Ceiling Guard: If an entry accumulates more than 64 structured fields during atypical burst operations, its underlying array is discarded rather than returned to the pool, preventing long-lived memory spikes.
- Formatter JSON Self-Healing: The
JSONFormatteruses reflection and trial-marshaling to intercept unmarshalable types (channels, function pointers, cyclic references). Problematic fields are sanitized into string descriptions rather than aborting the logging pipeline.
4.4 Size-Based Rolling Invariants
- Pre-Write Evaluation: If
currentSize + len(payload) > maxSize, rotation occurs before writing the new payload. - Post-Write Safety Check: If write completion leaves the file exceeding
maxSize, an immediate rotation occurs. - Retention Ceiling (
maxFiles): Archival log files matching<name>-<timestamp>-<rotation>.extare indexed and sorted chronologically. Any files exceedingmaxFilesare pruned immediately.
5. Verification, Troubleshooting, and Runbook
5.1 Subsystem Verification
Execute the test suite with race detection and statement coverage tracking:
# Run unit, race, and coverage tests for pkg/logging
go test -race -cover ./pkg/logging/...
# Generate detailed statement coverage breakdown
go test -coverprofile=coverage.out ./pkg/logging/...
go tool cover -func=coverage.out
5.2 Common Operational Failure Scenarios & Remedies
| Symptom / Failure Mode | Root Cause Analysis | Remediation Protocol |
|---|---|---|
| MCP Client Disconnection / JSON-RPC Parse Error | A command or daemon wrote raw diagnostic text to os.Stdout instead of using the logging framework or GetCommandOutputWriter. |
Ensure all CLI handlers query logging.GetCommandOutputWriter(ctx). Never call fmt.Println or os.Stdout.Write directly on diagnostic paths. |
| Missing Tail Logs on Process Exit | Process exited before the 1-second shared background ticker flushed buffered writers. | Ensure root commands register defer logging.CloseGlobalRouter() and call logging.TryCloseLoggerDestinations(logger) during graceful shutdown. |
| Log Disk Space Exhaustion | Destination configured with "none" rolling strategy or oversized maxFiles retention. |
Apply DefaultRollingPolicy() (10MB max size, 5 files retention) or specify explicit size-based rolling in configuration. |
| JSON Formatting Fallback Warnings | Unmarshalable complex Go types (pointers to channels, closures) attached as field values. | Ensure custom objects passed to .Fields(...) implement fmt.Stringer or serialize to standard JSON primitives. |
| File Descriptor Leakage in Tests or Long-Running Daemons | Dynamic decisionContextLogger instances created repeatedly without closing underlying writers. |
Call logger.Close() on decision context loggers when discarding component execution contexts. |
6. Subsystem Architectural Alignment
| Capability | Implementation Primitive | Architectural Contract |
|---|---|---|
| Thread-Safe Shared Flusher | sharedBufferedFlusher |
Single process-wide ticker; zero parked OS thread growth |
| Zero-Allocation Recycling | sync.Pool with 64-field cap |
Bounded heap churn on hot validation loops |
| Protocol Safety Membrane | shouldSuppressStdioForMCPProtocol |
Fail-closed protection for JSON-RPC standard output |
| Credential Redaction | zqkenv.IsSensitiveKey |
Automatic masking of secrets before disk formatting |
| Statement Coverage | Unit & Fault Injection Suites | Maintained >= 85.0% statement coverage invariant |