Context Events Pipeline, Wire Event Serialization & JSON Codec Multiplexing Subsystem Operational Architecture Guide
1. Executive Architectural Summary
The Context Events Pipeline and Wire Codec Subsystem (pkg/contextevents) provides append-only, line-delimited JSON (JSONL) durable signaling across the ZQK Knowledge Kernel execution plane (POL-OBS-001). While transient, high-frequency in-process pub/sub event fan-out is delegated to pkg/coordination and pkg/events, pkg/contextevents represents the canonical durable evidence pathway persisted under .zqk/metrics/context_events.jsonl.
This subsystem captures lightweight inter-subsystem context signalsβincluding verification matrix outcomes, criteria assertions, convergence session snapshots, job scheduler telemetry, and operator diagnostic notes. By coupling strict schema validation, defensive rune/byte sizing bounds (4,096-rune note truncation, 64KB payload ceiling), thread-safe serialized write operations, and backwards-compatible JSON codec multiplexing, pkg/contextevents ensures resilient, uncorrupted, and auditable event streams across multi-agent swarms and CLI execution harnesses.
2. Component Topology & Subsystem Architecture
The subsystem spans core primitives within pkg/contextevents and interfaces directly with the CLI system commands:
- Storage Membrane & Path Resolver (
ContextEventsJSONLPath): Encapsulates the canonical filesystem path resolution for.zqk/metrics/context_events.jsonlwithin the project root. - Append-Only Pipeline Core (
Append): Enforces static floor invariants, normalizes slice collections, truncates diagnostic notes, applies UTC RFC3339 timestamps, and serializes concurrent appends via package-level mutex locking. - Payload Merger & Memory Boundary (
MergePayloadJSON): Validates raw JSON byte slices against a strict 64KB capacity ceiling (payloadJSONMaxBytes = 65536), merging object keys into existing destination maps safely. - Wire Serialization & Demultiplexing Codec (
MarshalJSON/UnmarshalJSON): Aligns JSON object keys with canonical domain ontology field constants (objects.FieldKey*) while transparently decoding legacy wire keys (backlog_item_ids,convergence_session_id).
flowchart TD
subgraph EmissionPlane["Producers & Emission Sources"]
CLI["CLI: zqk system emit-context-event"]
Matrix["Verification Matrix Engine"]
Scheduler["Background Job Scheduler"]
Convergence["Convergence Loop Controllers"]
end
subgraph ContextEventsSubsystem["pkg/contextevents Subsystem"]
Record["Record Struct\n(Event, Source, Correl, CVS, Slices, Payload)"]
AppendPipeline["Append(projectRoot, rec)"]
Mutex[("appendMu sync.Mutex")]
Truncator["Note Truncator\n(<= 4,096 runes)"]
Normalizer["Slice Normalizer\n(trim & filter empty)"]
PayloadEngine["MergePayloadJSON\n(<= 64KB check)"]
Codec["JSON Codec\n(MarshalJSON / UnmarshalJSON)"]
end
subgraph StorageMembrane["Durable Persistence Membrane"]
TargetFile[(".zqk/metrics/context_events.jsonl")]
FileUtil["fileutil.AppendJSONLine"]
end
subgraph ConsumerPlane["Telemetry Consumers & Ingestion"]
Auditor["Telemetry & Audit Harvesters"]
Rollup["Metrics Rollup & Vitality Evaluators"]
Dashboard["TUI & Web Studio Observers"]
end
EmissionPlane -->|Construct| Record
Record -->|Invoke| AppendPipeline
AppendPipeline --> PayloadEngine
AppendPipeline --> Truncator
AppendPipeline --> Normalizer
AppendPipeline --> Mutex
Mutex --> Codec
Codec --> FileUtil
FileUtil --> TargetFile
TargetFile -.->|Stream Ingestion| ConsumerPlane
3. Interface & Contract Specifications
The subsystem exports a minimal, highly robust functional API:
// Path and Default Constants
const (
SchemaVersionV1 = "1"
DefaultSourceCLI = "cli"
)
// Sentinel Error Indicators
var (
ErrPayloadTooLarge = errors.New("payload: JSON exceeds max size")
ErrEmptyEventType = errors.New("event_type: required")
)
// Canonical Context Event Record
type Record struct {
TsRFC3339 string
SchemaVersion string
EventType string
Source string
CorrelationID string
JobID string
Note string
CriteriaRefs []string
BacklogItemRefs []string
ConvergenceSessionRef string
Payload map[string]any
}
// Subsystem API Primitives
func ContextEventsJSONLPath(projectRoot string) string
func Append(projectRoot string, rec *Record) error
func MergePayloadJSON(dst map[string]any, raw []byte) (map[string]any, error)
Key Behavioral Contracts
ContextEventsJSONLPath(projectRoot): - Returns empty string""whenprojectRootis empty. - Resolves to<projectRoot>/.zqk/metrics/context_events.jsonlutilizing standardized paths frompkg/paths.Append(projectRoot, rec): - Graceful No-Op: Returnsnilimmediately whenprojectRoot == ""orrec == nil. - Invariant Gate: Fails closed returningErrEmptyEventTypewhenEventTypeis empty or composed exclusively of whitespace. - Defaults Seeding: Automatically appliesSchemaVersionV1if unpopulated, and assigns current UTC timestamp intime.RFC3339Nanoformat ifTsRFC3339is absent. - Memory & String Boundaries: TruncatesNoteto a maximum of 4,096 Unicode runes. - Slice Hygiene: Cleanses leading/trailing whitespace from slice elements (CriteriaRefs,BacklogItemRefs) and strips empty strings. If all entries are empty, normalizes slice reference tonil. - Concurrency Serialization: Guards target file writes using package-levelsync.Mutex(appendMu) to prevent interleaved line fragments.MergePayloadJSON(dst, raw): - Returns existingdstunmodified whenlen(raw) == 0or input consists purely of whitespace. - Enforces a strict 64KB boundary (payloadJSONMaxBytes = 65536), returningErrPayloadTooLargewithout mutatingdst. - Deserializes into a newly allocated map whendst == nil. - Overwrites existing keys with incoming JSON keys idempotently.
4. Wire Format & JSON Codec Multiplexing Invariants
The wire protocol enforces rigorous alignment between runtime structs and the persistent JSONL representation.
Field Key Mapping Matrix
| Record Struct Field | Wire JSON Key Constant | Source Origin | Omission Invariant |
|---|---|---|---|
TsRFC3339 |
"ts_rfc3339" |
WireTsRFC3339 (POL-OBS-001) |
Mandatory in wire output |
SchemaVersion |
"schema_version" |
objects.FieldKeySchemaVersion |
Mandatory in wire output ("1") |
EventType |
"event_type" |
objects.FieldKeyEventType |
Mandatory in wire output |
Source |
"source" |
objects.FieldKeySource |
Omitted if empty |
CorrelationID |
"correlation_id" |
WireCorrelationID |
Omitted if empty |
JobID |
"job_id" |
WireJobID |
Omitted if empty |
Note |
"note" |
objects.FieldKeyNote |
Omitted if empty |
CriteriaRefs |
"criteria_refs" |
objects.FieldKeyCriteriaRefs |
Omitted if empty or nil |
BacklogItemRefs |
"backlog_item_refs" |
objects.FieldKeyBacklogItemRefs |
Omitted if empty or nil |
ConvergenceSessionRef |
"convergence_session_ref" |
objects.FieldKeyConvergenceSessionRef |
Omitted if empty |
Payload |
"payload" |
objects.FieldKeyPayload |
Omitted if empty or nil |
Backwards-Compatible Multiplexing & Legacy Demultiplexing
To guarantee continuous telemetry ingestion during rolling upgrades across different versions of ZQK agents and tools, UnmarshalJSON implements fault-tolerant multiplexing:
- Convergence Session Reference: Reads
"convergence_session_ref"; if empty or missing, falls back to legacy"convergence_session_id". - Backlog Item References: Evaluates
"backlog_item_refs"; if missing or malformed, falls back to legacy"backlog_item_ids". - Null Safety: Safely consumes explicit JSON
nullvalues across string, slice, and object fields without panic or type conversion exceptions.
flowchart LR
subgraph JSONLine["Incoming Wire Frame"]
Raw["JSON Line Raw Message"]
end
subgraph CodecParser["UnmarshalJSON Multiplexer"]
ParseJSON["json.Unmarshal(data, &raw)"]
CheckCVS{"convergence_session_ref\npresent?"}
FallbackCVS["Fallback to convergence_session_id"]
CheckBLI{"backlog_item_refs\npresent & valid?"}
FallbackBLI["Fallback to backlog_item_ids"]
ParsePayload["Parse payload map[string]any"]
end
subgraph Materialized["Materialized Record"]
TargetRecord["Record Struct"]
end
Raw --> ParseJSON
ParseJSON --> CheckCVS
CheckCVS -->|No| FallbackCVS
CheckCVS -->|Yes| TargetRecord
FallbackCVS --> TargetRecord
ParseJSON --> CheckBLI
CheckBLI -->|No / Malformed| FallbackBLI
CheckBLI -->|Yes| TargetRecord
FallbackBLI --> TargetRecord
ParseJSON --> ParsePayload
ParsePayload --> TargetRecord
5. Event Publishing & Subscriber Ingestion Dynamics
The following sequence details an operational context event being emitted via the CLI, marshaled through pkg/contextevents, written to disk, and consumed by audit observers.
sequenceDiagram
autonumber
actor Operator as CLI / Subsystem Producer
participant CLI as cmd/zqk/system
participant Core as pkg/contextevents
participant Codec as json_codec.go
participant Disk as .zqk/metrics/context_events.jsonl
actor Observer as Telemetry Harvester
Operator->>CLI: zqk system emit-context-event --event-type criteria_eval ...
CLI->>Core: MergePayloadJSON(nil, payloadBytes)
Note over Core: Validates payload <= 64KB
Core-->>CLI: validated map[string]any
CLI->>Core: Append(projectRoot, &rec)
Note over Core: Validate EventType != ""<br/>Apply SchemaVersion = "1"<br/>Apply TsRFC3339 Nano<br/>Truncate Note <= 4096 runes<br/>Trim string slices
Core->>Core: appendMu.Lock()
Core->>Codec: MarshalJSON()
Codec-->>Core: Compact JSON byte line
Core->>Disk: fileutil.AppendJSONLine(path, line)
Core->>Core: appendMu.Unlock()
Core-->>CLI: nil (success)
CLI-->>Operator: Emit complete (JSONL path)
Note over Disk,Observer: Asynchronous Harvest
Observer->>Disk: Stream Read JSONL lines
Observer->>Codec: UnmarshalJSON(lineBytes)
Codec-->>Observer: Materialized Record
6. CLI Ergonomics & Operational Recipes
Operators and autonomous subagents can emit and inspect context events using standard CLI invocations.
Emitting Context Events via CLI
# 1. Emit a basic criteria evaluation signal with correlation
zqk system emit-context-event \
--event-type criteria_verified \
--source "qa_runner" \
--corr "run-9842" \
--criteria "crit-perf-floor,crit-zero-leak" \
--note "Automated static verification passed with zero leaks"
# 2. Emit an event bound to a convergence session and backlog item
zqk system emit-context-event \
--event-type iteration_checkpoint \
--source "tpm_orchestrator" \
--cvs "cvs-session-42" \
--bli "bli-refactor-worker" \
--payload-json '{"iteration": 3, "remaining_tasks": 0, "status": "nominal"}'
# 3. Emit a job scheduler diagnostic event
zqk system emit-context-event \
--event-type scheduler_job_completed \
--job "job-nightly-vet" \
--source "scheduler" \
--payload-json '{"duration_ms": 1420, "exit_code": 0}'
Inspecting Context Events
# Tail the active context events stream
tail -n 20 .zqk/metrics/context_events.jsonl | jq .
# Filter events by event type
jq 'select(.event_type == "criteria_verified")' .zqk/metrics/context_events.jsonl
# Correlate events belonging to a specific session
jq 'select(.convergence_session_ref == "cvs-session-42")' .zqk/metrics/context_events.jsonl
7. Operational Troubleshooting Matrix
| Symptom / Error | Root Cause | Impact | Recommended Remediation |
|---|---|---|---|
event_type: required |
Invocations provided an empty string or whitespace only for EventType. |
Call fails closed without appending. | Ensure --event-type is supplied and contains non-whitespace identifiers. |
payload: JSON exceeds max size |
Supplied payload exceeds payloadJSONMaxBytes (64KB). |
Mutation rejected without altering state. | Compress payload, strip binary blobs, or log summarized metrics rather than raw outputs. |
| Malformed JSON error on merge | Input to MergePayloadJSON contains unparseable syntax. |
Parse error propagated to caller. | Validate JSON formatting prior to invocation using jq or schema linters. |
| Notes truncated unexpectedly | Note parameter exceeded 4,096 Unicode runes. | Data beyond 4,096 runes discarded gracefully. | Store verbose output in dedicated artifact files and record relative file path in the note or payload. |
| Concurrency lock delays on append | Extremely high emission contention on .zqk/metrics/context_events.jsonl. |
Minor latency in thread serialization. | Expected behavior under appendMu to prevent interleaved line corruption; use asynchronous workers if non-blocking operation is required. |
| Unrecognized legacy fields | Ingestion from older writers containing backlog_item_ids. |
None (automatically demultiplexed). | Verify consumer unmarshals through pkg/contextevents.Record which natively supports legacy fallbacks. |
8. Verification & Continuous Testing Invariants
The pkg/contextevents package is maintained under strict continuous verification invariants:
- 100.0% Statement Coverage Floor: All conditional branches, fallback unmarshaling paths, string slice trimmers, note rune truncations, and boundary limits are verified.
- Race Condition Immunity: Verified under high concurrency using
go test -raceacross concurrent worker pools. - Open-Core Payload Integrity: Strictly free of literal nanosecond kernel IDs, maintaining full compliance with open-core redistribution standards.