Error Formatting, Actionable Diagnostics & Poison Pill Protection Subsystem Guide
1. Executive Architectural Summary
The Error Formatting, Actionable Diagnostics & Poison Pill Protection Subsystem (pkg/errfmt) provides standardized, chainable error construction, root-cause wrapping, operator-actionable guidance, and agent-electrocution protection across the ZQK Knowledge Kernel.
In distributed systems and autonomous AI agent swarms, uncaught or vaguely phrased errors degrade operational efficiency and derail autonomous execution loops. Generic error messages such as "project root not found" leave operators guessing the next remediation step. Concurrently, when multi-agent orchestrators signal unrecoverable fatal halts using electrocution sentinels (⚡️ BZZZT!), intermediate helper functions that blindly prefix errors can obscure the sentinel or break fast-path fail-closed tripwires.
The pkg/errfmt subsystem resolves these architectural challenges through five fundamental guarantees:
- Root-Cause Preservation &
%wSupport: First-class support for Go 1.13+ error wrapping via%w, enabling downstream consumers to inspect errors usingerrors.Isanderrors.As. - Agent Poison Pill Defense: Unconditional preservation of electrocution fatal signals (
⚡️ BZZZT!). When an electrocution error is passed into formatting or wrapping routines, the subsystem returns the underlying error untouched without applying generic prefix boilerplate. - Actionable Diagnostic Rewriting: Context-aware expansion of common runtime omissions (such as missing project roots) into clear, step-by-step remediation instructions for operators and AI agents.
- Fluent, Multi-Part Builder API: The
ErrorStringBuilderenables composable construction of structured diagnostic messages with conditional segments (AndIfTrue) and fallback defaults (AndOrDefault). - Static Floor Coverage Invariant: A strict statement test coverage invariant (\(\ge 85.0\%\), verified at \(100.0\%\)) ensuring all format combinations, poison pill bypasses, and edge branches are continually validated.
2. Component Topology & Subsystem Architecture
The subsystem is positioned at the foundation of the kernel utility stack, imported by CLI commands, storage layers, and agent runtime harnesses:
flowchart TD
subgraph Consumers["Kernel Consumers & Agent Harnesses"]
CLI["CLI Commands\n(cmd/zqk)"]
Storage["Storage & CAS\n(pkg/storage)"]
AgentRunner["Autonomous Agent Runner\n(pkg/agentdelivery)"]
Workflow["Workflow Progression Engine\n(pkg/pipeline)"]
end
subgraph ErrFmt["Error Formatting Subsystem (pkg/errfmt)"]
direction TB
EntryRouter["Entry Point\n(Errorf / Newf / NewErrorStringBuilder)"]
PoisonGate{"Poison Pill Gate\nContains '⚡️ BZZZT!'?"}
RawPass["Raw Error Bypass\n(Return unmodified)"]
ActionableGate{"Actionable Sentinel?\n'project root not found'"}
ActionableRewriter["Guidance Rewriter\nActionableProjectRootNotFound"]
Builder["ErrorStringBuilder\n(Parts Slice & Chain Methods)"]
WrapEngine["Wrap Engine\n(fmt.Errorf %s: %w)"]
end
Consumers -->|"Format string & args / errors"| EntryRouter
EntryRouter --> PoisonGate
PoisonGate -- Yes --> RawPass
PoisonGate -- No --> ActionableGate
ActionableGate -- Yes (0 args) --> ActionableRewriter
ActionableGate -- No --> Builder
ActionableRewriter --> Builder
RawPass --> Consumers
Builder --> WrapEngine
Builder -->|"Build() / String() / Error() / Format()"| Consumers
WrapEngine --> Consumers
Component Responsibilities
| Component | Identifier | Primary Responsibility |
|---|---|---|
| Errorf Wrapper | Errorf(format, args...) |
Drop-in replacement for fmt.Errorf with built-in poison pill interception, %w wrapping validation, and actionable diagnostic rewriting. |
| ErrorStringBuilder | ErrorStringBuilder |
Composable error message builder aggregating diagnostic segments into delimited chains. Implements fmt.Stringer, error, and fmt.Formatter. |
| Poison Pill Sentinel Gate | ⚡️ BZZZT! |
Intercepts electrocution messages from autonomous agent execution pods and returns them unmodified to preserve fatal interrupt signals. |
| Actionable Diagnostics Engine | ActionableProjectRootNotFound |
Rewrites raw missing-root errors into actionable operator remediation text containing initialization commands and environment variable flags. |
| Fluent Chain Modifiers | With, AndOrDefault, AndIfTrue |
Provides declarative message segment composition with whitespace trimming and conditional evaluation. |
3. Error Composition & Wrapping Lifecycle
The lifecycle of error construction in pkg/errfmt guarantees deterministic output formatting, safe argument evaluation, and full compatibility with Go standard library inspection protocols (errors.Is / errors.As).
sequenceDiagram
autonumber
actor Caller as Kernel Subsystem / Agent Pod
participant API as errfmt.Errorf / Newf
participant Poison as Poison Pill Sentinel Gate
participant Rewriter as Diagnostic Rewriter
participant Builder as ErrorStringBuilder
participant GoFmt as Go Runtime fmt
Caller->>API: Errorf("context %s: %w", "auth", rootErr)
API->>Poison: Scan args for error containing "⚡️ BZZZT!"
alt Poison Pill Detected
Poison-->>Caller: Return rootErr (Unwrapped & Unmodified)
else Normal Execution
API->>Rewriter: Check format == "project root not found" && len(args) == 0
alt Matches Missing Root Sentinel
Rewriter-->>API: Substitute ActionableProjectRootNotFound
end
API->>GoFmt: fmt.Errorf(format, args...)
GoFmt-->>Caller: Standard wrapped error (Supports errors.Is/As)
end
Unwrapping and Tree Inspection
When errors are wrapped using Errorf or ErrorStringBuilder.Wrap, the underlying error is formatted using %w. This preserves the complete causality chain across abstraction boundaries:
rootErr := &StorageWriteError{Path: "/data/cas/blob"}
wrappedErr := errfmt.Errorf("pipeline commit failed: %w", rootErr)
// Downstream inspection preserves root types
var target *StorageWriteError
if errors.As(wrappedErr, &target) {
// Target fields accessible without string parsing
log.Printf("Failed storage path: %s", target.Path)
}
4. Poison Pill Defense Mechanism: Preserving Electrocution Signals
In autonomous multi-agent environments, agents operate under strict supervisor loops. When an agent violates execution constraints, triggers security barriers, or exceeds budget thresholds, the supervisor issues an electrocution signal:
The Masking Vulnerability
Standard error-handling wrappers often wrap incoming errors inside layer-specific descriptions:
// Problematic anti-pattern in generic wrappers:
return fmt.Errorf("storage layer failed to commit: %w", err)
If an error containing "⚡️ BZZZT!" is wrapped with generic prefixes, naive regex checks or log monitors looking for root-level sentinels may misinterpret the failure as a transient storage glitch rather than an intentional abort. Furthermore, stack-unwinding mechanisms can mistake the error for a recoverable operational error and trigger inappropriate retry loops.
Subsystem Defense Implementation
Both errfmt.Errorf and (*ErrorStringBuilder).Wrap actively intercept electrocution errors:
// ⚡️ AGENT POISON PILL: Ensure electrocution messages are never wrapped in generic prefixes
for _, arg := range args {
if err, ok := arg.(error); ok && strings.Contains(err.Error(), "⚡️ BZZZT!") {
return err
}
}
When an error argument matches the electrocution sentinel, pkg/errfmt aborts formatting and returns the error directly. This guarantees:
- Fatal signals remain unpolluted.
- Supervisor watchdogs trip instantly without false retry attempts.
- Non-error arguments mentioning the string in diagnostic logs are not inadvertently aborted.
5. Actionable Diagnostics & Developer Experience
A core design principle of the ZQK Knowledge Kernel is that errors should provide unambiguous next actions rather than cryptic statements of fact.
Project Root Resolution Failure
One of the most frequent initialization errors occurs when a CLI command is run outside of an initialized workspace. Standard Unix tools typically emit:
error: project root not found
An operator or new agent receiving this message must deduce whether to run an initialization command, change directories, or set an environment variable. pkg/errfmt intercepts this condition and produces an actionable diagnostic:
const ActionableProjectRootNotFound = "project root not found. Run 'zqk system init --project-name <name>' to initialize, or set ZQK_PROJECT_ROOT"
Transformation Rules
- Bare Sentinel Match: Calling
errfmt.Errorf("project root not found")with zero arguments replaces the format string withActionableProjectRootNotFound. - Builder Parity: Calling
errfmt.Newf("project root not found")performs identical replacement. - Context Preservation: Calling
errfmt.Errorf("project root not found: %s", path)with arguments preserves the explicit format and arguments, ensuring custom diagnostics are not clobbered.
6. ErrorStringBuilder API Reference & Recipes
The ErrorStringBuilder struct enables fluent composition of complex error messages across sequential validation stages.
Method Index
| Method | Signature | Description |
|---|---|---|
Newf |
Newf(format string, args ...any) *ErrorStringBuilder |
Initializes builder with a formatted base message segment. |
NewErrorStringBuilder |
NewErrorStringBuilder(template string, args ...any) *ErrorStringBuilder |
Canonical alias for Newf. |
With |
With(template string, args ...any) *ErrorStringBuilder |
Appends a new formatted segment. Automatically supports %w within templates. |
AndOrDefault |
AndOrDefault(value, defaultValue string) *ErrorStringBuilder |
Appends value if non-empty after trimming; otherwise appends defaultValue. |
AndIfTrue |
AndIfTrue(cond bool, template string, args ...any) *ErrorStringBuilder |
Appends segment only when cond evaluates to true. |
Build |
Build() string |
Joins all segments using ": " as delimiter. Safe on nil receivers. |
String |
String() string |
Satisfies fmt.Stringer. Equivalent to Build(). |
Error |
Error() string |
Satisfies error interface. Allows builder to be returned directly as an error. |
Format |
Format(s fmt.State, verb rune) |
Satisfies fmt.Formatter for %s, %v, and %q verbs. |
Wrap |
Wrap(err error) error |
Joins segments and wraps err using fmt.Errorf("%s: %w", b.Build(), err). Honors poison pill. |
Practical Recipes
Recipe A: Multi-Stage Pipeline Validation
b := errfmt.Newf("pipeline stage %q", stageName).
With("attempt %d", attempt).
AndIfTrue(dryRun, "mode=dry-run").
AndOrDefault(userTag, "default-tenant")
if err := executeStage(); err != nil {
return b.Wrap(err)
}
Recipe B: Formatting with verbs in Logging
b := errfmt.Newf("transaction aborted").With("account %s", acctID)
// Compatible with standard format verbs:
fmt.Sprintf("%s", b) // "transaction aborted: account 123"
fmt.Sprintf("%q", b) // "\"transaction aborted: account 123\""
Recipe C: Returning Builder as Direct Error
func validateConfig(cfg *Config) error {
if cfg.Host == "" {
return errfmt.Newf("invalid configuration").
With("missing required parameter %q", "host")
}
return nil
}
7. Concurrency & Memory Model
- Immutable Construction per Goroutine: Builders are intended for sequential composition within a single goroutine stack frame.
- Thread-Safe Concurrent Reads: Once constructed, calling
Build(),String(),Error(), orFormat()on an*ErrorStringBuilderis fully thread-safe and re-entrant across concurrent goroutines. - Nil Safety: All receiver methods (
Build,String,Error,Format) safely guard against nil pointers, returning empty strings or no-ops without panicking. - Allocation Efficiency: Segments are aggregated in a compact slice of strings, minimizing intermediate heap allocations during multi-part assembly.
8. Verification & Static Coverage Invariant
The subsystem enforces a strict verification standard:
- Static Floor Invariant: Statement test coverage must strictly satisfy \(\ge 85.0\%\). Current test suites in
pkg/errfmtmaintain \(100.0\%\) statement coverage across all branches. - Three-Fold Proof Coverage:
1. Static Floor Invariant: Verified via unit tests covering all functions, receiver methods, and nil receiver guards.
2. Operational Dynamic Proof: End-to-end verification of
%wunwrapping (errors.Is/errors.As), multi-argument formatting, and fluent chaining. 3. Negative Adversarial Boundary: Poison pill non-wrapping of"⚡️ BZZZT!"electrocution sentinels, write error handling infmt.Formatter, and whitespace fallback resolution inAndOrDefault. - Zero Swallowed Errors Invariant: Every error returned by standard library callers is checked and handled.