Diagnostic Capture, Stagnation Detection & Runtime Profile Subsystem Architectural & Operational Guide
1. Executive Summary & Design Philosophy
The Diagnostic Capture, Stagnation Detection & Runtime Profile Subsystem (pkg/diagnostics) delivers deterministic failure introspection, non-blocking runtime state extraction, automated stagnation loop prevention, and signal-driven profiling across the ZQK Knowledge Kernel and daemon services.
When distributed processes, worker swarms, or background schedulers stall, loop endlessly, or encounter hard errors, naive retry harnesses waste computational budgets, mask root causes, and produce catastrophic cascade failures. The diagnostics subsystem addresses this through four core design pillars:
- Deterministic Structural Failure Attribution: Every execution failure is classified into standard, project-agnostic categories (
success,timeout,execution_failure,constraint_violation,context_cancelled,verification_failed,unknown) and fingerprinted with a cryptographically deterministic SHA-256 signature invariant to timestamps. - Entropy-Driven Stagnation Detection: Sequential retry attempts are evaluated for diagnostic delta. If successive failures produce zero entropy (identical failure signatures or repeated failing constraint patterns), the loop is immediately flagged as stagnant to trigger strategy pivoting or supervisor escalation rather than blind looping.
- Pluggable & Fail-Closed Diagnostic Extractors: Subsystems register isolated diagnostic extractors with a thread-safe registry. Extractors run sequentially with strict timeout bounds; a failure in any individual extractor never halts the report run.
- Non-Blocking Runtime Profiling & Signal Handling: Low-overhead Unix signal listeners (
SIGUSR1for diagnostic snapshots,SIGUSR2for 30-second CPU profiling) and an embedded, localhost-boundpprofHTTP server provide instant operational observability into live production binaries without restarts.
2. Subsystem Architectural Overview
The diagnostics subsystem bridges runtime OS telemetry, memory and thread profiling, failure classification, and retry feedback.
flowchart TD
subgraph Triggers["Diagnostic & Profiling Triggers"]
SIGUSR1["SIGUSR1 Signal (SetupSignalHandler)"]
SIGUSR2["SIGUSR2 Signal (StartCPUProfileOnSIGUSR2)"]
CronTimer["Scheduler Diagnostics Interval"]
TaskFailure["Task / BLI Execution Failure"]
PprofHTTP["Pprof HTTP Ingress (localhost:6060)"]
end
subgraph CaptureEngine["pkg/diagnostics Capture Core"]
CaptureDiag["CaptureDiagnostics"]
CaptureLock["captureMu Mutex Serializer"]
GoroutineDump["captureGoroutineDump (_goroutines.txt)"]
GoroutinePB["captureGoroutineProfileProto (_goroutines.pb.gz)"]
HeapProfile["captureHeapProfile (_heap.prof)"]
ProcInfo["captureProcessInfo (_processes.txt ps/top/lsof)"]
ThreadDump["captureThreadDump (_threads.txt)"]
PruneEngine["pruneDiagnosticsCaptures (Retention <= 10)"]
end
subgraph ExtractorRegistry["Pluggable Extractor Registry"]
Registry["DiagnosticExtractorRegistry"]
ExtTimeout["WithTimeout Wrapper"]
ReportBuilder["DiagnosticReport & ResultRecords"]
end
subgraph AdaptiveIntelligence["Loop Intelligence & Disambiguation"]
Classify["ClassifyError & ExtractFailureEnvelope"]
SigEngine["ComputeSignature (SHA-256)"]
Stagnation["EvaluateStagnation & StagnationPolicy"]
Checkpoints["BuildCheckpoints & Entropy Tracking"]
Disambiguation["GenerateDisambiguation & Negative Constraints"]
PromptFormat["FormatDisambiguationPrompt"]
end
subgraph StorageOutputs["Output Artifacts & Signals"]
DiagFiles[".zqk/scheduler/diagnostics/"]
CPUFiles[".zqk/logs/cpu-*.prof"]
PromptInjection["Adaptive Retry Prompt Injection"]
EscalationAlert["Supervisor Escalation / Pivot Trigger"]
end
SIGUSR1 --> CaptureLock
CronTimer --> CaptureLock
CaptureLock --> CaptureDiag
CaptureDiag --> GoroutineDump
CaptureDiag --> GoroutinePB
CaptureDiag --> HeapProfile
CaptureDiag --> ProcInfo
CaptureDiag --> ThreadDump
CaptureDiag --> PruneEngine
GoroutineDump --> DiagFiles
GoroutinePB --> DiagFiles
HeapProfile --> DiagFiles
ProcInfo --> DiagFiles
ThreadDump --> DiagFiles
SIGUSR2 --> CPUFiles
PprofHTTP --> CaptureEngine
TaskFailure --> Classify
Classify --> SigEngine
SigEngine --> Stagnation
Stagnation --> Checkpoints
Stagnation --> Disambiguation
Disambiguation --> PromptFormat
PromptFormat --> PromptInjection
Stagnation --> EscalationAlert
Registry --> ExtTimeout
ExtTimeout --> ReportBuilder
ReportBuilder --> Classify
3. Core Component Deep Dive
3.1 Failure Envelopes & Deterministic Signatures
The FailureEnvelope (pkg/diagnostics/envelope.go) encapsulates execution outcomes across attempts:
type FailureEnvelope struct {
Attempt int `json:"attempt"`
ExitCode int `json:"exit_code"`
ErrorMessage string `json:"error_message"`
Category string `json:"category"`
FailingConstraints []string `json:"failing_constraints,omitempty"`
ExecutedActions []string `json:"executed_actions,omitempty"`
DiagnosticReports []DiagnosticArtifact `json:"diagnostic_artifacts,omitempty"`
Signature string `json:"signature"`
Timestamp time.Time `json:"timestamp"`
}
Deterministic Cryptographic Fingerprint (ComputeSignature)
Failure signatures are calculated using SHA-256 over normalized structural characteristics:
- Failure category
- Numerical exit code
- Trimmed and bounded error message (capped at
DefaultMaxErrorLength = 4096bytes) - Alphabetically sorted constraints
- Ordered executed actions sequence
Because timestamp is intentionally excluded, two execution attempts that encounter the identical structural root cause produce byte-for-byte identical signatures.
Error Classification Categories
The ClassifyError function parses error texts and exit codes into unambiguous taxonomy:
CategorySuccess:err == nil,exitCode == 0, and zero failing constraints.CategoryConstraintViolation: One or more invariant or criteria constraints failed.CategoryTimeout: Deadlines exceeded or timed-out context errors.CategoryContextCancelled: Explicit caller or worker cancellation.CategoryVerificationFailed: Automated verifier or test assertion rejection.CategoryExecutionFailure: Process exited with non-zero exit code.CategoryUnknown: Unclassified fallbacks.
3.2 Loop Stagnation Detection & Adaptive Disambiguation
Retrying failed operations blindly when the underlying state has not changed burns token budgets and wastes compute. The stagnation detector (pkg/diagnostics/stagnation.go) and disambiguation engine (pkg/diagnostics/disambiguation.go) implement fail-fast feedback.
Stagnation Policy & Assessment Rules
A StagnationPolicy defines loop boundaries:
MaxStagnantAttempts: Consecutive zero-entropy retry attempts before escalating (default: 2).MaxIdenticalSignatures: Cumulative count of identical signatures across full history before escalating (default: 2).ZeroDeltaFailFast: Whentrue, any consecutive identical failure triggers an immediatepivot_strategyrecommendation.
stateDiagram-v2
[*] --> InitialExecution
InitialExecution --> Assessment: Attempt Fails
Assessment --> Continue: Non-zero Entropy Delta
Assessment --> PivotStrategy: Consecutive Zero Delta = 1 (FailFast)
Assessment --> EscalateSupervisor: Consecutive Zero Delta >= MaxStagnant
Assessment --> EscalateSupervisor: Signature Repeats >= MaxIdentical
PivotStrategy --> DisambiguationPrompt
DisambiguationPrompt --> RetryExecution
RetryExecution --> Assessment: Attempt Fails
Continue --> RetryExecution
EscalateSupervisor --> [*]: Loop Halted
Adaptive Disambiguation Prompts
When a retry is necessary, GenerateDisambiguation synthesizes actionable guidance:
- Negative Constraints: Extracts failing action patterns, ranking them by violation frequency to explicitly instruct the agent:
AVOID repeating action "X": <reason>. - Diagnostic Differential: Contrasts attempt $N-1$ and \(N\), documenting category or exit code shifts or highlighting zero-delta stagnation.
- Actionable Guidance: Synthesizes unsatisfied constraints and remediation steps, strictly bounded to
MaxPromptBytes(default: 2048 bytes).
3.3 Diagnostic Capture Pipeline (CaptureDiagnostics)
The CaptureDiagnostics function (pkg/diagnostics/capture.go) executes a comprehensive system dump:
sequenceDiagram
autonumber
participant Caller as Caller (Signal / Scheduler)
participant Lock as captureMu (Mutex)
participant Capture as CaptureDiagnostics
participant Pprof as Go Runtime / pprof
participant OS as OS Commands (ps/top/lsof)
participant Prune as pruneDiagnosticsCaptures
participant Disk as Filesystem (.zqk/)
Caller->>Lock: Lock()
Lock-->>Capture: Acquired
Capture->>Pprof: Write goroutine panic-style stacks
Pprof-->>Disk: *_goroutines.txt
Capture->>Pprof: Write gzip pprof profile proto
Pprof-->>Disk: *_goroutines.pb.gz
Capture->>Pprof: Write heap memory profile
Pprof-->>Disk: *_heap.prof
Capture->>OS: Execute ps, top, lsof (8s timeout each)
OS-->>Disk: *_processes.txt
Capture->>Pprof: captureThreadDump (runtime.Stack)
alt Goroutines >= SkipThreshold
Capture->>Disk: Write skip notice to *_threads.txt
else Goroutines < SkipThreshold
Capture->>Disk: Write full stacks to *_threads.txt
end
Capture->>Prune: Prune oldest capture sets
Prune->>Disk: Retain newest 10 sets, unlink older
Capture-->>Lock: Unlock()
Goroutine Skip Threshold Invariant
Under heavy scheduler load, running runtime.Stack on thousands of active goroutines can allocate 30MB+ buffers and stall execution.
- If the active goroutine count exceeds
threadDumpSkipRuntimeStackMinGoroutines(default: 5000, configurable viaZQK_DIAGNOSTICS_THREAD_STACK_SKIP_MIN_GOROUTINES), the thread dump body is safely skipped with a logged notice. - Complete goroutine stack traces remain available in the accompanying
*_goroutines.txtand*_goroutines.pb.gzartifacts.
Retention Pruning Invariant
Each capture invocation calls pruneDiagnosticsCaptures (pkg/diagnostics/capture_prune.go):
- Keeps the newest 10 capture sets (
diagnosticsRetentionMaxCaptureSets). - Unmatched or non-diagnostic files are left completely untouched.
- Cleanups are fail-safe: I/O errors during pruning never fail the diagnostic capture call.
3.4 Runtime Profiling & Embedded Pprof Server
SIGUSR1 Signal Handler (SetupSignalHandler)
Registers a background listener for syscall.SIGUSR1. Upon receipt, it spawns an asynchronous, bounded goroutine to invoke CaptureDiagnostics, writing artifacts to <projectRoot>/.zqk/diagnostics/<prefix>/. The main listener continues monitoring for subsequent signals.
SIGUSR2 CPU Profiler (StartCPUProfileOnSIGUSR2)
Registers a background listener for syscall.SIGUSR2. On trigger:
- Checks
cpuProfileActivemutex to prevent concurrent overlapping dumps. - Spawns a 30-second
pprof.StartCPUProfilewriting to<logsDir>/cpu-<timestamp>.prof. - Flushes and halts profiling cleanly upon duration expiry.
Embedded Localhost Pprof Server (StartPprofServerIfEnabled)
When enabled via environment variables, starts an HTTP server serving standard Go /debug/pprof/ routes:
- Binds strictly to
localhost:<port>(never public interfaces). - Built with
BuildPprofServerconfiguring timeouts:ReadHeaderTimeout = 5s,ReadTimeout = 30s,WriteTimeout = 60s,IdleTimeout = 60s.
4. Configuration & Environment Variables
| Variable | Fallback Keys | Default | Description |
|---|---|---|---|
ZQK_DIAGNOSTICS_THREAD_STACK_SKIP_MIN_GOROUTINES |
None | 5000 |
Minimum goroutine count at which runtime.Stack is skipped in _threads.txt. Set to 0 or negative to disable threshold skipping. |
ZQK_PPROF |
ZQK_SCHEDULER_PPROF, ZQK_STABLE_PPROF |
unset (false) |
Set to 1, true, or TRUE to start the embedded pprof HTTP server on bootstrap. |
ZQK_PPROF_PORT |
ZQK_SCHEDULER_PPROF_PORT, ZQK_STABLE_PPROF_PORT |
6060 |
TCP port for embedded pprof HTTP server (valid range 1–65535). |
5. Operational Runbooks
Runbook 1: On-Demand Diagnostic Snapshot via Signal
When a scheduler daemon or CLI process appears unresponsive:
# 1. Identify target process PID
PID=$(pgrep -f "zqk-scheduler" || pgrep -f "zqk")
# 2. Trigger diagnostic capture
kill -USR1 "$PID"
# 3. Inspect generated files
ls -lt .zqk/scheduler/diagnostics/
# Output includes:
# scheduler_daemon_<timestamp>_goroutines.txt
# scheduler_daemon_<timestamp>_goroutines.pb.gz
# scheduler_daemon_<timestamp>_heap.prof
# scheduler_daemon_<timestamp>_processes.txt
# scheduler_daemon_<timestamp>_threads.txt
Runbook 2: 30-Second On-Demand CPU Profile
To profile CPU bottlenecks without restarting the process:
# 1. Send SIGUSR2 to active daemon
kill -USR2 "$PID"
# 2. Wait 30 seconds for sampling duration to conclude
sleep 31
# 3. Analyze profile using standard go pprof
LATEST_PROF=$(ls -t .zqk/logs/cpu-*.prof | head -n 1)
go tool pprof -http=:8080 "$LATEST_PROF"
Runbook 3: Interactive Pprof HTTP Inspection
When ZQK_PPROF=1 is configured:
# 1. Check live goroutine counts
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -n 20
# 2. Download 30-second CPU trace
curl -o trace.out http://localhost:6060/debug/pprof/trace?seconds=30
go tool trace trace.out
# 3. Analyze heap memory allocation in browser
go tool pprof -http=:8081 http://localhost:6060/debug/pprof/heap
Runbook 4: Resolving Stagnant Retry Loops
When an agent or task repeatedly hits loop stagnation detected:
-
Check the failure envelope signature:
json { "is_stagnant": true, "consecutive_zero_delta": 2, "recommendation": "escalate_to_supervisor" } -
Inspect
failing_constraintsin the envelope to determine which precondition or postcondition is failing. - Review the generated
Negative Constraintsin the prompt injection to ensure the agent is not repeating prohibited commands or file mutations. - If the failure is non-transient, halt the execution cycle and route to an operator.
6. Verification & Invariants
| Invariant ID | Subsystem Invariant | Verification Mechanism |
|---|---|---|
| INV-DIAG-001 | Capture serialization: at most one CaptureDiagnostics runs concurrently. |
Mutex lock captureMu validated in unit tests. |
| INV-DIAG-002 | Goroutine stack skip threshold prevents out-of-memory under high concurrency. | threadStackSkipThresholdFromEnv tested with custom thresholds. |
| INV-DIAG-003 | Artifact pruning limits directory growth to newest 10 capture sets. | pruneDiagnosticsCaptures validated with mock datasets. |
| INV-DIAG-004 | Pluggable extractors fail-closed; single extractor failure never aborts run. | DiagnosticExtractorRegistry.Run tested with failing extractors. |
| INV-DIAG-005 | Cryptographic failure signatures are deterministic and invariant to timestamp. | ComputeSignature tested with identical parameters and varying times. |
| INV-DIAG-006 | Statement test coverage across pkg/diagnostics strictly satisfies \(\ge 85.0\%\). |
Automated CI gate: go test -race -coverprofile=coverage.out ./pkg/diagnostics/.... |