Tutorials, Demos & Guides
Docs » Tutorials, Demos & Guides » Developer & Agent Guide: Knowledge Kernel Diagnostics, Auto-Remedies & Self-Healing
Raw .md

Developer & Agent Guide: Knowledge Kernel Diagnostics, Auto-Remedies & Self-Healing

1. Executive Summary

In high-concurrency environments with multiple autonomous processes and agents mutating the repository simultaneously, transient failures can occur:

  • Abandoned lock files caused by abruptly killed processes.
  • Orphaned temporary files from interrupted file writes.
  • Content-Addressed Storage (CAS) hash mismatches or quarantine overflows.
  • Write-Ahead Log (WAL) journal fragmentation.

The ZQK Diagnostics & Self-Healing Engine provides continuous, multi-layered telemetry and deterministic auto-remedy recipes to detect, isolate, and repair system anomalies without manual human intervention.

2. The 4-Layer Inspection Model (zqk system check)

Running zqk system check performs a 4-layer holistic audit of the repository state:

┌────────────────────────────────────────────────────────┐
│ Layer 0: CAS Structural Integrity & Invariant Blockers │
├────────────────────────────────────────────────────────┤
│ Layer 1: Entities Needing Fixes (Inside Membrane)      │
├────────────────────────────────────────────────────────┤
│ Layer 2: Entities Ready for Execution / Promotion      │
├────────────────────────────────────────────────────────┤
│ Layer 3: Completed & Validated Entities                │
├────────────────────────────────────────────────────────┤
│ Layer 4: I/O Resource Hygiene & Storage Telemetry      │
└────────────────────────────────────────────────────────┘

Layer Telemetry Breakdown

  • Layer 0 (Integrity Blockers): Checks for corrupted YAML objects, malformed frontmatter, and broken cryptographic CAS digests.
  • Layer 1 (Schema & Lifecycle Deficiencies): Flags entities with missing required attributes, unbound criteria, or severed lineage.
  • Layer 2 (Ready Pipeline): Counts objects currently actionable by agents (planned items ready to claim, in_progress ready to promote).
  • Layer 3 (Historical Done Pipeline): Audits completed artifacts, archived items, and satisfied QA criteria.
  • Layer 4 (I/O & Storage Telemetry): Monitors open file descriptors, storage volume footprint, stale .lock files, and orphaned .tmp staging buffers.

3. Deterministic Auto-Remedies (--auto-remedy)

When anomalies are detected in Layer 4 (e.g., stale locks or orphaned temp files), the system outputs deterministic remedy recipes. Instead of manual removal, invoke:

zqk system check --auto-remedy

How Auto-Remedy Works

  1. Deadlock Detection: Examines file modification timestamps of locks in .zqk/lock/, .zqk/scheduler/locks/, and .zqk/scheduler/state/locks/.
  2. Process Liveness Verification: Checks whether the PID that created the lock is still alive. If the owning process has exited, the lock is classified as abandoned.
  3. Atomic Purge: Removes stale lock files and sweeps orphaned .tmp staging files.
  4. Receipt Generation: Outputs a verification receipt detailing every removed artifact.

4. Deep System Diagnostics (zqk system check --details & zqk feed doctor)

For comprehensive environment verification across compiler versions, git configurations, and daemon sockets:

zqk system check --details
zqk feed doctor

Diagnostic Check Categories

  • Toolchain & Storage Health: Verifies CAS integrity, draft plane objects, file descriptors, and storage volume footprint.
  • Daemon Telemetry: Tests connectivity to the background scheduler daemon (zqk scheduler health-check).
  • Storage Backend: Confirms file-based storage permissions and Content-Addressable Storage (CAS) digest verification.
  • Git Hook Integrity: Verifies that pre-commit and pre-push check-valves are executable and properly linked to zqk-vet.

5. Recovering from CAS Corruptions & Quarantine

If an unexpected power loss or interrupted disk write results in a CAS hash mismatch:

# 1. Detect corrupted objects and integrity blockers
zqk system check

# 2. Inspect quarantined objects
zqk system quarantine-report

# 3. Repair CAS filename and content hash mismatches
zqk system repair-cas-corruption

The repair engine re-parses the YAML payload, validates attributes against the schema registry, re-stamps a valid SHA-256 CAS hash, and re-admits the object into the authoritative storage plane.