Docman Documentation Registry & Cryptographic Verifier Architecture Guide

Executive Summary & System Overview

The pkg/docman subsystem is the core documentation management, metadata extraction, graph registration, and cryptographic verification engine within the ZQK Knowledge Kernel. In distributed software platforms and multi-agent autonomous engineering environments, documentation frequently suffers from semantic drift, broken references, stale runbooks, and unindexed filesystem sprawl.

pkg/docman bridges physical filesystem documentation with the Knowledge Kernel's relational graph. By discovering Markdown documents, parsing frontmatter and semantic AST structures, binding documents into versioned Content-Addressed Storage (doc_entry graph objects), and enforcing continuous cryptographic integrity (SHA-256 hashes and content byte sizes), pkg/docman ensures documentation remains an immutable, verifiable, and queryable asset across the entire development lifecycle.

flowchart TD
    subgraph Filesystem["Project Filesystem Plane"]
        MDDocs["Markdown Documents<br/>(docs/**/*.md)"]
        Shipped["Shipped Subtrees<br/>(architecture, best-practices, onboarding)"]
    end

    subgraph DocmanCore["Docman Subsystem (pkg/docman)"]
        Discoverer["Discoverer<br/>Subtree Traversal & Path Filtering"]
        Parser["Parser<br/>AST Metadata & Heading Extraction"]
        Registry["Registry<br/>Graph Registration & CAS Lifecycle"]
        Updater["Updater<br/>Reference Normalization & Hash Backfill"]
        Verifier["Verifier<br/>Cryptographic Drift Detection & Auto-Sealing"]
    end

    subgraph KernelStorage["Knowledge Kernel Storage Plane"]
        StorageFactory["StorageFactory & ObjectStorageProvider"]
        DocEntries["doc_entry Graph Objects<br/>(DOC-001, DOC-002, ...)"]
        GraphIndex["Path Alias Index & Read Model Cache"]
    end

    Filesystem -->|Scan directories| Discoverer
    Discoverer -->|MarkdownFile list| Registry
    Registry -->|Parse content| Parser
    Parser -->|DocumentMetadata| Registry
    Registry -->|CAS Create & Draft Promotion| StorageFactory
    Updater -->|Backfill Schema Fields| StorageFactory
    StorageFactory -->|Persist| DocEntries
    DocEntries -->|Index Paths| GraphIndex
    Verifier -->|Read doc_entry| StorageFactory
    Verifier -->|Verify disk integrity| Filesystem

Core Responsibilities & System Invariants

1. Subsystem Responsibilities

Component Primary Responsibility Key Output / Artifact
Discoverer Recursive filesystem scanner with exclusion heuristics, AppleDouble suppression, and subtree scoping. Slice of *MarkdownFile containing absolute and normalized relative paths.
Parser Extract semantic titles (H1), overview summaries, lifecycle statuses, and ontological groups/categories. *DocumentMetadata containing structured document properties.
Registry Manages doc_entry creation, sequential ID allocation, draft-plane promotion, and path alias registration. Versioned doc_entry graph objects persisted in Kernel CAS storage.
Updater Schema migration and normalization tool; backfills missing reference arrays and cryptographic fields. Updated doc_entry records with zero-cost idempotent patching.
Verifier Cryptographic verification engine; audits on-disk SHA-256 and byte sizes against graph declarations. *VerificationResult detailing passing, drifted, unsealed, and missing docs.

2. Mandatory System Invariants

  1. Cryptographic Sealing Invariant: Every published doc_entry object in the Knowledge Kernel MUST declare cryptographic sealing fields (content_hash containing a 64-character lowercase hexadecimal SHA-256 digest, and content_size denoting exact byte length).
  2. Fail-Closed Shipped Documentation Invariant: During system workspace initialization (RegisterShippedDocs), all Markdown files located within core shipped documentation subtrees (docs/architecture, docs/best-practices, docs/onboarding) MUST successfully register into the kernel graph. If any document is unreadable, malformed, or missing from storage, system initialization halts fail-closed.
  3. Draft-First Promotion Membrane: Object creation in CreateDocEntry parks new entries on the preliminary draft plane. Any document whose parsed status is empty, draft, or conceptual is automatically promoted to review status upon creation to ensure visibility in standard queries while safeguarding authoritative release gates.
  4. Path Canonicalization & Cache Invariant: Document paths are normalized to POSIX-compliant slash delimiters (paths.NormalizeDocEntryPathForKey) and registered into the project path cache (paths.AddPathAlias), allowing instantaneous resolution across absolute paths, relative paths, and prefix: path-cache aliases.
  5. Thread-Safety & Labeled Concurrency: All background scans, batch updates, and verifier sweeps MUST maintain thread safety across concurrent goroutines using mutex synchronization and labeled goroutine primitives (goroutinelabels.NewGoroutine).

Document Discovery & Path Filtering Hierarchy

The Discoverer struct scans documentation directories while preventing index pollution from vendor directories, version control trees, cache directories, or operating system metadata.

flowchart LR
    Start([Discover Invoked]) --> CheckMode{Subtrees Specified?}
    CheckMode -- Yes --> ScanSubtrees[Scan explicit subtrees]
    CheckMode -- No --> ScanDocs[Scan prefix:docs directory]

    ScanSubtrees --> Walk[filepath.Walk]
    ScanDocs --> Walk

    Walk --> CheckDir{Is Directory?}
    CheckDir -- Yes --> CheckExDir{Is Excluded Dir?}
    CheckExDir -- Yes --> SkipDir[filepath.SkipDir]
    CheckExDir -- No --> Continue[Continue Traversal]

    CheckDir -- No --> CheckExFile{Is Excluded Path?}
    CheckExFile -- Yes --> Ignore[Ignore]
    CheckExFile -- No --> CheckApple{AppleDouble ._*?}
    CheckApple -- Yes --> Ignore
    CheckApple -- No --> CheckExt{Has .md Extension?}
    CheckExt -- Yes --> Collect[Append to MarkdownFile List]
    CheckExt -- No --> Ignore

Discovery Heuristics & Exclusion Rules

The discoverer applies strict path-filtering rules:

  • Default Excluded Directories: .git, node_modules, __pycache__, .zqk (project data dir), archive, _archive.
  • Case-Insensitive Match: Directory segments are inspected with strings.EqualFold, ensuring cross-platform consistency between case-sensitive Linux filesystems and case-insensitive macOS/Windows mounts.
  • AppleDouble Suppression: On macOS hosts, hidden metadata fork files matching ._* are detected and dropped via appledouble.SkipPathInTreeWalk.
  • Subtree Scoping: Scans can be constrained to arbitrary directory slices (e.g., docs/architecture or docs/onboarding) using NewDiscovererWithSubtrees.

Markdown AST Parsing & Metadata Extraction

The Parser struct extracts semantic frontmatter and structural metadata from raw Markdown content without external heavy dependencies.

sequenceDiagram
    participant D as Discoverer
    participant P as Parser
    participant R as Registry

    D->>R: Deliver MarkdownFile (Path, RelPath)
    R->>P: Parse(file.Path)
    P->>P: Read file bytes & split lines
    P->>P: extractTitle (find first H1 '# Title', strip 'v1.0')
    P->>P: extractSummary (parse '## Overview', clean Markdown, take first sentence)
    P->>P: extractStatus (parse '**Status**: ...', map to active/deprecated)
    P->>P: determineGroupAndCategory (path heuristic mapping)
    P-->>R: Return DocumentMetadata struct

Metadata Extraction Logic

  1. Title Extraction (extractTitle): - Scans line-by-line for the first Markdown H1 heading (# ...). - Strips semantic version suffixes (e.g., # Core Architecture v1.0 -> Core Architecture). - Fallback: If no H1 exists, converts the filename base into title case, converting hyphens and underscores to spaces (e.g., system-design_overview.md -> System Design Overview).

  2. Summary Synthesis (extractSummary): - Targets the ## Overview heading and extracts body content up to the next ## heading. - Fallback: Extracts the first non-metadata paragraph following the title, skipping bold header lines like **Author**: Agent. - Markdown Sanitation: Strips bold markers (**), backticks (`), links ([text](url) -> text), and heading prefixes (#). - Sentence Isolation: Bounded to the first complete sentence using delimiter regex [.!?]\s+ and capped at 200 characters.

  3. Status Mapping (extractStatus): - Parses **Status**: <status> metadata. - Maps active, implemented, design complete, and research complete to canonical active. - Maps deprecated and superseded to canonical deprecated.

  4. Ontological Group & Category Determination (determineGroupAndCategory): - Analyzes relative filesystem path tokens to classify documents into standard domains:

    • docs/architecture/* -> Group: architecture, Category: validation, graph-backend, storage, cli, migration, audit, or check.
    • docs/onboarding/* -> Group: onboarding, Category: "".
    • docs/marketing/* or docs/strategic/* -> Group: other, Category: strategic.
    • docs/process/_internal/* -> Group: process, Category: internal.

Documentation Registry Lifecycle & Mutation Safety

The Registry coordinates document registration into the Knowledge Kernel's Content-Addressed Storage provider.

sequenceDiagram
    participant Client as Invoking Client (CLI / System Init)
    participant Reg as Registry
    participant Disc as Discoverer
    participant Store as ObjectStorageProvider
    participant Cache as Path Alias Cache

    Client->>Reg: RegisterAll / RegisterSubtrees(subtrees, dryRun)
    Reg->>Disc: Discover()
    Disc-->>Reg: []*MarkdownFile
    Reg->>Store: List(KindDocEntry)
    Store-->>Reg: Existing doc_entries & maxID

    loop For each MarkdownFile
        Reg->>Reg: Check deduplication (normalized key & relPath)
        alt Already exists
            Reg->>Reg: Increment skipped counter
        else Needs creation
            Reg->>Reg: Parse metadata & compute DOC-xxx ID
            Reg->>Reg: Compute SHA-256 & byte size
            alt dryRun == true
                Reg->>Reg: Log dry-run creation, increment count
            else dryRun == false
                Reg->>Store: Create(entry) [parks on draft plane]
                Reg->>Store: Update(id, status=review) [promotes off draft plane]
                Reg->>Cache: AddPathAlias(relPath, relPath)
                Reg->>Reg: TouchMeaningfulActivity()
            end
        end
    end
    Reg-->>Client: (registeredCount, skippedCount, err)

Two-Phase Draft Promotion & Deduplication

  • Deduplication Check: Before invoking the parser or attempting creation, the registry checks both paths.NormalizeDocEntryPathForKey(file.RelPath) and file.RelPath against the existing cache.
  • Draft Membrane: Objects created via r.storageProvider.Create originate at the draft lifecycle origin. The registry immediately promotes new entries with status draft or conceptual to review status via r.storageProvider.Update, ensuring immediate discoverability across kernel read models.
  • Path Alias Injection: Upon successful creation, paths.AddPathAlias updates the in-memory and on-disk path resolution index so that subsequent references using relative paths resolve in sub-millisecond time.

Schema Normalization & Backfilling (Updater)

The UpdateExistingEntries method runs idempotent schema migrations across all existing doc_entry graph objects:

  • Missing Relational Fields: Ensures empty string slices exist for goal_refs, workstream_refs, milestone_refs, and requirement_refs.
  • Missing Cryptographic Sealing: Computes missing content_hash and content_size fields for legacy objects where on-disk files are present.

Cryptographic Verifier & Auto-Sealing Engine

The Verifier audits documentation health, detecting broken filesystem links, tampered contents, and unsealed documents.

stateDiagram-v2
    [*] --> EvaluatingEntry: Inspect doc_entry

    EvaluatingEntry --> TargetMissing: File does not exist on disk
    EvaluatingEntry --> TargetUnreadable: File permissions unreadable
    EvaluatingEntry --> Unsealed: content_hash empty in doc_entry
    EvaluatingEntry --> HashMismatch: Disk SHA-256 != doc_entry hash
    EvaluatingEntry --> SizeMismatch: Disk size != doc_entry size
    EvaluatingEntry --> Verified: Hash & Size match perfectly

    TargetMissing --> ReportViolation: Mark Missing
    TargetUnreadable --> ReportViolation: Mark Missing

    Unsealed --> CheckAutoSeal: AutoSeal Enabled?
    HashMismatch --> CheckAutoSeal: AutoSeal Enabled?
    SizeMismatch --> CheckAutoSeal: AutoSeal Enabled?

    CheckAutoSeal --> RecomputeHash: Yes & Target Readable
    CheckAutoSeal --> ReportViolation: No (Drifted / Unsealed)

    RecomputeHash --> UpdateCAS: Compute SHA-256 & byte size
    UpdateCAS --> Verified: doc_entry updated successfully
    UpdateCAS --> ReportViolation: CAS Update Failed

    Verified --> [*]: Result Passed
    ReportViolation --> [*]: Result Violation Recorded

Verification Result Taxonomy

The verifier aggregates results into a VerificationResult report:

  • TotalChecked: Count of examined doc_entry objects.
  • Passed: Count of documents passing cryptographic verification.
  • Drifted: Count of documents where disk hash or size differed from stored metadata.
  • Unsealed: Count of documents missing cryptographic sealing declarations.
  • Missing: Count of documents where target disk files are missing or unreadable.
  • AutoSealed: Count of drifted/unsealed documents successfully re-sealed.
  • Violations: Detailed list of individual storage.DocIntegrityViolation records.

Fail-Closed Check-Valve in System Boot

During system startup, RegisterShippedDocs invokes a fail-closed post-init verification check:

// Fail-closed verification: ensure every discovered markdown file in ShippedInitDocSubtrees has a doc_entry
discoverer := NewDiscovererWithSubtrees(projectRoot, ShippedInitDocSubtrees)
files, err := discoverer.Discover()
...
for _, f := range files {
    key := paths.NormalizeDocEntryPathForKey(f.RelPath)
    if _, ok := existing[key]; !ok {
        return reg, skip, errfmt.Errorf("fail-closed: shipped doc %q has no registered doc_entry in kernel graph", f.RelPath)
    }
}

If a developer deletes or renames a core architecture document without updating the kernel registry, the workspace boot pipeline will immediately fail closed with a diagnostic trace.


Production Operations & Runbooks

Runbook 1: Registering All Documentation

To scan and register newly added Markdown documentation across the workspace:

# Preview documents to be registered without writing to storage
zqk docman register --dry-run

# Register all workspace documentation
zqk docman register

# Register only core shipped documentation subtrees
zqk docman register --shipped-only

Runbook 2: Cryptographic Drift Auditing & Auto-Sealing

To verify workspace documentation integrity:

# Run strict cryptographic verification (fails closed if drift detected)
zqk docman verify --strict

# Audit specific subtrees
zqk docman verify --subtrees docs/architecture,docs/onboarding

# Re-seal documentation after authorized modifications
zqk docman verify --auto-seal

Runbook 3: Backfilling Legacy Entries

When upgrading documentation schema contracts or adding reference fields:

# Update existing doc_entries with missing reference arrays and SHA-256 seals
zqk docman register --update-existing

Runbook 4: Disaster Recovery & Unsealing Drift Remediation

Symptom: CI pre-commit or release gate fails with DocIntegrityCodeHashMismatch or fail-closed: shipped doc has no registered doc_entry.

Root Cause: A Markdown document was edited directly on disk or through a pull request without updating the cryptographic hash in its corresponding doc_entry graph object.

Remediation Steps:

  1. Check git status to confirm modified files: bash git status docs/

  2. Run auto-sealing to recompute hashes: bash zqk docman verify --auto-seal

  3. Verify all violations have been resolved: bash zqk docman verify --strict

  4. Commit updated doc_entry objects alongside the Markdown edits.


Verification & Conformance Evidence

Statement test coverage across pkg/docman is rigorously maintained above the 85.0% architectural floor:

Subsystem File Target Responsibility Statement Coverage Conformance Status
discoverer.go Filesystem scanning, exclude filters, AppleDouble skipping 87.5% - 100.0% PASS
parser.go Markdown AST parsing, title/summary extraction, classification 97.2% - 100.0% PASS
registry.go Graph entry synthesis, ID allocation, fail-closed shipped docs 82.1% - 100.0% PASS
updater.go Schema backfilling, reference normalization, hash calculation 97.8% - 100.0% PASS
verifier.go Cryptographic verification, drift detection, auto-sealing 98.6% - 100.0% PASS
Total pkg/docman Comprehensive Documentation Subsystem 95.9% PASS (Exceeds 85% Target)