Docman Documentation Registry & Cryptographic Verifier Architecture Guide
Executive Summary & System Overview
The pkg/docman subsystem is the core documentation management, metadata extraction, graph registration, and cryptographic verification engine within the ZQK Knowledge Kernel. In distributed software platforms and multi-agent autonomous engineering environments, documentation frequently suffers from semantic drift, broken references, stale runbooks, and unindexed filesystem sprawl.
pkg/docman bridges physical filesystem documentation with the Knowledge Kernel's relational graph. By discovering Markdown documents, parsing frontmatter and semantic AST structures, binding documents into versioned Content-Addressed Storage (doc_entry graph objects), and enforcing continuous cryptographic integrity (SHA-256 hashes and content byte sizes), pkg/docman ensures documentation remains an immutable, verifiable, and queryable asset across the entire development lifecycle.
flowchart TD
subgraph Filesystem["Project Filesystem Plane"]
MDDocs["Markdown Documents<br/>(docs/**/*.md)"]
Shipped["Shipped Subtrees<br/>(architecture, best-practices, onboarding)"]
end
subgraph DocmanCore["Docman Subsystem (pkg/docman)"]
Discoverer["Discoverer<br/>Subtree Traversal & Path Filtering"]
Parser["Parser<br/>AST Metadata & Heading Extraction"]
Registry["Registry<br/>Graph Registration & CAS Lifecycle"]
Updater["Updater<br/>Reference Normalization & Hash Backfill"]
Verifier["Verifier<br/>Cryptographic Drift Detection & Auto-Sealing"]
end
subgraph KernelStorage["Knowledge Kernel Storage Plane"]
StorageFactory["StorageFactory & ObjectStorageProvider"]
DocEntries["doc_entry Graph Objects<br/>(DOC-001, DOC-002, ...)"]
GraphIndex["Path Alias Index & Read Model Cache"]
end
Filesystem -->|Scan directories| Discoverer
Discoverer -->|MarkdownFile list| Registry
Registry -->|Parse content| Parser
Parser -->|DocumentMetadata| Registry
Registry -->|CAS Create & Draft Promotion| StorageFactory
Updater -->|Backfill Schema Fields| StorageFactory
StorageFactory -->|Persist| DocEntries
DocEntries -->|Index Paths| GraphIndex
Verifier -->|Read doc_entry| StorageFactory
Verifier -->|Verify disk integrity| Filesystem
Core Responsibilities & System Invariants
1. Subsystem Responsibilities
| Component | Primary Responsibility | Key Output / Artifact |
|---|---|---|
Discoverer |
Recursive filesystem scanner with exclusion heuristics, AppleDouble suppression, and subtree scoping. | Slice of *MarkdownFile containing absolute and normalized relative paths. |
Parser |
Extract semantic titles (H1), overview summaries, lifecycle statuses, and ontological groups/categories. | *DocumentMetadata containing structured document properties. |
Registry |
Manages doc_entry creation, sequential ID allocation, draft-plane promotion, and path alias registration. |
Versioned doc_entry graph objects persisted in Kernel CAS storage. |
Updater |
Schema migration and normalization tool; backfills missing reference arrays and cryptographic fields. | Updated doc_entry records with zero-cost idempotent patching. |
Verifier |
Cryptographic verification engine; audits on-disk SHA-256 and byte sizes against graph declarations. | *VerificationResult detailing passing, drifted, unsealed, and missing docs. |
2. Mandatory System Invariants
- Cryptographic Sealing Invariant: Every published
doc_entryobject in the Knowledge Kernel MUST declare cryptographic sealing fields (content_hashcontaining a 64-character lowercase hexadecimal SHA-256 digest, andcontent_sizedenoting exact byte length). - Fail-Closed Shipped Documentation Invariant: During system workspace initialization (
RegisterShippedDocs), all Markdown files located within core shipped documentation subtrees (docs/architecture,docs/best-practices,docs/onboarding) MUST successfully register into the kernel graph. If any document is unreadable, malformed, or missing from storage, system initialization halts fail-closed. - Draft-First Promotion Membrane: Object creation in
CreateDocEntryparks new entries on the preliminary draft plane. Any document whose parsed status is empty,draft, orconceptualis automatically promoted toreviewstatus upon creation to ensure visibility in standard queries while safeguarding authoritative release gates. - Path Canonicalization & Cache Invariant: Document paths are normalized to POSIX-compliant slash delimiters (
paths.NormalizeDocEntryPathForKey) and registered into the project path cache (paths.AddPathAlias), allowing instantaneous resolution across absolute paths, relative paths, andprefix:path-cache aliases. - Thread-Safety & Labeled Concurrency: All background scans, batch updates, and verifier sweeps MUST maintain thread safety across concurrent goroutines using mutex synchronization and labeled goroutine primitives (
goroutinelabels.NewGoroutine).
Document Discovery & Path Filtering Hierarchy
The Discoverer struct scans documentation directories while preventing index pollution from vendor directories, version control trees, cache directories, or operating system metadata.
flowchart LR
Start([Discover Invoked]) --> CheckMode{Subtrees Specified?}
CheckMode -- Yes --> ScanSubtrees[Scan explicit subtrees]
CheckMode -- No --> ScanDocs[Scan prefix:docs directory]
ScanSubtrees --> Walk[filepath.Walk]
ScanDocs --> Walk
Walk --> CheckDir{Is Directory?}
CheckDir -- Yes --> CheckExDir{Is Excluded Dir?}
CheckExDir -- Yes --> SkipDir[filepath.SkipDir]
CheckExDir -- No --> Continue[Continue Traversal]
CheckDir -- No --> CheckExFile{Is Excluded Path?}
CheckExFile -- Yes --> Ignore[Ignore]
CheckExFile -- No --> CheckApple{AppleDouble ._*?}
CheckApple -- Yes --> Ignore
CheckApple -- No --> CheckExt{Has .md Extension?}
CheckExt -- Yes --> Collect[Append to MarkdownFile List]
CheckExt -- No --> Ignore
Discovery Heuristics & Exclusion Rules
The discoverer applies strict path-filtering rules:
- Default Excluded Directories:
.git,node_modules,__pycache__,.zqk(project data dir),archive,_archive. - Case-Insensitive Match: Directory segments are inspected with
strings.EqualFold, ensuring cross-platform consistency between case-sensitive Linux filesystems and case-insensitive macOS/Windows mounts. - AppleDouble Suppression: On macOS hosts, hidden metadata fork files matching
._*are detected and dropped viaappledouble.SkipPathInTreeWalk. - Subtree Scoping: Scans can be constrained to arbitrary directory slices (e.g.,
docs/architectureordocs/onboarding) usingNewDiscovererWithSubtrees.
Markdown AST Parsing & Metadata Extraction
The Parser struct extracts semantic frontmatter and structural metadata from raw Markdown content without external heavy dependencies.
sequenceDiagram
participant D as Discoverer
participant P as Parser
participant R as Registry
D->>R: Deliver MarkdownFile (Path, RelPath)
R->>P: Parse(file.Path)
P->>P: Read file bytes & split lines
P->>P: extractTitle (find first H1 '# Title', strip 'v1.0')
P->>P: extractSummary (parse '## Overview', clean Markdown, take first sentence)
P->>P: extractStatus (parse '**Status**: ...', map to active/deprecated)
P->>P: determineGroupAndCategory (path heuristic mapping)
P-->>R: Return DocumentMetadata struct
Metadata Extraction Logic
-
Title Extraction (
extractTitle): - Scans line-by-line for the first Markdown H1 heading (# ...). - Strips semantic version suffixes (e.g.,# Core Architecture v1.0->Core Architecture). - Fallback: If no H1 exists, converts the filename base into title case, converting hyphens and underscores to spaces (e.g.,system-design_overview.md->System Design Overview). -
Summary Synthesis (
extractSummary): - Targets the## Overviewheading and extracts body content up to the next##heading. - Fallback: Extracts the first non-metadata paragraph following the title, skipping bold header lines like**Author**: Agent. - Markdown Sanitation: Strips bold markers (**), backticks (`), links ([text](url)->text), and heading prefixes (#). - Sentence Isolation: Bounded to the first complete sentence using delimiter regex[.!?]\s+and capped at 200 characters. -
Status Mapping (
extractStatus): - Parses**Status**: <status>metadata. - Mapsactive,implemented,design complete, andresearch completeto canonicalactive. - Mapsdeprecatedandsupersededto canonicaldeprecated. -
Ontological Group & Category Determination (
determineGroupAndCategory): - Analyzes relative filesystem path tokens to classify documents into standard domains:docs/architecture/*-> Group:architecture, Category:validation,graph-backend,storage,cli,migration,audit, orcheck.docs/onboarding/*-> Group:onboarding, Category:"".docs/marketing/*ordocs/strategic/*-> Group:other, Category:strategic.docs/process/_internal/*-> Group:process, Category:internal.
Documentation Registry Lifecycle & Mutation Safety
The Registry coordinates document registration into the Knowledge Kernel's Content-Addressed Storage provider.
sequenceDiagram
participant Client as Invoking Client (CLI / System Init)
participant Reg as Registry
participant Disc as Discoverer
participant Store as ObjectStorageProvider
participant Cache as Path Alias Cache
Client->>Reg: RegisterAll / RegisterSubtrees(subtrees, dryRun)
Reg->>Disc: Discover()
Disc-->>Reg: []*MarkdownFile
Reg->>Store: List(KindDocEntry)
Store-->>Reg: Existing doc_entries & maxID
loop For each MarkdownFile
Reg->>Reg: Check deduplication (normalized key & relPath)
alt Already exists
Reg->>Reg: Increment skipped counter
else Needs creation
Reg->>Reg: Parse metadata & compute DOC-xxx ID
Reg->>Reg: Compute SHA-256 & byte size
alt dryRun == true
Reg->>Reg: Log dry-run creation, increment count
else dryRun == false
Reg->>Store: Create(entry) [parks on draft plane]
Reg->>Store: Update(id, status=review) [promotes off draft plane]
Reg->>Cache: AddPathAlias(relPath, relPath)
Reg->>Reg: TouchMeaningfulActivity()
end
end
end
Reg-->>Client: (registeredCount, skippedCount, err)
Two-Phase Draft Promotion & Deduplication
- Deduplication Check: Before invoking the parser or attempting creation, the registry checks both
paths.NormalizeDocEntryPathForKey(file.RelPath)andfile.RelPathagainst the existing cache. - Draft Membrane: Objects created via
r.storageProvider.Createoriginate at the draft lifecycle origin. The registry immediately promotes new entries with statusdraftorconceptualtoreviewstatus viar.storageProvider.Update, ensuring immediate discoverability across kernel read models. - Path Alias Injection: Upon successful creation,
paths.AddPathAliasupdates the in-memory and on-disk path resolution index so that subsequent references using relative paths resolve in sub-millisecond time.
Schema Normalization & Backfilling (Updater)
The UpdateExistingEntries method runs idempotent schema migrations across all existing doc_entry graph objects:
- Missing Relational Fields: Ensures empty string slices exist for
goal_refs,workstream_refs,milestone_refs, andrequirement_refs. - Missing Cryptographic Sealing: Computes missing
content_hashandcontent_sizefields for legacy objects where on-disk files are present.
Cryptographic Verifier & Auto-Sealing Engine
The Verifier audits documentation health, detecting broken filesystem links, tampered contents, and unsealed documents.
stateDiagram-v2
[*] --> EvaluatingEntry: Inspect doc_entry
EvaluatingEntry --> TargetMissing: File does not exist on disk
EvaluatingEntry --> TargetUnreadable: File permissions unreadable
EvaluatingEntry --> Unsealed: content_hash empty in doc_entry
EvaluatingEntry --> HashMismatch: Disk SHA-256 != doc_entry hash
EvaluatingEntry --> SizeMismatch: Disk size != doc_entry size
EvaluatingEntry --> Verified: Hash & Size match perfectly
TargetMissing --> ReportViolation: Mark Missing
TargetUnreadable --> ReportViolation: Mark Missing
Unsealed --> CheckAutoSeal: AutoSeal Enabled?
HashMismatch --> CheckAutoSeal: AutoSeal Enabled?
SizeMismatch --> CheckAutoSeal: AutoSeal Enabled?
CheckAutoSeal --> RecomputeHash: Yes & Target Readable
CheckAutoSeal --> ReportViolation: No (Drifted / Unsealed)
RecomputeHash --> UpdateCAS: Compute SHA-256 & byte size
UpdateCAS --> Verified: doc_entry updated successfully
UpdateCAS --> ReportViolation: CAS Update Failed
Verified --> [*]: Result Passed
ReportViolation --> [*]: Result Violation Recorded
Verification Result Taxonomy
The verifier aggregates results into a VerificationResult report:
TotalChecked: Count of examineddoc_entryobjects.Passed: Count of documents passing cryptographic verification.Drifted: Count of documents where disk hash or size differed from stored metadata.Unsealed: Count of documents missing cryptographic sealing declarations.Missing: Count of documents where target disk files are missing or unreadable.AutoSealed: Count of drifted/unsealed documents successfully re-sealed.Violations: Detailed list of individualstorage.DocIntegrityViolationrecords.
Fail-Closed Check-Valve in System Boot
During system startup, RegisterShippedDocs invokes a fail-closed post-init verification check:
// Fail-closed verification: ensure every discovered markdown file in ShippedInitDocSubtrees has a doc_entry
discoverer := NewDiscovererWithSubtrees(projectRoot, ShippedInitDocSubtrees)
files, err := discoverer.Discover()
...
for _, f := range files {
key := paths.NormalizeDocEntryPathForKey(f.RelPath)
if _, ok := existing[key]; !ok {
return reg, skip, errfmt.Errorf("fail-closed: shipped doc %q has no registered doc_entry in kernel graph", f.RelPath)
}
}
If a developer deletes or renames a core architecture document without updating the kernel registry, the workspace boot pipeline will immediately fail closed with a diagnostic trace.
Production Operations & Runbooks
Runbook 1: Registering All Documentation
To scan and register newly added Markdown documentation across the workspace:
# Preview documents to be registered without writing to storage
zqk docman register --dry-run
# Register all workspace documentation
zqk docman register
# Register only core shipped documentation subtrees
zqk docman register --shipped-only
Runbook 2: Cryptographic Drift Auditing & Auto-Sealing
To verify workspace documentation integrity:
# Run strict cryptographic verification (fails closed if drift detected)
zqk docman verify --strict
# Audit specific subtrees
zqk docman verify --subtrees docs/architecture,docs/onboarding
# Re-seal documentation after authorized modifications
zqk docman verify --auto-seal
Runbook 3: Backfilling Legacy Entries
When upgrading documentation schema contracts or adding reference fields:
# Update existing doc_entries with missing reference arrays and SHA-256 seals
zqk docman register --update-existing
Runbook 4: Disaster Recovery & Unsealing Drift Remediation
Symptom: CI pre-commit or release gate fails with DocIntegrityCodeHashMismatch or fail-closed: shipped doc has no registered doc_entry.
Root Cause: A Markdown document was edited directly on disk or through a pull request without updating the cryptographic hash in its corresponding doc_entry graph object.
Remediation Steps:
-
Check git status to confirm modified files:
bash git status docs/ -
Run auto-sealing to recompute hashes:
bash zqk docman verify --auto-seal -
Verify all violations have been resolved:
bash zqk docman verify --strict -
Commit updated
doc_entryobjects alongside the Markdown edits.
Verification & Conformance Evidence
Statement test coverage across pkg/docman is rigorously maintained above the 85.0% architectural floor:
| Subsystem File | Target Responsibility | Statement Coverage | Conformance Status |
|---|---|---|---|
discoverer.go |
Filesystem scanning, exclude filters, AppleDouble skipping | 87.5% - 100.0% | PASS |
parser.go |
Markdown AST parsing, title/summary extraction, classification | 97.2% - 100.0% | PASS |
registry.go |
Graph entry synthesis, ID allocation, fail-closed shipped docs | 82.1% - 100.0% | PASS |
updater.go |
Schema backfilling, reference normalization, hash calculation | 97.8% - 100.0% | PASS |
verifier.go |
Cryptographic verification, drift detection, auto-sealing | 98.6% - 100.0% | PASS |
Total pkg/docman |
Comprehensive Documentation Subsystem | 95.9% | PASS (Exceeds 85% Target) |