Architecture & Foundation
Docs » Architecture & Foundation » Tiered Storage, Subtree Flattening, and Archival Lifecycle
Raw .md

Tiered Storage, Subtree Flattening, and Archival Lifecycle

Technical Specification: TSP-TIERED-STORAGE-ARCHIVE-001
Governing Policy: POL-STORAGE-RETENTION-001
Status: Approved / Active Architecture
Target Audience: Core Storage Architects, DevOps, Agent Operators, Platform Engineers

1. Executive Summary & Scaling Mandate

In high-velocity AI coding environments, autonomous agent swarms produce immense volumes of process entities: goals, milestones, requirements, acceptance criteria, backlog items, tasks, and verification receipts.

Historically, setting an entity's status to archived was purely a metadata label—the underlying YAML blob remained loose on local disk in .zqk/process/<kind>/<hash>.yaml. As the graph expands past tens of thousands of objects:

  • Filesystem & Inode Exhaustion: Directory walking across 20,000+ files degrades filesystem caching and inflates I/O wait.
  • System Check Scan Penalties: Discovery routines must evaluate thousands of dead objects during every validation pass.
  • Active Edge Ambiguity: Completed historical nodes clutter active graph traversal queries.

This architecture introduces Tiered Storage and Subgraph Archival, converting loose historical objects into immutable Semantic Capsules across a 4-tier lifecycle: Hot → Warm → Cold → Abyss.

2. The 4-Tier Storage Topology

flowchart TD
    subgraph Tier0["Tier 0: HOT STORAGE (Active CAS Plane)"]
        H1[".zqk/process/[kind]/[hash].yaml"]
        H2["Loose CAS Blobs - Mutable Edges - 0ms Read/Write"]
    end

    subgraph Tier1["Tier 1: WARM STORAGE (Local Capsule Archive)"]
        W1[".zqk/archive/bundles/[ROOT-ID].capsule.zst"]
        W2["Apoptotic Plane - Strictly Immutable - Transparent Read-Through"]
        W3["85-92% Storage and File Reduction"]
    end

    subgraph Tier2["Tier 2: COLD STORAGE (Detached Remote Vault)"]
        C1["Remote Object Store (S3 / GCS / Git LFS / ~/.zqk/cold-vault)"]
        C2["0 Bytes Local Payload - Tombstone Locator in Index"]
        C3["On-Demand Fetch: Remote vault rehydration / zqk system state-restore"]
    end

    subgraph Tier3["Tier 3: THE ABYSS (A-Bits / Cryptographic Purge)"]
        A1["Erased Data Payload (0 Bytes Local and Remote)"]
        A2["Immutable Merkle Receipt in .zqk/streams/ WAL"]
        A3["Non-Repudiation and Cryptographic Audit Proof"]
    end

    Tier0 -->|Inactivity / Storage Watermark Trigger| Tier1
    Tier1 -->|Warm Retention Expiry / Local Quota Breach| Tier2
    Tier2 -->|Policy Expiration / Retention Purge| Tier3

Storage Tier Specification

Tier Lifecycle Plane Storage Media Index Footprint Read SLA Mutation Policy
Hot promoted Loose CAS YAML (.zqk/process/) Full In-Memory Cache < 1ms direct disk Fully mutable (via Mode A or Mode B socket)
Warm apoptotic Compressed Capsule (.capsule.zst) Tombstone Pointer in CAS Index 1–2ms in-memory stream Forbidden (Mathematically Immutable)
Cold apoptotic Detached Vault (S3/GCS/FS) Remote URL Locator Network fetch on-demand Forbidden
Abyss purged Zeroed (WAL Audit stream only) Historical Merkle Hash Non-recoverable (Proof only) Permanent Eviction

3. Subtree Flattening & Semantic Compression

Archival operates at subgraph granularity, not object isolation. A Goal roots a directed acyclic graph (DAG):

flowchart LR
    Goal["Goal"] --> Milestones["Milestones"] --> Requirements["Requirements"] --> Criteria["Criteria / BLIs"] --> Tasks["Tasks"] --> QARecords["QA Records"]

Archival Pipeline

flowchart LR
    subgraph Loose["Loose CAS Objects (100+ files)"]
        direction TB
        L_GOAL["GOAL-001 (Root)"]
        L_MIL["- MIL-001, MIL-002"]
        L_REQ["- REQ-010..030"]
        L_BLI["- BLI-100..150"]
        L_QA["- QA-001..090"]
    end

    subgraph Capsule["Immutable Semantic Capsule"]
        direction TB
        C_CAP["GOAL-001.capsule.zst"]
        C_MAN["- manifest.json"]
        C_TOP["- topology.json"]
        C_NAR["- narrative_summary.md"]
        C_BLOB["- blobs.bin.zst"]
    end

    Loose -->|Archival Compactor: Flatten and Compress| Capsule
    Loose -.->|Pruned from .zqk/process/| Pruned["Clean Working Tree"]
    Capsule -.->|Lightweight Tombstone Pointer| Index["Index Entry in .zqk/process/goals/.goal.index"]

3.1 Pipeline Execution Stages

  1. Cascade Boundary Traversal: The compactor starts at GOAL-xxx and recursively collects all transitive child references.
  2. Terminal Invariant Gate: Archival fails closed if any child object is non-terminal (e.g. in_progress, open, under_review).
  3. Capsule Artifact Compilation: - manifest.json: Root metadata, schema version, object inventory, and SHA-256 digests. - topology.json: Inter-object reference graph preserving causal lineage. - narrative_summary.md: Synthesized human- and LLM-readable summary detailing objectives delivered, completion dates, test pass metrics, and final commit SHAs. - blobs.bin.zst: Zstandard level 19 compressed stream of all raw YAML files.
  4. Hot Storage Pruning: Loose YAML files are deleted from .zqk/process/.

  5. CAS Index Tombstone: A tombstone record replaces the loose file entry: json { "id": "GOAL-001", "tier": "warm", "capsule_ref": ".zqk/archive/bundles/GOAL-001.capsule.zst", "capsule_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "plane": "apoptotic", "archived_at": "2026-09-19T10:15:00Z" }

  6. Apoptotic Plane Isolation: The object is assigned to dna.PlaneApoptotic. Apoptotic objects are strictly rejected from participating in active graph edges.

4. Configurable Retention Windows (Zero Hardcoding)

Per policy POL-STORAGE-RETENTION-001, no retention duration or tier boundary may be hardcoded. Environments exhibit vastly different throughput:

  • Ephemeral CI / Testbeds: High churn; hot retention measured in hours.
  • Developer Workstations: Medium churn; hot retention measured in days.
  • Regulated Enterprise Clusters: Low churn, high audit retention; cold storage retained for years.

Dynamic Configuration File: config/archive_policy.yaml

# ZQK Archival Lifecycle & Tiered Storage Configuration
version: "1.0"

tier_transitions:
  hot:
    # Time an object must remain in a terminal state before being eligible for warm compaction
    inactivity_threshold: "${ZQK_ARCHIVE_HOT_INACTIVITY:-30d}"
    terminal_grace_period: "${ZQK_ARCHIVE_HOT_GRACE:-7d}"

    # Capacity watermarks that trigger early proactive compaction sweeps
    max_volume_bytes: "${ZQK_ARCHIVE_MAX_HOT_BYTES:-524288000}"   # 500 MiB
    max_object_count: "${ZQK_ARCHIVE_MAX_HOT_OBJECTS:-5000}"
    auto_compact_on_watermark: true

  warm:
    # Duration capsules remain in local warm storage before offloading to cold vault
    retention_threshold: "${ZQK_ARCHIVE_WARM_RETENTION:-90d}"
    max_local_archive_bytes: "${ZQK_ARCHIVE_MAX_WARM_BYTES:-5368709120}" # 5 GiB
    compression:
      algorithm: "zstd"
      level: 19

  cold:
    # Duration objects remain in external cold storage before abyss purge
    retention_threshold: "${ZQK_ARCHIVE_COLD_RETENTION:-365d}" # e.g. '7y' or 'infinite'
    vault:
      backend: "${ZQK_COLD_VAULT_BACKEND:-local_vault}" # local_vault | s3 | gcs | git_lfs
      endpoint: "${ZQK_COLD_VAULT_ENDPOINT:-~/.zqk/cold-vault}"
      bucket: "${ZQK_COLD_VAULT_BUCKET:-zqk-archive-cold}"
      prefix: "vault/capsules"
      auto_offload_trigger: "quota_breach" # immediate | scheduled | quota_breach

  abyss:
    # Window before final eviction from cold vault
    purge_after_duration: "${ZQK_ABYSS_PURGE_DURATION:-never}" # 'never' preserves cold vault indefinitely
    preserve_audit_merkle_receipt: true # Guarantees non-repudiation in .zqk/streams/

5. Developer & Agent Experience (Access Model)

Archival must never result in dangling pointers or confusing "Object Not Found" errors.

5.1 Transparent Read-Through (zqk object get)

When an agent or user retrieves an archived entity:

./bin/zqk object get GOAL-001

The CLI detects the tombstone record in the index:

  1. Warm Capsule: The engine streams the capsule header, parses GOAL-001, and outputs the object with an archival badge: [WARM ARCHIVE: .zqk/archive/bundles/GOAL-001.capsule.zst | Apoptotic Plane | Immutable] id: GOAL-001 title: "Complete Core Microkernel DNA Architecture" status: archived ...

  2. Cold Storage: If the local bundle was offloaded: ⚠️ Object GOAL-001 resides in COLD STORAGE (Vault: s3://zqk-archive-cold/vault/capsules/GOAL-001.capsule.zst) Payload not present on local disk. Restore state snapshot or fetch capsule: zqk system state-restore --snapshot-file <path>

5.2 Executive Timeline Inspection (zqk object get / zqk system state-diff)

Rather than sifting through hundreds of raw YAMLs, operators inspect the synthesized narrative or compare snapshots:

./bin/zqk object get GOAL-001 --format yaml
./bin/zqk system state-diff --snapshot-file .zqk-state/system-state.csnap

Outputs the compiled executive history:

  • Objective and final deliverables
  • All satisfied requirements and graduation test suites
  • Velocity analytics (commit hashes, contributors, duration)
  • Cryptographic SHA-256 CAS verification tree

5.3 Fail-Closed Mutation Shield

Attempts to modify an archived entity:

./bin/zqk object update GOAL-001 --field title="Altered Goal"

Fails closed with immediate rejection:

Error: plane boundary violation: object GOAL-001 resides in plane 'apoptotic'.
Warm capsules are mathematically immutable. To modify this object, unarchive or restore:
  zqk system state-restore --snapshot-file <path>

5.4 Rehydration / State Restoration Workflow (zqk system state-restore)

If a project is revived or requires new active iteration:

./bin/zqk system state-restore --snapshot-file .zqk-state/system-state.csnap
  1. Decompresses the snapshot archive.
  2. Restores individual objects back into .zqk/process/<kind>/.
  3. Computes CAS hashes and restores active index pointers.
  4. Transitions plane from apoptotic back to staged or promoted.
  5. Removes the tombstone pointer.

6. Verification and Operational Hygiene

The tiered storage engine integrates into Layer 4 Storage & I/O Telemetry:

  • zqk system check reports:
  • Total Hot CAS files and volume size.
  • Number of local Warm Capsules and volume reduction ratio.
  • Number of Cold Vault references.
  • Verification that 0 apoptotic objects participate in active edges.

7. Traceability & Backlog Verification

This architecture is implemented and verified by pkg/resourcehygiene and CLI commands under zqk system resource-hygiene.

8. Hot Storage Resource Caching & Concurrency Hardening

Within the active Tier 0 (Hot) storage plane, storage.ResourceCache[T] manages lazily initialized, long-lived resources (such as ObjectStorageProvider routing instances and CAS file handles).

Concurrency Invariants & Promise Model

  1. Channel Promise Synchronization (ready chan struct{}): Rather than utilizing standard sync.Once which permanently records errors on initial failure, ResourceCache employs channel promise synchronizers. Waiting concurrent goroutines block on <-cached.ready or the caller's context (select { case <-ctx.Done(): ... case <-cached.ready: ... }).
  2. Atomic Transient Error Eviction (CompareAndDelete): If an initFunc encounters a transient timeout, context cancellation, or I/O failure, the failed entry is evicted atomically via sync.Map.CompareAndDelete(key, cached). This guarantees that subsequent or concurrent callers unwedge and retry initialization cleanly without poisoning the process-lifetime cache or deleting entries initialized by competing contenders.
  3. In-Flight Get() Guarding: Non-blocking Get(key) queries evaluate promise readiness via select { case <-cached.ready: ... default: return zero, false }. If initialization is in flight, Get() reports a cache miss (false) rather than returning zero-value uninitialized state as a false cache hit.