Disk Usage & Storage Capacity Telemetry Subsystem Architectural Guide

1. Executive Architectural Summary

The Disk Usage & Storage Capacity Telemetry Subsystem (pkg/diskusage) provides pure-Go directory traversal, size rollup aggregation, human-readable byte conversions, and storage capacity telemetry primitives for the ZQK Knowledge Kernel.

Storage monitoring is critical in high-throughput autonomous multi-agent environments. Background task schedulers, git worktree checkouts, ephemeral scratch workspaces, append-only WAL streams, and CAS object graphs continuously allocate storage across project roots. Unmonitored storage growth leads to cascading disk exhaustion, failed atomic file swaps, corrupted databases, and stalled agent swarms.

Rather than invoking OS-dependent shell binaries like du(1), df(1), or find(1)β€”which introduce shell injection attack vectors, fork-exec latency, and POSIX/Windows output divergenceβ€”pkg/diskusage provides:

  1. Deterministic Pure-Go Tree Scanning: Symlink-safe, recursive aggregation with customizable depth limits, byte thresholds, and file-level inclusion.
  2. Standardized Binary Byte Formatting & Parsing: Bi-directional formatting and parsing supporting IEC and metric representations (KiB, MiB, GiB, TiB, PiB).
  3. Cross-Platform Volume Telemetry: Kernel statfs and volume-level capacity reporting distinguishing unprivileged available space from superuser-reserved blocks.
  4. Resilient Threshold Sentinel Evaluation: Deterministic multi-tier alert classification protecting long-running background loops against storage starvation.

2. Component Topology & Storage Telemetry

The storage subsystem operates across three coordinated functional layers:

flowchart TD
    subgraph ClientLayer ["Client & Command Layer"]
        CLI["zqk system disk-usage\n(cmd/zqk/system)"]
        HealthSentinel["Health & Doctor Checks\n(pkg/operational)"]
        AgentScheduler["Task Scheduler & Swarm\n(pkg/scheduler)"]
    end

    subgraph CoreEngine ["pkg/diskusage Engine"]
        Scanner["Scan(root, opts)\nDirectory Tree Walker"]
        Formatter["FormatBytes(bytes)\nBinary Unit Formatter"]
        Parser["ParseSize(str)\nHuman Size Parser"]
    end

    subgraph OSMembrane ["OS Abstraction & Volume Telemetry"]
        StatfsPOSIX["statfs(2) / statvfs(3)\nLinux & Darwin"]
        WinAPI["GetDiskFreeSpaceExW\nWindows Win32 API"]
        ContainerQuota["cgroup / Mount Quota\nUnprivileged Containers"]
    end

    CLI --> Scanner
    CLI --> Formatter
    CLI --> Parser
    HealthSentinel --> Scanner
    AgentScheduler --> Formatter

    Scanner --> OSMembrane
    HealthSentinel --> OSMembrane

Architectural Responsibilities

Component Module Core Functionality
Directory Tree Walker Scan Pure-Go recursive traversal, directory size rollup, depth bounding, symlink avoidance, and top-N sorting.
Binary Byte Formatter FormatBytes Normalizes byte values into human-readable strings (e.g. 1.5KiB, 25GiB) with 1024-based binary scaling.
Human Size Parser ParseSize Parses string size inputs (500M, 1.5GiB, 1024, 2TB) into exact int64 byte counts with strict error checking.
Statfs Volume Sentinel Volume Inspection Queries filesystem block availability, total capacity, and tracks consumption against alert thresholds.

3. Directory Tree Aggregation & Pure-Go Walk Engine

The core scanner performs recursive aggregation while protecting the host system from unbounded memory usage, directory loops, and permission faults.

flowchart TD
    Start(["Scan(root, opts) Entry"]) --> Clean["filepath.Clean & filepath.Abs(root)"]
    Clean --> DepthClamp{"opts.MaxDepth < 0?"}
    DepthClamp -- "Yes" --> SetZero["opts.MaxDepth = 0"]
    DepthClamp -- "No" --> InitWalk["Initialize Walk State\n(dirSizes, fileEntries, skipped)"]
    SetZero --> InitWalk

    InitWalk --> WalkEntry["filepath.WalkDir(root, fn)"]
    WalkEntry --> CheckErr{"walkErr != nil?"}

    CheckErr -- "Yes" --> IncSkip["skipped++"]
    IncSkip --> IsDirErr{"d != nil && d.IsDir()?"}
    IsDirErr -- "Yes" --> SkipDir["return fs.SkipDir"]
    IsDirErr -- "No" --> ContWalk["return nil"]

    CheckErr -- "No" --> SymlinkCheck{"d.Type() is Symlink?"}
    SymlinkCheck -- "Yes" --> SymDir{"d.IsDir()?"}
    SymDir -- "Yes" --> SkipSymDir["return fs.SkipDir"]
    SymDir -- "No" --> ContSymFile["return nil (Leaf Skip)"]

    SymlinkCheck -- "No" --> DirCheck{"d.IsDir()?"}
    DirCheck -- "Yes" --> RecordDir["dirsVisited++\ndirSizes[path] = 0"]
    RecordDir --> NextItem["return nil"]

    DirCheck -- "No" --> FileStat["d.Info()"]
    FileStat -- "Error" --> SkipInfo["skipped++\nreturn nil"]
    FileStat -- "OK" --> IncFiles["filesVisited++\nsz = info.Size()"]

    IncFiles --> IncludeCheck{"opts.IncludeFiles &&\ndepth <= opts.MaxDepth?"}
    IncludeCheck -- "Yes" --> RecordFile["fileEntries[path] = sz"]
    IncludeCheck -- "No" --> Rollup
    RecordFile --> Rollup["Rollup sz to Ancestors\nup to root"]
    Rollup --> NextItem

    NextItem --> Finished{"Walk Complete?"}
    Finished -- "No" --> WalkEntry
    Finished -- "Yes" --> FilterEntries["Filter by MaxDepth & MinBytes"]
    FilterEntries --> SortEntries["Sort Descending by SizeBytes\nTie-break Alphabetical by Path"]
    SortEntries --> LimitEntries{"opts.Limit > 0?"}
    LimitEntries -- "Yes" --> SliceLimit["entries = entries[:opts.Limit]"]
    LimitEntries -- "No" --> ReturnResult["Assemble & Return *Result"]
    SliceLimit --> ReturnResult

Cumulative Size Rollup Contract

Unlike naive file listings, disk usage reporting requires that every parent directory reflects the cumulative storage of all transitive descendant files:

  1. When visiting leaf file root/a/b/c/data.bin with size \(S\), \(S\) is added to dirSizes["root/a/b/c"], dirSizes["root/a/b"], dirSizes["root/a"], and dirSizes["root"].
  2. Directory nodes themselves contribute 0 direct bytes; their reported size is the strict sum of transitive file bytes.
  3. Symlinks are not followed: directory symlinks are skipped with fs.SkipDir to prevent infinite loops on recursive directory cycles, while file symlinks are treated as leaves without dereferencing targets.

4. Cross-Platform System Calls & Volume Telemetry

To calculate true storage limits and remaining headroom, ZQK interfaces directly with kernel filesystem telemetry:

sequenceDiagram
    autonumber
    participant App as ZQK Runtime / SRE Sentinel
    participant Provider as Storage Telemetry Provider
    participant Kernel as Operating System Kernel

    App->>Provider: QueryCapacity(path)
    alt POSIX Platform (Linux / Darwin)
        Provider->>Kernel: statfs(path, &st)
        Kernel-->>Provider: Statfs_t (Bsize, Blocks, Bfree, Bavail)
        Provider->>Provider: TotalBytes = Blocks * Bsize
        Provider->>Provider: FreeBytes = Bfree * Bsize
        Provider->>Provider: AvailableBytes = Bavail * Bsize
    else Windows Platform
        Provider->>Kernel: GetDiskFreeSpaceExW(lpPath, &avail, &total, &free)
        Kernel-->>Provider: Available, Total, TotalFree Bytes
        Provider->>Provider: Map Direct uint64 Byte Values
    else Container / OverlayFS Quota
        Provider->>Kernel: Read /sys/fs/cgroup or Mount Quotas
        Kernel-->>Provider: Bounded Quota Limits
    end
    Provider-->>App: StorageCapacityTelemetry{Total, Free, Available, Used}

Platform-Specific Implementations

  1. Linux (sys_statfs / statvfs): - In Linux kernels, statfs(2) yields block counts and fundamental block size (f_bsize or f_frsize). - Root filesystem reservations (f_bfree - f_bavail) are accounted for directly.
  2. macOS / Darwin (statfs): - BSD-derived statfs returns blocks in terms of optimal transfer block size. - Standard non-root processes receive available blocks according to APFS or HFS+ container limits.
  3. Windows (GetDiskFreeSpaceExW): - Win32 API returns caller-accessible bytes directly, automatically accounting for per-user NTFS quotas and volume mounts.
  4. Unprivileged Containers & OverlayFS: - In containerized execution environments (e.g. Docker, Kubernetes Pods), overlay mounts inherit the host volume's capacity unless cgroup storage limits or container volume quotas are enforced.

5. Filesystem Boundaries, Reserved Blocks & Quotas

A critical failure mode in capacity management is confusing Free Bytes (f_bfree) with Available Bytes (f_bavail).

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        Total Filesystem Blocks                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚            Allocated / Used             β”‚  Available   β”‚   Reserved    β”‚
β”‚                 Blocks                  β”‚ (Unpriv User)β”‚ (Superuser 5%)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                          β–²              β–²
                                          β”‚              β”‚
                               AvailableBytes        FreeBytes

The Superuser Margin Invariant

  • FreeBytes (\(B_{free} \times S_{block}\)): The total unallocated blocks on the storage volume. This includes blocks reserved strictly for the operating system superuser (root) to prevent system deadlocks.
  • AvailableBytes (\(B_{avail} \times S_{block}\)): The unallocated blocks accessible to unprivileged user processes.
  • Invariant: \(B_{avail} \le B_{free}\). In standard ext4 filesystems, 5% of blocks are reserved for root. If an agent or operator computes remaining capacity using FreeBytes, writes will fail with ENOSPC (No space left on device) while reporting positive headroom.
  • Subsystem Contract: All threshold calculations and capacity percentages in ZQK MUST compute against AvailableBytes.

6. Human-Readable Formatting & Bidirectional Size Parsing

The pkg/diskusage package provides zero-dependency formatting and parsing in format.go.

Format Specification (FormatBytes)

  • Values below 1024 bytes are formatted as bare integers (0B, 512B, 1023B).
  • Scaled units use standard binary suffixes: KiB (\(1024^1\)), MiB (\(1024^2\)), GiB (\(1024^3\)), TiB (\(1024^4\)), PiB (\(1024^5\)).
  • Precision rule: values \(< 10.0\) are formatted with 1 decimal place (e.g. 1.5KiB, 9.9MiB); values \(\ge 10.0\) are formatted with 0 decimal places (e.g. 10KiB, 50GiB).
  • Negative values clamp safely to 0B.
  • Values exceeding PiB clamp to the maximum unit suffix (PiB) to prevent index out-of-range faults.

Parse Specification (ParseSize)

  • Strips leading/trailing whitespace and internal spaces.
  • Accepts both binary (KiB, MiB, GiB, TiB) and common metric/shorthand suffixes (K, KB, M, MB, G, GB, T, TB, B).
  • Case-insensitive (1.5gib == 1.5GIB).
  • Validates fractional float values (0.5GiB \(\to 536,870,912\)).
  • Rejects malformed strings, negative sizes, suffix-only inputs, and trailing alphabetical noise.

7. Threshold Sentinels & Storage Health Monitoring

Autonomous continuous loops monitor storage health using multi-tier sentinels:

stateDiagram-v2
    [*] --> Healthy: Available >= 10% AND >= 5 GiB
    Healthy --> Warning: Available < 10% OR < 5 GiB
    Warning --> Healthy: Cleanup Completed (Reclaimed > Threshold)
    Warning --> Critical: Available < 5% OR < 2.5 GiB
    Critical --> HaltSwarm: Emergency Circuit Breaker
    Critical --> Warning: Manual / Automated Pruning Reclaims Space

Alert Classification Matrix

Alert Tier Threshold Condition Subsystem Action
OK Available \(\ge 10\%\) and \(\ge 5\text{ GiB}\) Normal continuous operations; scheduled background tasks execute unimpeded.
WARNING Available \(< 10\%\) or \(< 5\text{ GiB}\) Emit diagnostic log warning; trigger background cache and temp file sweep.
CRITICAL Available \(< 5\%\) or \(< 2.5\text{ GiB}\) Halt new task dispatches; invoke circuit breaker; initiate emergency WAL compaction.

8. Verification, Concurrency & Statement Coverage Invariant

The subsystem enforces strict operational invariants verified via automated unit and boundary suites:

  1. Statement Coverage Invariant (\(\ge 85.0\%\)): - The test suite pkg/diskusage/diskusage_coverage_elevation_cycle160_test.go achieves \(94.7\%\) statement coverage across all package routines.
  2. Concurrency Safety: - Multiple goroutines can execute Scan, FormatBytes, and ParseSize concurrently without data races or lock contention.
  3. Resilience & Zero Swallowed Errors: - All filesystem traversal errors increment SkippedErrors counters; no error is silently discarded.
  4. Execution Command: bash ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/diskusage