Disk Usage & Storage Capacity Telemetry Subsystem Architectural Guide
1. Executive Architectural Summary
The Disk Usage & Storage Capacity Telemetry Subsystem (pkg/diskusage) provides pure-Go directory traversal, size rollup aggregation, human-readable byte conversions, and storage capacity telemetry primitives for the ZQK Knowledge Kernel.
Storage monitoring is critical in high-throughput autonomous multi-agent environments. Background task schedulers, git worktree checkouts, ephemeral scratch workspaces, append-only WAL streams, and CAS object graphs continuously allocate storage across project roots. Unmonitored storage growth leads to cascading disk exhaustion, failed atomic file swaps, corrupted databases, and stalled agent swarms.
Rather than invoking OS-dependent shell binaries like du(1), df(1), or find(1)βwhich introduce shell injection attack vectors, fork-exec latency, and POSIX/Windows output divergenceβpkg/diskusage provides:
- Deterministic Pure-Go Tree Scanning: Symlink-safe, recursive aggregation with customizable depth limits, byte thresholds, and file-level inclusion.
- Standardized Binary Byte Formatting & Parsing: Bi-directional formatting and parsing supporting IEC and metric representations (KiB, MiB, GiB, TiB, PiB).
- Cross-Platform Volume Telemetry: Kernel statfs and volume-level capacity reporting distinguishing unprivileged available space from superuser-reserved blocks.
- Resilient Threshold Sentinel Evaluation: Deterministic multi-tier alert classification protecting long-running background loops against storage starvation.
2. Component Topology & Storage Telemetry
The storage subsystem operates across three coordinated functional layers:
flowchart TD
subgraph ClientLayer ["Client & Command Layer"]
CLI["zqk system disk-usage\n(cmd/zqk/system)"]
HealthSentinel["Health & Doctor Checks\n(pkg/operational)"]
AgentScheduler["Task Scheduler & Swarm\n(pkg/scheduler)"]
end
subgraph CoreEngine ["pkg/diskusage Engine"]
Scanner["Scan(root, opts)\nDirectory Tree Walker"]
Formatter["FormatBytes(bytes)\nBinary Unit Formatter"]
Parser["ParseSize(str)\nHuman Size Parser"]
end
subgraph OSMembrane ["OS Abstraction & Volume Telemetry"]
StatfsPOSIX["statfs(2) / statvfs(3)\nLinux & Darwin"]
WinAPI["GetDiskFreeSpaceExW\nWindows Win32 API"]
ContainerQuota["cgroup / Mount Quota\nUnprivileged Containers"]
end
CLI --> Scanner
CLI --> Formatter
CLI --> Parser
HealthSentinel --> Scanner
AgentScheduler --> Formatter
Scanner --> OSMembrane
HealthSentinel --> OSMembrane
Architectural Responsibilities
| Component | Module | Core Functionality |
|---|---|---|
| Directory Tree Walker | Scan |
Pure-Go recursive traversal, directory size rollup, depth bounding, symlink avoidance, and top-N sorting. |
| Binary Byte Formatter | FormatBytes |
Normalizes byte values into human-readable strings (e.g. 1.5KiB, 25GiB) with 1024-based binary scaling. |
| Human Size Parser | ParseSize |
Parses string size inputs (500M, 1.5GiB, 1024, 2TB) into exact int64 byte counts with strict error checking. |
| Statfs Volume Sentinel | Volume Inspection | Queries filesystem block availability, total capacity, and tracks consumption against alert thresholds. |
3. Directory Tree Aggregation & Pure-Go Walk Engine
The core scanner performs recursive aggregation while protecting the host system from unbounded memory usage, directory loops, and permission faults.
flowchart TD
Start(["Scan(root, opts) Entry"]) --> Clean["filepath.Clean & filepath.Abs(root)"]
Clean --> DepthClamp{"opts.MaxDepth < 0?"}
DepthClamp -- "Yes" --> SetZero["opts.MaxDepth = 0"]
DepthClamp -- "No" --> InitWalk["Initialize Walk State\n(dirSizes, fileEntries, skipped)"]
SetZero --> InitWalk
InitWalk --> WalkEntry["filepath.WalkDir(root, fn)"]
WalkEntry --> CheckErr{"walkErr != nil?"}
CheckErr -- "Yes" --> IncSkip["skipped++"]
IncSkip --> IsDirErr{"d != nil && d.IsDir()?"}
IsDirErr -- "Yes" --> SkipDir["return fs.SkipDir"]
IsDirErr -- "No" --> ContWalk["return nil"]
CheckErr -- "No" --> SymlinkCheck{"d.Type() is Symlink?"}
SymlinkCheck -- "Yes" --> SymDir{"d.IsDir()?"}
SymDir -- "Yes" --> SkipSymDir["return fs.SkipDir"]
SymDir -- "No" --> ContSymFile["return nil (Leaf Skip)"]
SymlinkCheck -- "No" --> DirCheck{"d.IsDir()?"}
DirCheck -- "Yes" --> RecordDir["dirsVisited++\ndirSizes[path] = 0"]
RecordDir --> NextItem["return nil"]
DirCheck -- "No" --> FileStat["d.Info()"]
FileStat -- "Error" --> SkipInfo["skipped++\nreturn nil"]
FileStat -- "OK" --> IncFiles["filesVisited++\nsz = info.Size()"]
IncFiles --> IncludeCheck{"opts.IncludeFiles &&\ndepth <= opts.MaxDepth?"}
IncludeCheck -- "Yes" --> RecordFile["fileEntries[path] = sz"]
IncludeCheck -- "No" --> Rollup
RecordFile --> Rollup["Rollup sz to Ancestors\nup to root"]
Rollup --> NextItem
NextItem --> Finished{"Walk Complete?"}
Finished -- "No" --> WalkEntry
Finished -- "Yes" --> FilterEntries["Filter by MaxDepth & MinBytes"]
FilterEntries --> SortEntries["Sort Descending by SizeBytes\nTie-break Alphabetical by Path"]
SortEntries --> LimitEntries{"opts.Limit > 0?"}
LimitEntries -- "Yes" --> SliceLimit["entries = entries[:opts.Limit]"]
LimitEntries -- "No" --> ReturnResult["Assemble & Return *Result"]
SliceLimit --> ReturnResult
Cumulative Size Rollup Contract
Unlike naive file listings, disk usage reporting requires that every parent directory reflects the cumulative storage of all transitive descendant files:
- When visiting leaf file
root/a/b/c/data.binwith size \(S\), \(S\) is added todirSizes["root/a/b/c"],dirSizes["root/a/b"],dirSizes["root/a"], anddirSizes["root"]. - Directory nodes themselves contribute 0 direct bytes; their reported size is the strict sum of transitive file bytes.
- Symlinks are not followed: directory symlinks are skipped with
fs.SkipDirto prevent infinite loops on recursive directory cycles, while file symlinks are treated as leaves without dereferencing targets.
4. Cross-Platform System Calls & Volume Telemetry
To calculate true storage limits and remaining headroom, ZQK interfaces directly with kernel filesystem telemetry:
sequenceDiagram
autonumber
participant App as ZQK Runtime / SRE Sentinel
participant Provider as Storage Telemetry Provider
participant Kernel as Operating System Kernel
App->>Provider: QueryCapacity(path)
alt POSIX Platform (Linux / Darwin)
Provider->>Kernel: statfs(path, &st)
Kernel-->>Provider: Statfs_t (Bsize, Blocks, Bfree, Bavail)
Provider->>Provider: TotalBytes = Blocks * Bsize
Provider->>Provider: FreeBytes = Bfree * Bsize
Provider->>Provider: AvailableBytes = Bavail * Bsize
else Windows Platform
Provider->>Kernel: GetDiskFreeSpaceExW(lpPath, &avail, &total, &free)
Kernel-->>Provider: Available, Total, TotalFree Bytes
Provider->>Provider: Map Direct uint64 Byte Values
else Container / OverlayFS Quota
Provider->>Kernel: Read /sys/fs/cgroup or Mount Quotas
Kernel-->>Provider: Bounded Quota Limits
end
Provider-->>App: StorageCapacityTelemetry{Total, Free, Available, Used}
Platform-Specific Implementations
- Linux (
sys_statfs/statvfs): - In Linux kernels,statfs(2)yields block counts and fundamental block size (f_bsizeorf_frsize). - Root filesystem reservations (f_bfree - f_bavail) are accounted for directly. - macOS / Darwin (
statfs): - BSD-derivedstatfsreturns blocks in terms of optimal transfer block size. - Standard non-root processes receive available blocks according to APFS or HFS+ container limits. - Windows (
GetDiskFreeSpaceExW): - Win32 API returns caller-accessible bytes directly, automatically accounting for per-user NTFS quotas and volume mounts. - Unprivileged Containers & OverlayFS: - In containerized execution environments (e.g. Docker, Kubernetes Pods), overlay mounts inherit the host volume's capacity unless cgroup storage limits or container volume quotas are enforced.
5. Filesystem Boundaries, Reserved Blocks & Quotas
A critical failure mode in capacity management is confusing Free Bytes (f_bfree) with Available Bytes (f_bavail).
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Total Filesystem Blocks β
βββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββ¬ββββββββββββββββ€
β Allocated / Used β Available β Reserved β
β Blocks β (Unpriv User)β (Superuser 5%)β
βββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ΄ββββββββββββββββ
β² β²
β β
AvailableBytes FreeBytes
The Superuser Margin Invariant
FreeBytes(\(B_{free} \times S_{block}\)): The total unallocated blocks on the storage volume. This includes blocks reserved strictly for the operating system superuser (root) to prevent system deadlocks.AvailableBytes(\(B_{avail} \times S_{block}\)): The unallocated blocks accessible to unprivileged user processes.- Invariant: \(B_{avail} \le B_{free}\). In standard ext4 filesystems, 5% of blocks are reserved for
root. If an agent or operator computes remaining capacity usingFreeBytes, writes will fail withENOSPC(No space left on device) while reporting positive headroom. - Subsystem Contract: All threshold calculations and capacity percentages in ZQK MUST compute against
AvailableBytes.
6. Human-Readable Formatting & Bidirectional Size Parsing
The pkg/diskusage package provides zero-dependency formatting and parsing in format.go.
Format Specification (FormatBytes)
- Values below 1024 bytes are formatted as bare integers (
0B,512B,1023B). - Scaled units use standard binary suffixes:
KiB(\(1024^1\)),MiB(\(1024^2\)),GiB(\(1024^3\)),TiB(\(1024^4\)),PiB(\(1024^5\)). - Precision rule: values \(< 10.0\) are formatted with 1 decimal place (e.g.
1.5KiB,9.9MiB); values \(\ge 10.0\) are formatted with 0 decimal places (e.g.10KiB,50GiB). - Negative values clamp safely to
0B. - Values exceeding PiB clamp to the maximum unit suffix (
PiB) to prevent index out-of-range faults.
Parse Specification (ParseSize)
- Strips leading/trailing whitespace and internal spaces.
- Accepts both binary (
KiB,MiB,GiB,TiB) and common metric/shorthand suffixes (K,KB,M,MB,G,GB,T,TB,B). - Case-insensitive (
1.5gib==1.5GIB). - Validates fractional float values (
0.5GiB\(\to 536,870,912\)). - Rejects malformed strings, negative sizes, suffix-only inputs, and trailing alphabetical noise.
7. Threshold Sentinels & Storage Health Monitoring
Autonomous continuous loops monitor storage health using multi-tier sentinels:
stateDiagram-v2
[*] --> Healthy: Available >= 10% AND >= 5 GiB
Healthy --> Warning: Available < 10% OR < 5 GiB
Warning --> Healthy: Cleanup Completed (Reclaimed > Threshold)
Warning --> Critical: Available < 5% OR < 2.5 GiB
Critical --> HaltSwarm: Emergency Circuit Breaker
Critical --> Warning: Manual / Automated Pruning Reclaims Space
Alert Classification Matrix
| Alert Tier | Threshold Condition | Subsystem Action |
|---|---|---|
OK |
Available \(\ge 10\%\) and \(\ge 5\text{ GiB}\) | Normal continuous operations; scheduled background tasks execute unimpeded. |
WARNING |
Available \(< 10\%\) or \(< 5\text{ GiB}\) | Emit diagnostic log warning; trigger background cache and temp file sweep. |
CRITICAL |
Available \(< 5\%\) or \(< 2.5\text{ GiB}\) | Halt new task dispatches; invoke circuit breaker; initiate emergency WAL compaction. |
8. Verification, Concurrency & Statement Coverage Invariant
The subsystem enforces strict operational invariants verified via automated unit and boundary suites:
- Statement Coverage Invariant (\(\ge 85.0\%\)):
- The test suite
pkg/diskusage/diskusage_coverage_elevation_cycle160_test.goachieves \(94.7\%\) statement coverage across all package routines. - Concurrency Safety:
- Multiple goroutines can execute
Scan,FormatBytes, andParseSizeconcurrently without data races or lock contention. - Resilience & Zero Swallowed Errors:
- All filesystem traversal errors increment
SkippedErrorscounters; no error is silently discarded. - Execution Command:
bash ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/diskusage