I/O Resource Hygiene, File Descriptor Leak Detection & Storage Telemetry Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The I/O Resource Hygiene, File Descriptor Leak Detection & Storage Telemetry Subsystem (pkg/resourcehygiene) provides comprehensive runtime resource protection, storage telemetry, and autonomous garbage collection for the ZQK Knowledge Kernel.
As a high-performance local developer microkernel orchestrating concurrent AI agent swarms, background daemons, scheduled jobs, and ephemeral worktrees, ZQK core operations generate high-velocity filesystem artifacts. These include content-addressed blobs, write-ahead logs (WAL), lock files, temporary staging buffers, execution outputs, and telemetry chunks.
Without automated resource hygiene and continuous monitoring, sustained autonomous execution faces severe failure modes:
- File Descriptor Exhaustion: Open handle accumulation across subprocess pipelines and background workers exhausts the operating system
RLIMIT_NOFILE, causing cascading system-wide I/O failures. - Ephemeral Disk Bloat: Stale temporary staging files (
.tmp-*,.yaml.tmp,pclimits-*) and unpruned execution logs consume disk capacity and inflate git/cas directory tree scans. - Orphaned Process Drift: Abandoned background worker processes reparented to
init(PID 1) after an unexpected termination consume CPU and retain file descriptor locks. - Stale Lock Contention: Abandoned
.lockfiles left by crashed processes prevent downstream worker dispatch.
The pkg/resourcehygiene package addresses these concerns through deterministic point-in-time telemetry inspection (InspectIOResources), non-blocking advisory flock probing, cross-platform file descriptor monitoring (GetProcessFDUsage), and coordinated multi-phase resource reclamation (ExecuteHygiene).
2. Component Topology & Subsystem Architecture
The subsystem is partitioned into four cooperating functional domains:
flowchart TD
subgraph ResourceHygieneSubsystem["pkg/resourcehygiene Subsystem"]
FDUsage["FD Usage Monitor\n(GetProcessFDUsage)"]
Telemetry["Storage Telemetry Collector\n(InspectIOResources)"]
ReaperEngine["Resource Reaper Engine\n(ExecuteHygiene)"]
Types["Configuration & Reporting\n(HygieneOptions / Report)"]
end
subgraph KernelConsumers["Kernel Callers & Autonomous Engines"]
AmbientDaemon["Ambient Compute Daemon"]
Scheduler["Scheduler Watchdog"]
CLI["CLI Commands (zqk system check)"]
Swarm["Agent Swarm Orchestrator"]
end
subgraph FileSystem["Local Filesystem Topology (.zqk)"]
LocksDir[".zqk/locks & daemon_locks"]
CacheDir[".zqk/cache (.tmp-*)"]
LogsDir[".zqk/logs & scheduler logs"]
ScorecardsDir[".zqk/scorecards"]
MetricsDir[".zqk/metrics"]
ProcFS["/proc/self/fd or /dev/fd"]
end
KernelConsumers -->|Configure & Execute| ReaperEngine
KernelConsumers -->|Poll Vitality| Telemetry
Telemetry -->|Query NOFILE & Handles| FDUsage
FDUsage -->|Scan| ProcFS
Telemetry -->|Walk| FileSystem
ReaperEngine -->|Purge Stale| LocksDir
ReaperEngine -->|Sweep Abandoned| CacheDir
ReaperEngine -->|Prune/Truncate| LogsDir
ReaperEngine -->|Reap Orphaned| ScorecardsDir
ReaperEngine -->|Retire Aged| MetricsDir
ReaperEngine -->|Synthesize| Types
Component Roles
- Resource Reaper Engine (
reaper.go): Drives atomic sweeps across lock files, temporary files, logs, orphaned scorecards, aged metric chunks, and reparented child processes. - Storage Telemetry Collector (
telemetry.go): Generates aggregated point-in-time snapshots of open file descriptors, directory tree size, storage volume capacity, and hygiene anomaly counts. - File Descriptor Usage Monitor (
fd_usage_unix.go/fd_usage_windows.go): Provides low-overhead inspection of open file descriptors and softNOFILElimits across Unix and Windows platforms. - Configuration & Reporting Contracts (
types.go): DeclaresHygieneOptions,DefaultHygieneOptions,IOResourceTelemetry, andHygieneExecutionReport.
3. Resource Sweep & Leak Detection Lifecycle
The execution lifecycle of a hygiene sweep is orchestrated deterministically. Sweeps support both dry-run mode (reporting without mutating) and live reclamation.
sequenceDiagram
autonumber
participant Caller as Orchestrator / SRE
participant Reaper as ExecuteHygiene
participant FD as GetProcessFDUsage
participant FS as Local Filesystem (.zqk)
participant PS as Process Sentinel
Caller->>Reaper: ExecuteHygiene(projectRoot, opts)
alt ReapTemp Enabled
Reaper->>FS: ReapOrphanedTempFiles(threshold, dryRun)
FS-->>Reaper: Count, Bytes Reclaimed, Reaped Paths
end
alt ReapLocks Enabled
Reaper->>FS: ReapStaleLocks(threshold, dryRun)
Note over Reaper,FS: Non-blocking flock probe ensures no active holder
FS-->>Reaper: Count, Reaped Locks
end
alt EnforceRetention Enabled
Reaper->>FS: EnforceLogRetention(maxAge, maxSize, dryRun)
Note over Reaper,FS: Prunes aged logs and truncates oversized streams
FS-->>Reaper: Pruned Count, Bytes Freed, Truncated Paths
end
alt ReapScorecards Enabled
Reaper->>FS: ReapOrphanedScorecards(dryRun)
Note over Reaper,FS: Unlinks scorecards whose source file was deleted
FS-->>Reaper: Scorecard Count, Bytes Reclaimed
end
alt ReapMetrics Enabled
Reaper->>FS: ReapAgedMetrics(metricsMaxAge, dryRun)
Note over Reaper,FS: Unlinks chunk and metric files older than threshold
FS-->>Reaper: Metrics Count, Bytes Reclaimed
end
alt ReapProcesses Enabled
Reaper->>PS: ReapOrphanedProcesses(dryRun)
Note over Reaper,PS: Identifies PPID=1 orphaned project processes; sends SIGTERM/SIGKILL
PS-->>Reaper: Process Count, Terminated PIDs
end
Reaper-->>Caller: HygieneExecutionReport{Timestamp, DryRun, Counts, Bytes, Errors}
4. Platform-Specific File Descriptor Inspection
Operating systems manage file descriptors and handle tables using divergent kernel interfaces. The subsystem abstracts these differences behind a unified contract:
func GetProcessFDUsage() (int, int, error)
flowchart TD
Entry(["GetProcessFDUsage()"]) --> OSCheck{"runtime.GOOS"}
OSCheck -- "linux" --> LinuxFD["Read /proc/self/fd"]
OSCheck -- "darwin / bsd" --> UnixFD["Read /dev/fd"]
OSCheck -- "windows" --> WinStub["Return (-1, -1, nil)\n(POSIX FD not exposed)"]
LinuxFD --> OpenDir["fileutil.Open(fdDir)"]
UnixFD --> OpenDir
OpenDir -- "Error" --> RetErr["Return (-1, -1, err)"]
OpenDir -- "Success" --> ReadNames["f.Readdirnames(-1)"]
ReadNames -- "Error" --> RetErr
ReadNames -- "Success" --> CountFDs["openCount = len(names) - 1\n(Exclude descriptor for dir itself)"]
CountFDs --> GetRLimit["syscall.Getrlimit(RLIMIT_NOFILE, &rLimit)"]
GetRLimit -- "Success" --> SetMax["maxLimit = int(rLimit.Cur)"]
GetRLimit -- "Error" --> MaxFallback["maxLimit = -1"]
SetMax --> FinalRet["Return (openCount, maxLimit, nil)"]
MaxFallback --> FinalRet
Invariants:
- Descriptor Self-Correction: When scanning
/dev/fdor/proc/self/fd, the directory descriptor opened byfileutil.Openis subtracted fromlen(names)to ensure telemetry accurately reflects baseline application consumption. - Windows Handle Isolation: On Windows, file handles are non-numeric kernel objects;
GetProcessFDUsagereturns(-1, -1, nil)safely without throwing runtime errors or panicking.
5. Orphaned Temporary Files & Stale Lock Safety
Temporary Artifact Detection Pattern
Temporary files created during atomic writes or ephemeral operations follow strict naming conventions verified by IsTempFileName:
- Suffix
.tmpor.yaml.tmp - Infix
.tmp-or.tmp. - Prefix
.tmp-orpclimits-
Any temporary artifact with modification time strictly older than DefaultTempOrphanAge (30 minutes by default) is categorized as orphaned and queued for reclamation.
Advisory Flock Probing Protocol
A critical flaw in naive lock file deletion is unlinking a file that is actively held by a running process. In POSIX filesystems, unlinking a locked file causes inode detachment: subsequent callers open a newly created inode, breaking mutual exclusion while the original process is still running.
ReapStaleLocks solves this through a non-blocking lock probe:
- Verify modification time is older than the configured lock stale age (default 15 minutes).
- Skip any lock inside
.zqk/state/daemon_locks/(permanent singleton daemon reservations). - Open the lock file via
fileutil.OpenFile(path, fileutil.O_RDWR, paths.FilePerm600). - Attempt non-blocking exclusive flock:
syscallutil.FileFlock(file, syscall.LOCK_EX|syscall.LOCK_NB). - If the lock probe fails (e.g.
EAGAINorEWOULDBLOCK), the lock is actively held by a live process and must never be touched. - If the lock is acquired, release it immediately (
LOCK_UN), close the handle, and remove the file.
stateDiagram-v2
[*] --> ModTimeCheck
ModTimeCheck --> RetainActive: ModTime >= cutoff (Fresh)
ModTimeCheck --> DaemonCheck: ModTime < cutoff (Candidate)
DaemonCheck --> RetainDaemon: In /state/daemon_locks/
DaemonCheck --> NonBlockingProbe: User / Worktree Lock
NonBlockingProbe --> OpenHandle: fileutil.OpenFile
OpenHandle --> RetainUnreadable: Open Error (Permission/Missing)
OpenHandle --> FlockAttempt: syscallutil.FileFlock(LOCK_EX | LOCK_NB)
FlockAttempt --> RetainHeld: Lock Contended (Live Process)
FlockAttempt --> ReleaseAndUnlink: Lock Acquired
ReleaseAndUnlink --> FileRemoved: fileutil.Remove
RetainActive --> [*]
RetainDaemon --> [*]
RetainUnreadable --> [*]
RetainHeld --> [*]
FileRemoved --> [*]
6. Storage Telemetry, Sizing & Volume Breakdown
The InspectIOResources API aggregates operational storage telemetry across the active project root:
| Telemetry Metric | Field Name | Description |
|---|---|---|
| Open Descriptors | open_file_descriptors |
Current process open descriptor count. |
| Soft Limit | max_file_descriptors |
Configured RLIMIT_NOFILE soft limit. |
| Kernel Files | total_zqk_files |
Total count of files under .zqk/. |
| Kernel Bytes | total_zqk_bytes |
Total byte footprint under .zqk/. |
| Volume Capacity | volume_total_bytes |
Underlying storage volume total capacity. |
| Volume Available | volume_available_bytes |
Underlying storage volume free capacity. |
| Folder Usage Breakdown | zqk_children |
Subdirectory breakdown (e.g., logs, cas, blobs, metrics). |
| Stale Lock Anomaly | stale_locks_count |
Number of unheld abandoned locks. |
| Orphaned Temp Anomaly | orphaned_temp_count |
Number of abandoned temporary staging files. |
| Orphaned Processes | orphaned_process_count |
Count of reparented child processes. |
7. Log Retention, Scorecard Reaping & Aged Metrics Maintenance
Log Retention & Truncation
Log maintenance enforces both maximum age and maximum size bounds under .zqk/logs, .zqk/mcp/logs, and .zqk/scheduler:
- Age Pruning: Files older than
opts.LogMaxAge(default 48 hours) are deleted. - Proportional Size Truncation: Files exceeding
opts.LogMaxSize(default 10MB) are truncated to retain the most recent line records: - If lines > 10,000: Retains trailing 10,000 lines.
- If lines > 100: Retains trailing 50% of lines.
- If lines > 1: Retains the final line.
- Protected File Exemptions: State files such as
issues.json,*-summary.json,.pid,.keepalive, and directories namedstate,locks,triggers, orhourglassare never deleted by log retention sweeps.
Orphaned Scorecards
Scorecard files located under .zqk/scorecards/ represent code quality and test coverage results mapped to source code files. When source files are moved, renamed, or deleted, scorecards become orphaned:
- Decodes the source file relative path from the filename (e.g.,
pkg__component.go.jsonmaps topkg/component.go). - Verifies existence of the source file. If missing, falls back to inspecting the embedded
file_pathJSON attribute. - Unlinks the orphaned scorecard if the source target no longer exists on disk.
Aged Metrics
TSDB chunks and operational metric logs stored under .zqk/metrics/ (command_metrics, object_volume, filesystem_snapshot, stream_volume) older than opts.MetricsMaxAge (default 14 days) are automatically pruned.
8. Concurrency, Goroutine Labeling & Thread-Safety Invariants
All file inspection and sweep routines in pkg/resourcehygiene are re-entrant and concurrency-safe:
- Zero Mutex Contention on Inspection:
InspectIOResourcesruns without global locks, relying on filesystem snapshot isolation. - Goroutine Labeling Compliance: Background cleanup loops and async sweepers MUST be dispatched using
goroutinelabels.NewGoroutineto ensure proper categorization, panic containment, and lifecycle tracing:
builder := goroutinelabels.NewGoroutine("resourcehygiene_reaper", "automated periodic storage hygiene sweep")
builder.StartSimple(func() {
report, err := resourcehygiene.ExecuteHygiene(projectRoot, opts)
if err != nil {
// handle or log error
}
})
- Filesystem Abstraction Invariant: All filesystem mutations and lookups strictly utilize
pkg/utils/fileutil, ensuring safe cross-platform descriptor management and preventing file descriptor leaks in test and production environments.
9. Verification & Test Coverage Invariant
Statement test coverage for pkg/resourcehygiene is enforced by CI check-valves and must strictly satisfy the static floor invariant:
Targeted verification command:
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/resourcehygiene