Host Load Sampling & Adaptive Concurrency Subsystem Architectural Guide (pkg/hostload)
1. Executive Summary & Design Philosophy
The Host Load Sampling & Adaptive Concurrency Subsystem (pkg/hostload) provides real-time CPU saturation telemetry, dynamic pressure classification, and adaptive concurrency throttling across the ZQK Knowledge Kernel execution plane.
In modern multi-agent systems, autonomous execution loops, parallel test runners, and DAG pipelines can easily overwhelm workstation or CI environments if concurrency is calculated solely as a static multiple of runtime.NumCPU(). If background agents, compilation jobs, or external system processes (such as antivirus or IDE indexing) spike CPU consumption, static worker budgets lead to CPU starvation, high latency, context-switching thrashing, and test timeouts.
The pkg/hostload subsystem solves this through dynamic telemetry and adaptive throttling:
- Real-Time Telemetry Sampling: Continuous, lightweight sampling of 1-minute load averages and CPU tick ratios (user, system, idle, iowait).
- EMA (Exponential Moving Average) Smoothing: Low-pass filtering (\(50\%\) weight) prevents momentary CPU spikes or GC pauses from inducing thrashing in concurrency budgets.
- Three-Tier Pressure Classification: Maps continuous telemetry into discrete operational tiers:
LevelHeadroom,LevelTight, andLevelStarve. - Self-Heat Compensation:
OverBudgetExcludingSelfprevents self-inflicted backpressure by subtracting an agent's or worker's own held core occupancy from observed host load. - Fail-Open Non-Blocking Guarantee: When platform hardware telemetry is unavailable, incomplete, or disabled, the subsystem fails open to
LevelHeadroom, ensuring execution never stalls due to missing sensors. - Zero-CGO & Cross-Platform Support: Native telemetry collection on Linux via
/procand macOS viaSysctlRaw/Mach statistics, with graceful no-CGO and OS stub fallbacks.
2. Component Topology & Architecture
The subsystem is organized into modular layers separating hardware sensing, mathematical smoothing, pressure evaluation, and client throttling decisions:
flowchart TD
subgraph ClientLayer["Execution Clients"]
Swarm["Agent Worker Swarms"]
Pipeline["Pipeline DAG Stages"]
Scheduler["Job Scheduler"]
Validator["System Check Runner"]
end
subgraph HostloadSubsystem["pkg/hostload Subsystem"]
API["Public Entrypoints<br/>CurrentLevel(), Scale(), OverBudget()"]
Excl["Self-Occupancy Accounting<br/>excludingSelf(selfUnits)"]
Snap["Snapshot & Level Classifier<br/>Snapshot.Level()"]
Sampler["OS Sampler & EMA Filter<br/>osSampler.snapshotFromSample()"]
end
subgraph PlatformTelemetry["Platform Hardware Sensors"]
LinuxSensors["Linux Telemetry<br/>/proc/stat, /proc/loadavg"]
DarwinSensors["macOS Telemetry<br/>vm.loadavg, Mach ticks"]
StubSensors["Fallback / Stub Sensors<br/>Fail-Open Headroom"]
end
Swarm --> API
Pipeline --> API
Scheduler --> API
Validator --> API
API --> Excl
Excl --> Snap
Snap --> Sampler
Sampler --> LinuxSensors
Sampler --> DarwinSensors
Sampler --> StubSensors
Component Roles
-
SamplerInterface: Defines the hardware sampling contract:go type Sampler interface { Snapshot() Snapshot } -
osSampler: Process-wide singleton maintaining historical tick counters and the moving average accumulator (idleEMA). Thread-safe via internal mutex protection. -
Snapshot: Immutable capture of point-in-time system state (NCPU,Load1,IdleFraction,SampledAt). -
LevelClassifier: Evaluates CPU ticks and load average against strict thresholds, electing the more conservative (higher pressure) rating.
3. Pressure Classification & Threshold Semantics
Host pressure is classified into three discrete operational states:
stateDiagram-v2
[*] --> LevelHeadroom: Initialized / Normal Operation
LevelHeadroom --> LevelTight: Idle < 25% OR (Load1 / NCPU) >= 1.0
LevelHeadroom --> LevelStarve: Idle < 8% OR (Load1 / NCPU) >= 1.6
LevelTight --> LevelHeadroom: Idle >= 25% AND (Load1 / NCPU) < 1.0
LevelTight --> LevelStarve: Idle < 8% OR (Load1 / NCPU) >= 1.6
LevelStarve --> LevelTight: (Idle >= 8% AND Idle < 25%) AND (Load1 / NCPU) < 1.6
LevelStarve --> LevelHeadroom: Idle >= 25% AND (Load1 / NCPU) < 1.0
Threshold Constants
| Metric Boundary | LevelTight Threshold | LevelStarve Threshold | Behavior |
|---|---|---|---|
Idle Fraction (IdleFraction) |
\(< 0.25\) (\(25\%\) idle) | \(< 0.08\) (\(8\%\) idle) | Leaves \(\approx 1\) core of slack on a 10-core workstation. |
Normalized Load (Load1 / NCPU) |
\(\ge 1.0\) (\(100\%\) capacity) | \(\ge 1.6\) (\(160\%\) capacity) | Host run-queue is substantially overloaded. |
Evaluation Rules:
- Worst-Case Wins: If
IdleFractionindicatesLevelTightbutLoad1 / NCPUindicatesLevelStarve, the snapshot evaluates toLevelStarve. - Missing Sensor Fallback: Negative values for
IdleFractionorLoad1indicate missing data. If both are negative, the subsystem yieldsLevelHeadroom. - Zero Core Protection: If
NCPU <= 0, load average calculations are ignored, and pressure relies exclusively on idle tick fraction.
4. Exponential Moving Average (EMA) Smoothing
Raw CPU tick counters can fluctuate drastically from millisecond to millisecond due to thread scheduling, garbage collection, and short command executions. To prevent high-frequency oscillations in concurrency limits, pkg/hostload applies Exponential Moving Average smoothing on consecutive tick samples.
Mathematical Formulation
Given successive tick samples with delta total ticks \(\Delta \text{Tot} = \text{Tot}_t - \text{Tot}_{t-1}\) and delta idle ticks \(\Delta \text{Idle} = \text{Idle}_t - \text{Idle}_{t-1}\):
When an existing moving average exists (\(haveEMA = true\)):
When uninitialized, the initial sample seeds the baseline: \(\text{EMA}_0 = \text{Sample}_0\).
sequenceDiagram
participant Worker as Execution Engine
participant Sampler as osSampler
participant OS as OS Kernel Telemetry
Worker->>Sampler: Snapshot()
Sampler->>OS: readCPUTicks()
OS-->>Sampler: idle, total, tickOK
Sampler->>OS: readLoadAvg()
OS-->>Sampler: load1, loadOK
alt First Sample (prevOK == false)
Sampler->>Sampler: Record prevIdle, prevTot, set prevOK=true
Sampler-->>Worker: Snapshot(IdleFraction = -1.0) [Fail-Open]
else Successive Sample (total > prevTot)
Sampler->>Sampler: Calculate Sample = dIdle / dTot
Sampler->>Sampler: EMA = 0.5 * Sample + 0.5 * prevEMA
Sampler-->>Worker: Snapshot(IdleFraction = EMA)
end
5. Adaptive Concurrency Throttling & Self-Exclusion
Dynamic Scaling (Scale)
The Scale(n int, level Level) int function dynamically adjusts a desired concurrency count $n$:
LevelHeadroom: Returns $n$ unchanged.LevelTight: Returns \(\lfloor n / 2 \rfloor\) (minimum $1$).LevelStarve: Returns $1$.- Invariant: When \(n \le 1\),
Scalealways returns $n$, ensuring forward progress is never completely halted.
Budget Checking (OverBudget vs OverBudgetExcludingSelf)
// OverBudget checks raw host pressure (including current process load)
func OverBudget(inUse, capacity int) bool
// OverBudgetExcludingSelf subtracts held self-occupancy from host pressure
func OverBudgetExcludingSelf(want, capacity int) bool
Self-Occupancy Accounting (excludingSelf)
When a high-concurrency job (e.g. parallel AST linting or compilation) consumes substantial CPU, live host sensors will naturally observe high load and low idle time. If the subsystem did not account for this, the worker would detect its own resource usage as foreign pressure and erroneously throttle or kill its own tasks.
excludingSelf(selfUnits) adjusts the snapshot before level evaluation:
- Calculates self-fraction: \(\text{frac} = \frac{\text{selfUnits}}{\text{NCPU}}\)
- Adds \(\text{frac}\) back to
IdleFraction(clamped to a maximum of $1.0$). - Subtracts \(\text{selfUnits}\) from
Load1(clamped to a minimum of $0.0$).
This ensures that only foreign host contention (external daemons, other developer tasks) causes the worker to yield slots.
6. Cross-Platform Hardware Telemetry
The subsystem provides cross-platform hardware telemetry implementations using Go build tags:
| Platform | Build Tags | Telemetry Source | Implementation Notes |
|---|---|---|---|
| Linux | //go:build linux |
/proc/loadavg/proc/stat |
Direct zero-allocation parsing with fileutil.ReadFile and bytes.Fields. Extracts user, nice, system, idle, and iowait. |
| macOS (Darwin) with CGO | //go:build darwin && cgo |
vm.loadavggopsutil/v3/cpu |
unix.SysctlRaw("vm.loadavg") extracts load average. Mach host_processor_info via gopsutil extracts tick millisecond counters. |
| macOS (Darwin) without CGO | //go:build darwin && !cgo |
vm.loadavg |
Load average via Sysctl remains active; CPU tick sampling returns ok = false (fails open to Headroom unless load average indicates pressure). |
| Windows / Other Stubs | //go:build !darwin && !linux |
Stub implementations | Returns 0, false for both sensors; transparently fails open to LevelHeadroom. |
Test Scoping & CI Isolation
To prevent test flakes on noisy developer workstations or multi-tenant CI runners, underGoTest() inspects test flags (test.v). When executing under go test, live hardware polling is bypassed, returning a synthetic Snapshot{IdleFraction: 1.0, Load1: 0.0} unless explicit mock samplers (setSamplerForTest) are injected.
Administrative Opt-Out
Telemetry sampling can be completely disabled process-wide by setting the environment variable:
export ZQK_HOSTLOAD_DISABLE=1
When disabled, Disabled() returns true, and all calls to CurrentLevel() and CurrentLevelExcluding() unconditionally return LevelHeadroom.
7. Verification, Testing & Statement Coverage Invariant
The subsystem enforces a strict statement test coverage invariant (\(\ge 85.0\%\)).
Coverage Verification Command
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/hostload
Key Verification Scenarios
- Dynamic Scaling Limits: Exhaustive verification of
Scale()under negative, zero, single-core, and multi-core bounds across all levels. - Matrix Threshold Tests: Verifying all permutations of
IdleFractionandLoad1transitions, boundary inclusiveness ($0.25, 0.08, 1.0, 1.6$), and fail-open fallbacks. - EMA Blending Proof: Verifying multi-sample decay, initialization seeding, stagnant tick handling, and sensor error isolation.
- Concurrent Thread Safety: Multi-goroutine concurrent execution against
osSamplerandactiveSamplerunder thread contention. - Self-Heat Exemption Proof: Verifying that worker slots are preserved under self-inflicted load but correctly throttled under foreign load spikes.