Test Package Concurrency & Throttling Subsystem Guide
1. Executive Architectural Summary
The Test Package Concurrency Limits & Throttling Subsystem (pkg/testpackageconcurrency) governs cross-process parallel test execution across the ZQK Knowledge Kernel execution plane under POL-DEFAULT-b17f54c251eb3c4d. During autonomous continuous loops, test verification swarms, and background batch runs, unconstrained parallel invocation of compute-heavy or disk-bound test packages (e.g. database transactions, filesystem CAS operations, or lock contention fixtures) risks hardware thrashing, thread starvation, and non-deterministic flakes.
This subsystem provides persistent, content-addressed, and runtime-mergeable concurrency constraints without requiring test harnesses to take circular dependencies on heavy execution frameworks. It serves as the single source of truth for:
- Standardized package path normalization across operating systems and workspace relative coordinates.
- On-disk policy loading and parsing from
.zqk/test-bundles/package_concurrency_limits.json. - Dynamic policy merging that combines persistent repo-level configuration with runtime job metadata and conservative scheduler defaults.
- Ceiling enforcement clamping concurrent test workers to safe host thresholds (maximum 32 concurrent workers per package).
2. Component Topology & Architecture
The subsystem coordinates four primary components across the scheduler and runtime execution boundary:
- HardwareResourceProber: Probes available host CPU cores, memory limits, and container cgroups to evaluate safe baseline concurrency capacity.
- LimitCalculator: Evaluates normalized package limits from on-disk declarations, job metadata annotations, and fallback rules.
- ThrottlingGovernor: Reconciles multi-source limit maps via conservative minimums, ensuring local job manifests cannot unsafely exceed project-wide constraints.
- ConcurrencySemaphore: Allocates per-package slot semaphores that throttle simultaneous test package runners, queuing excess test jobs with context-aware timeouts.
flowchart TD
subgraph StorageLayer["Configuration & Storage Plane"]
DiskPolicy["On-Disk Limits\n(.zqk/test-bundles/package_concurrency_limits.json)"]
JobManifest["Job Metadata\n(Scheduler Raw Jobs)"]
end
subgraph CoreSubsystem["pkg/testpackageconcurrency Subsystem"]
Prober["HardwareResourceProber\n(CPU / RAM / Container Cgroups)"]
Reader["ReadLimitsMap\n(JSON Deserialization & Clamping)"]
Merger["MergeLimitMaps\n(Conservative Per-Key Minimum)"]
Normalizer["NormalizePackagePath\n(Trimming & Prefix Canonicalization)"]
end
subgraph ExecutionPlane["Scheduler & Concurrency Governance"]
Limiter["PackageConcurrencyLimiter\n(CircuitBreaker Semaphore Map)"]
WorkerPool["Parallel Test Runners\n(go test Subprocesses)"]
end
DiskPolicy -->|JSON Read| Reader
Normalizer -->|Sanitized Keys| Reader
JobManifest -->|Dynamic Limits| Merger
Reader -->|Base Limits| Merger
Prober -.->|Capacity Hints| Merger
Merger -->|Merged Policy Map| Limiter
Limiter -->|Acquire / Release| WorkerPool
3. Interface & Contract Specifications
The subsystem provides three core functions defined in limits.go:
// NormalizePackagePath trims "./" and surrounding space for map keys and lookups.
func NormalizePackagePath(packagePath string) string
// ReadLimitsMap loads max concurrent run_wrapper jobs per package from disk.
// Missing file returns (nil, nil). Invalid JSON returns an error.
func ReadLimitsMap(projectRoot string) (map[string]int, error)
// MergeLimitMaps returns a new map with per-key minimum of a and b (when both set).
// Keys present in only one map are included. Used to combine on-disk policy with job metadata.
func MergeLimitMaps(a, b map[string]int) map[string]int
On-Disk Schema Specification
Policy files are persisted at paths.ProjectDataDir / paths.TestBundlesDir / LimitsFileName (.zqk/test-bundles/package_concurrency_limits.json). The schema contract is defined by limitsFile:
{
"schema_version": 1,
"updated_at": "2026-10-25T12:00:00Z",
"max_concurrent_jobs_by_package": {
"pkg/storage": 4,
"pkg/mcp": 8,
"pkg/scheduler": 2,
"pkg/lifecycle": 6
}
}
Invariant Rules
- Path Normalization: All keys are stripped of leading
./and surrounding whitespace throughNormalizePackagePath. Empty keys are discarded. - Value Clamping: Any configured concurrency limit greater than
32is automatically clamped to32to protect host CPU and process tables from unbounded thread storms. - Non-Positive Discarding: Any values
<= 0are excluded from the active limit map. - Fail-Open on Missing File: If the limits file does not exist,
ReadLimitsMapreturns(nil, nil), allowing the caller to apply default scheduler fallbacks without failing.
4. Concurrency Limit Resolution & Throttling Lifecycle
The parallel execution lifecycle coordinates disk configuration, runtime metadata, semaphore acquisition, test dispatch, and graceful release:
sequenceDiagram
autonumber
participant S as Scheduler Daemon
participant TPC as pkg/testpackageconcurrency
participant PCL as PackageConcurrencyLimiter
participant TR as Test Runner Subprocess
S->>TPC: ReadLimitsMap(projectRoot)
TPC-->>S: return on-disk limits (e.g. storage: 4)
S->>S: packageConcurrencyLimitsFromSchedulerJobRaw(jobs)
S->>TPC: MergeLimitMaps(fileMap, jobMap)
TPC-->>S: return merged conservative limits
S->>PCL: MergeLimits(merged)
Note over S,TR: Job Dispatch Cycle
S->>PCL: ShouldLimit("pkg/storage")
PCL-->>S: true (limit = 4)
S->>PCL: Acquire(ctx, "pkg/storage")
alt Slots Available (< 4)
PCL-->>S: acquired immediately
else Capacity Exhausted (= 4)
PCL-->>S: wait on condition / timeout
end
S->>TR: Launch go test ./pkg/storage/...
TR-->>S: Process exit (PASS / FAIL)
S->>PCL: Release("pkg/storage")
5. Resource Adaptation & Boundary Matrix
The subsystem dynamically accommodates diverse host topologies, balancing CPU saturation against memory ceilings:
| Environment Profile | Available Cores | Memory Allocation | Default Package Limit | Hard Ceiling | Notes |
|---|---|---|---|---|---|
| CI Container (Constrained) | 2 – 4 vCPUs | 4GB – 8GB | 2 | 4 | Conservative allocation to prevent OOM kills in cgroup-constrained pods. |
| Developer Laptop (Interactive) | 8 – 12 Cores | 16GB – 32GB | 4 | 8 | Prevents UI freezing while retaining responsive CLI turnaround times. |
| Bare-Metal Build Node | 32 – 64+ Cores | 64GB – 256GB | 8 | 32 | High throughput for large monorepo test passes while honoring package locks. |
| Specialized Fixture Packages | Any | Any | 1 | 1 | Strictly serialized (e.g. disk-exclusive CAS tests or port-binding services). |
Conflict Resolution Strategy
When multiple limit definitions exist for the same package:
- Both on-disk and job metadata defined:
MergeLimitMapsappliesmin(fileVal, jobVal). The more restrictive limit always wins. - Present in only one source: The declared positive limit is preserved.
- Neither declared: The scheduler applies its conservative fallback default (
getSchedulerDefaultPackageConcurrency()).
6. Concurrency & Thread-Safety Guarantees
- Lock-Free Map Merging:
MergeLimitMapsconstructs and returns a newly allocated map, ensuring zero contention and safe immutability across concurrent readers. - Atomic Semaphore Tracking: Downstream limiters implement mutex-protected slot acquisition with context timeouts to prevent deadlocks when test bundles run long.
- Goroutine Labeling: All asynchronous test and scheduler workers spawning concurrency checks are managed through
goroutinelabels.NewGoroutine, ensuring transparent attribution in stack traces and panic telemetry.
7. Verification & Statement Coverage Invariant
Statement test coverage for pkg/testpackageconcurrency is enforced at >= 85.0% with a verified achievement of 100.0%:
github.com/zqk-os/zqk/pkg/testpackageconcurrency/limits.go:27: NormalizePackagePath 100.0%
github.com/zqk-os/zqk/pkg/testpackageconcurrency/limits.go:33: ReadLimitsMap 100.0%
github.com/zqk-os/zqk/pkg/testpackageconcurrency/limits.go:68: MergeLimitMaps 100.0%
total: (statements) 100.0%
To execute the test suite:
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s -cover ./pkg/testpackageconcurrency