Goroutine Identity & Concurrency Labels Subsystem Operational Architecture Guide

Comprehensive architectural manual, synchronization invariants specification, and operational guide for the ZQK Goroutine Identity & Concurrency Labels Subsystem (pkg/goroutinelabels).


1. Overview & Architectural Compass

In an autonomous multi-agent operating kernel where background schedulers, CAS validation pipelines, git worktree managers, and telemetry collectors run simultaneously, uncoordinated goroutines create severe operational liabilities:

  • Anonymous Goroutine Sprawl: Stack dumps during incidents or profiler captures (pprof) yield unlabelled goroutine entries with identical anonymous signatures (runtime.goexit), obfuscating root causes.
  • Unbounded Concurrency & Thread Starvation: Spawning goroutines without budget caps saturates the Go runtime scheduler, induces CPU context switching overhead, and exhausts operating system file descriptors.
  • Silent Panic Swallowing or Process Death: An unhandled panic in an unmonitored background goroutine crashes the entire process kernel or silently halts background processing without trace telemetry.
  • Queue Deadlocks & Worker Leaks: Naive worker pools without graceful drain mechanisms risk deadlocking producers or dropping in-flight transactional work.

The Goroutine Identity & Concurrency Labels Subsystem (pkg/goroutinelabels) resolves these challenges by providing:

  1. Pprof Runtime Categorization: Automatic tagging via runtime/pprof.Labels attaching semantic identity (name, purpose, and subsystem category) to every goroutine.
  2. Global Concurrency Budgeting: Process-wide slot tracking (Budget) preventing runaway concurrency while preserving dedicated control-plane reserve capacity.
  3. Bounded Resilient Worker Pools: Dynamic worker pools with graceful drain phases, queue pressure observability, and deterministic fallback pools.
  4. Fail-Closed Panic Telemetry: Structured panic recovery capturing full stack traces, emitting alerts, and updating monotonic telemetry counters.
  5. Ergonomic Test Builders: Synchronized test fixtures ensuring deterministic goroutine lifecycle termination without race flakes.

2. Subsystem Architecture & Execution Flows

flowchart TD
    Client["Subsystem Caller (Scheduler / Validator / Pipeline)"] --> BuilderInit["NewGoroutine(name, purpose)"]
    BuilderInit --> BudgetCheck{"WithBudget Configured?"}

    BudgetCheck -->|"Yes"| ControlPlaneCheck{"IsControlPlane / AsCleanup?"}
    ControlPlaneCheck -->|"Yes"| SkipBudget["Bypass Budget Reservation"]
    ControlPlaneCheck -->|"No"| ReserveSlot["Budget.Reserve(1)"]

    ReserveSlot -->|"Success"| TagPprof["Tag Goroutine via runtime/pprof"]
    ReserveSlot -->|"Exceeded & Handler Present"| OnExceeded["Invoke onBudgetExceeded() -> Abort"]
    ReserveSlot -->|"Exceeded & No Handler"| FallbackNotify["notifyBudgetExceededFallback() -> Emergency Run"]

    SkipBudget --> TagPprof
    FallbackNotify --> TagPprof
    BudgetCheck -->|"No"| TagPprof

    TagPprof --> ExecScope["Execute Wrapped Task Function"]
    ExecScope --> PanicRecovery{"Panic Detected?"}

    PanicRecovery -->|"Yes"| IncrementPanic["Increment unhandledPanicsTotal"]
    IncrementPanic --> CustomPanicCheck{"Custom Handler Configured?"}
    CustomPanicCheck -->|"Yes"| CallPanicHandler["Execute onPanic(r) / defaultPanicHandler"]
    CustomPanicCheck -->|"No"| StderrTelemetry["Emit Structured Stack to Stderr"]

    PanicRecovery -->|"No"| Cleanups["Execute PreCleanup -> CleanupFunc -> PostCleanup"]
    CallPanicHandler --> Cleanups
    StderrTelemetry --> Cleanups
    Cleanups --> ReleaseBudget["Release Budget Slot (defer)"]
    ReleaseBudget --> SignalExit["Signal WaitGroup / Exit Channel"]

Worker Pool Dispatch Sequence

sequenceDiagram
    autonumber
    actor Caller as Subsystem Client
    participant Pool as Pool (pkg/goroutinelabels)
    participant Budget as Concurrency Budget
    participant Channel as Work Queue Channel
    participant Worker as Background Worker

    Caller->>Pool: NewPool(budget, name, purpose, workers, queueCap)
    Pool->>Budget: Reserve(workerCount)
    alt Budget Exceeded or Nil
        Budget-->>Pool: Reservation Error / Declined
        Pool->>Pool: Instantiate Single-Worker Fallback Pool
        Pool-->>Caller: Return Fallback Pool (IsFallback=true)
    else Reservation Succeeded
        Budget-->>Pool: Release Function Handle
        Pool-->>Caller: Return Bounded Pool (IsFallback=false)
    end

    Caller->>Pool: Start(ctx)
    Pool->>Worker: Launch N Goroutines with Pprof Labels

    Caller->>Pool: Submit / SubmitNonBlocking(ctx, task)
    alt Queue Full (Non-Blocking)
        Pool-->>Caller: ErrPoolFull
    else Task Accepted
        Pool->>Channel: Enqueue Task
        Channel-->>Worker: Dequeue Task
        Worker->>Worker: Execute Safely (runWorkItemSafely)
        Pool-->>Caller: Nil (Success)
    end

    Caller->>Pool: Stop()
    Pool->>Channel: Close Work Channel (submitMu locked)
    Worker->>Worker: Drain Remaining Queued Tasks
    Worker-->>Pool: Workers Exit (sync.WaitGroup)
    Pool->>Budget: Invoke Budget Release Function

3. Subsystem Taxonomy & Core Mechanics

3.1 Named Goroutines & Runtime Categorization

Goroutine labels are assigned using Go's runtime/pprof infrastructure, allowing live profiling tools (go tool pprof, execution tracer, and thread dump analyzers) to group running goroutines by logical responsibility:

package example

import (
    "context"
    "github.com/zqk-os/zqk/pkg/goroutinelabels"
)

func LaunchValidationWorker(ctx context.Context) {
    goroutinelabels.NewGoroutine("validation_daemon", "validating CAS objects").
        WithContext(ctx).
        Start(func() error {
            // Task executes with labels:
            // "name"    = "validation_daemon"
            // "purpose" = "validating CAS objects"
            return processQueue(ctx)
        })
}

Key builder options include:

  • WithContext(ctx) / WithLifecycleContext(ctx): Binds cancellation lifecycle to the goroutine.
  • WithWaitGroup(wg): Automatically adds (wg.Add(1)) before launching and calls defer wg.Done(). Callers do not call wg.Add(1) manually.
  • WithSignalOnExit(chan bool): Sends exit acknowledgment to a buffered completion channel.
  • WithPreCleanup(fn), WithCleanup(fn), WithPostCleanup(fn): Multi-stage deferral hooks guaranteed to execute even during panics.
  • WithShutdownCheck(fn): Pre-execution guard; if fn() returns true, the goroutine terminates before entering task logic.

3.2 Global Concurrency Budgeting (Budget)

The Budget component provides a deterministic ceiling on concurrent goroutines across the application. Unlike naive semaphores, Budget distinguishes between normal data-plane tasks and mission-critical control-plane processes.

// Initialize global cap (e.g. max 500 concurrent goroutines)
budget := goroutinelabels.NewBudget(goroutinelabels.BudgetConfig{
    MaxTotal: 500,
})

// Set as process default
goroutinelabels.SetDefaultBudget(budget)

Reservation Mechanics

  • b.Reserve(n): Atomically reserves n slots. Returns a thread-safe, idempotent release closure. If requested slots exceed Available(), returns an error without allocating slots.
  • b.Reserve(0) / b.Reserve(negative): Handled safely as no-ops.
  • b.Available(): Computes remaining slots clamped to 0 even under anomalous saturation conditions.
  • b.MaxTotal() == 0: Configures unbounded mode where reservations always succeed.

3.3 Control-Plane & Cleanup Immunity

Critical kernel routines (such as WAL finalizers, resource release hooks, signal handlers, and lock sweepers) must never fail to run due to background worker budget exhaustion.

Marking a goroutine with .AsControlPlane() or .AsCleanup() guarantees budget bypass:

goroutinelabels.NewGoroutine("wal_finalizer", "flushing write-ahead log").
    AsControlPlane().
    WithBudget(budget). // Budget reservation is safely bypassed
    StartSimple(func() {
        wal.FlushSync()
    })

3.4 Bounded Worker Pools (Pool)

The Pool abstraction combines budget reservation, channel queuing, and safe drain mechanics:

// Create worker pool reserving 8 slots from the budget with a buffer of 100 tasks
pool := budget.NewPool("indexer_pool", "indexing graph edges", 8, 100)
pool.Start(ctx)
defer pool.Stop()

// Submit blocking task
err := pool.Submit(ctx, func(taskCtx context.Context) error {
    return indexEdge(taskCtx, edge)
})

// Submit non-blocking task (fail fast on saturation)
err = pool.SubmitNonBlocking(ctx, func(taskCtx context.Context) error {
    return indexEdge(taskCtx, edge)
})
if errors.Is(err, goroutinelabels.ErrPoolFull) {
    // Handle queue saturation
}

Fallback Pool Safety

If NewPool is called with a nil budget, an invalid worker count (<= 0), or when the budget is completely exhausted, the subsystem does not panic or leave work unhandled. Instead:

  1. It transparently constructs a single-worker fallback pool (isFallback == true).
  2. It notifies the registered PoolCreationDeclinedNotifier callback for telemetry logging.
  3. It limits resource consumption to exactly 1 bounded worker.

Graceful Drain on Shutdown

Calling pool.Stop() executes a structured 3-phase shutdown:

  1. Locks submission mutex (submitMu) and closes the incoming work channel.
  2. Cancels worker runtime context (runCtx) to wake any waiting tasks.
  3. Awaits all active workers draining queued tasks before releasing reserved budget slots.
  4. Calling pool.Stop() multiple times is safe and strictly idempotent.

3.5 Panic Recovery & Telemetry Invariants

Goroutine execution is encapsulated in fail-closed panic recovery:

  1. Every panic is intercepted by recoverGoroutine.
  2. The global atomic counter unhandledPanicsTotal is incremented.
  3. If a custom onPanic callback is provided, it receives the panic value.
  4. If a global defaultPanicHandler is configured via SetDefaultPanicHandler, it receives the name, purpose, value, and captured stack trace.
  5. If no handlers are defined, a structured diagnostic dump is written to os.Stderr.
  6. Defers and cleanup callbacks are guaranteed to run regardless of panic state.
goroutinelabels.SetDefaultPanicHandler(func(name, purpose string, r any, stack []byte) {
    telemetry.RecordPanic(name, purpose, r, string(stack))
})

3.6 Test Fixtures & Mock Builders

The subsystem provides ergonomic test helpers in test_builder.go:

  • NewTestGoroutine(name, purpose): Constructs a synchronized test goroutine with automatic Wait() barrier.
  • StartTestGoroutine(name, purpose, fn): Launches an asynchronous worker returning a *sync.WaitGroup.
  • StartTestGoroutines(t, count, name, purpose, fn): Spawns a parameterized swarm of test goroutines with indexed worker IDs and unified completion tracking.

4. Verification & Statement Coverage Invariant

In alignment with kernel quality standards:

  • Statement Coverage Invariant: The pkg/goroutinelabels package enforces a strict static floor of >= 85.0% statement coverage (currently operating at 97.0%).
  • Zero Swallowed Errors: All returned error values must be checked explicitly. Blank identifier assignments (_ = ...) to error returns are prohibited.
  • Deterministic Teardown: All spawned goroutines must join before test termination to prevent asynchronous race contamination across test cases.

Verification Recipes

# Execute isolated package test suite within 30s timeout
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -timeout 30s ./pkg/goroutinelabels

# Verify package statement coverage against the 85.0% floor
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -cover ./pkg/goroutinelabels

# Verify full repository tree hygiene and AST invariants
./bin/zqk-vet -suite hygiene,tree

5. Open-Core Compliance Invariant

This guide adheres strictly to the Open-Core Public Release Gateway Standards. In compliance with scripts/open-core/check-public-release-payload.sh:

  • All identifiers in this guide are generic architectural patterns or exported symbols from pkg/goroutinelabels.
  • No internal ephemeral nanosecond IDs, private project references, or prohibited governing tags are present.
  • All code examples are copy-pasteable and functional within the public kernel codebase.