Batch AF (Aggregation & Orphaned Processing) Subsystem Architectural & Operational Guide
1. Executive Architectural Summary
The Batch AF (Aggregation & Orphaned Processing) Subsystem (pkg/batchaf) delivers high-throughput stream aggregation, dual-trigger batch dispatching, resilient worker pool execution, exponential backoff retries, and automated orphaned batch reclamation across the ZQK Knowledge Kernel.
In distributed autonomous systems, high-concurrency event ingestion poses severe operational challenges. Unbuffered granular writes overwhelm storage engines, trigger database lock contention, and degrade transaction throughput. Simultaneously, unhandled worker exits, network partitions, or transient external API timeouts can leave batches orphaned in an indeterminate state, risking data loss or silent pipeline stall.
Governed by core platform policies (including POL-DEFAULT-142277d3), the pkg/batchaf subsystem addresses these failure modes through five core design invariants:
- Dual-Trigger Batch Dispatch: Batches are automatically flushed when either the capacity limit (
MaxBatchSize) is reached or the idle interval (FlushInterval) elapses via a dedicatedTimerSentinel. - Bounded Concurrency & Worker Dispatch: A dedicated
WorkerPoolprocesses flushed batches across a bounded number of worker goroutines, insulating downstream systems from concurrency spikes. - Resilient Retry & Exponential Backoff: Transient batch execution failures trigger automated retries with deterministic exponential backoff, isolating transient faults without crashing caller threads.
- Active Orphan Tracking & Reclamation: The
ReclamationQueuecontinuously tracks in-flight batches. Any batch whose processing duration exceedsBatchTimeoutis isolated, marked for reclamation, and routed to dead-letter sinks or retry queues. - Static Floor Coverage Invariant: A strict statement test coverage invariant (\(\ge 85.0\%\), verified at \(>96\%\)) ensuring that all concurrency branches, timeout handlers, retry loops, and error scenarios remain rigorously verified.
2. Component Topology & Subsystem Architecture
The subsystem coordinates stream ingestion, threshold dispatching, worker execution, and error telemetry through five core components:
flowchart TD
subgraph Producers["Stream Producers & Subsystems"]
CLI["CLI Command Ingestion"]
AgentFeed["Agent Coordination Feeds"]
ProcessEngine["Orphaned Requirement Sweeper"]
end
subgraph BatchAF["Batch AF Subsystem (pkg/batchaf)"]
direction TB
IngestChannel["Item Buffer Queue\n(itemCh chan T)"]
subgraph AggregatorCore["BatchAggregator Core"]
ActiveSlice["Active Buffer Slice\n([]T with Mutex Boundary)"]
Sentinel["TimerSentinel\n(Ticker Tick Dispatch)"]
ControlLoop["Ingest & Flusher Control Loop\n(runLoop)"]
end
subgraph WorkerSubsystem["Worker Dispatch & Reclamation"]
Pool["WorkerPool\n(Bounded Worker Routines)"]
Reclamation["ReclamationQueue\n(In-Flight Map & Orphan Sweeper)"]
ErrCollector["ErrorCollector\n(Thread-Safe Diagnostics)"]
end
end
subgraph Sinks["Downstream Consumers & Sinks"]
Callback["FlushCallback Handler"]
DeadLetter["Dead-Letter Callback Sink"]
end
Producers -->|"Add() / Submit() / Push()"| IngestChannel
IngestChannel --> ControlLoop
Sentinel -->|"Interval Tick"| ControlLoop
ControlLoop -->|"Buffer Items"| ActiveSlice
ControlLoop -->|"Dispatch Batch"| Reclamation
ControlLoop -->|"Queue Job"| Pool
Pool -->|"Execute Flush"| Callback
Reclamation -->|"Sweep Exceeded Timeouts"| DeadLetter
Pool -->|"Collect Failures"| ErrCollector
Component Responsibilities
| Component | Architecture Role | Key Responsibilities |
|---|---|---|
BatchAggregator[T] |
Root Controller | Manages intake channels, enforces capacity ceilings, runs the primary select loop, and coordinates graceful shutdown. |
WorkerPool[T] |
Execution Engine | Distributes flushed batches among worker goroutines (goroutinelabels.NewGoroutine) with non-blocking job queuing. |
TimerSentinel |
Temporal Heartbeat | Emits periodic interval ticks to trigger partial batch flushes during low-throughput or sparse ingest periods. |
ReclamationQueue[T] |
Fault Containment | Maintains thread-safe registry of in-flight batches, measures processing durations, and reclaims abandoned or timed-out batches. |
ErrorCollector |
Diagnostics Aggregator | Thread-safe collector capturing worker dispatch errors, enabling runtime inspection and health checks. |
3. Aggregation & Flush Lifecycle
The ingestion and dispatch lifecycle balances maximum aggregation efficiency against low event latency:
sequenceDiagram
autonumber
actor Producer as Event Producer
participant Ingest as itemCh (Buffer)
participant Loop as Aggregator runLoop
participant Sentinel as TimerSentinel
participant RecQueue as ReclamationQueue
participant Pool as WorkerPool
participant Processor as FlushCallback
Producer->>Ingest: Add(ctx, item) / Submit(item)
Ingest-->>Loop: Receive item
Loop->>Loop: Append item to active buffer
alt Buffer reaches MaxBatchSize
Loop->>RecQueue: RegisterInFlight(batch)
Loop->>Pool: Dispatch(batch)
else FlushInterval expires
Sentinel-->>Loop: Periodic Tick
Loop->>RecQueue: RegisterInFlight(batch)
Loop->>Pool: Dispatch(batch)
else Manual Flush requested
Producer->>Loop: Flush()
Loop->>Loop: Drain pending items
Loop->>RecQueue: RegisterInFlight(batch)
Loop->>Pool: Dispatch(batch)
end
Pool->>Processor: FlushCallback(ctx, batch.Items)
alt Execution Successful
Processor-->>Pool: nil (success)
Pool->>RecQueue: MarkSuccess(batch.ID)
else Transient Failure (Retryable)
Processor-->>Pool: error
Pool->>Pool: Sleep exponential backoff
Pool->>Processor: Retry FlushCallback
else Max Retries Exhausted
Pool->>RecQueue: MarkFailed(batch.ID, err)
Pool->>Producer: DeadLetterCallback(batch, err)
end
4. Failure Modes & Orphaned Batch Reclamation
A primary requirement of distributed stream processing is handling catastrophic worker halts, processor deadlocks, or persistent external service failures. The ReclamationQueue manages batch states across a robust state machine:
stateDiagram-v2
[*] --> Active: Item Appended to Active Slice
Active --> InFlight: Capacity reached / Interval ticked
InFlight --> Succeeded: FlushCallback succeeds
InFlight --> Retrying: FlushCallback returns error (attempts <= MaxRetries)
Retrying --> InFlight: Exponential backoff elapsed
Retrying --> Failed: Max retries exhausted
Failed --> DeadLetter: Routed to DeadLetterCallback
InFlight --> Reclaimed: Processing duration >= BatchTimeout (SweepOrphaned)
Reclaimed --> DeadLetter: Orphaned batch escalated to DeadLetterCallback
Succeeded --> [*]
DeadLetter --> [*]
Reclamation Sweep Protocol
- Checkout Registration: When a batch is dispatched,
ReclamationQueue.RegisterInFlightrecords the batch with an initial timestamp andBatchStatusInFlightstatus. - Success Acknowledgement: Upon successful execution of
FlushCallback, the worker invokesMarkSuccess(batchID), removing the batch from active tracking. - Periodic / On-Demand Sweep:
ReclaimOrphaned(ctx)scans active in-flight batches. Any batch whose elapsed time since last update exceedsBatchTimeoutis transitioned toBatchStatusReclaimed, removed from the active map, and forwarded to the configuredDeadLetterCallback.
5. Concurrency Model & Thread-Safety Guarantees
The pkg/batchaf subsystem enforces rigorous concurrency invariants to operate safely in multi-threaded environments:
Lock Invariants & Data Isolation
- Channel Ingestion Boundary: Producers submit items across buffered Go channels (
itemCh), providing lock-free ingestion under normal load. - Mutex Protected Active Slice: Modifications to the in-memory item slice (
activeItems) are guarded bysync.MutexinsideappendItemanddispatchCurrentBatch. - Read-Write Mutex on Reclamation: State transitions in
ReclamationQueueusesync.RWMutex, allowing concurrent status queries (InFlightCount,Get) without blocking batch registrations. - Atomic Telemetry Counters: All throughput statistics (
ItemsSubmitted,BatchesFlushed,BatchesSucceeded,BatchesRetried,BatchesReclaimed,DeadLettered) use lockless atomic operations.
Goroutine Governance
All background threads (the timer sentinel, worker pool routines, and main aggregator event loop) are spawned exclusively using goroutinelabels.NewGoroutine, ensuring adherence to kernel telemetry policies and runtime profiling invariants.
Deterministic Graceful Drain
When Close() is invoked:
closeChis closed, halting acceptance of new interval ticks.TimerSentinel.Stop()terminates the interval ticker.drainRemaining()sweeps any remaining items initemChand dispatches final batches.WorkerPool.Stop()waits for all active worker routines to finish processing in-flight jobs.
6. Configuration & Operational Recipes
Default Configuration Baseline
opts := batchaf.DefaultOptions[string]()
opts.MaxBatchSize = 100
opts.FlushInterval = 250 * time.Millisecond
opts.WorkerConcurrency = 4
opts.BufferCapacity = 2000
opts.BatchTimeout = 10 * time.Second
opts.MaxRetries = 3
opts.BackoffBase = 50 * time.Millisecond
opts.FlushCallback = func(ctx context.Context, batch []string) error {
return persistBatchToStorage(ctx, batch)
}
opts.DeadLetterCallback = func(batch *batchaf.Batch[string], err error) {
log.Printf("dead-lettered batch %s (size: %d): %v", batch.ID, len(batch.Items), err)
}
aggregator, err := batchaf.NewBatchAggregator(opts)
if err != nil {
log.Fatalf("failed to initialize batch aggregator: %v", err)
}
defer aggregator.Close()
High-Throughput Non-Blocking Submissions
// Non-blocking submission: returns false if buffer capacity is saturated
submitted := aggregator.Submit("item-payload")
if !submitted {
// Apply backpressure or route to spillover cache
}
Context-Aware Blocking Submission
// Context-bounded submission: waits for buffer availability or aborts on cancellation
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
if err := aggregator.Add(ctx, "critical-item-payload"); err != nil {
// Handle context cancellation or closed aggregator
}
7. Verification & Coverage Invariant
Statement test coverage for pkg/batchaf is governed by continuous integration gates enforcing a static floor of \(\ge 85.0\%\).
To verify package test coverage locally:
ZQK_ALLOW_FOREGROUND_GO_TEST=1 go test -v -race -timeout 30s -cover ./pkg/batchaf
Verified Test Matrix
The test suite in pkg/batchaf/batchaf_coverage_elevation_cycle180_test.go exercises:
- Configuration validation (zero/negative sizes, missing callbacks, invalid intervals).
- Dual-trigger flushes (capacity boundary vs. timer sentinel ticks).
- High-concurrency multi-producer ingestion with zero race conditions.
- Retry loops with exponential backoff and dead-letter routing.
- Orphaned batch identification, status migration, and reclamation sweeps.
- Graceful drain, shutdown idempotency, and post-close error responses.