Subscriber Backpressure, Adaptive Concurrency Throttling, and Circuit Breaking
Executive Summary
In distributed event-driven systems and asynchronous swarm architectures, slow, blocked, or failing subscribers can easily destabilize the entire callback dispatcher. Without proactive backpressure buffering, concurrency throttling, and circuit-breaking isolation, a single misbehaving subscriber blocks dispatch loops, triggers unbounded memory consumption, and causes cascading failure cascades across the entire cluster.
This specification details the architecture, design, and mathematical mechanics of:
ResilientSubscriber: A resilient decorator wrapping anySubscriberwith integrated backpressure queuing, concurrency limits, and circuit breaker trip guards (cmd/zqk/callback/resilient_subscriber.go).AdaptiveRingBufferQueue: A thread-safe, bounded circular buffer supporting deterministic drop policies (DropOldestvsDropNewest).CircuitBreaker: A three-state trip-and-reset circuit breaker (Closed,Open,HalfOpen) with probe recovery and adaptive cool-off intervals.
Architectural Topology
flowchart TD
subgraph CallbackBus ["Knowledge Kernel Callback Bus"]
IncomingEvents["Callback Entries / Swarm Events"] --> ResilientSub["ResilientSubscriber (Decorator)"]
end
subgraph BackpressureLayer ["Adaptive Backpressure & Queueing"]
ResilientSub --> CBCheck{"Circuit Breaker State"}
CBCheck -->|"Open (Shed Load)"| FailFast["ErrCircuitBreakerOpen (Fast Reject)"]
CBCheck -->|Closed / Half-Open| Queue["Bounded Ring Buffer Queue"]
Queue -->|Capacity Reached| DropPolicy{"Drop Strategy"}
DropPolicy -->|DropOldest| EvictOld["Evict Head & Enqueue Tail"]
DropPolicy -->|DropNewest| EvictNew["Reject Incoming Entry (ErrQueueFull)"]
end
subgraph ConcurrencyLayer ["Concurrency Throttler"]
EvictOld --> Semaphore["Worker Semaphore (Max Concurrency)"]
Queue -->|Within Watermarks| Semaphore
Semaphore --> Worker["Execute Inner Subscriber.Notify()"]
end
subgraph CircuitBreakerStateMachine ["Circuit Breaker Trip & Recovery"]
Worker -->|Success| RecordSuccess["Record Success & Reset Failure Counter"]
RecordSuccess --> TransitionClosed["State -> Closed"]
Worker -->|Failure / Timeout| RecordFail["Increment Consecutive Failures"]
RecordFail --> Threshold{"Failures >= Threshold?"}
Threshold -->|Yes| TripOpen["State -> Open (Trip Breaker)"]
TripOpen --> Timer["Cool-Off Interval (Sleep)"]
Timer --> HalfOpen["State -> Half-Open (Canary Probe)"]
HalfOpen --> Worker
end
1. Bounded Backpressure & Drop Strategies
The ResilientSubscriber isolates the callback dispatcher from slow consumer processing speeds using a bounded in-memory buffer:
Drop Strategies
DropOldest: - When queue capacity is reached, the oldest unconsumed element at the buffer head is discarded. - Ideal for real-time telemetry, terminal state streaming, and sensor feeds where freshness takes precedence over historical replay.DropNewest: - Rejects the incoming element immediately, returningErrQueueFull. - Protects existing in-flight work batches from being overwritten when strict processing order is required.
2. Concurrency Throttling & Watermark Limits
To prevent memory leaks and thread starvation under sudden event spikes:
- Semaphore-Bounded Concurrency:
- Limits simultaneous executions of the inner subscriber to
MaxConcurrency. - Acquires a token before callingNotify(ctx, entry)and releases it upon return, preventing runaway goroutine proliferation. - Execution Timeout:
- Enforces a per-invocation timeout deadline (
Timeout). - If the subscriber does not complete within the deadline, execution is canceled withcontext.DeadlineExceededand recorded as a failure.
3. Trip-and-Reset Circuit Breaking
The circuit breaker protects upstream systems when a subscriber is experiencing persistent outages:
State Transitions
Closed(Normal Operation): - All events are dispatched to the inner subscriber. - Failures increment the consecutive failure counter. - If consecutive failures reachFailureThreshold(default: 5), the breaker trips toOpen.Open(Fault Isolation): - All incoming events are immediately rejected withErrCircuitBreakerOpenwithout invoking the subscriber or enqueueing. - Starts a cool-off timer (RecoveryTimeout, default: 5s).Half-Open(Canary Recovery): - When the cool-off timer elapses, a single probe event is allowed through to test subscriber health. - If the canary succeeds, the breaker resets toClosed. - If the canary fails, the breaker trips back toOpenfor another recovery period.