Distributed Event Mesh Telemetry Aggregation and Adaptive Backpressure Throttling
Overview
The Distributed Event Mesh Telemetry and Backpressure subsystem (cmd/zqk/callback/mesh_telemetry_backpressure.go) provides unified, multi-dimensional execution observability and adaptive flow control for asynchronous event subscribers across the Knowledge Kernel. It combines bounded sliding-window time-series bucketing with credit-based token bucket rate limiting and AIMD (Additive Increase / Multiplicative Decrease) dynamic congestion control to protect against queue starvation, memory exhaustion, and distributed event storms.
Architecture
+-----------------------------------------------------------------------------------+
| Distributed Event Mesh Telemetry & Backpressure Flow |
| |
| +--------------------+ Acquire() +-----------------------------+ |
| | Event Dispatcher | -----------------------> | MeshBackpressureThrottler | |
| +--------------------+ | (Credit Budgeting & AIMD) | |
| | +--------------+--------------+ |
| | Notify() | |
| v v |
| +--------------------+ Dimensional Snapshots +-----------------------------+ |
| | MeshTelemetryAgg | <----------------------- | Subscriber Worker Pools | |
| | (SlidingWindowBuf) | | (Policy: Reject|Wait|Drop) | |
| +--------------------+ +-----------------------------+ |
+-----------------------------------------------------------------------------------+
Core Components
1. SlidingWindowBuffer & Time-Series Bucketing
Aggregates high-throughput telemetry metrics across sliding sub-buckets:
- Discretized Sub-Buckets: Configurable window durations (default 10s) and bucket counts (default 10) providing sub-second temporal resolution.
- Latency Percentile Quantization: Maintains discretized latency histograms enabling accurate estimation of P50, P95, and P99 latency percentiles without unbounded memory allocation.
- Throughput Rate Normalization: Computes sliding-window throughput (
TotalDispatched / WindowDuration) and drop rates (DropRate() = TotalDropped / TotalDispatched) dynamically.
2. MeshTelemetryAggregator
Subscriber-aware dimensional telemetry store:
- Multi-Dimensional Tagging: Slices metrics by topic, broker, subscriber pool, and execution status (
success,dropped,error). - CallbackSubscriber Integration: Implements the canonical
CallbackSubscriberinterface (Notify()) to observe wire events without adding dispatch latency. - Credit Debt Telemetry: Exposes system-wide credit debt and queue buffer utilization ratios for ambient health monitoring.
3. MeshBackpressureThrottler
Credit-based adaptive flow control engine:
- Credit Budgets: Individual subscriber pools maintain independent initial, current, and maximum credit ceilings.
- Backpressure Policies:
PolicyReject: ReturnsErrCreditExhaustedimmediately when credits drop to zero.PolicyWait: Blocks caller until credits are replenished or context expires.PolicyDropOldest: Drops the oldest enqueued event to admit high-priority new arrivals.- Replenishment & Acknowledgment: Callers replenish credits via
Replenish()or notify completed dispatches viaAcknowledge(credits, latency).
4. AdaptiveRateLimiter (AIMD Congestion Control)
Dynamic rate regulation adapting to downstream subscriber health:
- Additive Increase: Gradually expands rate limits by a constant step upon consecutive successful acknowledgments below latency thresholds.
- Multiplicative Decrease: Swiftly halves rate limits upon credit exhaustion, dropped events, or SLA latency violations.
- Ceiling and Floor Bounds: Clamps rates strictly between configurable minimum and maximum bandwidth limits.
Boundary & Error Handling Specifications
- Invalid Configuration Guards: Rejects zero or negative window durations, bucket counts, or credit limits fail-closed (
ErrInvalidConfiguration,ErrInvalidCreditAmount). - Missing Pool Protection: Operations targeting unregistered subscriber pools return
ErrSubscriberNotFoundwithout panicking. - Graceful Shutdown Invariant: Closing the throttler or controller unblocks waiting callers immediately with
ErrThrottlerClosed. - Saturation & Clamping: Credit replenishment cannot exceed
MaxCredits, preventing unbounded buffer inflation.