Multi-Scenario Batch Evaluation Runner & Real-Time Telemetry Streaming
Overview
The Multi-Scenario Batch Evaluation Runner provides high-throughput concurrent evaluation execution across heterogeneous model endpoints and scenario definitions. Built for zero data-race concurrency and strict token budgeting, the runner bounds memory and thread growth via dedicated worker pools while publishing incremental progress events over server-sent event (SSE) channels.
Architecture & Data Flow
+----------------------+
| BatchConfig |
| (Scenarios, Pool, |
| Token Budget) |
+----------+-----------+
|
v
+----------------------+
| BatchRunner |
+----------+-----------+
|
+----------------------+----------------------+
| | |
v v v
+--------------+ +--------------+ +--------------+
| Worker 1 | | Worker 2 | | Worker N |
| (Scenario A) | | (Scenario B) | | (Scenario K) |
+-------+------+ +-------+------+ +-------+------+
| | |
+----------------------+----------------------+
|
+-----------------+-----------------+
| |
v v
+----------------------+ +----------------------+
| Token Aggregator | | SSE Event Stream |
| (Atomic Telemetry) | | (Real-time State) |
+----------------------+ +----------------------+
Key Components
1. BatchRunner
The central coordinator that validates static configuration constraints, manages bounded worker pools, and aggregates scenario outcomes into a cohesive BatchResult.
2. Token Budgeting & Circuit Breaking
Every scenario execution reports prompt and completion tokens into an atomic accumulator. If the cumulative token consumption reaches or exceeds MaxTokens, the runner halts in-flight work and returns ErrTokenBudgetExceeded, preventing runaway cost and quota starvation.
3. Real-Time Telemetry Event Stream
Scenario lifecycles emit typed StreamEvent notifications over buffered channels:
batch_started: Dispatched upon worker initialization.scenario_progress: Dispatched as each scenario starts execution.scenario_completed: Dispatched when an individual scenario passes evaluation.scenario_failed: Dispatched when a scenario encounters evaluation or model errors without halting peer workers.batch_completed: Emitted upon drain of all scenarios or budget exhaustion.
Verification & Test Proofs
- Static Floor Invariant: Bounded concurrency must be positive; scenarios list non-empty; token budgets non-zero.
- Operational Dynamic Proof: Bounded concurrent execution verified under heavy parallel load; SSE stream event ordering guaranteed.
- Negative Adversarial Boundary: Pre-cancelled contexts and token budget exhaustion gracefully halt workers without deadlocks or resource leaks.