ADR: Core Generic Scenario Evaluation Framework & Convergence Matrix Engine
Status
Accepted
Context
In multi-agent collaborative workflows and automated swarms, verifying system convergence across multiple distinct model providers and autonomous actors requires deterministic, structured evaluation mechanics. Previously, ad-hoc scenario assertions lacked aggregate statistical variance modeling and multi-model convergence reporting.
To guarantee that code transformations and architectural refactorings converge toward defined invariants, the kernel requires a first-class scenario evaluation framework and matrix convergence engine.
Decision
- Implement and standardize
MatrixEngineunderpkg/eval/to orchestrate multi-scenario, multi-model evaluation matrices. - Provide statistical aggregation across evaluation runs: - Run counts, passed counts, and pass rates per model. - Mean composite quality scores and score variance calculation. - Aggregate remediation recommendation synthesis.
- Integrate declarative CLI surface
zqk eval matrixsupporting structured output formats (json,yaml,stream). - Validate compliance using automated Three-Fold Proof verification covering static floor invariants, operational dynamic proof, and negative adversarial error boundaries.
Consequences
- Multi-agent convergence sessions obtain reliable, quantitative scoring across multiple models.
- Model performance variances and regression drifts are surfaced before code promotion.