Scenario Convergence Matrix Engine & Multi-Model Evaluation Benchmark
Status
- Author: PER-COMMUNITY-SYSTEM-ARCHITECT
- Phase: Release Candidate Readiness
- Status: Approved / Implemented
Architectural Context & Problem Statement
In early engineering iterations, scenario evaluations (zqk eval run) supported single-template evaluations against targeted file glob patterns or Go packages. In multi-agent and swarm coordination topologies, verifying convergence across multiple distinct model providers (e.g., Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) required executing disconnected commands and manual variance comparison.
Furthermore, convergence sessions (CVS-*) require structured, aggregate metrics across multiple dimensions (HCODE, determinism, syntactic hygiene, negative boundary enforcement) to detect model drift or regressions before promoting code.
Design & Implementation
1. Matrix Configuration & Aggregation Architecture
The MatrixEngine provides a high-throughput, deterministic evaluation harness that orchestrates:
- Multiple evaluation templates (
EvalTemplate) across targeted codebases. - Cross-model scoring variance and mean score aggregation.
- Pass/fail rate calculation per model provider.
- Automatic aggregation of remediation recommendations.
type MatrixConfig struct {
Templates []*EvalTemplate `json:"templates" yaml:"templates"`
Models []string `json:"models" yaml:"models"`
Targets []string `json:"targets" yaml:"targets"`
SessionID string `json:"session_id,omitempty" yaml:"session_id,omitempty"`
}
type ModelScoreSummary struct {
Model string `json:"model"`
RunsCount int `json:"runs_count"`
PassedCount int `json:"passed_count"`
PassRate float64 `json:"pass_rate"`
MeanScore float64 `json:"mean_score"`
ScoreVariance float64 `json:"score_variance"`
}
2. Variance Computation
Score variance is computed over $n$ evaluation runs per model:
3. CLI Integration
The zqk eval matrix command binds directly to MatrixEngine:
zqk eval matrix --session-id CVS-001 --template ast-hygiene --json
Verification & Conformance
- Test Suite:
pkg/eval/scenario_matrix_engine_test.go - Coverage: Multi-model variance aggregation, remediation recommendation synthesis, negative empty model/template boundary verification.