Scenario Convergence Matrix Engine & Multi-Model Evaluation Benchmark

Status

  • Author: PER-COMMUNITY-SYSTEM-ARCHITECT
  • Phase: Release Candidate Readiness
  • Status: Approved / Implemented

Architectural Context & Problem Statement

In early engineering iterations, scenario evaluations (zqk eval run) supported single-template evaluations against targeted file glob patterns or Go packages. In multi-agent and swarm coordination topologies, verifying convergence across multiple distinct model providers (e.g., Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) required executing disconnected commands and manual variance comparison.

Furthermore, convergence sessions (CVS-*) require structured, aggregate metrics across multiple dimensions (HCODE, determinism, syntactic hygiene, negative boundary enforcement) to detect model drift or regressions before promoting code.


Design & Implementation

1. Matrix Configuration & Aggregation Architecture

The MatrixEngine provides a high-throughput, deterministic evaluation harness that orchestrates:

  • Multiple evaluation templates (EvalTemplate) across targeted codebases.
  • Cross-model scoring variance and mean score aggregation.
  • Pass/fail rate calculation per model provider.
  • Automatic aggregation of remediation recommendations.
type MatrixConfig struct {
    Templates []*EvalTemplate `json:"templates" yaml:"templates"`
    Models    []string        `json:"models" yaml:"models"`
    Targets   []string        `json:"targets" yaml:"targets"`
    SessionID string          `json:"session_id,omitempty" yaml:"session_id,omitempty"`
}

type ModelScoreSummary struct {
    Model         string  `json:"model"`
    RunsCount     int     `json:"runs_count"`
    PassedCount   int     `json:"passed_count"`
    PassRate      float64 `json:"pass_rate"`
    MeanScore     float64 `json:"mean_score"`
    ScoreVariance float64 `json:"score_variance"`
}

2. Variance Computation

Score variance is computed over $n$ evaluation runs per model:

\[s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2 \quad (n > 1)\]

3. CLI Integration

The zqk eval matrix command binds directly to MatrixEngine:

zqk eval matrix --session-id CVS-001 --template ast-hygiene --json

Verification & Conformance

  • Test Suite: pkg/eval/scenario_matrix_engine_test.go
  • Coverage: Multi-model variance aggregation, remediation recommendation synthesis, negative empty model/template boundary verification.