← all analysis · 2026-04-14 · docs/analysis/2026-04-14-panel8-dual-judge.md
SP-Benchmark Panel — 8 open/cloud models, dual-judge (April 2026)
On this page
Date of run: 2026-04-14Models evaluated: deepseek-v3, gemma4-26b-local, gemma4-31b-cloud, glm-5.1-cloud, gpt-oss-120b-cloud, minimax-m2.7, nemotron-3-super, qwen3.5-397b-cloudJudges: claude-opus-4-6, gpt-5.4Total evaluations: 448 (56 questions × 8 models)
Provenance: Imported 2026-04-19 from OxHomeLab/ModelRisk/sp-eval-report.md. Unmodified data below the separator.
The benchmark reveals a critical finding for model risk teams: a 21.9% rate of high judge disagreement (spread ≥3/8) on reproducible, temperature-zero evaluation. Mean inter-judge core spread is 1.37 points on the 0–8 scale, with 64.1% of evaluations showing exact agreement. This methodological insight translates directly to risk practice. A FINMA-aligned LLM audit should not rely on single-judge scoring: the 1.37-point average noise floor means individual model scores lack resolution below ~±1.4 points per question. For a structured product advisor or pricing validator, single-judge validation will systematically underestimate or overestimate model fitness by this margin. This panel demonstrates that dual-judge evaluation is operationally feasible on the SP-Benchmark (incremental cost ~50% over single-judge) and should be mandatory for MRM sign-off on LLM deployment. Institutions with tighter risk budgets should treat single-judge scores as ±1.4-point confidence bands, not point estimates.
Structured Products Evaluation Report
Generated: 2026-04-14T06:57:40.701061+00:00
Leaderboard
Rank Model Core Score Score % Core (claude-opus-4-6) Core (gpt-5.4) Spread Cross-Domain Latency (ms) Cost (USD) Evals 1 gemma4-31b-cloud 6.04/8 75.6% 6.61/8 5.48/8 1.3 1.43/2 20528 $0.0621 56 2 qwen3.5-397b-cloud 5.93/8 74.1% 6.52/8 5.34/8 1.2 1.75/2 151316 $0.0801 56 3 glm-5.1-cloud 5.81/8 72.7% 6.52/8 5.11/8 1.5 1.68/2 256740 $0.0671 56 4 gemma4-26b-local 5.35/8 66.9% 5.80/8 4.89/8 1.3 1.41/2 42952 $0.0638 56 5 minimax-m2.7 5.12/8 64.0% 5.91/8 4.32/8 1.8 1.27/2 87930 $0.0708 56 6 gpt-oss-120b-cloud 4.89/8 61.2% 5.75/8 4.04/8 1.9 1.54/2 27274 $0.0970 56 7 nemotron-3-super 4.64/8 58.0% 5.27/8 4.02/8 1.4 1.36/2 28152 $0.0944 56 8 deepseek-v3 3.56/8 44.5% 3.50/8 3.62/8 0.7 0.23/2 37324 $0.0528 56
deepseek-v3
Overall: 3.56/8 (std: 1.84) | Cross-domain: 0.23/2
Timing: answer 37324ms avg | output 587 tokens avg | total answer tokens: 37,345 | total judge tokens: 178,302
By Domain
Domain Core Score Std D1 1.80/8 1.79 D10 3.75/8 2.25 D11 4.25/8 0.65 D12 3.12/8 0.85 D13 4.12/8 2.25 D2 4.50/8 1.78 D3 4.38/8 1.38 D4 1.50/8 1.78 D5 4.12/8 2.02 D6 5.33/8 1.29 D7 2.25/8 2.22 D8 2.70/8 0.76 D9 4.25/8 0.65
By Level
Level Core Score Std ADV 3.50/8 1.96 L1 3.12/8 2.00 L2 3.21/8 1.45 L3 4.42/8 1.82 L4 3.75/8 2.47
Failure Modes
Mode Description Rate FM1 Surface knowledge 48.2% FM2 Isolated knowledge 60.7% FM3 Directional confusion 21.4%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 0.00/8 0/8 0/8 0 Q-D1-L1-01 0.00/8 0/8 0/8 0 Q-D1-L2-01 3.00/8 3/8 3/8 0 Q-D1-L3-01 4.00/8 4/8 4/8 0 Q-D1-L4-01 2.00/8 2/8 2/8 0 Q-D10-ADV-01 4.00/8 4/8 4/8 0 Q-D10-L1-01 0.50/8 1/8 0/8 1 Q-D10-L2-01 5.50/8 5/8 6/8 1 Q-D10-L3-01 5.00/8 3/8 7/8 4 Q-D11-ADV-01 4.00/8 3/8 5/8 2 Q-D11-L1-01 5.00/8 5/8 5/8 0 Q-D11-L2-01 4.50/8 4/8 5/8 1 Q-D11-L3-01 3.50/8 3/8 4/8 1 Q-D12-ADV-01 3.50/8 3/8 4/8 1 Q-D12-L1-01 4.00/8 4/8 4/8 0 Q-D12-L2-01 3.00/8 2/8 4/8 2 Q-D12-L3-01 2.00/8 2/8 2/8 0 Q-D13-ADV-01 5.50/8 5/8 6/8 1 Q-D13-L1-01 4.00/8 5/8 3/8 2 Q-D13-L2-01 1.00/8 1/8 1/8 0 Q-D13-L3-01 6.00/8 6/8 6/8 0 Q-D2-ADV-01 2.50/8 3/8 2/8 1 Q-D2-L1-01 6.00/8 6/8 6/8 0 Q-D2-L2-01 3.50/8 4/8 3/8 1 Q-D2-L3-01 6.00/8 7/8 5/8 2 Q-D3-ADV-01 3.00/8 2/8 4/8 2 Q-D3-L1-01 3.50/8 4/8 3/8 1 Q-D3-L2-01 5.00/8 5/8 5/8 0 Q-D3-L3-01 6.00/8 6/8 6/8 0 Q-D4-ADV-01 0.00/8 0/8 0/8 0 Q-D4-L1-01 0.00/8 0/8 0/8 0 Q-D4-L2-01 2.50/8 2/8 3/8 1 Q-D4-L3-01 3.50/8 3/8 4/8 1 Q-D5-ADV-01 2.00/8 2/8 2/8 0 Q-D5-L1-01 5.00/8 5/8 5/8 0 Q-D5-L2-01 3.00/8 3/8 3/8 0 Q-D5-L3-01 6.50/8 7/8 6/8 1 Q-D6-ADV-01 6.00/8 5/8 7/8 2 Q-D6-ADV-02 6.50/8 6/8 7/8 1 Q-D6-L1-01 3.50/8 4/8 3/8 1 Q-D6-L2-01 4.00/8 5/8 3/8 2 Q-D6-L3-01 6.50/8 6/8 7/8 1 Q-D6-L4-01 5.50/8 6/8 5/8 1 Q-D7-ADV-01 5.00/8 5/8 5/8 0 Q-D7-L1-01 3.00/8 3/8 3/8 0 Q-D7-L2-01 0.00/8 0/8 0/8 0 Q-D7-L3-01 1.00/8 1/8 1/8 0 Q-D8-ADV-01 3.50/8 3/8 4/8 1 Q-D8-L1-01 1.50/8 2/8 1/8 1 Q-D8-L2-01 3.00/8 3/8 3/8 0 Q-D8-L2-02 3.00/8 3/8 3/8 0 Q-D8-L3-01 2.50/8 3/8 2/8 1 Q-D9-ADV-01 3.50/8 4/8 3/8 1 Q-D9-L1-01 4.50/8 4/8 5/8 1 Q-D9-L2-01 4.00/8 4/8 4/8 0 Q-D9-L3-01 5.00/8 5/8 5/8 0
gemma4-26b-local
Overall: 5.35/8 (std: 1.86) | Cross-domain: 1.41/2
Timing: answer 42952ms avg | output 1040 tokens avg | total answer tokens: 64,474 | total judge tokens: 231,519
By Domain
Domain Core Score Std D1 5.50/8 1.22 D10 5.25/8 1.04 D11 4.50/8 1.35 D12 5.12/8 2.17 D13 5.38/8 2.87 D2 4.38/8 1.49 D3 5.88/8 1.44 D4 7.00/8 1.00 D5 6.38/8 1.18 D6 5.17/8 2.36 D7 7.25/8 0.96 D8 3.50/8 2.40 D9 4.75/8 1.55
By Level
Level Core Score Std ADV 5.43/8 1.63 L1 5.08/8 2.26 L2 4.50/8 1.80 L3 6.35/8 1.41 L4 6.00/8 2.12
Failure Modes
Mode Description Rate FM1 Surface knowledge 23.2% FM2 Isolated knowledge 26.8% FM3 Directional confusion 12.5%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 5.50/8 6/8 5/8 1 Q-D1-L1-01 4.50/8 5/8 4/8 1 Q-D1-L2-01 5.50/8 7/8 4/8 3 Q-D1-L3-01 7.50/8 8/8 7/8 1 Q-D1-L4-01 4.50/8 5/8 4/8 1 Q-D10-ADV-01 4.00/8 4/8 4/8 0 Q-D10-L1-01 5.00/8 5/8 5/8 0 Q-D10-L2-01 5.50/8 7/8 4/8 3 Q-D10-L3-01 6.50/8 7/8 6/8 1 Q-D11-ADV-01 5.00/8 5/8 5/8 0 Q-D11-L1-01 5.50/8 6/8 5/8 1 Q-D11-L2-01 5.00/8 5/8 5/8 0 Q-D11-L3-01 2.50/8 2/8 3/8 1 Q-D12-ADV-01 4.00/8 3/8 5/8 2 Q-D12-L1-01 8.00/8 8/8 8/8 0 Q-D12-L2-01 3.00/8 3/8 3/8 0 Q-D12-L3-01 5.50/8 6/8 5/8 1 Q-D13-ADV-01 8.00/8 8/8 8/8 0 Q-D13-L1-01 5.00/8 5/8 5/8 0 Q-D13-L2-01 1.50/8 1/8 2/8 1 Q-D13-L3-01 7.00/8 7/8 7/8 0 Q-D2-ADV-01 4.00/8 6/8 2/8 4 Q-D2-L1-01 3.00/8 3/8 3/8 0 Q-D2-L2-01 4.00/8 5/8 3/8 2 Q-D2-L3-01 6.50/8 7/8 6/8 1 Q-D3-ADV-01 7.00/8 8/8 6/8 2 Q-D3-L1-01 5.50/8 6/8 5/8 1 Q-D3-L2-01 4.00/8 4/8 4/8 0 Q-D3-L3-01 7.00/8 8/8 6/8 2 Q-D4-ADV-01 7.50/8 8/8 7/8 1 Q-D4-L1-01 7.50/8 8/8 7/8 1 Q-D4-L2-01 5.50/8 6/8 5/8 1 Q-D4-L3-01 7.50/8 8/8 7/8 1 Q-D5-ADV-01 5.50/8 7/8 4/8 3 Q-D5-L1-01 5.50/8 7/8 4/8 3 Q-D5-L2-01 8.00/8 8/8 8/8 0 Q-D5-L3-01 6.50/8 8/8 5/8 3 Q-D6-ADV-01 5.50/8 6/8 5/8 1 Q-D6-ADV-02 2.00/8 2/8 2/8 0 Q-D6-L1-01 2.50/8 3/8 2/8 1 Q-D6-L2-01 6.50/8 8/8 5/8 3 Q-D6-L3-01 7.00/8 8/8 6/8 2 Q-D6-L4-01 7.50/8 8/8 7/8 1 Q-D7-ADV-01 7.00/8 8/8 6/8 2 Q-D7-L1-01 8.00/8 8/8 8/8 0 Q-D7-L2-01 6.00/8 8/8 4/8 4 Q-D7-L3-01 8.00/8 8/8 8/8 0 Q-D8-ADV-01 6.00/8 6/8 6/8 0 Q-D8-L1-01 0.00/8 0/8 0/8 0 Q-D8-L2-01 3.00/8 1/8 5/8 4 Q-D8-L2-02 3.00/8 2/8 4/8 2 Q-D8-L3-01 5.50/8 7/8 4/8 3 Q-D9-ADV-01 5.00/8 6/8 4/8 2 Q-D9-L1-01 6.00/8 6/8 6/8 0 Q-D9-L2-01 2.50/8 4/8 1/8 3 Q-D9-L3-01 5.50/8 6/8 5/8 1
gemma4-31b-cloud
Overall: 6.04/8 (std: 1.34) | Cross-domain: 1.43/2
Timing: answer 20528ms avg | output 957 tokens avg | total answer tokens: 58,927 | total judge tokens: 221,471
By Domain
Domain Core Score Std D1 6.30/8 1.30 D10 5.62/8 1.25 D11 5.25/8 1.26 D12 6.00/8 1.35 D13 6.38/8 1.11 D2 5.25/8 0.65 D3 6.00/8 1.58 D4 6.62/8 0.85 D5 6.50/8 1.58 D6 6.17/8 1.99 D7 6.75/8 0.65 D8 6.00/8 2.26 D9 5.62/8 0.85
By Level
Level Core Score Std ADV 5.89/8 1.20 L1 5.81/8 1.76 L2 6.04/8 1.08 L3 6.42/8 1.35 L4 6.25/8 1.77
Failure Modes
Mode Description Rate FM1 Surface knowledge 7.1% FM2 Isolated knowledge 21.4% FM3 Directional confusion 7.1%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 5.00/8 6/8 4/8 2 Q-D1-L1-01 8.00/8 8/8 8/8 0 Q-D1-L2-01 7.00/8 8/8 6/8 2 Q-D1-L3-01 6.50/8 8/8 5/8 3 Q-D1-L4-01 5.00/8 5/8 5/8 0 Q-D10-ADV-01 5.00/8 5/8 5/8 0 Q-D10-L1-01 5.00/8 5/8 5/8 0 Q-D10-L2-01 5.00/8 6/8 4/8 2 Q-D10-L3-01 7.50/8 8/8 7/8 1 Q-D11-ADV-01 5.00/8 5/8 5/8 0 Q-D11-L1-01 7.00/8 7/8 7/8 0 Q-D11-L2-01 5.00/8 5/8 5/8 0 Q-D11-L3-01 4.00/8 3/8 5/8 2 Q-D12-ADV-01 5.50/8 6/8 5/8 1 Q-D12-L1-01 8.00/8 8/8 8/8 0 Q-D12-L2-01 5.00/8 6/8 4/8 2 Q-D12-L3-01 5.50/8 6/8 5/8 1 Q-D13-ADV-01 7.50/8 7/8 8/8 1 Q-D13-L1-01 5.00/8 6/8 4/8 2 Q-D13-L2-01 6.00/8 8/8 4/8 4 Q-D13-L3-01 7.00/8 7/8 7/8 0 Q-D2-ADV-01 5.00/8 6/8 4/8 2 Q-D2-L1-01 5.50/8 6/8 5/8 1 Q-D2-L2-01 6.00/8 8/8 4/8 4 Q-D2-L3-01 4.50/8 5/8 4/8 1 Q-D3-ADV-01 4.50/8 5/8 4/8 1 Q-D3-L1-01 6.50/8 8/8 5/8 3 Q-D3-L2-01 5.00/8 6/8 4/8 2 Q-D3-L3-01 8.00/8 8/8 8/8 0 Q-D4-ADV-01 6.50/8 8/8 5/8 3 Q-D4-L1-01 7.00/8 7/8 7/8 0 Q-D4-L2-01 5.50/8 5/8 6/8 1 Q-D4-L3-01 7.50/8 8/8 7/8 1 Q-D5-ADV-01 7.50/8 8/8 7/8 1 Q-D5-L1-01 6.00/8 8/8 4/8 4 Q-D5-L2-01 8.00/8 8/8 8/8 0 Q-D5-L3-01 4.50/8 5/8 4/8 1 Q-D6-ADV-01 8.00/8 8/8 8/8 0 Q-D6-ADV-02 4.50/8 5/8 4/8 1 Q-D6-L1-01 3.00/8 4/8 2/8 2 Q-D6-L2-01 6.50/8 8/8 5/8 3 Q-D6-L3-01 7.50/8 8/8 7/8 1 Q-D6-L4-01 7.50/8 8/8 7/8 1 Q-D7-ADV-01 6.00/8 7/8 5/8 2 Q-D7-L1-01 6.50/8 8/8 5/8 3 Q-D7-L2-01 7.00/8 7/8 7/8 0 Q-D7-L3-01 7.50/8 8/8 7/8 1 Q-D8-ADV-01 7.00/8 8/8 6/8 2 Q-D8-L1-01 2.00/8 2/8 2/8 0 Q-D8-L2-01 6.50/8 6/8 7/8 1 Q-D8-L2-02 7.50/8 8/8 7/8 1 Q-D8-L3-01 7.00/8 8/8 6/8 2 Q-D9-ADV-01 5.50/8 6/8 5/8 1 Q-D9-L1-01 6.00/8 6/8 6/8 0 Q-D9-L2-01 4.50/8 5/8 4/8 1 Q-D9-L3-01 6.50/8 8/8 5/8 3
glm-5.1-cloud
Overall: 5.81/8 (std: 2.08) | Cross-domain: 1.68/2
Timing: answer 256740ms avg | output 3261 tokens avg | total answer tokens: 187,396 | total judge tokens: 250,286
By Domain
Domain Core Score Std D1 5.50/8 2.62 D10 6.75/8 1.50 D11 6.12/8 1.75 D12 6.50/8 1.73 D13 5.88/8 2.17 D2 5.38/8 1.31 D3 6.75/8 1.32 D4 6.38/8 1.31 D5 5.62/8 1.49 D6 3.67/8 3.34 D7 6.75/8 1.55 D8 5.90/8 2.86 D9 5.50/8 1.78
By Level
Level Core Score Std ADV 6.21/8 2.14 L1 5.50/8 2.41 L2 5.46/8 2.44 L3 6.19/8 1.38 L4 5.00/8 0.71
Failure Modes
Mode Description Rate FM1 Surface knowledge 10.7% FM2 Isolated knowledge 14.3% FM3 Directional confusion 12.5%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 8.00/8 8/8 8/8 0 Q-D1-L1-01 1.50/8 2/8 1/8 1 Q-D1-L2-01 7.50/8 8/8 7/8 1 Q-D1-L3-01 6.00/8 8/8 4/8 4 Q-D1-L4-01 4.50/8 5/8 4/8 1 Q-D10-ADV-01 5.00/8 5/8 5/8 0 Q-D10-L1-01 8.00/8 8/8 8/8 0 Q-D10-L2-01 8.00/8 8/8 8/8 0 Q-D10-L3-01 6.00/8 7/8 5/8 2 Q-D11-ADV-01 8.00/8 8/8 8/8 0 Q-D11-L1-01 7.00/8 7/8 7/8 0 Q-D11-L2-01 5.50/8 6/8 5/8 1 Q-D11-L3-01 4.00/8 3/8 5/8 2 Q-D12-ADV-01 7.00/8 8/8 6/8 2 Q-D12-L1-01 8.00/8 8/8 8/8 0 Q-D12-L2-01 4.00/8 4/8 4/8 0 Q-D12-L3-01 7.00/8 8/8 6/8 2 Q-D13-ADV-01 8.00/8 8/8 8/8 0 Q-D13-L1-01 5.50/8 7/8 4/8 3 Q-D13-L2-01 3.00/8 4/8 2/8 2 Q-D13-L3-01 7.00/8 7/8 7/8 0 Q-D2-ADV-01 4.50/8 7/8 2/8 5 Q-D2-L1-01 4.00/8 5/8 3/8 2 Q-D2-L2-01 6.50/8 7/8 6/8 1 Q-D2-L3-01 6.50/8 7/8 6/8 1 Q-D3-ADV-01 5.00/8 7/8 3/8 4 Q-D3-L1-01 6.50/8 8/8 5/8 3 Q-D3-L2-01 8.00/8 8/8 8/8 0 Q-D3-L3-01 7.50/8 8/8 7/8 1 Q-D4-ADV-01 6.50/8 8/8 5/8 3 Q-D4-L1-01 7.50/8 8/8 7/8 1 Q-D4-L2-01 4.50/8 5/8 4/8 1 Q-D4-L3-01 7.00/8 7/8 7/8 0 Q-D5-ADV-01 7.00/8 8/8 6/8 2 Q-D5-L1-01 6.00/8 8/8 4/8 4 Q-D5-L2-01 6.00/8 8/8 4/8 4 Q-D5-L3-01 3.50/8 5/8 2/8 3 Q-D6-ADV-01 0.00/8 0/8 0/8 0 Q-D6-ADV-02 6.00/8 8/8 4/8 4 Q-D6-L1-01 2.50/8 3/8 2/8 1 Q-D6-L2-01 0.00/8 0/8 0/8 0 Q-D6-L3-01 8.00/8 8/8 8/8 0 Q-D6-L4-01 5.50/8 8/8 3/8 5 Q-D7-ADV-01 7.50/8 8/8 7/8 1 Q-D7-L1-01 8.00/8 8/8 8/8 0 Q-D7-L2-01 4.50/8 5/8 4/8 1 Q-D7-L3-01 7.00/8 8/8 6/8 2 Q-D8-ADV-01 7.50/8 8/8 7/8 1 Q-D8-L1-01 1.50/8 2/8 1/8 1 Q-D8-L2-01 8.00/8 8/8 8/8 0 Q-D8-L2-02 8.00/8 8/8 8/8 0 Q-D8-L3-01 4.50/8 5/8 4/8 1 Q-D9-ADV-01 7.00/8 8/8 6/8 2 Q-D9-L1-01 5.50/8 6/8 5/8 1 Q-D9-L2-01 3.00/8 5/8 1/8 4 Q-D9-L3-01 6.50/8 8/8 5/8 3
gpt-oss-120b-cloud
Overall: 4.89/8 (std: 2.14) | Cross-domain: 1.54/2
Timing: answer 27274ms avg | output 3164 tokens avg | total answer tokens: 185,764 | total judge tokens: 425,756
By Domain
Domain Core Score Std D1 4.30/8 2.31 D10 4.88/8 2.59 D11 4.38/8 0.63 D12 4.50/8 2.61 D13 5.00/8 1.91 D2 4.62/8 2.69 D3 6.12/8 0.85 D4 6.38/8 1.25 D5 5.12/8 1.89 D6 4.67/8 2.11 D7 3.75/8 3.07 D8 4.60/8 3.80 D9 5.62/8 1.03
By Level
Level Core Score Std ADV 5.36/8 1.95 L1 4.08/8 2.49 L2 4.71/8 2.08 L3 5.23/8 2.15 L4 6.00/8 0.71
Failure Modes
Mode Description Rate FM1 Surface knowledge 26.8% FM2 Isolated knowledge 28.6% FM3 Directional confusion 14.3%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 1.00/8 1/8 1/8 0 Q-D1-L1-01 3.50/8 2/8 5/8 3 Q-D1-L2-01 4.00/8 5/8 3/8 2 Q-D1-L3-01 6.50/8 8/8 5/8 3 Q-D1-L4-01 6.50/8 8/8 5/8 3 Q-D10-ADV-01 6.00/8 7/8 5/8 2 Q-D10-L1-01 1.00/8 2/8 0/8 2 Q-D10-L2-01 6.50/8 8/8 5/8 3 Q-D10-L3-01 6.00/8 7/8 5/8 2 Q-D11-ADV-01 4.50/8 5/8 4/8 1 Q-D11-L1-01 4.50/8 4/8 5/8 1 Q-D11-L2-01 5.00/8 5/8 5/8 0 Q-D11-L3-01 3.50/8 4/8 3/8 1 Q-D12-ADV-01 6.50/8 8/8 5/8 3 Q-D12-L1-01 6.50/8 8/8 5/8 3 Q-D12-L2-01 4.00/8 4/8 4/8 0 Q-D12-L3-01 1.00/8 1/8 1/8 0 Q-D13-ADV-01 6.50/8 8/8 5/8 3 Q-D13-L1-01 4.50/8 5/8 4/8 1 Q-D13-L2-01 2.50/8 4/8 1/8 3 Q-D13-L3-01 6.50/8 8/8 5/8 3 Q-D2-ADV-01 5.00/8 8/8 2/8 6 Q-D2-L1-01 1.50/8 2/8 1/8 1 Q-D2-L2-01 4.00/8 6/8 2/8 4 Q-D2-L3-01 8.00/8 8/8 8/8 0 Q-D3-ADV-01 5.00/8 6/8 4/8 2 Q-D3-L1-01 6.00/8 6/8 6/8 0 Q-D3-L2-01 7.00/8 8/8 6/8 2 Q-D3-L3-01 6.50/8 8/8 5/8 3 Q-D4-ADV-01 6.50/8 8/8 5/8 3 Q-D4-L1-01 8.00/8 8/8 8/8 0 Q-D4-L2-01 5.00/8 5/8 5/8 0 Q-D4-L3-01 6.00/8 8/8 4/8 4 Q-D5-ADV-01 3.50/8 4/8 3/8 1 Q-D5-L1-01 7.00/8 8/8 6/8 2 Q-D5-L2-01 6.50/8 8/8 5/8 3 Q-D5-L3-01 3.50/8 5/8 2/8 3 Q-D6-ADV-01 6.00/8 6/8 6/8 0 Q-D6-ADV-02 2.50/8 4/8 1/8 3 Q-D6-L1-01 1.50/8 2/8 1/8 1 Q-D6-L2-01 6.50/8 8/8 5/8 3 Q-D6-L3-01 6.00/8 8/8 4/8 4 Q-D6-L4-01 5.50/8 7/8 4/8 3 Q-D7-ADV-01 8.00/8 8/8 8/8 0 Q-D7-L1-01 4.00/8 5/8 3/8 2 Q-D7-L2-01 1.50/8 3/8 0/8 3 Q-D7-L3-01 1.50/8 2/8 1/8 1 Q-D8-ADV-01 7.50/8 8/8 7/8 1 Q-D8-L1-01 0.00/8 0/8 0/8 0 Q-D8-L2-01 1.00/8 1/8 1/8 0 Q-D8-L2-02 8.00/8 8/8 8/8 0 Q-D8-L3-01 6.50/8 8/8 5/8 3 Q-D9-ADV-01 6.50/8 8/8 5/8 3 Q-D9-L1-01 5.00/8 5/8 5/8 0 Q-D9-L2-01 4.50/8 5/8 4/8 1 Q-D9-L3-01 6.50/8 8/8 5/8 3
minimax-m2.7
Overall: 5.12/8 (std: 1.94) | Cross-domain: 1.27/2
Timing: answer 87930ms avg | output 3741 tokens avg | total answer tokens: 214,821 | total judge tokens: 274,340
By Domain
Domain Core Score Std D1 5.70/8 0.91 D10 5.25/8 2.22 D11 5.25/8 2.02 D12 4.75/8 3.28 D13 5.75/8 2.02 D2 3.88/8 1.31 D3 5.62/8 2.56 D4 5.50/8 2.71 D5 5.00/8 1.47 D6 5.67/8 1.86 D7 3.75/8 1.89 D8 5.00/8 2.72 D9 5.00/8 0.00
By Level
Level Core Score Std ADV 5.82/8 1.67 L1 5.08/8 2.40 L2 4.75/8 1.71 L3 4.58/8 1.99 L4 6.50/8 0.00
Failure Modes
Mode Description Rate FM1 Surface knowledge 16.1% FM2 Isolated knowledge 19.6% FM3 Directional confusion 16.1%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 6.50/8 8/8 5/8 3 Q-D1-L1-01 5.00/8 5/8 5/8 0 Q-D1-L2-01 6.00/8 8/8 4/8 4 Q-D1-L3-01 4.50/8 5/8 4/8 1 Q-D1-L4-01 6.50/8 8/8 5/8 3 Q-D10-ADV-01 7.00/8 8/8 6/8 2 Q-D10-L1-01 2.00/8 2/8 2/8 0 Q-D10-L2-01 6.00/8 7/8 5/8 2 Q-D10-L3-01 6.00/8 7/8 5/8 2 Q-D11-ADV-01 7.00/8 8/8 6/8 2 Q-D11-L1-01 7.00/8 7/8 7/8 0 Q-D11-L2-01 3.50/8 2/8 5/8 3 Q-D11-L3-01 3.50/8 4/8 3/8 1 Q-D12-ADV-01 6.50/8 8/8 5/8 3 Q-D12-L1-01 8.00/8 8/8 8/8 0 Q-D12-L2-01 4.00/8 5/8 3/8 2 Q-D12-L3-01 0.50/8 1/8 0/8 1 Q-D13-ADV-01 7.50/8 8/8 7/8 1 Q-D13-L1-01 5.50/8 7/8 4/8 3 Q-D13-L2-01 3.00/8 4/8 2/8 2 Q-D13-L3-01 7.00/8 7/8 7/8 0 Q-D2-ADV-01 4.50/8 8/8 1/8 7 Q-D2-L1-01 2.00/8 3/8 1/8 2 Q-D2-L2-01 5.00/8 5/8 5/8 0 Q-D2-L3-01 4.00/8 5/8 3/8 2 Q-D3-ADV-01 2.00/8 1/8 3/8 2 Q-D3-L1-01 8.00/8 8/8 8/8 0 Q-D3-L2-01 6.00/8 7/8 5/8 2 Q-D3-L3-01 6.50/8 8/8 5/8 3 Q-D4-ADV-01 6.50/8 8/8 5/8 3 Q-D4-L1-01 7.50/8 8/8 7/8 1 Q-D4-L2-01 1.50/8 2/8 1/8 1 Q-D4-L3-01 6.50/8 8/8 5/8 3 Q-D5-ADV-01 3.50/8 4/8 3/8 1 Q-D5-L1-01 7.00/8 8/8 6/8 2 Q-D5-L2-01 5.00/8 5/8 5/8 0 Q-D5-L3-01 4.50/8 5/8 4/8 1 Q-D6-ADV-01 8.00/8 8/8 8/8 0 Q-D6-ADV-02 5.50/8 8/8 3/8 5 Q-D6-L1-01 2.50/8 3/8 2/8 1 Q-D6-L2-01 6.50/8 8/8 5/8 3 Q-D6-L3-01 5.00/8 6/8 4/8 2 Q-D6-L4-01 6.50/8 8/8 5/8 3 Q-D7-ADV-01 5.00/8 6/8 4/8 2 Q-D7-L1-01 5.00/8 6/8 4/8 2 Q-D7-L2-01 4.00/8 5/8 3/8 2 Q-D7-L3-01 1.00/8 1/8 1/8 0 Q-D8-ADV-01 7.00/8 8/8 6/8 2 Q-D8-L1-01 1.50/8 2/8 1/8 1 Q-D8-L2-01 3.00/8 5/8 1/8 4 Q-D8-L2-02 8.00/8 8/8 8/8 0 Q-D8-L3-01 5.50/8 7/8 4/8 3 Q-D9-ADV-01 5.00/8 6/8 4/8 2 Q-D9-L1-01 5.00/8 5/8 5/8 0 Q-D9-L2-01 5.00/8 6/8 4/8 2 Q-D9-L3-01 5.00/8 5/8 5/8 0
nemotron-3-super
Overall: 4.64/8 (std: 2.45) | Cross-domain: 1.36/2
Timing: answer 28152ms avg | output 2645 tokens avg | total answer tokens: 153,526 | total judge tokens: 415,326
By Domain
Domain Core Score Std D1 3.80/8 2.97 D10 4.75/8 2.25 D11 2.62/8 2.29 D12 4.00/8 2.55 D13 6.62/8 2.43 D2 5.50/8 1.96 D3 5.62/8 2.63 D4 5.25/8 3.01 D5 4.38/8 2.46 D6 5.42/8 1.93 D7 4.25/8 2.02 D8 5.10/8 2.82 D9 2.75/8 2.63
By Level
Level Core Score Std ADV 4.57/8 2.53 L1 4.46/8 2.77 L2 5.11/8 2.26 L3 4.54/8 2.53 L4 3.75/8 2.47
Failure Modes
Mode Description Rate FM1 Surface knowledge 26.8% FM2 Isolated knowledge 32.1% FM3 Directional confusion 23.2%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 0.00/8 0/8 0/8 0 Q-D1-L1-01 3.50/8 3/8 4/8 1 Q-D1-L2-01 6.50/8 7/8 6/8 1 Q-D1-L3-01 7.00/8 8/8 6/8 2 Q-D1-L4-01 2.00/8 2/8 2/8 0 Q-D10-ADV-01 4.50/8 5/8 4/8 1 Q-D10-L1-01 2.00/8 3/8 1/8 2 Q-D10-L2-01 7.50/8 8/8 7/8 1 Q-D10-L3-01 5.00/8 5/8 5/8 0 Q-D11-ADV-01 5.00/8 5/8 5/8 0 Q-D11-L1-01 0.00/8 0/8 0/8 0 Q-D11-L2-01 1.50/8 2/8 1/8 1 Q-D11-L3-01 4.00/8 4/8 4/8 0 Q-D12-ADV-01 6.50/8 8/8 5/8 3 Q-D12-L1-01 5.00/8 5/8 5/8 0 Q-D12-L2-01 4.00/8 4/8 4/8 0 Q-D12-L3-01 0.50/8 1/8 0/8 1 Q-D13-ADV-01 7.50/8 8/8 7/8 1 Q-D13-L1-01 8.00/8 8/8 8/8 0 Q-D13-L2-01 3.00/8 4/8 2/8 2 Q-D13-L3-01 8.00/8 8/8 8/8 0 Q-D2-ADV-01 6.00/8 8/8 4/8 4 Q-D2-L1-01 8.00/8 8/8 8/8 0 Q-D2-L2-01 3.50/8 5/8 2/8 3 Q-D2-L3-01 4.50/8 5/8 4/8 1 Q-D3-ADV-01 2.00/8 2/8 2/8 0 Q-D3-L1-01 5.50/8 6/8 5/8 1 Q-D3-L2-01 8.00/8 8/8 8/8 0 Q-D3-L3-01 7.00/8 8/8 6/8 2 Q-D4-ADV-01 6.50/8 8/8 5/8 3 Q-D4-L1-01 8.00/8 8/8 8/8 0 Q-D4-L2-01 1.00/8 2/8 0/8 2 Q-D4-L3-01 5.50/8 7/8 4/8 3 Q-D5-ADV-01 2.00/8 4/8 0/8 4 Q-D5-L1-01 6.50/8 7/8 6/8 1 Q-D5-L2-01 6.50/8 8/8 5/8 3 Q-D5-L3-01 2.50/8 3/8 2/8 1 Q-D6-ADV-01 7.00/8 8/8 6/8 2 Q-D6-ADV-02 5.50/8 8/8 3/8 5 Q-D6-L1-01 2.00/8 2/8 2/8 0 Q-D6-L2-01 5.00/8 5/8 5/8 0 Q-D6-L3-01 7.50/8 8/8 7/8 1 Q-D6-L4-01 5.50/8 7/8 4/8 3 Q-D7-ADV-01 6.00/8 8/8 4/8 4 Q-D7-L1-01 4.00/8 4/8 4/8 0 Q-D7-L2-01 5.50/8 7/8 4/8 3 Q-D7-L3-01 1.50/8 2/8 1/8 1 Q-D8-ADV-01 5.50/8 6/8 5/8 1 Q-D8-L1-01 0.50/8 1/8 0/8 1 Q-D8-L2-01 6.50/8 8/8 5/8 3 Q-D8-L2-02 8.00/8 8/8 8/8 0 Q-D8-L3-01 5.00/8 7/8 3/8 4 Q-D9-ADV-01 0.00/8 0/8 0/8 0 Q-D9-L1-01 5.00/8 6/8 4/8 2 Q-D9-L2-01 5.00/8 5/8 5/8 0 Q-D9-L3-01 1.00/8 0/8 2/8 2
qwen3.5-397b-cloud
Overall: 5.93/8 (std: 1.86) | Cross-domain: 1.75/2
Timing: answer 151316ms avg | output 5751 tokens avg | total answer tokens: 327,469 | total judge tokens: 319,351
By Domain
Domain Core Score Std D1 7.00/8 1.17 D10 6.38/8 1.18 D11 3.62/8 2.14 D12 6.38/8 1.65 D13 6.38/8 2.93 D2 5.00/8 1.08 D3 7.00/8 1.22 D4 7.25/8 1.19 D5 5.75/8 1.66 D6 5.83/8 2.32 D7 6.12/8 2.39 D8 5.10/8 1.98 D9 5.25/8 1.04
By Level
Level Core Score Std ADV 6.00/8 1.70 L1 5.92/8 1.98 L2 5.54/8 1.94 L3 6.15/8 2.09 L4 6.75/8 1.06
Failure Modes
Mode Description Rate FM1 Surface knowledge 21.4% FM2 Isolated knowledge 17.9% FM3 Directional confusion 10.7%
Per Question
Question Core Score claude-opus-4-6 gpt-5.4 Spread Q-D1-ADV-01 5.00/8 5/8 5/8 0 Q-D1-L1-01 7.00/8 7/8 7/8 0 Q-D1-L2-01 7.50/8 8/8 7/8 1 Q-D1-L3-01 8.00/8 8/8 8/8 0 Q-D1-L4-01 7.50/8 8/8 7/8 1 Q-D10-ADV-01 5.50/8 6/8 5/8 1 Q-D10-L1-01 5.50/8 6/8 5/8 1 Q-D10-L2-01 6.50/8 8/8 5/8 3 Q-D10-L3-01 8.00/8 8/8 8/8 0 Q-D11-ADV-01 4.00/8 5/8 3/8 2 Q-D11-L1-01 6.50/8 6/8 7/8 1 Q-D11-L2-01 2.00/8 2/8 2/8 0 Q-D11-L3-01 2.00/8 2/8 2/8 0 Q-D12-ADV-01 7.50/8 8/8 7/8 1 Q-D12-L1-01 8.00/8 8/8 8/8 0 Q-D12-L2-01 4.50/8 5/8 4/8 1 Q-D12-L3-01 5.50/8 6/8 5/8 1 Q-D13-ADV-01 7.50/8 8/8 7/8 1 Q-D13-L1-01 8.00/8 8/8 8/8 0 Q-D13-L2-01 2.00/8 2/8 2/8 0 Q-D13-L3-01 8.00/8 8/8 8/8 0 Q-D2-ADV-01 4.50/8 7/8 2/8 5 Q-D2-L1-01 4.00/8 5/8 3/8 2 Q-D2-L2-01 5.00/8 5/8 5/8 0 Q-D2-L3-01 6.50/8 8/8 5/8 3 Q-D3-ADV-01 6.50/8 8/8 5/8 3 Q-D3-L1-01 5.50/8 7/8 4/8 3 Q-D3-L2-01 8.00/8 8/8 8/8 0 Q-D3-L3-01 8.00/8 8/8 8/8 0 Q-D4-ADV-01 8.00/8 8/8 8/8 0 Q-D4-L1-01 8.00/8 8/8 8/8 0 Q-D4-L2-01 5.50/8 6/8 5/8 1 Q-D4-L3-01 7.50/8 8/8 7/8 1 Q-D5-ADV-01 4.50/8 4/8 5/8 1 Q-D5-L1-01 6.00/8 8/8 4/8 4 Q-D5-L2-01 8.00/8 8/8 8/8 0 Q-D5-L3-01 4.50/8 5/8 4/8 1 Q-D6-ADV-01 8.00/8 8/8 8/8 0 Q-D6-ADV-02 3.00/8 4/8 2/8 2 Q-D6-L1-01 3.00/8 4/8 2/8 2 Q-D6-L2-01 7.00/8 8/8 6/8 2 Q-D6-L3-01 8.00/8 8/8 8/8 0 Q-D6-L4-01 6.00/8 8/8 4/8 4 Q-D7-ADV-01 8.00/8 8/8 8/8 0 Q-D7-L1-01 8.00/8 8/8 8/8 0 Q-D7-L2-01 5.50/8 7/8 4/8 3 Q-D7-L3-01 3.00/8 3/8 3/8 0 Q-D8-ADV-01 7.00/8 8/8 6/8 2 Q-D8-L1-01 2.00/8 2/8 2/8 0 Q-D8-L2-01 6.50/8 8/8 5/8 3 Q-D8-L2-02 5.50/8 7/8 4/8 3 Q-D8-L3-01 4.50/8 5/8 4/8 1 Q-D9-ADV-01 5.00/8 6/8 4/8 2 Q-D9-L1-01 5.50/8 6/8 5/8 1 Q-D9-L2-01 4.00/8 6/8 2/8 4 Q-D9-L3-01 6.50/8 8/8 5/8 3
Analysis
Domain Difficulty (lower = harder)
Domain Mean Score D11 4.50/8 D8 4.74/8 D2 4.81/8 D9 4.84/8 D1 4.99/8 D12 5.05/8 D7 5.11/8 D6 5.24/8 D10 5.33/8 D5 5.36/8 D13 5.69/8 D4 5.73/8 D3 5.92/8
Level Difficulty
Level Mean Score L4 5.50/8 L3 5.49/8 ADV 5.35/8 L2 4.92/8 L1 4.88/8
Failure Mode Prevalence (all models)
Mode Rate FM1 22.5% FM2 27.7% FM3 14.7%
Inter-Judge Agreement
Judges: claude-opus-4-6, gpt-5.4
Metric Value Mean core_total spread 1.37 Exact agreement rate 64.1% Perfect agreement (spread=0) 33.0% High disagreement (spread>=3) 21.9%