Alloy Benchmark
We evaluate models on questions about real robot logs, telemetry and mission data, answered inside the Alloy agent. Higher scores are better.
Try Alloy ↗| Model | Score | Cost / question | Time / question | |
|---|---|---|---|---|
| 1 |  Claude Fable 5.1 Medium | 88.7% | $0.97 | 2.0 min |
| 2 | Claude Fable 5.1 High | 87.0% | $1.41 | 2.6 min |
| 3 |  GPT-6 Astra High | 85.8% | $1.28 | 2.4 min |
| 4 | GPT-6 Astra Medium | 84.8% | $1.08 | 1.8 min |
| 5 | GPT-6 Astra Low | 84.6% | $0.98 | 2.2 min |
| 6 | GPT-5.6 Sol Medium | 84.3% | $0.88 | 4.0 min |
| 7 | Claude Opus 5 Medium | 81.7% | $1.50 | 2.9 min |
| 8 |  Kimi K3 High | 81.7% | $0.53 | 2.6 min |
| 9 |  GLM 5.3 High | 81.5% | $0.41 | 4.1 min |
| 10 | GPT-6.1 Sol Medium | 81.3% | $0.13 | 1.4 min |
| 11 | GPT-5.6 Sol Extra High | 80.6% | $0.91 | 4.1 min |
| 12 | GLM 5.3 Flash High | 80.5% | $0.030 | 4.6 min |
| 13 | GPT-6 Sol Medium | 80.2% | $0.22 | 1.9 min |
| 14 | GPT-5.6 Sol High | 80.0% | $0.66 | 4.0 min |
| 15 | Claude Opus 5 Extra High | 80.0% | $1.78 | 4.7 min |
| 16 | Claude Fable 5.1 Low | 79.5% | $0.91 | 1.9 min |
| 17 | Claude Haiku 5.5 High | 78.7% | $0.11 | 1.1 min |
| 18 | GPT-5.6 Terra High | 76.2% | $0.18 | 1.0 min |
| 19 |  Gemini 3.7 Flash Medium | 75.8% | $2.38 | 2.7 min |
| 20 | GPT-6 Luna Medium | 75.4% | $0.011 | 1.3 min |
| 21 | GPT-5.6 Terra Max | 75.2% | $0.36 | 2.1 min |
| 22 | GPT-5.6 Terra Extra High | 75.2% | $0.26 | 1.3 min |
| 23 | Claude Haiku 5.5 Medium | 73.9% | $0.092 | 1.0 min |
| 24 | GPT-6 Luna High | 73.9% | $0.012 | 1.4 min |
| 25 | GPT-5.6 Terra Low | 73.8% | $0.17 | 0.9 min |
| 26 | GPT-5.6 Luna Medium | 73.8% | $0.029 | 1.5 min |
| 27 | GPT-6 Luna Extra High | 72.3% | $0.014 | 1.5 min |
| 28 | GPT-5.6 Terra Medium | 72.1% | $0.18 | 1.0 min |
| 29 | Claude Haiku 5.5 Low | 68.1% | $0.082 | 0.8 min |
| 30 | GPT-6 Luna None | 66.8% | $0.014 | 1.2 min |
| 31 | GPT-6 Luna Low | 66.2% | $0.008 | 0.8 min |
| 32 | GPT-5.6 Luna Low | 65.5% | $0.017 | 1.0 min |
Choose by preference
Balance evaluated quality, cost, and speed to find the model that suits your needs. Each model is shown at its best-scoring setting.
What matters most to you?
Questions are representative of the work robotics teams we partner with do in the field: motion, geolocation, fleet-wide coordination, and cross-mission and cross-fleet analysis. Every question is weighted equally. Every model runs inside the same Alloy agent with the same tools and data, and each answer is graded against a rubric written for its question. Avg cost / question is the recorded agent spend on model calls. Time / question is the median time to a finished answer. Results are subject to variance; small differences in scores may not be statistically meaningful.