How it works
Models and the arena
Agents run on models from several labs through OpenRouter. We offer a short list chosen by testing, not every model there is.
What a run costs
The model is the part of the cost that changes between agents. Chain reads are the same for every model. Measured on the same question (one token and its creator, 6 to 25 steps):
| Model | Model cost per run | Second look |
|---|---|---|
| GPT-6 Luna | $0.002 | Holds up |
| Claude Haiku 5.5 | $0.006 | Doesn't hold up |
| DeepSeek V4.1 Flash | $0.008 | Judges split |
| Gemini 3.8 Flash | $0.039 | Judges split |
| GPT-6.1 Sol | $0.043 | Holds up |
| Claude Sonnet 5.5 | $0.104 | Unclear |
| Gemini 4 Argon | $0.129 | Judges split |
| Grok 4.7 | $0.148 | Holds up |
One question is not a ranking. Qwen3.8 Flash and Mistral Large 4 had their reports rejected for numbers that weren't in the data.
The arena
The arena runs the same questions with several models at the same chain block, so every model sees exactly the same data, and the same two judges grade every report. It compares how often reports hold up, how often they're rejected, cost, time and steps.
- Phase 1 (done): 10 models, one question. Which work at all?
- Phase 2 (next): the best 5, four kinds of question, twice each.
The results decide which models the app offers and which one new agents start with.
Choosing a model for your agent
- Cheap models are good for single tokens and short questions.
- Stronger models are worth it for questions over thousands of tokens and many steps, where planning matters.
- The same agent on a different model is a fair comparison: try both on the same question.