On this page
We run a data-analysis agent in production for our customers. It answers questions like "which deals are stuck in stage three?". Instead of writing SQL from scratch, it calls tools on an MCP server for data. Those tools read a semantic model, the shared definition of each metric.
We started the agent on Gemini and kept it there for a long time. We looked at Claude, but its per-token prices were higher, and with no way to measure quality we had no reason to pay more. That changed once we had evals: automated checks that score the agent's answer to a fixed question. For the first time we could compare models on how well they do the job, not on the price list. So we ran the same agent on Claude to see. The result was not what we expected.
We tested Claude vs Gemini for AI agents the only way that settles it. Same agent, same tools, same evals, 99 turns per model, on 26 October 2026. Short version: for this workload Claude was more accurate, about 3x faster at p90 (the time nine turns in ten finish within), and about half the cost per turn. This post is just the results.
The setup#
- One agent, two variants. Same code, same tool set, same instructions. Each variant pairs a small model with a bigger one. The small model handles routing (the short steps that decide what the agent does next) and simple turns; the bigger model handles the rest. Gemini: 3.5 Flash Lite + 3.8 Flash. Claude: Haiku 5.5 + Sonnet 5.5. Both ran with prompt caching, which lets the model reuse the unchanged part of a long conversation instead of re-reading it at full price. The Gemini pair was our production config going into the test.
- Three eval suites, three runs each per variant. The eval system is still a proof of concept. The suites are the customer's own agent evals, a set of shared evals we run on our deployments, and questions real users had asked. A turn is one question through to the agent's final answer, and it usually takes several model calls. Three runs of every suite came to 99 turns per model.
- Writes stubbed. No run touched production data.
- Real prices. Cost comes from our LLM gateway's spend logs at real Vertex AI EU prices.
Is Claude better than Gemini for AI agents? The results#
For this agent, yes. Claude passed 95% of the customer evals to Gemini's 82%, ran at 48 s versus 161 s at p90, and cost $0.067 versus $0.139 per turn.
| Metric, 99 turns per variant | Gemini (3.5 Flash Lite + 3.8 Flash) | Claude (Haiku 5.5 + Sonnet 5.5) |
|---|---|---|
| Customer evals passed, 3 runs | 84% / 79% / 84% (avg 82%) | 95% / 95% / 95% (avg 95%) |
| Shared evals passed, 3 runs | 83% / 83% / 83% (avg 83%) | 83% / 100% / 100% (avg 94%) |
| Real user questions answered, 3 runs | 100% / 100% / 100% | 88% / 100% / 100% (avg 96%) |
| Flaky evals (pass in some runs, fail in others) | 6 | 2 |
| Turn latency, median / p90 | 14 s / 161 s | 16 s / 48 s |
| Customer suite wall time (start to finish) | 4.7 to 10.2 min | 1.0 to 1.5 min |
| Model calls per turn (requests to the model) | 16.6 | 7.0 |
| Input tokens per turn (share served from cache) | 514K (86%) | 140K (77%) |
| Cost per turn | $0.139 | $0.067 |
| Cost per turn without caching | $0.536 | $0.175 |
| Total, 99 turns | $13.72 | $6.62 |
Three of the real user questions, side by side. Here we count tool calls: each time the model asks the agent to run a tool, such as a metric query.
| Question | Gemini | Claude |
|---|---|---|
| Customer deals, all stages | Correct, 94 s | Correct, 47 s; separates linked from name-only matches |
| Revenue last quarter | Correct, 31 s | Correct, 12 s |
| Explain a deal's progression | Right stage, then speculates; 209 s, 40 tool calls | Right stage, then asks what is meant; 51 s, 14 tool calls |
Where the gap came from: accuracy, speed, cost#
Accuracy. Gemini missed the same two checks in every run and was flaky on revenue versus order intake. Those are the exact mistakes users had reported before the test, so the evals are catching real behavior. Claude's one steady failure is an eval whose pattern match is too narrow; the answer itself is fine.
Speed. The median turn is close (14 s versus 16 s). The tail is not. Gemini loops: 2.4x more model calls per turn, most of them tiny Flash Lite routing steps that go round again. On top of that, a single Gemini 3.8 Flash call took about 2.6x longer than a Sonnet 5.5 call in our traces. The "explain a deal's progression" row shows both effects at once: 40 tool calls and 209 s to reach the stage Claude identified in 14 tool calls and 51 s.
Cost. Gemini is cheaper per token. It is twice as expensive per turn because it reads 3.7x more input per turn, and every extra model call re-reads the whole conversation. Caching helps both: it saved Claude about 60% and Gemini about 74%. Even so, Claude without caching ($0.175) came in not far above Gemini with caching ($0.139), and far under Gemini without it ($0.536). For an agent, cost per token is the wrong unit. Cost per finished turn is what shows up on the invoice. It is the same lesson we drew about token capital: the budget buys answers, not tokens.
No mix wins. The cost breakdown rules out a Gemini-for-routing, Claude-for-reasoning split. Haiku 5.5 is already cheaper than Gemini 3.5 Flash Lite and makes up only 6% of the Claude variant's cost. Sonnet 5.5 is the other 94%, and caching is already on. There is no cheaper piece to swap in.
Caveats#
- Claude is not spotless. In one run it sent a read query through the write tool. Writes were stubbed, so nothing happened, but we are adding a line to the tool descriptions and watching for it in production.
- Two customer evals are too strict. Their pattern matches reject answers that are correct. One of them is Claude's single steady failure, so Claude's pass rate is a little lower than its answers deserve. Both models were scored against the same checks, so the comparison holds. We are widening them.
- This is one workload. A data agent that makes many tool calls against a semantic model is not every agentic workload, and the ratios will differ elsewhere. We have argued that the harness matters more than the model. On this harness the model still moved eval pass rates by 11 to 13 points, p90 latency by 3x, and cost by 2x.
What we are changing in production#
We are switching the agent to Claude by default: Haiku 5.5 for routing and simple turns, Sonnet 5.5 for the rest, caching on. We expected to confirm the cheaper models and move on. Instead, the smaller Claude models made fewer mistakes and needed fewer tokens to finish a task, so they were cheaper too. The test turned "we assume this is right" into numbers we can show a customer who asks why. If you are choosing a model for an agent of your own, copy the shape of the test rather than the winner: the same agent, the evals you already trust, real prices, and cost and latency per turn.
Frequently asked questions
- Is Claude better than Gemini for AI agents?
- For our agentic workload, yes. On a production data-analysis agent, Claude (Haiku 5.5 for routing, Sonnet 5.5 for the rest) was more accurate, about 3x faster at p90, and about half the cost per turn compared with Gemini 3.5 Flash Lite + 3.8 Flash. Much of the speed and cost gap came from agent behavior: Gemini looped through 2.4x more model calls per turn. Other workloads will differ, so run your own evals.
- Is Gemini cheaper than Claude for an AI agent?
- Per token, yes. Per turn, no, at least in our test. An agent's cost is tokens per turn times price per token, and Gemini used 3.7x more input tokens per turn (514K versus 140K). It made 16.6 model calls per turn against Claude's 7.0, and each call re-reads the growing context. Per-token price is the wrong unit for an agent; cost per completed turn is what you pay.
- How much does prompt caching save on an agentic workload?
- On our Claude variant, caching cut cost per turn from $0.175 to $0.067, about 60%. On the Gemini variant it cut cost from $0.536 to $0.139. Caching matters more for agents than for single prompts because every tool call re-sends the conversation so far, and most of it is unchanged.
- Would a mix of Gemini and Claude models be cheaper?
- Not in our setup. Haiku 5.5 handled routing and simple turns for only 6% of Claude's total cost, and it was already cheaper than Gemini Flash Lite. Sonnet 5.5 made up the other 94%, and caching was already on. There was no cheaper piece to swap in.
- Is Gemini better than Claude for AI agents?
- Not in our test, but it did three things well. Its median turn was slightly faster than Claude's (14 s versus 16 s). It answered every real user question in every run, where Claude missed one once. And its per-token price is lower, which is why we ran on it in the first place. Its problems were in the tail: looping tool calls that pushed p90 latency to 161 s, and two checks it missed in every run.
- How did you measure accuracy for the agent?
- With a proof-of-concept eval system: three suites, each run three times per model. One is the customer's own agent evals, one is a set of shared evals we run on our deployments, and one is built from real user questions. Writes were stubbed so no run touched production data. Cost came from our LLM gateway's spend logs at real Vertex AI EU prices.
Read more about revenue operations, growth strategies, and metrics in our blog and follow us on LinkedIn and Youtube.
All articles