GPT-6 Astra vs Claude vs Llama: We Tested the New Model on Real Work

GPT-6 Astra appeared on OpenRouter on the 4th of September. Days later it is the thing every AI forum is arguing about, mostly on the strength of benchmark tables and vibes.

We had a more specific question. We run a stack where different models do different jobs: one plans and judges, another writes code, a third reviews it because it fails differently. So: does GPT-6 Astra beat Claude? Does it beat Llama? And where, if anywhere, does it belong?

We tested it the same day, on our own work, for less than two pounds. Here is the whole thing including the parts that make us look silly.

The test

Three briefs, each a real job our stack does daily rather than a puzzle:

1. Planning. Hand a builder agent a customer-facing feature, given a stack that has caught builders reporting work they never did. Decompose it, say what gets written before any code, and say how you will know each part is genuinely done.

2. Idea generation and self-critique. A real conversion problem from a real product, with a real constraint (no headcount, no dishonest urgency). Produce three approaches that differ in mechanism, then attack your own three and say what evidence would settle it.

3. Design system. Build the visual system for a landing page aimed at buyers who have been burned by AI tools that overpromise. Palette with roles, type pairing, rhythm, the one engineered peak, and three things you refuse to do because they read as AI-generated.

Three models got identical briefs: GPT-6 Astra at high reasoning effort, Claude Fable 5.1, and Llama 4 Maverick, Meta's strongest generally available model at the time of writing.

Two rules made it a test rather than a demo. Blind: the three answers were shuffled into candidates A, B and C, and the mapping was written to a file nobody read until scoring finished. A fourth lineage judged: a different vendor's model did the scoring against a fixed rubric, because a model grading its own family is not a review. We are not neutral about Claude, which is exactly why we did not let ourselves hold the pen.

The result

The same order on every task, with no ties.

Average rubric scores out of 10, judged blind:

Three points to note before anyone quotes the table.

Astra is close on two of three, and level on design. On the design brief it tied on average and lost on the ranking. This is not a rout.

Planning is where the gap is real. 10.0 against 6.3 is a different order of answer, not a worse one. The judge's reason: the winner gave "a highly specific, code-level blueprint with explicit sabotage-testing and falsifiability criteria rather than generic process instructions."

Llama 4 was not close. On design the judge singled it out for "glowing teal accents and an animation of turning gears, embodying the exact luminous-AI and theatre-of-productivity clichés that the sceptical target buyer has already been burned by." The brief explicitly asked for three things it would refuse to do because they read as AI-generated. It proposed two of them.

Does reasoning effort fix it?

We ran Astra twice, because our first pass used the low reasoning setting our code assistant defaults to, and that would have been an unfair headline.

At low effort it produced no reasoning tokens at all. At high effort it produced 1,500 to 2,000 per task and its planning answer grew by a third. The score moved from 7.5 to 8.0 out of 10 on a straight two-way comparison against Claude's 9.5.

So: more reasoning made it better, and did not make it win. Worth knowing before anyone assumes the default setting is the story. Every number in the tables above is the high-effort run.

What Astra is genuinely better at

This is the part a ranking table hides, and it is the reason Astra now has a job here.

Across both rounds, in different words each time, the judge credited Astra with catching data-layer failure modes that the winner missed. In one round it warned that a transactional rollback inside a database mutation would erase the very warning state the feature exists to store. In another it named the split between transactional writes and network calls in our backend framework, which is precisely the mistake that sends a builder down a wrong path for an afternoon.

Both are real bugs our stack has actually hit. Neither appeared in the answer that won.

On the idea task, its sharpest line was self-directed: that habitual use of the free feature might be "procrastination dressed as marketing" — users generating output without ever posting it. That is a better piece of self-criticism than the winner produced, and it is the kind of observation that changes a roadmap.

Its consistent weakness is the mirror image. It stays terse where the brief rewards specificity: 874 words against 1,172 on the design task, and the rubric repeatedly rewarded exact named values, real timestamps and falsifiable instructions over correct but general prose. Correct and general loses to correct and concrete, every time, in work that someone has to act on.

Where it sits in our stack

Our stack already separates the model that plans from the model that builds from the model that reviews, on the principle that a single model is a poor judge of its own work. This test moved Astra into one of those seats, and not the one it was auditioned for.

Not the planner. It lost that job on the widest margin, and planning is where a bad call costs the most downstream.

Yes, the reviewer. Its repeatable edge is exactly what a review should catch: concurrency, rollback and data-layer mistakes that a plan-shaped mind glosses over. Our cross-lineage review gate already routes Claude-built work to a different vendor's model, so Astra became the analyst rather than the architect. We also raised its reasoning effort for review specifically, per-run rather than globally, since a builder wants speed and a reviewer wants thought.

And the cost answer surprised us. Priced through an API, Astra is 10 dollars per million tokens in and 50 out. That sounds fine until you price the planner's job honestly: we measured our own long sessions at 1.7 to 2.7 billion cached input tokens, because every turn re-reads the whole history. One session, billed that way, comes to $2,377 at the standard rate and $4,672 once it crosses the long-context threshold, which our sessions do constantly.

It runs on a subscription instead. Our code assistant is signed in with subscription credentials and Astra was already its configured model, so in a terminal pane it is a fixed cost, exactly like the Claude sessions beside it. Which model you can afford to think with depends less on its price per token than on how you are billed for it — and we nearly got that backwards in public.

What this cost, and what it does not prove

The entire exercise: $1.67 of model calls and about $0.05 of judging. Llama's three answers cost two tenths of a cent in total, which is its own kind of answer.

The honest limits, stated because a table without them is marketing:

• Three tasks, one judge, one day. It steers our routing. It is not a benchmark paper, and one rubric has one set of blind spots.
• One vendor per lineage. We tested Llama 4 Maverick, not every open-weights model, and a Chinese open-weights model does well in our review panel where Llama did not do well here.
• Our briefs favour our stack. They ask for the things we have learned to need. A team with different failure modes would write different briefs and could get a different order.
• Days old. Models get better in service, and reasoning-mode variants exist that we did not separately test.

What we would say plainly: GPT-6 Astra clearly beats Llama 4 on this work, does not beat Claude on planning, matches it on design, and is worth a seat as a reviewer rather than a planner. If you already pay for it through a subscription, that seat is free, and it is the most useful free thing we have added this month.

The method here is the same one we use on ourselves, and it matters more than the result: give the models real work, hide the labels, and let something else do the scoring. Our full approach to proving agent work is genuinely done is in How We Verify Agent Work, and the earlier study that set the pattern is Which Claude Should Write Code.