Are You Using the Best Models for Your Agentic SDLC?

Picking a model on just a single factor (retail price, benchmark scores, industry hype) rarely predicts the true cost or quality of agentic development. I had a project that required an agentic development lifecycle and some free cycles, so the team ran an experiment: the same starting package (architecture and technical design docs, a product brief, an initial backlog, UX designs and mockups) through five different model combinations in GitHub Copilot. Each run planned its own work from the backlog, implemented every task it created, and tracked its consumption in AI Credits (AICs), GitHub's billing unit for Copilot usage.

All five runs completed everything they planned, yet the bill spanned 27x from cheapest to most expensive, and the model that benchmark sites measure as one of the cheapest per task produced the most expensive run of the exercise. The surprise, however, was that the combination I initially picked as the leader based on cost alone finished below the runner-up when factoring in code quality. And the most expensive, top-tier model didn't even produce the best output.

Quick findings

• Cost spread. Five model combinations ran the same exercise, and all five completed everything they planned. The bill ranged from 304 to 8,185 AICs: a 27x spread, or 6.4x excluding the Auto experiment. Same work, same definition of done, wildly different cost.

• Benchmarks measure their scenario, not yours. The model Artificial Analysis measures as one of the cheapest per task ($0.43) produced the most expensive run of the exercise. Benchmarks answer generalized scenarios; your prompts, context, and workflow are specific. Prove out cost and quality yourself, in your scenario.

• The cost leader finished below the runner-up on quality. The most cost-efficient serious run (GPT-5.6 Sol + Terra, 1,283 AICs) looked like the winner on cost and visual output, but code quality measures scored it below the runner-up (Grok 4.5, 2,277 AICs): 18 of 35 versus 27 of 35. That code quality could come back later in re-work and maintenance costs.

• Auto trades control for a discount. The Auto run was the cheapest and fastest of the exercise (304 AICs, about 14 minutes) and produced the weakest output by far, confirmed by both the review scores and the linter (14 errors where every other run had zero). Auto promises the lowest cost plus a 10% discount, but you give up all control and blindly trust the vendor and their router. The 10% discount likely costs you 10% more.

The setup

Every run started from the same package: architecture and technical design documents, a product brief, an initial backlog, and UX designs with mockups. Each run worked from a fresh branch, and issue tracking lived in local files so the tracking system itself stayed constant across runs.

Each run opened with a grilling exercise, where the agent interrogates the package to ground its understanding before writing anything. The architecture grilling had already happened ahead of this exercise, so cross-cutting concerns like concurrency, testing standards, and platform choices were generally resolved. What remained was product and feature understanding, plus the technical details specific to each feature: field lengths, validation rules on a form, that class of question. From the grilling, each run wrote a spec and broke the work into implementation tasks. This was all done in one session to maintain context fidelity and efficiency. That closed the planning phase.

Implementation meant working through the run's own task list until every task was marked complete; that was the definition of done for every run. Models could be switched between phases or mid-implementation, which is where the combinations come in. We tracked consumption in AICs throughout, recorded by hand or self-reported by the harness at the end of each session.

The team split the runs between us: the Grok and mixed-model runs, Opus 5, GPT Sol + Terra, and Auto.

The contenders

Five combinations made it into the exercise. Reasoning effort is listed per model because it moves both the bill and the output.

• The mixed run was supposed to be Opus planning with Sonnet implementation. Partway through I accidentally switched to Gemini 3.5 Flash and didn't notice for several sessions, which turned it into an accidental three-model comparison. Whoops.

• The Grok run earned its slot: Grok 4.5 had already put up Opus-class results at a fraction of the price in my own Ship Bench comparison, so it ran single-model at high effort for everything.

• The Opus 5 run put the flagship at its cheapest setting: low effort, with the 1M context window.

• The GPT run chased the Artificial Analysis cost-per-task numbers and the industry hype around GPT token efficiency: Sol for planning, Terra for implementation. The efficiency hype held up, at least on the bill.

• The Auto run handed model selection to the vendor. Auto promises the lowest cost and a 10% discount; we wanted to see what that promise actually delivers.

The runs

The table first, then the color. Costs are split by phase; wall clock is as captured, with one honest gap. After the runs, I had every output reviewed: static analysis with a pinned toolchain, plus an independent reviewer pass scoring seven quality dimensions out of 35, scoped to what the exercise required.

Auto ran last and carries the least weight, so it anchors the table despite being the cheapest row. The Mixed run's wall clock wasn't captured.

GPT Sol + Terra (1,283 AICs, about 47 minutes). The cheapest serious run, and the fastest of them by a wide margin; the GPT models were noticeably quick throughout. It looked like the winner early: a low bill, and visual output that matched the designs once the planning instructions told it to adhere to them (before that fix, the visuals lagged Grok's). The review pass was less kind: 18 of 35, with the client logic concentrated in one large component and thin architecture overall. Cheap, fast, pretty. The code needs more discipline, the kind our SDLC would normally enforce by putting quality tooling early in the process.

Grok (2,277 AICs, about four hours). The runner-up on cost and the quality leader: 27 of 35 on the review, with the cleanest architecture of the five. It adhered the strongest to the visual designs, especially once the updated planning instructions were in place. Two caveats on the experience. It felt slow: tokens trickled compared to running the same model through Grok Build, though throughput numbers like these are easily skewed by provider capacity and can vary wildly. And it exposed a process gap when I forgot to start Docker: agents should confirm their tooling and environment before starting. Its first move was writing its own monorepo launch script, which set the tone for a run that followed instructions incredibly well.

Mixed (4,378 AICs). The accidental three-model run. Opus planning was the most expensive planning of the exercise (622 AICs; the grilling alone cost 350, against Grok's 175 for a longer conversation). In implementation, the Sonnet sessions averaged 789 AICs against Gemini Flash's 348, though sessions aren't equal units of work and effort settings differed, so read that as directional. The run's real failure was process: the tickets never linked back to the UI designs, the visuals drifted, and a separate 783-AIC review pass (not counted above) was needed to bring it back. It also accumulated the most dependency surface, showing up as 11 vulnerable npm packages at scan time where every other run carried little beyond the one CVE they all inherited. Mid-pack on review at 24 of 35, on the biggest codebase of the five.

Opus 5 (8,185 AICs, about 127 minutes). The most expensive run by nearly double, and the one the benchmark tables missed hardest: Artificial Analysis measures Opus 5 at low effort as one of the cheapest models per task ($0.43, despite its higher retail price), largely on output efficiency. In a real agentic workflow with a 1M context window, it burned more than any other run; the scaffold ticket alone cost 1,855 AICs. It also shipped the only build that doesn't restore end to end: its own warnings-as-errors policy escalated a dependency vulnerability warning into a build error, and no one ran a full solution build to catch it. Discounting the build and restore issues, the review score tied Grok's at 27 of 35, and it came closest to the designs without any instruction help, which says something about its reasoning. But the build has to work.

Auto (304 AICs, about 14 minutes). Cheapest and fastest, and weakest by far: generic browser dialogs for delete confirmations, 14 lint errors where every other run had zero, 17 of 35 on the review. It completed everything it planned, which is worth saying. What it produced just isn't anything you'd ship.

What the benchmarks said

For an external reference point, here is what Artificial Analysis measured for these models (Intelligence Index v4.1.1, retrieved August 2026). These are API list-price benchmarks, not Copilot credits; the units don't convert, which is part of the point.

Benchmarked at max effort; our runs used medium. Auto has no benchmark entry, which tells you something on its own.

Two things stand out. The Opus 5 row is the exercise's reason to exist: benchmarked among the cheapest per task despite its premium retail price, it produced the most expensive run we had. And Terra, the cheapest per task on the board at $0.12, sits off the value frontier even in Artificial Analysis's own GPT-5.6 breakdown, which found Sol and Luna ahead of Terra at every effort level. Cheapest per task is not the same as best value. Benchmarks measure their scenario; the only cost and quality numbers that matter are the ones you prove out in yours.

What changes after running this

Test your own hypothesis. The benchmark tables got us to a shortlist and nothing further. If a benchmark's scenario is fixing bugs in a public repo and your scenario is building features from a design package, the numbers won't transfer. The only cost and quality figures that matter are the ones from your own process, your own prompts, your own codebase.

Discipline beats model choice. Every serious run completed everything it planned. What separated the good outcomes from the expensive ones wasn't the model card, it was the agentic SDLC around it: grounding the product understanding before writing code, instructions that actually carry the design intent, and quality tooling early enough to catch what the agent won't. With that in place, a variety of models can get you to success. The bill and the polish will vary; the outcome doesn't have to.

The archetypes rotate. Today's nicely balanced models (Grok), efficiency kings (the GPTs), and one-shot wonders (Opus 5, and really Claude Fable 5) won't be holding those titles next quarter. The lineup we ran here will be legacy by the time some readers find this article. Which is the actual takeaway: get into the habit of evaluating for yourself, because you'll be doing it again soon.

The benchmarks will get you to a shortlist. Your own runs tell you the truth.