Everyone Benchmarks AI Coding. Nobody Checks the Code. So We Made 9 Engines Build a Real App.

Series: Vibe Coding on Your Own Terms
Related: "Free Has a Ceiling", "Lovable vs. a $0 Alternative" and Post #1 — "Lovable Alternative: Free Vibe Coding with VSCode, Cline, Ollama & Supabase"

The problem with AI coding benchmarks

There are more AI coding benchmarks than ever, and they all measure the same strange thing: puzzle-solving. Pass@1 rates on algorithm snippets. Arena votes on chat answers. Patch rates on curated GitHub issues.

None of them measure the thing people actually use these models for in 2026: "build me a complete, working product."

Two blind spots in particular kept bothering me:

1. Nobody opens the app afterwards and clicks through every feature like a real user would.
2. Nobody reads the code the model wrote — the code you'll be living with, editing, and paying tokens on for the rest of the app's life.

So I built my own benchmark around a real deliverable, and graded both the product and the code.

The test

The task: a small but fully functional project management app. Dashboard with live progress stats, project CRUD with confirmation dialogs, a 3-column kanban board, milestones with overdue detection, a timeline view, global search, priority filters, full persistence, polished and mobile-ready down to 375px. A real product, not a toy.

The rules: every engine got the identical ~2,600-character prompt, verbatim, ONE SHOT. No follow-ups, no fixes, no mercy. Whatever state the first attempt shipped in — that's what got graded. (A local pipeline feeds the same prompt to each model and records everything automatically; how that works is its own story, for another day.)

The field: one model per provider family, plus Lovable as the commercial control.

Why exactly these nine? Fair question — where's Claude, GPT, Gemini?

The budget-builder constraint is the point of this whole blog. I'm documenting how far you can get building Lovable-class apps on a hobbyist budget — and all eight LLMs above are available under one $9.99/month flat-rate subscription. They're precisely the models a budget builder can hammer all day. Premium Western APIs answer a different question ("what does the best possible model produce?"); this benchmark answers "what does the $10 tier deliver?" — with Lovable as the commercial ceiling to beat.

One model per family keeps the comparison about different brains, not price tiers of the same brain. I made two deliberate exceptions: Moonshot got both its flagship (K3) and its code-tuned specialist (K2.7 Code) — a built-in experiment on whether "code-tuned" means anything. And DeepSeek got both Pro and Flash — the biggest price gap inside one family ($1.74 vs $0.14 per million input tokens), testing whether the cheap tier is a trap or a bargain.

And Lovable is a control, not a contestant: it's not a model, it's a product wrapping a model — the reference point for what the polished commercial experience delivers.

What we measured — five dimensions:

1. Delivery — did a working app come out, first try?
2. Acceptance — a human clicks through a 10-point checklist covering every spec feature, on desktop and at 375px. Score out of 10.
3. Code quality — static analysis of the model-written code only (template boilerplate excluded): compile gate, size, largest file, duplication, any-types, suppressed warnings, leftover debug code. Score out of 10.
4. Cost — reference cost at official per-token rates, with input, output, and cached tokens priced separately.
5. Consistency — every engine ran the full build twice, hours apart.

How to read "Best cost"

It's the app's delivery price tag: what it costs to go from pasting the prompt to a delivered app, counted as pure AI-inference cost — every token the model read and wrote during the build, priced at official pay-per-use rates. "Best" = the cheaper of each engine's two runs.

Three things to keep straight:

For the LLMs it's a reference price, not a bill. Under the $9.99 flat-rate subscription, the marginal cost of a build is $0 — the column shows what the same build would cost on a direct pay-per-token API key. For Lovable it IS the bill: 7.10 credits actually consumed ≈ $1.70.

It's the price of delivery, not of success. DeepSeek Pro's $0.29 bought a partially broken app. Cost and quality are separate columns on purpose.

Infrastructure is deliberately out of scope. Lovable's price includes a hosted platform — preview URLs, deployment, managed backend. None of that is being tested here. We are comparing the cost of the coding work, not the cost of the product around it — Lovable sells both, the LLMs sell only the first, and this table prices only the first. No hosting, no domains, no subscription base fees, no human time, on either side.

The scoreboard

The stat that surprised me most: 8 of 9 engines delivered a fully working 10/10 app. Only DeepSeek Pro shipped runtime defects. At the $10 tier, in mid-2026, correctness on a real app spec is nearly universal — the real differences live in code quality, cost, and consistency. (DeepSeek Flash also produced one total failure on a separate first attempt — more on that below; per protocol it's excluded from the table because the run was repeated before scoring.)

What the apps actually look like

Numbers are easier to trust when you can see what they produced. Every engine got the same spec and no design direction whatsoever — the naming, the palette and the layout are entirely the model's own choices. Here are four of the nine builds side by side.

The Lovable build is live, so you can click through it yourself instead of taking my word for the 10/10: open the Lovable "Northstar" demo app. It's the exact one-shot output that was graded — dashboard, project CRUD, kanban, milestones, timeline, search and filters, nothing added afterwards. It's the reference point the eight budget LLMs were measured against.

What the numbers hide

Every engine read millions of tokens to write one small app

The prompt was 2,600 characters. The models read up to 5.7 million tokens to fulfill it — the agent re-reads its own growing conversation on every step. 81–96% of that hits cheap cached rates, which is the only reason the economics work.

The code tells a different story than the clicks

Static analysis of the model-written code only (the scaffold every project starts from is excluded, so this is purely what each model typed):

Three findings jump out:

Same prompt, 2.3× the code. For the identical app, DeepSeek Pro wrote 1,355 lines and MiniMax M3 wrote up to 3,080 (day one). How much code an LLM writes for the same spec is its own personality trait — and every extra line is a token you'll pay to re-read on every future edit.

Lovable lands mid-pack on code quality. Five of the eight budget LLMs write statically cleaner code than the $1.70 control. What Lovable's price buys is not a smarter brain — it's the orchestration, the reliability, the polish (it was the only engine that set page titles and full SEO meta tags unprompted). The intelligence is now cheap; the product around it is what you pay for.

MiMo fails the compile gate in both runs — the same four unused-import errors each time, a consistent model habit. And yet…

Clean code ≠ working app — proven in both directions

The benchmark's best story is a symmetry:

DeepSeek Pro wrote the second-cleanest, leanest code of the field — and shipped the only broken app. Its summary claimed every feature done and the type-checker passing. Both true! But a React effect-ordering race meant the demo data silently self-destructed and every page reload wiped user data. "Compiles clean" is not "works."

MiMo Pro wrote the worst-scoring code of the field — and shipped a flawless app. Four compile-gate errors, hygiene debt everywhere, and every single feature works perfectly, wrapped in the field's best branding (it even named itself "Projevo" and set a proper page title).

Code quality and product correctness are different axes. Any benchmark that measures only one of them is telling you half the truth — which is exactly why this one measures both.

"Verification theater": the strangest failure of all

DeepSeek Flash deserves its own chapter. On its first attempt, it had the most diligent build process of any model: it ran the type-checker, ran the linter, fixed five lint errors with textbook patterns, updated the docs, and reported success.

The app crashed on its very first render.

The root cause is almost poetic: Flash found a type-checker warning, silenced it with non-null assertions ("the deepest ternary branch loses the null-narrowing — I'll use non-null assertions"), reported "TypeScript: 0 errors" — and that exact assertion crashed the app at runtime. It verified everything except actually opening the app. One character away from excellence.

Here's the twist: on its repeat run, Flash delivered the cleanest code of the entire field (9.8) AND a flawless 10/10 app, in 7 minutes, for three cents. The cheapest engine writes the best code — it just sometimes ships a corpse. At $0.03 per build, you can afford to build twice and check.

One more confession: the benchmark framed an innocent model

Kimi K2.7 initially scored 8/10 because "delete freezes the app" — reproduced three times. It turned out to be OUR testing setup: K2.7 is the only engine that uses native browser confirm dialogs, which block automated browser testing. A human just clicks OK. Rescored: 10/10, and a new protocol rule.

That was one of four times during this benchmark that the harness, not the model, was the problem. Every one got caught by the same instinct: when a reading looks uniform or weird across all engines, suspect the test, not the models. If you only remember one methodology lesson from this post, make it that one.

The verdicts — per use case, not one winner

Building an app you'll keep: Kimi K3. Best product + code combination (10/10, 9.6), the smallest largest-file of the field (best decomposition), and it never failed once across every run.

Maximum quality per dollar, if you can tolerate a retry: DeepSeek Flash. Cleanest code and cheapest delivery ($0.03) — with a documented 1-in-3 chance of shipping a crash. Pair it with an automated post-build check and the expected cost is still ~$0.05 including retries.

Raw speed for prototypes: MiniMax M3. Six minutes, fully working, most polished UI details — just know you inherit the heaviest codebase if the prototype graduates.

The intra-family experiments: code-tuned K2.7 matched flagship K3's 10/10 at 3.5× the speed on day one — "code-tuned" is real. And DeepSeek's cheap tier is a bargain with a temper, not a trap.

The control's lesson: Lovable delivered flawlessly, fast, with unmatched polish — and five budget LLMs beat its code quality at 1/30th the cost. You're not paying for the brain anymore. You're paying for everything around it.

Honest limitations

Two runs per engine, one app category, one-shot protocol. The code-quality score is static analysis — a deeper LLM-based blind code review is planned as layer 2. The one-shot rule punishes models that would self-correct with a second prompt. And our own harness produced four testing artifacts along the way — all caught, all documented, because a benchmark that doesn't audit itself is just vibes with tables.

The variance data alone deserves its own post: same model, same prompt, hours apart — build times swung up to 4.6× and costs up to 4.9×. That's coming next.

What's next

This is post one of a series built on this dataset: the non-determinism deep-dive (why every single-run benchmark you've read is a coin flip), whether AI code quality actually matters for your wallet, and the real economics of the $9.99 all-you-can-eat plan. The pipeline that ran all of this automatically — that's a story of its own.

Every number in this post comes from automatic telemetry: token counts from agent session metadata, costs computed at official published rates, code metrics from deterministic static analysis, and acceptance scores from manual click-throughs against a fixed 10-point checklist. Raw per-run data available on request.