Skip to content

The Spec Is the Code: A Loop for Agent-Driven Development

When an agent writes the code by following a plan, reviewing the code is reviewing the output. What you actually want to review is the plan. The spec is your "code" — the artifact you author and reason about, and its quality is what mostly determines everything downstream.

This is the title thesis, and I want to be honest that it isn't mine. acai.sh gets there through a thought experiment: if software were free and infinite and instant, "your criteria for acceptability is really the only thing of value. The spec." (acai.sh). GitHub's Spec Kit makes the same inversion concrete — spec-driven development "flips the script" on the old order where "code has been king — specifications were just scaffolding we built and discarded", so that now "specifications become executable" (Spec Kit). So the "flip code and spec" framing is already a named, tooled methodology. What I want to add isn't the inversion — it's wiring it into a closed loop.

So review the plan directly. Stress-test it. Hunt for the edge cases. Make sure it's comprehensive, with no unknowns left unaddressed or under-investigated — because those are often the gaps that turn into surprises mid-implementation, force the plan to change halfway through, and leave you with something incoherent or buggy. A good spec carries its own rollout and rollback plan, the metrics and logs to verify against, an argument for why it's correct, and tests to surface what it didn't anticipate.

That's the shape of the idea I keep coming back to: a loop of spec → exec → eval → bench → improve. A few components have to work together to make it real, and for most of them the question isn't whether the piece exists — it's whether anyone has wired them into one loop.

A spec format built for both humans and agents

First, the spec itself needs a format that is human-review-friendly, token-efficient, and AI-author-friendly all at once. It should ship with a renderer and collaboration features so reviewing a spec feels as natural as reviewing a diff. And it should live in the code — spec-in-code — to reduce drift between intent and implementation, and to make specs discoverable and easy to pull into context.

acai.sh's feature.yaml already nails most of this: a parseable spec where each requirement gets a stable ID (e.g. my-feature.ENG.2), a CLI, and a dashboard where you mark requirements completed, accepted, or rejected and leave comments — the renderer and collaboration layer I'm asking for. Crucially, an agent there started referencing those IDs (it calls them ACIDs) directly in the code and tests, which is the spec-in-code coupling that keeps intent and implementation from drifting. The promise is to "review requirements instead" with "No more file-by-file GitHub PR reviews" — exactly the "review the plan, not the output" move (acai.sh). So three of my four requirements are met today; the open one is the token-efficiency-plus-renderer angle, a format dense enough for an agent's context budget yet still pleasant for a human to read.

An agent swarm to execute

Once specs are reviewed and committed, an agent swarm picks up implementation and testing. That needs a sandbox or agent-fleet system underneath it — run each agent in "a Docker container or VM" and keep an emergency stop within reach (Self-Improving Coding Agents). Factory's Missions is the clearest working instance of the loop: an orchestrator writes a validation contract first, a runner spawns a fresh worker per feature that "writes tests first, then implements", and independent validators check the result until "This loop repeats until milestone validation passes." (How Missions Work). The mechanics are real, not hypothetical.

The part I'm less sure of is how much to take humans out. My instinct was that humans stay in the loop only to approve proposals for the next iteration — not to hand-hold every step. Vercel's "Agent responsibly" is a direct shot at that instinct, and I have to take it head-on. Their argument: "Green CI is no longer proof of safety", and "relying solely on review, whether human or synthetic, is a losing battle" — because the scarce resource is no longer writing code, "it's the judgment of what is safe to ship" (Agent responsibly, Vercel). Light-touch approval is precisely what they warn against; putting your name on a PR is supposed to mean "I have read this and I understand what it does."

They're right, and the fix is to be specific about where the human gates. Human judgment can lighten at the spec gate — that's the whole point of moving review up to the plan — but it can't evaporate at the ship gate. The eval and bench layers below are what would earn any reduction in per-diff review; until they exist and are trusted, "humans only approve proposals" is aspirational, not a license to stop owning what ships.

A self-improving harness

Then there's the part that makes the loop compound, and it's the piece I think is genuinely under-built. When a spec is locked in, it generates a benchmark test case. At the end of an implementation cycle, the system retrospects and improves itself — either as an outside observer or by asking the agent to do the retrospective. Those accumulated test cases are what you evaluate improvements against: they prevent regression and check that a change actually generalizes rather than overfitting to the last task.

Every ingredient here exists somewhere. LangChain's harness work is the cleanest real self-improving harness — they took a coding agent from Top 30 to Top 5 and "only changed the harness", using trace analysis to make targeted changes while watching that "Changes that overfit to a task are bad for generalization and can lead to regressions in other Tasks", with a middleware that forces "a verification pass against the Task spec" (Harness engineering, LangChain). Google's ReasoningBank does the compounding part, distilling "useful insights from both successful and failed experiences" until "simple checklists evolved into memories with compositional, preventative logic structures" (ReasoningBank, Google). Missions writes a per-mission validation contract of behavioral assertions before any features (How Missions Work).

But none of them quite do the thing I'm after. LangChain measures against an external benchmark (Terminal Bench), not one grown from each spec. ReasoningBank accumulates reasoning-pattern memories, not regression tests. Missions' contract is per-mission, not a benchmark that carries across tasks. The synthesis I haven't seen anywhere: generate a regression test case from each locked spec and accumulate them into a benchmark you reuse across tasks, so the harness's own changes are graded against the union of everything it has ever shipped. That's the part I'd claim as the contribution rather than pretend already exists.

Rollout as an executable handbook

Every spec should carry real, comprehensive rollout sections: how to design the experiment, how to roll back, which functional and non-functional metrics to monitor. These need step-by-step, concrete instructions — not a high-level gesture at "we'll monitor things," but a handbook someone (or some agent) can actually run.

Vercel describes exactly this and is worth borrowing wholesale. Their executable guardrails are "operational knowledge as runnable tools instead of documentation" — a safe-rollout skill that wires the flag, generates a rollback plan, and specifies how to verify behavior, rather than a Notion page. Deployments are self-driving: "If a canary deployment degrades, the rollout stops and rolls back automatically." And the payoff is concrete — a database failover they rehearsed in production is why "a real Azure outage this year was a non-event for our customers" (Agent responsibly, Vercel). That's the bar for "executable operational handbook": runnable tools and rehearsed failure, not prose.

Proving the thing is correct

Finally, the part that's easy to wave away. I originally wanted a formal validation — a theoretical proof of the algorithm's and the system's correctness. I should flag that this is the one piece with no support in anything I've read, and more pointedly, the field has converged on the empirical alternative instead. Missions encodes correctness as a validation contract — a checklist of behavioral assertions. acai.sh tracks "acceptance coverage" rather than aiming at a proof. Both substitute runnable acceptance criteria for a theorem. So formal proof stays on my list as the aspirational frontier, not the near-term plan; what's actually buildable today is continuous whitebox testing plus simulated failure.

Even there, my own sources push back on the framing. Missions' user-testing validators "exercise the system as a black box - using it the way a real user would" — the opposite of my whitebox instinct. The simulate-unpredictable-users half is solidly precedented; the whitebox part is a preference I'd have to justify, probably as a complement to black-box user simulation rather than a replacement. And for simulated system failure before real traffic, Vercel's chaos experiments and failover rehearsals are the existing version (Agent responsibly, Vercel).

What the loop adds

None of these pieces are exotic on their own — which is the honest takeaway, not a disappointment. This sits a level above the argument I made in Make Regenerating Code Safe, Not Generated Code Good: if the spec is the durable artifact and the code is regenerable cattle, then the spec needs the same machinery code has — versioning, review, and a test suite. The new claim here is that the test suite should be grown from the specs themselves and accumulated, so the harness improves against everything it has ever built.

The bet is that wiring spec, exec, eval, bench, and improve into a single closed loop — where each locked spec leaves behind a regression test, and the harness is graded against all of them — is what turns agent-driven development from "the agent wrote some code" into a system that gets better on its own. Most of the parts already exist in the open. What I haven't seen is the loop closed.

Sources

  • acai.sh — spec-driven development; the spec as "the only thing of value," feature.yaml/ACIDs, and requirement-level review.
  • github/spec-kit — Spec-Driven Development "flips the script"; specifications become executable.
  • How Missions Work — Factory.ai; orchestrator/worker/validator swarm, validation contract, black-box user testing.
  • Agent responsibly — Vercel; green CI isn't safety, judgment is the scarce resource, executable guardrails, canary auto-rollback, failover rehearsal.
  • Improving Deep Agents with harness engineering — LangChain; trace-driven self-improving harness, overfitting-vs-generalization, verification against the task spec.
  • ReasoningBank — Google Research; accumulating memories from success and failure, strategic maturity.
  • Self-Improving Coding Agents — Addy Osmani; planner-worker swarms, sandboxing, emergency stop.