Five lessons from running agents in production engineering

Plenty of teams treat AI agents the way they once treated cloud functions: drop them in, point them at a job, hope it works. That stops working at scale. Here's what we've learned running agents in real engineering orgs, including a few things we'd do differently.

1. Authorization boundaries beat capability boundaries

Don't ask "what can this agent do?" Ask "who authorized this action?" The first question scales badly. The second one generalizes to every new tool, model, and workflow you add later.

We learned this the slow way. Capability lists rot every time a model upgrade lands. The agent that couldn't write migrations in March writes them in June, and your controls are suddenly describing a tool that no longer exists. An authorization model survives the upgrade: which actions need a human sign-off, which run free, which are forbidden regardless of how confident the agent sounds. It's also the version your security team can actually review.

2. Specs are interfaces

The spec a person reads to do the work is the spec the agent reads. If your team can't write a spec the agent can act on, you have a process gap, and no tool will close it for you.

Teams in this situation keep shopping for a better prompt library when what they're missing is a decision about what the work is. Write the behavior, the constraints, and the definition of done, and most of the "prompting problem" evaporates.

3. Cost-per-outcome, not cost-per-call

Token spend per call is easy to chart and tells you almost nothing on its own. The number we care about is cost per shipped change, with rework included. On the engagements where we've tracked it, that figure has surprised people in both directions.

Instrumenting this matters because felt speed and measured speed diverge sharply. In METR's 2025 randomized trial, experienced developers using AI took 19% longer on real tasks from their own repositories while believing, even afterward, that they'd been sped up by about 20%. That's not an argument against agents. It's an argument against using vibes as telemetry. If you aren't measuring cost per shipped outcome, you're managing by anecdote, and the anecdotes are systematically optimistic.

4. The bottleneck moves to review

Agents speed up generation. They don't speed up the human judgment that decides whether to merge. The most common failure mode we see is teams pouring effort into generating more while doing nothing to make review faster or more trustworthy.

The distrust is widespread and, frankly, rational: the 2024 DORA report found 39% of practitioners have little or no trust in AI-generated code. When review is the only gate and review is drowning, distrust is the correct response. Making review keep up at agent volume means smaller changes, tests the reviewer can lean on, and specs the reviewer can verify against. That's the boring infrastructure that turns "looks plausible" into "provably matches intent."

5. Sunset is a feature

In our experience, most teams have never deliberately retired an AI tool, and that's worth noticing. Adding tools is easy. Choosing what to keep is the harder discipline, and it's the one that tells you what actually mattered. An annual pass to rationalize the stack is cheap insurance against sprawl.

The Sigao take

Agents reward the same boring things that reward people: clear authorization, written intent, honest measurement, review that can keep up, and the discipline to retire what isn't earning its keep. None of the five lessons above is about models. All of them are about the operating system around the models, which is exactly why they'll still be true after the next upgrade.

Sources

• Becker, Rush, Barnes, and Rein (METR), "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 2025.
• Google Cloud, "Announcing the 2024 DORA report", and DORA, Accelerate State of DevOps Report 2024.