The task was supposed to take about eight tool calls. When we found it the next morning, it had made a little over two thousand.
We'd built an autonomous agent to handle a class of internal operations work — multi-step account and billing issues that a support engineer normally resolves by looking things up across a few systems, cross-referencing them, and applying a fix. Exactly the kind of repetitive, tool-heavy workflow everyone wants to automate. The agent could plan, call internal tools, read the results, and decide what to do next. Most of the time it worked, and when it worked it felt great.
Then one task hit an ambiguous state overnight and never came back out. It called a lookup tool, misread the result, called it again, tried another tool, came back to the first one, and kept going — the same handful of calls in slightly different orders, hundreds of times, making steady progress toward nothing. By the time someone noticed, that single run had cost more than the feature had in the previous month.
The unsettling part was that nothing had technically failed. Every individual tool call was valid and every model response was reasonable in isolation. There was no exception, no crash, no obviously broken output. The runtime owned the loop, but the model effectively owned the transitions — what happened next, whether to continue, and when the task was done. We'd never given the system a hard reason to stop.
What we'd actually built
The agent was a fairly standard ReAct-style setup orchestrated in LangChain. The model reasoned about the task, chose a tool, we executed it, fed the result back, and let the model choose the next action. It had access to account lookups, billing data, and a few write actions that could actually change customer state, with a system prompt describing the job and when it should consider the task complete.
The whole appeal was autonomy. We didn't want to hard-code every resolution path because there were too many variations. We wanted to hand the model a goal and a toolbox and let it work out the path, and for clean cases that was exactly what happened. The problem was that the model wasn't just making judgments. It was also deciding the effective control flow: how many steps to take, which action could follow which, when to retry, and when to stop.
A language model can be useful at deciding what an ambiguous result means. It is not a reliable termination mechanism. You don't see that on the happy path, where a task resolves in six steps and everyone is impressed. The problem lives in the tail.
Cost is a function of steps
The runaway forced me to think about agent cost differently. With a normal model call, cost is reasonably bounded by input and output tokens. An agent changes that equation. Total cost is roughly cost per step × number of steps, and the number of steps is decided at runtime.
The same type of ticket might finish in six steps on one run and thirty on another. In the naive design, there was nothing preventing it from taking two thousand. The metric I'd been ignoring wasn't cost per call. It was steps per task, especially the tail of that distribution.
The average looked fine because most tasks were short. The p95 and p99 didn't. That matters because a runaway doesn't make every request a little more expensive; it makes most requests normal and a tiny number catastrophically expensive. Averages hide exactly the failure you care about.
That also told me where the fix wasn't. A cheaper model per step wasn't going to save a task doing two thousand unnecessary steps. The lever was the step count itself.
The options I actually evaluated
My first instinct was the obvious one: the model wasn't smart enough. If it were a little better, surely it would realize it was looping. I tried to challenge that assumption before changing the architecture.
A stronger model. It helped some trajectories, but it didn't change the shape of the failure. The model still decided whether to continue, and there was still no hard upper bound. Making a structural failure rarer isn't the same as removing it.
Better prompting. I tried variants of "if you find yourself repeating actions, stop." It helped on average. It wasn't something I was willing to treat as a safety mechanism. Asking the same component that's confused to reliably recognize its own confusion isn't a control system.
More tools and more autonomy. This only widened the surface area. Every new tool is another possible branch, another malformed argument, another way to circle back. If the tool changes state, the consequences get much worse.
The option that actually worked felt less exciting: give the model less to decide. Put the control flow back in code.
The fix was owning the control flow
We moved from an open-ended autonomous loop to an explicit state graph using LangGraph. The model still made judgment calls where judgment was useful, but the structure of the workflow — valid transitions, stopping conditions, and which tools were available where — became deterministic.
Most of the fix was ordinary software engineering. Every run got a hard step budget, wall-clock timeout, and cost ceiling. Hit any of them and the run stopped and escalated. That alone would have turned the two-thousand-call incident into a twenty-call annoyance. A limit enforced in code is worth more than an instruction asking the model to behave.
We also added loop and cycle detection by tracking recent states and tool-call signatures. If the workflow revisited the same state or repeated the same tool with the same arguments, we stopped the loop and either tried a different path or escalated. The model didn't get to decide whether it was looping; we could observe that from outside.
At each node, we reduced the choice space. Instead of giving every step access to every tool, each state exposed only the actions that made sense there, with validated structured arguments. We also added explicit preconditions and postconditions around transitions, so a model recommendation wasn't enough if the current state didn't satisfy the rules for that action.
The write paths got stricter. Anything that changed customer state required approval, and where possible the operation supported dry-run and idempotency. We attached idempotency keys to side effects so a retry or resumed run couldn't apply the same mutation twice. A read loop wastes money. A write loop corrupts data.
We also checkpointed workflow state after meaningful transitions. If a worker died halfway through a long run, the next process resumed from the last safe state instead of replaying the whole task and risking duplicate side effects.
None of this made the model worse at the work. It changed the envelope it was allowed to operate inside. The worst case was now bounded.
Optimizing cost the agent way
Once we owned the control flow, cost optimization looked very different from chatbot optimization. The biggest savings came from removing model decisions entirely.
If a step didn't require judgment, it became normal code. Parsing a known response shape doesn't need a reasoning model. Checking whether a field exists doesn't need a reasoning model. Following a deterministic state transition doesn't need a reasoning model. Every time we removed one of those decisions, we reduced both cost and failure surface.
Within a run, we cached tool results so revisiting a state didn't mean paying for the same lookup again. We also used different models by step type rather than by whole task. Planning or judgment nodes could use the stronger model; mechanical extraction nodes did not.
That granularity mattered. An agent isn't one model call. It's a workflow made of different kinds of work. The cost ceiling also became part of reliability: the same circuit breaker that stops a runaway caps worst-case spend per task, turning agent cost from an open-ended liability into something you can actually budget.
Knowing it worked meant measuring the tail
I stopped trusting averages after that incident. The useful metrics were the ones that described bad runs: p95 and p99 steps per task, percentage of runs hitting a budget ceiling, loop or repeated-call rate, cost variance between runs, and human escalation rate.
Full-trajectory tracing mattered just as much. "The agent failed" tells you almost nothing. "The run entered the billing lookup state at step 17, returned to the account lookup state three times, then repeated the same call signature from steps 24 through 31" is something you can fix.
Agent failures are often control-flow stories. You need the full path to see them.
If you take one thing from this
When an agent misbehaves, the instinct is to reach for a stronger model or a better prompt. Sometimes those help. They don't give you guarantees.
The useful shift for me was separating judgment from control. The model can decide what an ambiguous result means. It should not be the only thing deciding whether a workflow is allowed to continue forever, whether a write is safe to repeat, or whether the current state is valid.
You don't make an agent reliable by making the model smarter. You make it reliable by giving it less to decide: own the transitions, bound the steps, validate the state, make side effects idempotent, persist progress, and put circuit breakers around anything that costs money or changes data.
Our agent didn't do anything clever the night it ran up the bill. It just kept deciding what to do next, two thousand times, because we'd built a system where that decision had no hard boundary. The fix wasn't intelligence. It was control.
If you're running an agent in production and you can't tell me your p99 steps per task — or exactly what happens when a run doesn't terminate — that's the scenario I'd test first.