The Economics of Running Agents: Cost and Latency

Loops multiply everything

The economics of a single language-model call are simple and predictable: one request, one response, a known cost. The economics of an agent are not, and the reason is the loop. An agent that takes ten passes to finish a task makes ten model calls, and because each pass carries the accumulated history forward, every call is larger than the last. The result is that an agent can easily cost an order of magnitude more than a single completion, and take ten times as long, for the same underlying model.

This is why pricing assumptions that worked beautifully for a chatbot can quietly break for an agent. A per-message cost that seemed negligible becomes a per-task cost that does not, and it does so invisibly, because nobody was watching the multiplier. Cost and latency are not afterthoughts for agents; they are core design constraints, and the teams that treat them that way ship systems that are actually viable to run.

Know your cost per task

You cannot manage what you do not measure. Instrument every run for the numbers that matter: input tokens, output tokens, the number of tool calls, and total wall-clock time. From these, compute the metric that actually governs your business, cost per completed task, and track it the way you would any unit economic.

This matters because an agent feature can demo flawlessly and still lose money on every single use once real traffic hits. Without per-task cost visibility, you find that out from your provider bill at the end of the month, not from your dashboard in real time. With it, you can see immediately when a change to the prompt or model pushed your unit cost the wrong way, and you can make pricing and design decisions with eyes open.

Right-size the model

A common and expensive habit is to use your most capable, most costly model for every step of the loop. Most steps do not need it. Routing a request, classifying an intent, deciding which tool to call, extracting a field, these are easy decisions that a smaller, faster, cheaper model handles just as well.

Reserve the expensive model for the steps that genuinely require deep reasoning, and delegate the rest. A mixed-model agent, cheap models for the routine moves, a powerful one for the hard thinking, is frequently both cheaper and faster than an all-premium design, with no measurable loss in quality. The discipline is to be honest about which steps are actually hard.

Control the context

The single biggest hidden cost in most agents is context growth. By default, each pass re-sends the entire history, every prior message, every tool result, and you pay for all of it, again, on every call. By the tenth pass you may be paying ten times to re-process the same early turns that no longer affect the decision at hand.

Manage context actively. Summarise older turns into compact notes once their detail no longer matters. Prune raw tool output after you have extracted the relevant facts. Carry forward only what the next decision genuinely needs. This is not just a cost lever; a leaner context also tends to make the model more accurate, so you save money and improve quality at the same time. It is one of the rare wins with no real downside.

Cache what repeats

Many agent runs share a large, stable prefix: the system instructions, the tool definitions, the examples, the policy text. This preamble is identical on every call and often dwarfs the variable part. Prompt caching lets you avoid paying full price to re-process that fixed prefix each time, and the savings on a busy agent can be substantial.

To benefit, structure your prompts so the stable content comes first and the variable content, the current task and recent history, comes last. The more of the prompt that stays byte-for-byte identical across calls, the more the cache can do for you. It is a small architectural habit that pays off continuously.

Set hard budgets

Finally, put hard ceilings on iterations, token spend, and tool calls per task. These are usually framed as safety guardrails against runaway loops, and they are, but they are equally economic controls. A budget turns an open-ended, unbounded risk, "this task could cost anything", into a known, capped line item you can reason about and price around.

When a budget is reached, the agent should stop cleanly and, if appropriate, escalate, never silently spin or pretend it finished. With budgets in place, your worst-case cost per task is a number you chose, not a number you discover. That predictability is what makes it safe to put an autonomous, looping system in front of real users at scale.