Agents Aren’t Chatbots: Why Distributed Systems Matter

Last month I watched a production agent hang for six hours because state wasn’t persisted between API calls. The orchestrator restarted the LLM five times. Each time it started over. No backoff. No circuit breaker. Just hammering the same endpoint until rate limits bit back. That’s when I realized: most people building agents are still thinking like chatbot engineers.

Agents Are Stateful Systems Now

A chatbot is stateless. You send a message, get a response, done. An AI agent is a distributed system. It makes decisions, calls APIs, waits for results, retries on failure, manages concurrency, and tracks execution state across multiple services. The cognitive overhead isn’t just “what should the agent think”—it’s “what happens when the database is slow,” “how many concurrent tool calls can we safely spawn,” and “what data survives if the container dies halfway through.”

Google’s Ax orchestration framework and Agent Substrate treat agents as proper distributed systems. They have to. When you’re orchestrating real work—fetching data, writing databases, triggering deployments—the agent doesn’t just need to be smart. It needs to be reliable.

Where Chatbot Architecture Breaks

The usual pattern for an agentic LLM loop is simple: call model, parse response, execute tool, repeat. Chatbot teams ship this in an afternoon. But watch what happens under load or failure:

• No request deduplication. Two concurrent tool invocations hit the same endpoint. Both succeed. State diverges. Now you’ve got two conflicting versions of truth.
• No backoff semantics. HTTP 429 (rate limit) comes back. The agent either retries immediately (hammers harder) or fails (loses the work). There’s no exponential backoff, no jitter, no circuit breaker.
• No partial failure handling. Three concurrent calls, two succeed, one times out. Do you roll back the two successes? Retry just the one? Leave the state inconsistent? Most agent code has no answer.
• No checkpoint recovery. Container dies mid-execution. The work is lost. You restart from the beginning or lose it entirely.

Chatbot code never had to solve these problems. A user sent a message, the model responded, the session ended. No persistent state to corrupt. No cascading failures. Agents have both.

What Distributed Systems Get Right

Systems that have been solving this for twenty years—Kafka brokers, workflow orchestrators, task queues—have patterns. They’re boring. They work.

Idempotency. Every tool call includes a request ID. If the same request gets submitted twice (retry, network duplicate, whatever), the system recognizes it as identical and returns the cached result instead of executing twice. Most agent code doesn’t do this. Should.

Async state machines. Instead of a simple loop (“call model, call tool, repeat”), the agent tracks discrete steps: pending, in-progress, succeeded, failed, retry-scheduled. Each step can be checkpointed. If the process crashes, you know exactly where you were and resume from that point, not from scratch.

Retry policies with backoff. Not “try three times.” Exponential backoff with jitter. Different retry limits for different error codes. Transient failures (network timeout) retry. Permanent failures (404, auth error) fail immediately.

Observability hooks. Every state transition, every API call, every failure logged. Not as strings in logs—as structured events. So when something breaks, you don’t spend six hours guessing which retry loop got stuck. You have the trace.

The Real Cost

I’m not saying this to be difficult. I’m saying it because I’ve seen production agents that cost companies money every time they loop. A misconfigured retry policy that hammers an external API. No idempotency check that causes duplicate transactions. A failure mode that leaves state half-written and unrecoverable.

If you’re deploying an agent that makes API calls, changes databases, or triggers anything expensive, treat it like a real system. Add a request ID to every tool invocation. Log structured traces. Implement backoff. Design for failure recovery. It’s not optional overhead—it’s the baseline cost of running distributed code.

# Minimal idempotent retry with backoff
import time
import random

def call_with_backoff(tool_fn, request_id, max_retries=3):
    """Call tool with exponential backoff and idempotency."""
    for attempt in range(max_retries):
        try:
            # Request ID allows deduplication on the service side
            result = tool_fn(request_id=request_id)
            return result
        except Exception as e:
            if attempt < max_retries - 1:
                backoff = 2 ** attempt + random.uniform(0, 1)
                print(f"Retry {attempt+1} after {backoff:.2f}s: {e}")
                time.sleep(backoff)
            else:
                raise

The agent that doesn't crash under load, doesn't lose work on failure, and doesn't burn money on cascading retries—that's the difference between a demo and production. Build like you know the difference.