Contents · 9 sections↓
LLM agent orchestration is how you coordinate model calls, tools and sometimes several agents: who decides the next step, who runs it, what each part sees, and when to stop. Most of the decisions are about patterns: a single agent with tools, a fixed pipeline, a router, a supervisor with workers, handoffs between specialists. This guide explains each pattern, when it fits, and what the controller must enforce, with a real supervisor plan as an example. It's for developers deciding how to structure an LLM system before writing it.
What is LLM agent orchestration?
Orchestration is the control logic around model calls: deciding which model or agent acts next, passing it the right context and tools, running its actions, checking results and stopping. Anthropic's Building effective agents separates workflows, where code defines the path (chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer), from agents, where the model directs its own steps. In both, the model proposes and your code decides what actually runs.
Key takeaways:
Start with the simplest pattern that works, usually one agent with good tools.
Use code-defined workflows when the steps are known; use an agent loop when they aren't.
Add multiple agents for separation, not sophistication: different tools, permissions, context or parallel work.
The controller enforces the rules: step limits, budgets, permissions, validation, approvals.
Multi-agent systems cost more. Anthropic reports about 15x the tokens of a chat for theirs.
[02]
The orchestration patterns
Pattern | Who decides the next step | Use it when | Watch out for |
|---|---|---|---|
Single agent with tools (ReAct loop) | The model, one tool call at a time | Most tasks; steps depend on results | Too many tools in one agent |
Prompt chaining (pipeline) | Your code, fixed order | Steps are known: extract, then check, then write | Errors pass silently between steps without checks |
Routing | A classifier or rules, once | Inputs fall into distinct types with different handling | Misroutes on borderline inputs |
Parallelization (sectioning or voting) | Your code | Independent subtasks, or several attempts to vote on | Merging conflicting results |
Orchestrator-workers (supervisor) | A supervisor model plans and delegates | Subtasks aren't known until you see the input | Workers missing context; supervisor overhead |
Handoff (decentralized) | The current agent transfers control | Conversations that move between specialists (sales, billing, support) | Context lost at transfer; ping-pong between agents |
Evaluator-optimizer | A second model grades, the first revises | Clear quality criteria, revisions help | Endless loops; judge bias |
OpenAI's practical guide to building agents groups multi-agent systems into two of these: a manager pattern, where a central agent calls specialists as tools and keeps control, and a decentralized pattern, where agents hand off control to each other. In the OpenAI Agents SDK, a handoff is exposed to the model as a tool named like transfer_to_refund_agent, and calling it transfers the conversation; an agent used as a tool returns its result without transferring control.
[03]
How to choose a pattern
Ask these in order and stop at the first yes:
Can one model call do it? Then don't orchestrate.
Are the steps the same every time? Chain them in code, with a check between steps.
Does the input type decide the handling? Route, then use one of the simpler patterns per route.
Does the next step depend on tool results? One agent with tools in a loop. See the ReAct loop from scratch.
Do parts need different tools, permissions or context, or can they run in parallel? Then consider multiple agents.
Anthropic's advice in the same article is to find the "simplest solution possible, and only increasing complexity when needed". Their multi-agent research system shows both sides: it outperformed a single agent by 90.2% on their internal research evaluation, but used about 15x more tokens than a chat (single agents about 4x), and they note that domains where all agents need the same context, and most coding tasks, are poorer fits.
[04]
Worked example: a supervisor plan
A supervisor gets three workers and a request:
You are a supervisor. Split the request into subtasks for these workers:
- researcher: finds facts in the help center and the customer's account. Read-only.
- analyst: does calculations on numbers that the researcher found.
- writer: drafts the reply to the customer from the other results.
Each subtask names its worker, a self-contained instruction (the worker sees nothing else), the subtask ids it
depends on, and what it must return. Use as few subtasks as possible.Customer C-77 on the Team plan with 14 seats asks: what would we save per year by switching to annual
billing, and can we keep our current invoice date?With a JSON schema for the plan (worker as an enum, depends_on as a list), the model returned three subtasks (instructions trimmed):
research-1 researcher depends_on: []
"Look up the customer's account and help-center articles needed to answer the question. From the account,
retrieve: (a) current billing frequency for customer C-77; (b) current invoice date … Return a JSON object
with these fields: { "account_billing_frequency", "invoice_date", "plan_name", "seats",
"monthly_price_per_seat", "annual_price_per_seat", "annual_discount_percent", … "source_links" }."
analyse-1 analyst depends_on: [research-1]
"Using only the numbers returned by research-1, compute: (a) current annual cost if billed monthly …
(c) absolute savings per year … Return a JSON object: { "current_annual_cost", "annual_cost_if_switched",
"annual_savings", "annual_savings_percent", "first_year_proration_effect" }."
write-1 writer depends_on: [research-1, analyse-1]
"Draft a clear, customer-facing reply to C-77 that: (1) states the computed annual savings … (2) explains
whether they can keep their current invoice date … Use the exact numbers and policy language returned by
the researcher and analyst. Return the drafted email text as plain text."This is a good plan. Each instruction is self-contained, each worker returns a defined shape, the analyst works only from researched numbers, and the writer comes last. It also shows the controller's job: "Using only the numbers returned by research-1" works only if your code passes research-1's output into the analyst's prompt. The model wrote the dependency; the controller has to honor it.
[05]
What the controller must enforce
def run_plan(plan: dict, workers: dict, max_subtasks: int = 6) -> dict:
subtasks = plan["subtasks"]
if len(subtasks) > max_subtasks:
raise ValueError("plan too large")
results = {}
pending = {s["id"]: s for s in subtasks}
while pending:
ready = [s for s in pending.values() if all(d in results for d in s["depends_on"])]
if not ready:
raise ValueError(f"unresolvable dependencies: {list(pending)}")
for s in ready: # independent subtasks could run in parallel
context = {d: results[d] for d in s["depends_on"]}
results[s["id"]] = workers[s["worker"]](s["instruction"], context)
del pending[s["id"]]
return resultsBeyond dependency order, the controller is where these rules live, whatever the pattern:
Limits: maximum steps, subtasks, tokens and wall-clock time per run. See AI agent cost control.
Permissions per worker: the researcher gets read-only tools; only an approved step can write. Each agent should have the smallest set of tools it needs.
Validation between steps: check each worker's output against its declared shape before passing it on.
Approvals: risky actions pause for a person. See human-in-the-loop approval.
Tracing: every model call, tool call and handoff logged with the run id. See AI agent observability.
For a full, runnable coordinator with specialist agents, tracing and cost per run, see build a multi-agent system in Python.
[06]
Handoffs: passing control between specialists
In a handoff, the current agent decides another agent should take over and transfers the conversation. It suits customer-facing flows where the topic changes: a triage agent hands a refund question to a billing agent. Three rules keep handoffs reliable:
Pass a summary, not just the transcript: what the user wants, what's been checked, what's left.
Limit who can hand off to whom, and cap the number of handoffs per conversation to stop ping-pong.
Give each specialist only its own tools, so a handoff is also a permission boundary.
If a specialist only needs to answer a sub-question, call it as a tool instead and keep control in one place.
[07]
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
Multi-agent by default | More tokens, more failure points, harder debugging | Start with one agent; split only for a clear reason |
Workers that see only "do part 2" | They lack context the supervisor had | Self-contained instructions plus dependency outputs |
One agent with every tool | Tool choice degrades, permissions are too broad | Split tools by role or use tool search |
No limits in the controller | Loops and runaway spend | Step, token, time and handoff caps |
Unchecked outputs between steps | One bad result poisons everything after it | Validate each output against its expected shape |
Letting the model decide permissions | Prompts are not access control | Enforce permissions in code per agent and tool |
[08]
Frequently asked questions
What's the difference between an agent and a workflow?
In a workflow, your code defines the sequence of model calls and tools. In an agent, the model decides the next step in a loop, based on tool results. Workflows are more predictable; agents handle open-ended tasks. Many systems mix them: a fixed workflow with an agent inside one step.
When should I use multiple agents?
When parts of the task need different tools, permissions or context, or can run in parallel, such as several research threads at once. If one agent with the right tools can do it, it's cheaper and easier to debug. Anthropic reports its multi-agent system used about 15x the tokens of a chat.
What is the supervisor pattern?
A supervisor model breaks the task into subtasks, assigns each to a worker agent, and combines the results. It's called orchestrator-workers in Anthropic's terms and the manager pattern in OpenAI's. Use it when the subtasks depend on the input and can't be fixed in code.
What is an agent handoff?
A transfer of control from one agent to another, usually exposed to the model as a tool such as transfer_to_billing_agent. The receiving agent continues the conversation. Use handoffs when the conversation moves between specialists; use agents-as-tools when one agent should stay in charge.
Do I need a framework for orchestration?
Not to start. A loop, a plan format and a small controller like the one above go a long way. Frameworks help once you need persistence, tracing, approvals, retries and many agents. Pick one that lets you see and control every step.
[09]
Next steps
Build a coordinator with specialist agents: multi-agent system in Python.
Start with one agent: how to build an AI agent in Python.
What surrounds the agent loop: what is an agent harness?
The loop inside each agent: ReAct agent from scratch.
Plan, act and verify: plan-and-execute agents.
Routing between models: LLM routing: small vs large model.