From chatbot to decision system: when to use multi-agent architecture

Why this matters

The second agent added a coordination step, not a solution#

A team splits one reliable assistant into a planner and an executor expecting specialization. The planner now has to pass the request, constraints, and partial result to the executor. If one constraint is omitted, the executor cannot tell whether that was intentional; retries may then duplicate work. This passage of work from one agent to another is a handoff, and it creates a new place to lose information.

That failure is why agent count should follow a specific bottleneck, not a diagram. First name what one workflow cannot do well—for example, search two independent sources at the same time or keep database access away from the component that writes the final answer. Add another agent only when that separation improves a measured result enough to justify the extra coordination.

Before adding another role, place the task on a spectrum from a fixed sequence to genuinely independent decisions.

Architecture spectrum

Start with a fixed sequence; add agent choices only when needed#

Move right only when the task requires a choice that a fixed sequence cannot make well.

1. 01

   Fixed workflow

   The steps and branches are known in advance—for example: validate the request, query an approved source, then format the answer.

2. 02

   Single agent

   One agent chooses among approved capabilities, such as web search or a code runner, while completing one bounded task.

3. 03

   Multi-agent system

   Different agents own genuinely separate work, pass defined results between them, and combine those results at the end.

The spectrum narrows the options; concrete signals show whether a second agent has a job that justifies its cost.

When separation helps

Give each added agent a concrete reason to exist#

Separation can help when one component searches sources, another performs a calculation, another checks a policy, and a final component explains the evidence. It can also keep sensitive access—such as a production database credential—away from the component that does not need it, or run independent searches at the same time.

For each boundary, write what the next component receives, what it must return, which capabilities it may use, how long it may run, what happens after a failure, and who owns that failure. This makes the coordination testable. Simply allowing agents to exchange messages does not.

Once that job is named, choose the smallest coordination pattern that can perform it.

Topology menu

Choose a coordination pattern before choosing a framework#

These patterns solve different coordination problems. Select one from the information and authority each role needs, not from framework features.

1. 01

   Prompt chain

   Use a fixed sequence when each stage transforms and validates the previous result.

2. 02

   Parallel fan-out

   Run independent analyses together, then aggregate only evidence that meets the same contract.

3. 03

   Router and specialists

   Classify the request, send it to one bounded specialist, and keep an explicit fallback.

4. 04

   Evaluator and optimizer

   Let one component draft and another critique against a rubric, with a hard iteration limit.

A conversational analytics case makes those differences visible in one system.

Sanitized product pattern

A conversational analytics system is more than chat#

In a production conversational analytics product, a user request may require intent classification, data retrieval, statistical computation, comparison and explanation. The interface looks like one conversation, but the responsibilities behind it are different.

The useful pattern is not the number of agents. It is routing cheap or deterministic paths early, preserving continuity for follow-up questions, and requiring the final answer to carry the evidence produced by each specialist.

The case also exposes the next design question: what happens when a specialist, tool, or handoff fails?

Failure containment

Decide what happens when one component fails#

Every extra agent creates another place to lose a constraint, exceed the time limit, or contradict an earlier result. The coordinating component therefore needs a deadline, a limit on retries, and a useful fallback—for example, return the verified source list even if the synthesis step fails.

Record which agent received the task, which approved capability it used, and what result it returned. If the final answer is wrong, this execution record helps the team locate whether the request went to the wrong specialist, the evidence was weak, a calculation was wrong, or the final explanation distorted a correct result.

Failure handling adds cost, so the multi-agent design must now beat a simpler baseline on a measured outcome.

Architecture experiment

Make the complex design beat a simpler baseline#

Run the same representative set through a fixed sequence with defined inputs and outputs, the single-agent version, and the proposed multi-agent design. Compare task completion and critical failures first; then inspect p95 latency, cost per successful task, human intervention, and how often the system returns a partial but still useful result.

Adopt the multi-agent design only when the gain survives repeated runs and comes from specialization or failure containment. If quality is unchanged while latency, cost, and diagnosis worsen, keep the simpler design.

from dataclasses import dataclass
import json

@dataclass(frozen=True)
class Request:
    intent: str
    question: str

def generalist(request: Request) -> dict:
    return {"path": ["generalist"], "answer": "baseline", "evidence": []}

def data_specialist(request: Request) -> dict:
    if "timeout" in request.question.lower():
        raise TimeoutError("data specialist timed out")
    return {
        "path": ["router", "data_specialist"],
        "answer": "compare two validated segments",
        "evidence": ["segment_a", "segment_b"],
    }

def route(request: Request) -> dict:
    if request.intent != "compare_segments":
        return generalist(request)
    try:
        return data_specialist(request)
    except TimeoutError:
        return {
            "path": ["router", "data_specialist", "safe_fallback"],
            "answer": "comparison unavailable",
            "evidence": [],
            "degraded": True,
        }

cases = [
    Request("small_talk", "hello"),
    Request("compare_segments", "compare the top two"),
    Request("compare_segments", "force timeout"),
]
print(json.dumps([route(case) for case in cases], indent=2))

[
  {
    "path": ["generalist"],
    "answer": "baseline",
    "evidence": []
  },
  {
    "path": ["router", "data_specialist"],
    "answer": "compare two validated segments",
    "evidence": ["segment_a", "segment_b"]
  },
  {
    "path": ["router", "data_specialist", "safe_fallback"],
    "answer": "comparison unavailable",
    "evidence": [],
    "degraded": true
  }
]

The comparison turns architectural taste into a decision about the value bought by each extra boundary.

Decision rule

Ask what complexity buys#

Answer these questions with measurements from the same task and conditions. A diagram is not evidence of improvement.

• ✓Can a fixed sequence with defined inputs and outputs solve the task?
• ✓Does one agent have enough context and isolated access?
• ✓Are responsibilities meaningfully different?
• ✓Can the stages run independently or in parallel?
• ✓Can you observe what each component receives and returns?
• ✓Is it clear who responds when each specialist fails?
• ✓If one stage fails, can the system still return a useful partial result?
• ✓Does the evaluation compare the simpler and more complex design?

Even a justified multi-agent design cannot remove the ordinary limits of models, tools, and organizational ownership.

Conclusion

Add an agent only when it removes a proven bottleneck#

The conclusion is not that multi-agent systems are better. The default should remain the smallest architecture that satisfies the contract. Extra agents are justified only when context, permissions, tools, ownership, or genuinely independent work cannot be handled clearly inside one workflow.

Before adding a role, name the bottleneck it removes and the new failure it introduces. Type the handoff, preserve the evidence, set a retry and stop rule, and make one component responsible for the final state. If the team cannot isolate and recover a failed handoff, simplification is the safer architecture decision.