05. May 2026
AI Agents in Software Architecture: When the Overhead Is Worth It
The Use Case: Automating Product Research
In a client project, I was faced with the following task: Users enter product information via a form. Behind the scenes, this information should be automatically researched, enriched, evaluated, and returned as a normalized JSON structure. The user is unaware of the processing; to them, it’s a black box.
How do I set up this automation effectively?
First Approach: Multi-Agent System with SmolAgents
The principle of a multi-agent system is easy to explain: A higher-level manager agent distributes tasks to specialized sub-agents. Only the manager knows the other agents. In my case, there were:
- a research agent with access to a search API,
- an extraction agent that summarizes information, and
- an evaluation agent that validates the results.
This clear division of tasks prevents loss of context, a common problem when a single LLM is expected to handle too many steps at once.
For the framework, my first choice was SmolAgents by Hugging Face. It’s lightweight, LLM-agnostic, and follows a plug-and-play approach for models and tools. The OpenAI API was used as the LLM backend.
SmolAgents makes getting started with a multi-agent system pleasantly simple: With the @tool decorator, any Python function becomes a tool that an agent can use. The CodeAgent operates in a Think → Act → Observe loop and decides independently which tool to call and with which parameters.
What Worked Well
- Semantic Understanding: The agents were able to solve tasks that previously required manual research.
- Flexibility: If a search query returned no results, the agent rephrased it based on the context, without any additional programming.
- Quick Development: Thanks to SmolAgents, a working prototype was ready in no time.
Where It Fell Short
Every step the agent takes (“What is the query?”, “What tools do I have?”, “What do I already know?”, “Which tool should I call?”, “How do I evaluate the result?”) is a separate LLM call. This adds up: More agents mean more overhead, in terms of both latency and cost. For time-critical processes, this can become a real problem.
The Alternative: A Deterministic Pipeline with Targeted LLM Use
As the project progressed, it became clear that not every process step requires the flexibility of an agent. If a workflow can be clearly defined, a deterministic pipeline is often the better choice.
Specifically, this means: We know which search queries work for a particular product category. So we build the queries ourselves, fetch the results, and pass the extracted text to a single LLM call. The LLM knows from the prompt what to look for and how to structure the output.
The result: a clean architecture, normalized data structures, and the LLM’s semantic intelligence exactly where it’s needed—without the overhead of agent orchestration.
When the Overhead Is Worth It and When It Isn’t
The decision between agents and a pipeline is ultimately an architectural question. Agents are worthwhile where processes are exploratory and ambiguous, where search queries need to be rephrased, results interpreted, and special cases handled without knowing every path in advance. They provide real added value when the alternative would be to program dozens of if-else branches that would never cover all cases.
A pipeline, on the other hand, is the right choice when the flow is predictable. Fixed input fields, known data sources, clear extraction rules: Here, a deterministic process with a targeted LLM call ensures lower latency, lower costs, and easier maintainability.
In our project, the combination of both ultimately led to the best result: agents for the ambiguous, exploratory tasks; pipelines for everything that could be clearly defined.
Golden Rules from Real-World Experience
- Data contract first: Define your Pydantic model before building your first agent. Without a clear output structure, you’re building blindly.
- Fewer tools, clear tasks: Every additional tool is a new source of error for the agent. Keep your toolbox lean.
- Always measure: Latency, costs, completeness—without metrics, there’s no basis for deciding whether or not to use agents.
- Defense in Depth: Never rely on a single layer. Prompt guardrails, post-processing, and Pydantic validation together create a robust system.
Conclusion
AI agents are not a panacea, but where processes are ambiguous and semantic understanding is required, they demonstrate real strength. The key lies in consciously deciding where agents deliver added value and where a lean pipeline is the better architecture.
