Multi-Agent Magazine RAG Chatbot
From plain RAG to router-based agentic RAG, guided by layered evaluation
This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.
A chatbot over a magazine archive. It answers questions about articles with grounded answers and recommends magazines when the user is browsing. I built the first version as a plain RAG and then improved it in two ways. I moved the decisions into a router that runs before any answer, and I built an evaluation that shows which layer failed, so every VOC (voice of the customer: user reports and outages) could be traced to a cause and the update shipped in two days instead of two weeks.
Agentic RAG · RAGAS + golden sets
Same model, same golden sets: initial plain RAG vs final agentic RAG
Quality scores (0–1)
Article hit = the correct article is in the top 5. Conv. retrieval = share of 48 conversational messages (long context, worries described without a question) that went through retrieval.
Agentic RAG architecture
The router decides first, and only some requests reach retrieval. An input guard screens every message, then a router with structured output sends it down one of four routes: grounded answer, magazine recommendation, chitchat or refusal. Each sub-agent has its own prompt and tools. Chitchat and refusals are answered without retrieval.
Problem
The model decided whether to retrieve
Questions went straight to the LLM agent, with no input check.
One in four conversational messages skipped retrieval.
No step filtered out unsafe requests or chitchat.
Root cause
Decisions and answers mixed in one model
Whether to search, what to do and how to answer, all in one call.
When it failed, there was no way to tell which decision was wrong.
Forcing retrieval alone would answer unsafe requests with articles.
Fix
Guard → router → sub-agents
A structured-output router behind a NeMo Guardrails input guard.
Four routes, each agent with its own prompt and tools.
Chitchat and refusals answered without retrieval.
ResultArticle hit 0.55 → 0.70 · conversational retrieval 0.75 → 1.00 · unsafe and chitchat filtered 0.94
Agentic RAG — Instead of always retrieving once and answering, an LLM agent decides whether to retrieve, which tool to call and with what query. Splitting those decisions across a router and specialised agents makes each one testable on its own.
FIX 1–2 mark where the evaluation found a defect (see below). The search tool's description doubles as the query-rewriting prompt: the agent searches with the query the model writes into the tool call, not the user's message.
Why a router, not standard RAG
Standard RAG, which searches with the question as is and answers in one shot, reaches the same article hit rate of 0.70. But it searches every input, including the 32 that should never be searched, and answered 6 of 16 violent or sexual requests using the retrieved articles as grounds, without refusing. Retrieved context beats the model’s tendency to refuse. A layer that decides what not to retrieve is what the router adds.
Same model, same golden sets
| Pipeline | Article hit | Faithfulness | Conv. retrieval | Unsafe / chitchat filtered |
|---|---|---|---|---|
| Standard RAG | 0.70 | 0.96 | 1.00 | 0.00 |
| Plain RAG (initial) | 0.55 | 0.84 | 0.75 | 0.97 |
| Agentic RAG (final) | 0.70 | 0.97 | 1.00 | 0.94 |
Plain RAG's 0.97 on the unsafe set isn't good filtering. It skipped retrieval often, one in four conversational messages. Without separate layers the two can't be told apart.
Evaluation: golden sets and five layers
The first evaluation couldn’t tell me what to fix, so before changing the agent I rebuilt how it is measured.
Problem
One VOC took two weeks
There was no way to tell which layer got it wrong.
The only metric was one retrieval score: 507 questions written from articles, scored by chunk hit@5.
Nothing checked whether a fix broke something else.
Root cause
The evaluation didn't match real users
Questions came from the articles; users don't know the proper nouns in them.
A different chunk of the right article counted as a miss.
Empty answers, skipped retrieval and unsafe replies never moved the score.
Fix
Three golden sets, five RAGAS layers
Layered 60 · conversational 48 · negative 32, in real users' phrasing.
Routing, retrieval, context, generation and safety scored separately.
Every VOC added as a reproduction question, with a regression gate on every PR.
ResultVOC processing 2 weeks → 2 days · version comparison 2–3 days → 30 min
RAGAS (Es et al., 2023) — A framework for evaluating RAG pipelines with an LLM as judge.
Layered · 60
Article questions6 categories × easy · medium · hard
"What technologies are being discussed to ease fears about EV fires, and what should drivers do?"
Conversational · 48
How people actually talklong context · no question mark
"I'm an economics student. I worry a lot about my career."
Negative · 32
Inputs not to searchchitchat · violence · sexual · prompt injection
What the system should not retrieve is part of the golden set too.
Scoring is per article, not per chunk, and each layer is scored separately so a failure points to where it happened.
Five layers
| Layer | Metrics | Why separately |
|---|---|---|
| L1 Routing | Retrieval rate · empty answers · route | If retrieval never ran, nothing below means anything |
| L2 Retrieval | Article hit · MRR · hit@k · nDCG | Deterministic and free, so it gates every PR |
| L3 Context · RAGAS | Context Precision · Context Recall | Sees useful context beyond the gold article |
| L4 Generation · RAGAS | Faithfulness · Response Relevancy | Right retrieval can still give a wrong answer |
| L5 Safety | Negative-set accuracy | Does it refuse what it shouldn't search |
What the evaluation found
This is what made VOC processing faster. Every VOC now becomes a reproduction question in the golden set. The layered run points to the failing layer, the fix goes into the architecture or the prompt (versioned in Langfuse), and the same golden sets plus the PR gate confirm nothing regressed before the update ships. Reproduce, diagnose and check for regressions are one evaluation run, so a VOC that took two weeks now takes two days. Neither defect below showed up in the retrieval score.
① L1 · routing
44% of conversational messages skipped retrieval
The router read "what would be good?" as a magazine request and worries without a question mark as chitchat. All 17 sent to the recommender also came back empty, on a path the old golden set never took.
FixRecommend only on an explicit magazine request; advice and worries go to retrieval.
0.56 → 1.00conversational retrieval rate
② L2 · query rewriting
Search queries didn't carry the question
The agent searches with the query the model writes into the tool call, and the tool's one-line description was the only rewriting prompt.
FixCompared 12 rewriting instructions within the noise band and adopted "one-sentence question, keeping clues from the background".
0.62 → 0.70article hit rate
LangGraph · Milvus · vLLM · RAGAS · Langfuse · NeMo Guardrails · GitHub Actions · Prometheus