Multi-Agent Magazine RAG Chatbot

#langgraph#milvus#ragas#agentic-rag

From plain RAG to router-based agentic RAG, guided by layered evaluation

This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.

A chatbot over a magazine archive. It answers questions about articles with grounded answers and recommends magazines when the user is browsing. I built the first version as a plain RAG and then improved it in two ways. I moved the decisions into a router that runs before any answer, and I built an evaluation that shows which layer failed, so every VOC (voice of the customer: user reports and outages) could be traced to a cause and the update shipped in two days instead of two weeks.

Period
Aug 2025 – Dec 2025
Role
Agentic RAG design, evaluation framework, prompt engineering, serving observability
Scale
An archive of 6,000+ magazine titles · golden sets of 140 questions
Stack
LangGraph · Milvus · vLLM · RAGAS · Langfuse · NeMo Guardrails

Agentic RAG · RAGAS + golden sets

Same model, same golden sets: initial plain RAG vs final agentic RAG

Quality scores (0–1)

2 daysVOC processingfrom 2 weeks · report to shipped update, outages included
30 minComparing versionsfrom 2–3 days
5 layersMeasured separatelyfrom one retrieval score

Article hit = the correct article is in the top 5. Conv. retrieval = share of 48 conversational messages (long context, worries described without a question) that went through retrieval.

Agentic RAG architecture

The router decides first, and only some requests reach retrieval. An input guard screens every message, then a router with structured output sends it down one of four routes: grounded answer, magazine recommendation, chitchat or refusal. Each sub-agent has its own prompt and tools. Chitchat and refusals are answered without retrieval.

The model decided whether to retrieve

Questions went straight to the LLM agent, with no input check.

One in four conversational messages skipped retrieval.

No step filtered out unsafe requests or chitchat.

Decisions and answers mixed in one model

Whether to search, what to do and how to answer, all in one call.

When it failed, there was no way to tell which decision was wrong.

Forcing retrieval alone would answer unsafe requests with articles.

Guard → router → sub-agents

A structured-output router behind a NeMo Guardrails input guard.

Four routes, each agent with its own prompt and tools.

Chitchat and refusals answered without retrieval.

ResultArticle hit 0.55 → 0.70 · conversational retrieval 0.75 → 1.00 · unsafe and chitchat filtered 0.94

Agentic RAG — Instead of always retrieving once and answering, an LLM agent decides whether to retrieve, which tool to call and with what query. Splitting those decisions across a router and specialised agents makes each one testable on its own.

Agentic RAG architectureA user question passes the NeMo Guardrails input guard, then the answer router. The router sends explicit magazine requests to the magazine recommender, chitchat and refusals to a direct reply without retrieval, and everything else to the RAG answer agent, which calls the hybrid search tool with a rewritten query. Langfuse holds prompt versions and traces; vLLM and GPU health monitoring track throughput, time to first token and errors. Fixes 1 and 2 mark where the evaluation found and fixed a defect.PROMPTS · TRACESLangfuseprompt versions · run pathsRECOMMEND · FIX 1Magazine recommendergroup search · single-digit msMONITORINGvLLM + GPU healththroughput · TTFT · errorsINPUTUser questionStreamlit chat UIGUARDInput guardNeMo GuardrailsROUTER · FIX 1Answer routerstructured output · 4 routesRAG ANSWERRAG answer agentgrounded answer + sourcesNO RETRIEVALDirect replychitchat · refusalSEARCH TOOL · FIX 2Hybrid searchdense + keyword · rerankermagazine requestchitchat · rejectrewritten queryAgent · local LLM on vLLMInput · guard · toolOps toolAnswered without retrieval

FIX 1–2 mark where the evaluation found a defect (see below). The search tool's description doubles as the query-rewriting prompt: the agent searches with the query the model writes into the tool call, not the user's message.

Why a router, not standard RAG

Standard RAG, which searches with the question as is and answers in one shot, reaches the same article hit rate of 0.70. But it searches every input, including the 32 that should never be searched, and answered 6 of 16 violent or sexual requests using the retrieved articles as grounds, without refusing. Retrieved context beats the model’s tendency to refuse. A layer that decides what not to retrieve is what the router adds.

Same model, same golden sets

PipelineArticle hitFaithfulnessConv. retrievalUnsafe / chitchat filtered
Standard RAG0.700.961.000.00
Plain RAG (initial)0.550.840.750.97
Agentic RAG (final)0.700.971.000.94

Plain RAG's 0.97 on the unsafe set isn't good filtering. It skipped retrieval often, one in four conversational messages. Without separate layers the two can't be told apart.

Evaluation: golden sets and five layers

The first evaluation couldn’t tell me what to fix, so before changing the agent I rebuilt how it is measured.

One VOC took two weeks

There was no way to tell which layer got it wrong.

The only metric was one retrieval score: 507 questions written from articles, scored by chunk hit@5.

Nothing checked whether a fix broke something else.

The evaluation didn't match real users

Questions came from the articles; users don't know the proper nouns in them.

A different chunk of the right article counted as a miss.

Empty answers, skipped retrieval and unsafe replies never moved the score.

Three golden sets, five RAGAS layers

Layered 60 · conversational 48 · negative 32, in real users' phrasing.

Routing, retrieval, context, generation and safety scored separately.

Every VOC added as a reproduction question, with a regression gate on every PR.

ResultVOC processing 2 weeks → 2 days · version comparison 2–3 days → 30 min

RAGAS (Es et al., 2023) — A framework for evaluating RAG pipelines with an LLM as judge.

Layered · 60

Article questions6 categories × easy · medium · hard

"What technologies are being discussed to ease fears about EV fires, and what should drivers do?"

Conversational · 48

How people actually talklong context · no question mark

"I'm an economics student. I worry a lot about my career."

Negative · 32

Inputs not to searchchitchat · violence · sexual · prompt injection

What the system should not retrieve is part of the golden set too.

Scoring is per article, not per chunk, and each layer is scored separately so a failure points to where it happened.

Five layers

LayerMetricsWhy separately
L1 RoutingRetrieval rate · empty answers · routeIf retrieval never ran, nothing below means anything
L2 RetrievalArticle hit · MRR · hit@k · nDCGDeterministic and free, so it gates every PR
L3 Context · RAGASContext Precision · Context RecallSees useful context beyond the gold article
L4 Generation · RAGASFaithfulness · Response RelevancyRight retrieval can still give a wrong answer
L5 SafetyNegative-set accuracyDoes it refuse what it shouldn't search

What the evaluation found

This is what made VOC processing faster. Every VOC now becomes a reproduction question in the golden set. The layered run points to the failing layer, the fix goes into the architecture or the prompt (versioned in Langfuse), and the same golden sets plus the PR gate confirm nothing regressed before the update ships. Reproduce, diagnose and check for regressions are one evaluation run, so a VOC that took two weeks now takes two days. Neither defect below showed up in the retrieval score.

① L1 · routing

44% of conversational messages skipped retrieval

The router read "what would be good?" as a magazine request and worries without a question mark as chitchat. All 17 sent to the recommender also came back empty, on a path the old golden set never took.

FixRecommend only on an explicit magazine request; advice and worries go to retrieval.

0.56 → 1.00conversational retrieval rate

② L2 · query rewriting

Search queries didn't carry the question

The agent searches with the query the model writes into the tool call, and the tool's one-line description was the only rewriting prompt.

FixCompared 12 rewriting instructions within the noise band and adopted "one-sentence question, keeping clues from the background".

0.62 → 0.70article hit rate

LangGraph · Milvus · vLLM · RAGAS · Langfuse · NeMo Guardrails · GitHub Actions · Prometheus