Case 3 · Self Evaluation and Refinement
A seven-week study of summary quality. The goal was to compress each article to 80% of its original length without losing facts. General articles passed, but special formats such as interviews, scripts and news-brief roundups quietly lost their structure and factual accuracy.
Self-Refine · ExpeL · Reflexion
Per-format rules, self-repair, human gates
Silent defects (%)
Silent defect = an auto-delivered summary that falls outside the length window (75–120% of target) or scores below 0.6 on G-Eval.
The task: 80% of the length, every fact kept
The target length is 80% of the original, and a result within 75–120% of the target passes. The publication date is passed in so that relative dates like “last year” become absolute dates, and any number in the output that isn’t in the source is flagged. Articles that pass become training data for the client. The work started from the compressor that still runs as the production service’s compression endpoint.
Problem
Summaries quietly broke on special-format articles
Structure broke or the length drifted out of range.
16.9% of auto-delivered summaries were below standard (silent defects).
For interview and dialogue articles it was 25.5%.
Root cause
One rule for every format, delivery without checks
With no per-format rules, every article went through the same prompt.
Nothing evaluated quality before delivery. The only checks were length and numbers.
Text not meant to be compressed, like puzzles and rosters, was processed anyway.
Fix
Per-format rules, self-repair, human gates
ExpeL — build per-format rules into Format Cards and route to them.
Reflexion — rewrite only the chunks the Judge flags.
Two human gates, plus DSPy GEPA prompt tuning.
ResultA blind evaluation met all three pre-registered conditions — silent defects 16.9% → 10.0%
Self-Refine architecture
The pipeline is a Self-Refine loop: the Compression Agent drafts, the Judge Agent critiques and the Compression Agent revises, with one local LLM in every role.
Self-Refine (Madaan et al., 2023) — One LLM drafts an output, writes feedback on its own draft, then revises using that feedback. The loop repeats until the feedback says to stop or an iteration limit is reached. It needs no training, no extra models and no reinforcement learning.
I borrowed ideas, not frameworks. ExpeL’s experience pool became the Format Card library, and Reflexion’s verbal feedback became the repair loop.
ExpeL (Zhao et al., AAAI 2024) — An agent gathers experience over a set of training tasks and distills it into natural-language insights by comparing successes with failures. At inference time it recalls those insights and similar successful trajectories as in-context examples, with no fine-tuning.
Reflexion (Shinn et al., 2023) — After an attempt fails, as judged by an external signal such as tests or an evaluator, the agent writes a short verbal reflection on what went wrong. The reflection is kept in an episodic memory and fed into the next attempt, so the agent learns through text rather than weight updates.
The middle row is the automatic path and the bottom row is the path to a human. Pick an article type to keep only the path it takes.
Every agent is the same local LLM, served with vLLM. Both gates write to an xlsx review sheet. When the reviewer's accept, edit or reroute comes back, the next run resumes from that point. DSPy GEPA tunes the compression, repair and card prompts offline from run logs.
The Profile Agent extracts the source's structure, format label and repeating unit, then picks the matching Format Card. The Compression Agent compresses chunk by chunk under the card's rules and fits the length to 75–120% of the target. Short general articles take the paraphrase path. The Judge Agent gives a verdict per chunk and quotes the source for every issue it flags. Deterministic checks on dates, numbers and card structure run alongside.
If errors remain, the same Compression Agent takes the Judge's critique as feedback and rewrites only the flagged chunks (Reflexion). A rewrite is kept only if errors drop and the length still fits. Two rounds at most.
Puzzles, phone directories and subscriber lists stop at the profile step, and no summary is written.
With no matching card, the article is held at Gate A. Similar held articles are clustered into a new card draft. Once a person approves it, the card joins the library and those articles take the automatic path from then on. The cause is a missing rule, so the fix outlives the article as a new card. 10.0% of all articles.
If an article still fails after two repairs, its state is saved and it is held at Gate B. "A rule exists but repair couldn't fix it" is a different cause, so it gets its own gate. 3.2% of all articles.
Design choices
- One local LLM for every role. Compression, profiling, verdicts, repair, card drafts and scoring all run on the same local LLM, with no external APIs.
- Typed verdicts. Verdicts come back as Pydantic objects through LangChain and json_schema guided decoding. About 2,000 offline tests run against a stub LLM.
- Experience as a card library (ExpeL). Cards cover interviews, scripts, subheaded articles, news briefs and lists. Articles that fit no card are clustered into a new card draft, which goes for approval.
- Fixing with verbal feedback (Reflexion). The Judge’s critique, quoting the source, becomes the feedback for each chunk’s repair call.
- Holding with state (human gates). Gates A and B pause an article with its state, send out a review sheet and take the decision back.
- Offline tuning and measurement (DSPy · deepeval). GEPA tunes prompts offline, and a guardrail rejects any prompt that breaks placeholders or length behavior. Quality is measured with G-Eval and a blind pairwise judge.
DSPy (Khattab et al., 2023) — A framework that writes LLM pipelines as composable modules with declared input/output signatures. Its optimizers tune the prompts and few-shot examples against a metric you supply. GEPA is one of those optimizers.
Results
All three pre-registered conditions held. The blind judge preferred the new pipeline by +0.45 net, silent defects fell 6.9 pp, and human review stayed at 13.2%, under the 15% limit.
Net preference by criterion
Results by format
Preference gains concentrate in the special formats (+0.59 to +0.74). General articles, included for comparison, gained only +0.06.
Silent defects · old → new pipeline, with net preference (small value below: content only)
Interviews fell from 25.5% to 20.9% with cards alone, and to 11.6% with self-refinement. For news-brief roundups and subheaded articles, nearly all of the defect reduction came from self-refinement; the cards mainly preserved structure.