Case 2 · Classifier Distillation

#distillation#dual-teacher#llm-judge#confidence

The production tagging agent is a structured-output LLM that tags each article with labels from a set of ~150. It costs 3.2 seconds per article. Distilling it into a small classifier fixes the cost, but the classifier also learns the teacher’s mistakes: its pipeline limits which labels it can give at all, and its confidences can’t tell a weak tag from a strong one.

I distilled it into a confidence-aware encoder classifier and added a second teacher, an LLM judge. Teacher 1, the tagger, supplies the labels to distill. Teacher 2 re-labels only the article–label pairs that the student’s confidence flags, where Teacher 1 is structurally weak.

Knowledge distillation (Hinton et al., 2015) — A small student model is trained to reproduce a larger teacher model’s outputs, often its full probability distribution rather than only the hard labels. The student keeps much of the teacher’s accuracy at a fraction of the inference cost.

Problem: A 3.2-second LLM call per article, with labels capped by the teacher's own pipeline: it can't see past its 20 candidates, can't abstain, and can't say which of its tags are weak.
Solution: Dual teacher. Teacher 1, the LLM tagger, is distilled into a confidence-aware encoder. Teacher 2, an LLM judge, re-labels the doubtful pairs that the student's confidence selects.
Result: 3,244 ms → 2 ms per article. Judge-based F1 0.614 → 0.801, recall 0.543 → 0.796.

Dual-teacher distillation · confidence-aware student

Two LLM teachers train it; serving is one encoder forward pass

Per-article tagging latency · log scale

Teacher 1 · LLM taggerlocal LLM · 2 vLLM servers3,244 ms

Student classifierone GPU2.13 ms

0.801Judge-based F1from 0.614 when trained on Teacher 1 alone

0.796Recallfrom 0.543 · precision 0.806

~900KPairs re-labeled by Teacher 2chosen by the student's confidence

Teacher agreement (R@n_gold) measures how closely the student copies Teacher 1. F1 measures agreement with labels Teacher 2 corrected, so the two numbers use different yardsticks.

The problem: a slow teacher with structural limits

Teacher 1 retrieves the 20 closest of ~150 labels by embedding similarity, and a local LLM then picks 3–10 of them with a confidence for each. Labeling 520K articles took 6 days on 2 vLLM servers. Distillation removes that cost, but a student that copies Teacher 1 also copies the limits below.

Teacher 1 step by step, and the project timeline

Input

Articletitle + body

The whole article goes to both steps.

Step 1 · retrieval

Embeddings + FAISSnarrow the candidates

Computes cosine similarity against ~150 label descriptions and passes the top 20 on as candidates.

20candidate labels

Step 2 · selection

Local LLMstructured output

Picks 3–10 of the candidates and attaches a confidence and an explanation to each tag.

3,244 msaverage per article

Output

2.6M tags520K articles

Only 7 outputs failed the schema.

5.03tags per article

One month of work · Aug–Sep 2026

Aug 30–Sep 5 · Teacher 1 labeling→Sep 7–9 · KD design · baseline→Sep 12–17 · input · encoder→Sep 18–22 · confidence head→Sep 22–27 · Teacher 2→Sep 26–28 · Stage 2

An EDA of Teacher 1’s 2.6M tags found four limits that come from its design, not from the LLM’s skill. A student trained only on Teacher 1’s labels learns all four, and no amount of distillation removes them.

① Top-20 blind spot

Nothing outside the 20

Retrieval passes only 20 of ~150 labels on as candidates. Any other label can't be tagged, even when it fits.

FixTeacher 2 judges labels outside the 20 that the student scores high.

② Forced choice

No "none of these"

The output schema only allows the 20 candidates, so the model can't abstain. ~3.2K tags have confidence 0.0, and 25% of explanations for tags under 0.3 contain "not". It writes "this article is not about X" and picks X anyway.

FixTeacher 2 re-checks Teacher 1's tags.

③ Rounded confidence

Picked numbers, not probabilities

Only 31 unique values, with 0.70 as the mode. Teacher 1's confidence can't filter out weak tags.

FixThe student's own confidence, per label, picks the doubtful pairs.

④ Twin labels

Only one of a pair

26 label pairs are near-synonyms. Teacher 1 outputs only one of each pair, and which one is close to arbitrary.

FixTeacher 2 judges the siblings; unjudged siblings are left out of the loss.

Teacher 1's confidence distribution

Teacher 1's confidence distribution · share of 2.6M tags at each value (%)

0.1

0.4

0.1

0.5

3.1

3.6

0.3

.55

9.4

1.9

.65

15.2

5.2

.75

12.7

10.7

.85

13.1

10.5

.95

13.1

Grey bars are the values below 0.5, excluded from training (4.3%). Eleven values at 0.05 steps make up 95% of all tags: the model doesn't compute a probability, it picks a rounded number. 0.70 sits far above its neighbours (0.65 at 1.9%, 0.75 at 5.2%), so it looks like a default for when the model can't decide, not "70% sure".

The fix: a second teacher and a confidence-aware student

Distillation fixes the cost and copies everything else, so I built two pieces around it. The student got a confidence-aware head that reports, for every label, how sure it is. Teacher 2, an LLM judge, checks one article–label pair at a time, and the student’s confidence decides which pairs it sees. Its verdicts replace Teacher 1’s labels for those pairs in a second round of fine-tuning. Both teachers stay in training; serving is the student alone.

Each problem, and what handles it

Problem	Handled by	Result
3.2 s LLM call per article	Distilling Teacher 1 into an encoder	3,244 → 2.13 ms
Weak tags	Teacher 2 re-checks Teacher 1's tags	Agreement with Teacher 2: 0.71 → 0.84
Correct labels skipped	Teacher 2 judges labels the student wants but Teacher 1 dropped	0.67 → 0.82
Top-20 blind spot	Teacher 2 judges high-scoring labels outside the 20	48% accepted · 0.69 → 0.77
Twin labels	Teacher 2 judges the siblings; unjudged siblings get no loss	0.846 → 0.896

Agreement is the student's accuracy against Teacher 2's verdicts in each group of pairs, before and after Stage 2.

Training · Teacher 1's labels, then Teacher 2's verdicts on the pairs the student doubts

The encoder and the ranking head learn only from soft cross-entropy. The evidence head learns only from the Beta loss, behind a stop-gradient. Verdicts enter both targets; replay articles keep Teacher 1's labels.

Inference · no LLM, one encoder forward pass

Not in the serving path

~~Teacher 1 (LLM)Embedding top-20Teacher 2 (judge)LLM call per request~~

Both teachers' decisions are distilled into the weights.

Serving cost

Latency: ~2 ms per article in batches · ~4 ms one at a time
Memory: ~1.2 GB VRAM on one GPU
Coverage: All ~150 labels scored, no top-20 limit

① Distillation from Teacher 1

Problem

A 3.2-second LLM call per article

Teacher 1 averaged 3,244 ms per article.

Tagging 520K articles took 6 days on 2 vLLM servers.

Root cause

A generative LLM decodes for every article

It reads the article and 20 candidate labels, then generates a confidence and explanation per tag, token by token.

The task is only to choose, but it paid for generation.

Fix

LLM as offline teacher, encoder for serving

Knowledge distillation from 2.5M LLM labels.

An encoder scores all ~150 labels in one forward pass.

Result3,244 ms → 2.13 ms per article on one GPU · teacher agreement R@n_gold 0.723

The student, an encoder with an MLP head, learns Teacher 1’s full soft label distribution over all ~150 labels rather than its top-1 pick, so the second and third tags on an article still count. A 512-token input cut off 90% of test articles; the summary the service already makes fixed that and improved every metric. A summary only helps a model that can’t read the whole article; it did nothing for Teacher 1. I chose the large encoder on the summary.

Student target, training labels and metrics

title + summary→encoder→2-layer MLP head→scores for ~150 labels

Target: Soft cross-entropy against Teacher 1's confidences, normalized to sum to 1. The student learns Teacher 1's label ranking as is. Training on the top-1 label alone would throw away 4/5 of the supervision and teach that the other labels are wrong.

Teacher 1's tags kept at each confidence threshold

None100% · 5.03 per article

≥ 0.5chosen95.7% · 4.83

≥ 0.691.8% · 4.64

≥ 0.765.3% · 3.33

≥ 0.860.1% · 3.06

≥ 0.923.6% · 1.37

0.5 removes only the 0.0–0.4 range, where Teacher 1 was forced to pick with no fitting candidate, and drops just 4.3% of tags. Going from 0.6 to 0.7 removes 26% at once, because it cuts the 0.70 peak Teacher 1 uses as a default.

R@n_gold: Recall when taking as many labels per article as Teacher 1 gave (n). Predicting more labels doesn't inflate it. Used for model selection.
macro_R@5: R@5 averaged over labels, so rare labels count as much as common ones.
top1: Whether the top-1 label is a Teacher 1 label, and whether it is right at the top-level category.

Encoder comparison: baseline, size × input, accuracy vs latency

The baseline, a base-size encoder on the first 512 tokens of the body, matched a Teacher 1 label at top-1 90.5% of the time from little more than the first paragraph. 512 tokens cut off 90% of test articles, while Teacher 1 reads the whole article. Switching the input to the summary fixed that and also made training 1.7× faster. Of the five options below, the large encoder on the summary is the second-fastest and second-most-accurate.

0.690R@n_goldval 0.690, no overfitting

0.946Top-level category top1top1 0.905

90%Test articles cut offat 512 tokens · Teacher 1 reads the whole article

Encoder size × input · test set

Encoder	Params	Input	R@n_gold	macro_R@5	top1
Base	113M	body, 512 tok	0.6903	0.5483	0.9046
Base	113M	summary	0.7057	0.5684	0.9133
Large	338M	body, 512 tok	0.7121	0.5909	0.9198
Large	338M	summary	0.7219	0.6058	0.9241
Long-context	341M	body, 2048 tok	0.7336	0.6266	0.9257

Changing size alone (input fixed at 512) from 113M to 338M adds +0.022 R@n and +0.043 macro_R@5. Changing length alone (size fixed) from 512 to 2,048 tokens adds +0.021 and +0.036. The summary's gain doesn't stack (base +0.015, large +0.010), because both fix the same defect: reading only the start of the article.

Accuracy and cost rank in opposite orders · per-article latency (log scale)

Teacher 13,244 msreference

Long-context · body 2koutput varies with batch47.28 msR@n 0.734

Large · body3.27 msR@n 0.712

Base · body2.23 msR@n 0.690

Large · summarychosen2.13 msR@n 0.722

Base · summary1.27 msR@n 0.706

The long-context encoder is the most accurate but 21× slower, and its block-sparse attention makes the output depend on batch size. The summary input is the fastest because summaries are short.

The chosen model, the large encoder on the summary:

2.13 msPer-article latencyin batches · about 4 ms one at a time · Teacher 1 3,244 ms

0.723Teacher agreementR@n_gold · baseline 0.690 · top1 0.924

1.2 GBVRAM peakTeacher 1 uses 2 vLLM servers

② Confidence-aware student

Teacher 1’s confidences are rounded picks, so they can’t say which tags are doubtful. I gave the student a Beta-evidential head that outputs a probability and an evidence level for each of the ~150 labels, without changing the ranking. The confidence is honest: 97.1% of pairs scored 0.95–0.99 matched Teacher 1.

Measured against Teacher 1, though, it stopped at 40% of the oracle gain, because many disagreements are Teacher 1’s own errors. Measured against an independent judge, the same confidence predicts the verdicts at AUROC 0.890 (0.74 against Teacher 1). That made it the router that decides what Teacher 2 sees.

Beta head design, and routing signals compared against an oracle

Beta distribution — A probability distribution on [0, 1] with two shape parameters, α and β. A model that predicts α and β instead of a single probability can express both an estimate (the mean α/(α+β)) and how certain it is: the larger α+β, the more concentrated the distribution.

The head reads detached encoder features and predicts Beta(a, b) for each label. Its mean p = a/(a+b) is the label probability, and vacuity = 2/(a+b) measures how little evidence there is.

Rankingencoder features→MLP head · soft CE→label ranking · R@n comes from here

Evidencereads detached features→Beta(a, b) per label→p = a/(a+b) · vacuity = 2/(a+b)

Ranking: Because the features are detached, the gradients of the ranking loss don't change. R@n moves by ±0.001 or less on all three baselines.
Signal: On a model trained with softmax, the Beta mean was a better per-label signal than a calibrated probability (AUROC 0.727 vs 0.705, and ECE 0.013 with no calibration).
Why not EDL: Textbook evidential loss, which folds the logits into the Beta, removes the gradient from rare positives that the model gets confidently wrong.

The chart below sends the most ambiguous 20% of articles back to Teacher 1 and measures how much of the oracle's gain each signal recovers. The disagreements it can't recover are mostly Teacher 1's own errors: twins and labels outside the top 20.

Sending the most ambiguous 20% of articles to Teacher 1 · share of the oracle gain recovered

Random0%R@n 0.706

Expected errors33%0.731

MC dropout std34%0.732

Vacuity40%0.736

Oracle100%0.781

③ Teacher 2: an LLM judge

LLM-as-a-judge — An LLM is prompted to grade or compare outputs against stated criteria, in place of human annotators. It is cheap and scales well, but its agreement with humans has to be measured, because judges have known biases such as position, verbosity and self-preference.

Teacher 1 can’t verify its own weak tags or consider labels outside its 20 candidates. Teacher 2, an LLM judge built on G-Eval, scores one article–label pair at a time, so it has neither limit. Judging every pair would bring back the cost distillation removed, so the student’s confidence picks four groups of pairs, one for each kind of Teacher 1 error. Teacher 2 judged 30% of the training set this way, about 900K pairs. It rejected half of Teacher 1’s tags, mostly low-confidence ones, and accepted 48% of the candidates Teacher 1 never saw.

Pairs sent to Teacher 2 · chosen by the student's confidence

Pool	What is asked	Teacher 1 limit	Pairs	Yes
pos	Tags Teacher 1 assigned	② forced choice · ③ rounded confidence	199K	49%
neg_dis	In the top 20, dropped by Teacher 1, but the student says yes	Correct labels skipped	27K	31%
out_hi	Outside the top 20, scored high by the student	① top-20 blind spot	42K	37%
twin	Twin sibling of a chosen label	④ twin labels	31K	17%
floor	Undisputed negatives · val only, to watch for leakage	—	5K	—

Counts are from the first 10% of the training set. A further 20% followed, about 900K pairs in all.

Judge design, agreement checks and acceptance by Teacher 1's confidence

91%Verdict agreement between two promptspilot · score correlation 0.877

48%Out-of-top-20 candidates acceptedlabels Teacher 1 never saw

0.890AUROC for predicting judge verdictsfrom the student's Beta mean · 0.74 against Teacher 1

G-Eval (Liu et al., 2023) — An LLM is given an evaluation prompt with chain-of-thought evaluation steps and a form-filling output format, and scores text on criteria such as coherence or consistency. The score can be weighted by the probabilities of the output score tokens.

article + one label→evaluation steps + rubric→JSON · reason + score 0–10→logprob-weighted → 0–1

This follows G-Eval directly: each pair gets a score and a one-sentence reason, on the same local LLM as Teacher 1.

Share of Teacher 1's tags that Teacher 2 accepted · by Teacher 1's confidence

≤ 0.607%

0.60–0.7519%

0.75–0.8540%

0.85–0.9568%

> 0.9591%

Teacher 2 rejects Teacher 1's weak tags and keeps its confident ones. On the first 10% alone it changed labels by +29K / −101K, taking positives per article from 4.82 to 2.37.

④ Stage 2: retrain on the verdicts

Stage 2 retrains the best Stage 1 student on the judged articles plus as many replay articles, swapping in Teacher 2’s verdict only for judged pairs. A control arm trained on Teacher 1’s labels alone separates the effect of extra training from the effect of Teacher 2’s signal.

ΔF1 over Stage 1, by which of Teacher 2's verdicts are used

Control+0.002 Remove only−0.006 + disagreements+0.037 All pools · soft+0.082 All pools · hard+0.100

The control matches Stage 1, so the later gains come from Teacher 2's signal. Only removing wrong tags raises precision by 0.15 but cuts recall by 0.07, so F1 drops: the student learns only "no". Each step that fills in labels Teacher 1 missed adds 0.04 or more.

The label-correction arms in full

Label-correction arms · judged 10% · judged test

Arm	Pools that use verdicts	F1	P	R	ΔF1
Stage 1	—	0.614	0.705	0.543	0
Control	none (Teacher 1's labels only)	0.615	0.676	0.565	+0.002
Remove only	pos	0.608	0.855	0.471	−0.006
+ disagreements	pos · neg_dis	0.651	0.854	0.526	+0.037
All pools · soft	+ out_hi · twin, raw score	0.697	0.860	0.586	+0.082
All pools · hard	+ out_hi · twin, 1 / 0 at 0.7	0.714	0.892	0.595	+0.100

Hard targets cut at 0.7 beat raw soft scores (F1 +0.016, AUROC 0.929 → 0.938); scores in the 0.3–0.7 range were noise. The evaluation gold comes from Teacher 2, so this F1 is agreement with the judge. Two checks support it: 91% verdict agreement between two prompts, and accuracy of 0.98 or higher on undisputed negatives.

Judge-based F1 · judged test · click a step for details

Stage 1 (Teacher 1 only)0.614

Student distilled from Teacher 1's labels, threshold 0.7. Precision 0.705 · recall 0.543 · AUROC 0.887.

Teacher 2's verdicts0.714 +0.100

Judged 10%, hard targets. Teacher 2's signal itself is the biggest step.

Longer training0.737 +0.023

Twice the epochs on the same data.

Judged 30%0.746 +0.009

Judged articles from 10% to 30%. The bottleneck was the quality of the supervision signal more than the amount of data.

Twin mask0.771 +0.025

Unjudged twin siblings are left out of the loss instead of being taught as negatives. The twin pool moved for the first time.

Threshold 0.550.801 +0.030

Serving threshold chosen on judged val. Final precision 0.806 · recall 0.796 · AUROC 0.946.

Teacher agreement (R@n) fell from 0.723 to 0.697. Teacher 2 rejects half of Teacher 1's positives, so this drop is expected by design.

Takeaways

Distillation

Input length and encoder size: +0.02 each

A summary is a cheap stand-in for length, but gains less on a larger encoder.

Confidence-aware student

Confidence needs an independent signal

Against Teacher 1's labels: 40% of the oracle gain. Against Teacher 2's verdicts: AUROC 0.89, so it routes pairs.

Dual teacher

Signal quality beat data volume

Teacher 2's signal on the same articles: F1 +0.10. Three times the judged data: +0.009.

Dual teacher

Fix labels by removing and adding

Removing Teacher 1's wrong tags alone: F1 −0.006. Adding the labels it skipped or never saw: +0.10.

Final model: Large encoder (338M) + Beta head on title + summary · tag if Beta mean ≥ 0.55 · F1 0.801 (P 0.806, R 0.796) · about 2 ms per article in batches, 4 ms one at a time