Case 1 · MLOps: Monitoring and Recovery

#vllm#monitoring#dagster#recovery

A batch job translates and summarizes 24K articles on a local LLM served by vLLM. Running it surfaced two problems: the vLLM dashboard showed the GPU recomputing prompts it should have cached, and a single failed summary re-ran its whole article.

Efficiency: A prefix-cache hit rate of only 24.7% on the vLLM dashboard. Parallel processing and restructured prompts raised it to 45.7%.
Recovery: A failed summary re-ran its whole article, and failures were buried in the result files. One Dagster step per agent call made each failure visible, and only the failed step re-runs.
Result: Prefill compute −28%, prefill time −10%, TTFT −8%. Recovery tokens −37% versus re-running failed articles.

vLLM monitoring · Dagster step-level re-runs

Monitoring found both problems; restructuring the pipeline fixed them

Prefix-cache hit rate

Before24.7%

After45.7% +21.0 pp

Tokens spent recovering 20 failures

Whole batch1.61M · 446 s

Failed articles276k · 89 s

Failed stepsDagster174k · 56 s

−28%Prefill compute

−10%Prefill time

−8%Time to first token

−37%Recovery tokens

Efficiency: the prefix cache

vLLM exports Prometheus metrics such as prefix-cache queries and hits, cached prompt tokens, prefill time and time to first token. Together with GPU health, they feed a Grafana dashboard I watch during batch runs. During the summarization batch the prefix-cache hit rate sat at 24.7%, even though many requests share the same article text.

vLLM automatic prefix caching — vLLM keys each KV-cache block by a hash of its tokens and every token before them. A request that shares a prompt prefix with an earlier one reuses those blocks and skips their prefill, the pass over the prompt before the first output token. Unused blocks are evicted least-recently-used first, so request order decides how much is reused.

Problem

The same prompt prefix was recomputed

Many requests share the same article text.

Each one prefilled it again instead of reusing it from the cache.

Diagnosis

Found while monitoring vLLM and GPU health

The dashboard showed a prefix-cache hit rate of only 24.7%.

Shared prefixes were evicted before they could be reused.

Fix

Parallel processing and prompt restructuring

Changed the pipeline to parallel processing.

Restructured the prompts to share a longer common prefix.

ResultSame requests, same model, less GPU compute — prefill compute −28%, prefill time −10%, TTFT −8%

Fix: parallel processing and prompt restructuring

I changed the pipeline to parallel processing and restructured the prompts, so requests that share a prefix reuse it from the cache. The requests and the model stayed the same.

Same requests, before vs after

Metric	Before	After
Prefix-cache hit rate	24.7%	45.7%
Prompt tokens computed	18.24M	13.20M
Total prefill time	3,259 s	2,923 s
Time to first token	108 ms	99 ms

Recovery: re-run the failed step

If one step failed, the old batch script re-ran the whole article, redoing the translation and summaries that had already succeeded.

Problem

One failed summary re-ran the whole article

Usually one language failed, yet the re-run redid the translation and the good summaries.

Failures sat inside each article's result; finding them meant scanning.

Root cause

The recovery unit was the article

The batch script could only resume per article.

Per-step outputs could not be reused.

Fix

One agent call, one Dagster step

A branch per article, a step per agent call, article ID in the step key.

Good upstream outputs load from storage; only the failed step re-runs.

ResultLess time finding failures, fewer tokens recovering them

Comparing recovery granularity

Switch the recovery granularity to see the difference. Each tile is one article: left half English summary, right half Korean summary.

Recovery method Failures Re-run batch Failed articles Failed steps · Dagster

The Korean summary failed for 20 of 100 articles. Every English summary succeeded. The granularity at which you recover these failures sets the cost.

Re-running the whole batch regenerates the 80 successful articles and every successful English summary. You pay the original cost a second time.

Re-running only the failed articles still regenerates their 20 successful English summaries. This is how the batch script worked, since it could only resume per article.

Dagster re-runs only the 20 failed steps and loads the 20 successful English summaries from storage.

Recovering 20 failed steps	Requests	Tokens	Time
Re-run whole batch	1,915	1.61M	446 s
Re-run failed articles	338	276k	89 s
Dagster · failed steps only	205	174k	56 s