Magazine Recommendation System
Recommending articles by why they are read, with an LLM facet graph
This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.
This started with my own use of the company’s service, the Moazine magazine app. Reading on it, I kept running into the same problems: the recommendations barely changed for weeks, and finding the few articles worth reading in an issue was left to me. I proposed a different approach: an LLM reads every article and records who reads it and why, those answers are linked into a knowledge graph, and the reader is recommended articles that connect to what they already read. I first built a working demo over 5,000 articles, which earned the go-ahead to develop it on the company’s real magazine archive. That version, over the full archive, beat the recommender in production and was adopted as an official project.
LLM facet graph + embedding · three graders
Same 125 simulated readers: current recommender vs proposed
precision@10: share of 10 recommendations that match the reader's hidden interest
The only input is which articles a reader opened: no ratings, dwell time or search terms. "Current" reproduces the production recommender at article level.
Why not collaborative filtering
The app recommends whole magazines, and the usual fix, collaborative filtering (“people who read this also read…”), runs on exactly what the service lacks.
Problem
Recommendations went stale
New issues arrive once a month, so the list barely changes for weeks.
Readers read a few articles, but the unit of recommendation is a whole magazine.
There are few users to learn from.
Root cause
Collaborative filtering needs a crowd
It needs many users whose reading overlaps.
An article nobody has read has zero signal, and every monthly issue starts that way.
Its only reason is "similar readers saw this", so a rejection says nothing about what to change.
Fix
Understand the content instead
Recommend articles, not magazines.
An LLM records why each article is read, and a graph links articles that share those reasons.
Every slot shows its reason; rejecting one lowers that axis on the next screen.
Resultprecision@10 0.373 → 0.859 · new articles become candidates the moment they arrive
Understanding content needs no users at all, so the approach works with one reader, and a new article is ready to recommend before anyone opens it. The same method works for whole magazines too, since a magazine is a bundle of articles.
The demo
The proposal started as this working demo over 5,000 articles. The public version below runs on entirely synthetic articles, so it shows the method, not the company’s content. Pick a profile, or enter interests and add articles you have read (search, or click them in the graph), then press 추천하기 (Recommend). The interface is in Korean.
Architecture
The heavy work happens once per article, offline. When the app opens, two tracks look for candidates from the reader’s history, and a re-ranker merges them into one screen.
The two tracks answer different questions: the graph asks why an article is read, the embedding what it is about. Only the graph can give a reason for a slot and take a reader's correction, and the re-ranker also uses its facets to cap repeated themes and pick exploration topics.
Personalized PageRank — A random walk that keeps restarting from the reader’s own articles, so the nodes it visits most are the ones closest to what they read. Approximated with Forward Push, it takes about 0.02 s per request.
The LLM facet graph
Categories say what an article is about. The facet graph adds who reads it, in what situation and for what need, so two articles on different topics connect when they serve the same need.
① Extract
Facets per articleneed · situation · audience · topic · entity · season
The LLM fills a fixed JSON schema from the text alone: empty when there is no evidence, no guessing age or gender.
② Normalise
Fixed vocabulary, then a graphfree text doesn't connect
95.7% of the raw need values appeared in only one article. Mapping phrases onto a fixed vocabulary and merging spellings of the same entity is what makes articles meet.
③ Find themes
Leiden communitiesthemes no category has
The first run melted everything into one cluster around a few hub facets; leaving out the most common facets fixed it. The LLM then names each community.
95.9%of articles fall into a theme
Here is one article from the demo, with what the LLM wrote and the graph node each phrase becomes.
「속도 다른 크루에 50대가 합류할 때」 When someone in their 50s joins a faster running crew · synthetic article
| Facet | LLM output (raw) | Graph node (normalised) |
|---|---|---|
| Topic | 러닝 크루 어르신 합류 older runners joining a crew | 시니어 seniors |
| Topic | 동호회 속도 차이 pace gaps in a club | 동호회 clubs |
| Topic | 무리 없는 페이스 a pace that doesn't strain | 러닝 running |
| Topic | 관절 보호 스트레칭 stretching to protect joints | 스트레칭 stretching |
| Entity | 크루 리더 서도윤 crew leader Seo Do-yun | 서도윤 |
| Entity | 밤그늘러닝클럽 · 밤그늘 러닝클럽 two spellings across articles | 밤그늘러닝클럽 one node |
| Need | 속도 차이로 인한 갈등 줄이기 easing friction over pace | kept as written |
| Situation | 연령대 섞인 겨울 크루 러닝 a winter crew run across ages | kept as written |
| Audience | 50대 신입 크루원 a new crew member in their 50s | kept as written |
"어르신" (elders) is one of the aliases of 시니어, alongside 고령층, 노년층 and 실버 세대. That one node ties this sports article to 238 articles spanning 12 categories. In the demo only topics and entities are normalised; the production system maps need, situation and audience onto a fixed vocabulary too.
Evaluation: three graders
There is no human-labelled answer set yet, so I simulated 125 readers with hidden interests and scored both recommenders three independent ways in the same run. A conclusion counts only when all three point the same way.
Golden set
Against the hidden interestsprecision@10 · coverage · share KL
Did the screen match what the reader secretly likes, in the right proportions?
LLM as a judge
0–3 per articlescored twice, order reversed
Trusted only after it rejected irrelevant articles planted among the candidates.
Browsing agent
Agent4Recopen · skip · satisfaction
LLM agents page through the screen like users; judged by the paired 95% CI per reader.
Agent4Rec (Zhang et al., 2024) — LLM agents that imitate users, browsing recommendation pages and recording what they open and how satisfied they are.
Same 125 simulated readers, same run
| Metric | Current | Proposed | Change |
|---|---|---|---|
| precision@10 ↑ | 0.373 | 0.859 | 2.3× |
| Interest coverage ↑ | 0.597 | 0.822 | +38% |
| Share KL ↓ interest mix distortion | 0.366 | 0.068 | −81% |
| LLM as a judge ↑ 0–3 | 1.064 | 1.977 | +0.91 |
| Browsing satisfaction ↑ | 3.992 | 7.648 | +3.66 |
| Irrelevant articles opened ↓ | 0.106 | 0.072 | −32% |
For a reader whose interests split 79:21, 8.6 of 10 slots match a hidden interest, against 1.0 for random. The final verdict comes from an interleaving test in the live app.
LLM · Knowledge graph · Neo4j · Personalized PageRank · Leiden · Embedding · LLM as a judge · Agent4Rec