Magazine Recommendation System

#llm#knowledge-graph#neo4j#recommendation

Recommending articles by why they are read, with an LLM facet graph

This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.

This started with my own use of the company’s service, the Moazine magazine app. Reading on it, I kept running into the same problems: the recommendations barely changed for weeks, and finding the few articles worth reading in an issue was left to me. I proposed a different approach: an LLM reads every article and records who reads it and why, those answers are linked into a knowledge graph, and the reader is recommended articles that connect to what they already read. I first built a working demo over 5,000 articles, which earned the go-ahead to develop it on the company’s real magazine archive. That version, over the full archive, beat the recommender in production and was adopted as an official project.

Period
Jun 2026 – present
Role
Proposer and lead: problem definition, design, PoC, evaluation
Status
Adopted as an official company project

LLM facet graph + embedding · three graders

Same 125 simulated readers: current recommender vs proposed

precision@10: share of 10 recommendations that match the reader's hidden interest

2.3×Recommendation hit rate0.373 → 0.859
+3.66Browsing satisfactionAgent4Rec · 95% CI [3.18, 4.11]
0.78After only 10 articles readcurrent recommender: 0.35

The only input is which articles a reader opened: no ratings, dwell time or search terms. "Current" reproduces the production recommender at article level.

Why not collaborative filtering

The app recommends whole magazines, and the usual fix, collaborative filtering (“people who read this also read…”), runs on exactly what the service lacks.

Recommendations went stale

New issues arrive once a month, so the list barely changes for weeks.

Readers read a few articles, but the unit of recommendation is a whole magazine.

There are few users to learn from.

Collaborative filtering needs a crowd

It needs many users whose reading overlaps.

An article nobody has read has zero signal, and every monthly issue starts that way.

Its only reason is "similar readers saw this", so a rejection says nothing about what to change.

Understand the content instead

Recommend articles, not magazines.

An LLM records why each article is read, and a graph links articles that share those reasons.

Every slot shows its reason; rejecting one lowers that axis on the next screen.

Resultprecision@10 0.373 → 0.859 · new articles become candidates the moment they arrive

Understanding content needs no users at all, so the approach works with one reader, and a new article is ready to recommend before anyone opens it. The same method works for whole magazines too, since a magazine is a bundle of articles.

The demo

The proposal started as this working demo over 5,000 articles. The public version below runs on entirely synthetic articles, so it shows the method, not the company’s content. Pick a profile, or enter interests and add articles you have read (search, or click them in the graph), then press 추천하기 (Recommend). The interface is in Korean.

Open the demo in a new tab ↗

The first load downloads about 10 MB of graph data.

Architecture

The heavy work happens once per article, offline. When the app opens, two tracks look for candidates from the reader’s history, and a re-ranker merges them into one screen.

Recommendation architectureOffline, every article goes through two tracks: an LLM extracts facets that are normalised into a knowledge graph, and the article body is embedded into a vector index. When the app opens, the reader's history of opened articles drives a graph walk over the facet graph and a per-interest nearest-neighbour search over the index. A re-ranker merges both candidate lists with diversity, context and exploration rules and fills a 10-slot screen, each slot with its reason.TRACK 1 · OFFLINELLM facet graphextract · normalise · linkTRACK 1 · ON OPENGraph walkPPR · shared facets · themesINPUTArticlesevery article · new issuesREADERReading historyopened article ids onlyMERGE · RE-RANKRe-rankerdiversity · context · exploreTRACK 2 · OFFLINEEmbedding indexone vector per articleTRACK 2 · ON OPENPer-interest searchcluster history · kNN eachOUTPUTHome screen10 slots · a reason eachLLM stepSearch · rankingData

The two tracks answer different questions: the graph asks why an article is read, the embedding what it is about. Only the graph can give a reason for a slot and take a reader's correction, and the re-ranker also uses its facets to cap repeated themes and pick exploration topics.

Personalized PageRank — A random walk that keeps restarting from the reader’s own articles, so the nodes it visits most are the ones closest to what they read. Approximated with Forward Push, it takes about 0.02 s per request.

The LLM facet graph

Categories say what an article is about. The facet graph adds who reads it, in what situation and for what need, so two articles on different topics connect when they serve the same need.

① Extract

Facets per articleneed · situation · audience · topic · entity · season

The LLM fills a fixed JSON schema from the text alone: empty when there is no evidence, no guessing age or gender.

② Normalise

Fixed vocabulary, then a graphfree text doesn't connect

95.7% of the raw need values appeared in only one article. Mapping phrases onto a fixed vocabulary and merging spellings of the same entity is what makes articles meet.

③ Find themes

Leiden communitiesthemes no category has

The first run melted everything into one cluster around a few hub facets; leaving out the most common facets fixed it. The LLM then names each community.

95.9%of articles fall into a theme

Here is one article from the demo, with what the LLM wrote and the graph node each phrase becomes.

「속도 다른 크루에 50대가 합류할 때」 When someone in their 50s joins a faster running crew · synthetic article

FacetLLM output (raw)Graph node (normalised)
Topic러닝 크루 어르신 합류 older runners joining a crew시니어 seniors
Topic동호회 속도 차이 pace gaps in a club동호회 clubs
Topic무리 없는 페이스 a pace that doesn't strain러닝 running
Topic관절 보호 스트레칭 stretching to protect joints스트레칭 stretching
Entity크루 리더 서도윤 crew leader Seo Do-yun서도윤
Entity밤그늘러닝클럽 · 밤그늘 러닝클럽 two spellings across articles밤그늘러닝클럽 one node
Need속도 차이로 인한 갈등 줄이기 easing friction over pacekept as written
Situation연령대 섞인 겨울 크루 러닝 a winter crew run across ageskept as written
Audience50대 신입 크루원 a new crew member in their 50skept as written

"어르신" (elders) is one of the aliases of 시니어, alongside 고령층, 노년층 and 실버 세대. That one node ties this sports article to 238 articles spanning 12 categories. In the demo only topics and entities are normalised; the production system maps need, situation and audience onto a fixed vocabulary too.

Evaluation: three graders

There is no human-labelled answer set yet, so I simulated 125 readers with hidden interests and scored both recommenders three independent ways in the same run. A conclusion counts only when all three point the same way.

Golden set

Against the hidden interestsprecision@10 · coverage · share KL

Did the screen match what the reader secretly likes, in the right proportions?

LLM as a judge

0–3 per articlescored twice, order reversed

Trusted only after it rejected irrelevant articles planted among the candidates.

Browsing agent

Agent4Recopen · skip · satisfaction

LLM agents page through the screen like users; judged by the paired 95% CI per reader.

Agent4Rec (Zhang et al., 2024) — LLM agents that imitate users, browsing recommendation pages and recording what they open and how satisfied they are.

Same 125 simulated readers, same run

MetricCurrentProposedChange
precision@10 ↑0.3730.8592.3×
Interest coverage ↑0.5970.822+38%
Share KL ↓ interest mix distortion0.3660.068−81%
LLM as a judge ↑ 0–31.0641.977+0.91
Browsing satisfaction ↑3.9927.648+3.66
Irrelevant articles opened ↓0.1060.072−32%

For a reader whose interests split 79:21, 8.6 of 10 slots match a hidden interest, against 1.0 for random. The final verdict comes from an interleaving test in the live app.

LLM · Knowledge graph · Neo4j · Personalized PageRank · Leiden · Embedding · LLM as a judge · Agent4Rec