Speaker-Conditioned Deepfake Voice Detection
From “is this voice fake?” to “is this really them?”, backed by a PoC
This is company R&D in its PoC stage, so this page covers only what can be shared: the problem, the proposal and the PoC results.
The R&D plan was the voice version of the on-device deepfake image detector: one lightweight real/fake classifier on the phone that catches every synthetic voice. I doubted that would work while voice generators improve every few months, so before building it I measured a detector of that size. It missed about 80% of the latest cloned voices. I proposed a narrower alternative: build a profile of each contact’s voice and ask whether a call really sounds like them. A PoC compared both on the same audio, and management and the CTO changed the R&D plan to speaker-conditioned detection.
PoC · same audio, same threshold
Generic detector vs speaker-conditioned detection
Detection rate, with no real call from the person flagged
Generic = AASIST. Speaker = timbre + prosody compared with the contact's profile. 10 speakers, English read speech; 30 s = the voice sample given to the cloning tool.
Measuring the plan before building it
Problem
One generic detector on the phone
The plan: a lightweight real/fake classifier that catches every synthetic voice.
I tested a detector of that size on 800 Korean cloned voices first.
It missed 76.5–82.5% of the latest TTS.
Root cause
A chase it can't win
A generic detector is weak on generators it has never seen.
New generators ship every few months: new data, retraining, a new app release each time.
A phone's size budget means the model can't just grow.
Proposal
Ask "is it really them?", not "is it fake?"
Impersonation calls copy a voice the user already knows.
A profile from past calls with that contact gives a reference.
No need to know which generator made the fake.
ResultManagement and the CTO changed the R&D plan from a generic classifier to speaker-conditioned detection
Cloned voices missed by AASIST (~300K parameters), at 5% of real voices flagged
Korean cloned voices, 10 speakers. The smaller AASIST-L (~85K parameters) missed close to 90%. The newer the generator and the smaller the model, the more it misses.
AASIST (Jung et al., 2022) — The most widely used baseline for detecting spoofed speech. At ~300K parameters it fits on a phone, which made it the closest stand-in for the planned detector.
How the PoC made the case
A clone copies timbre, which voice the person has, more easily than prosody, how they speak: pitch and rhythm. In voice conversion, where a real person’s speech is given someone else’s voice, timbre moved most of the way to the target speaker while prosody stayed with the original. On the latest TTS, prosody alone wasn’t enough, so the final check uses both together.
I compared the two approaches on the same audio and threshold, and stated the weak points up front. The comparison, not an objection, is what changed the plan.
Detection rate, same audio, no real call from the person flagged
| Attack | Generic (AASIST) | Speaker-conditioned |
|---|---|---|
| Voice conversion · Seed-VC | 20% | 100% |
| XTTS-v2 · 30 s sample | 80% | 100% |
| Chatterbox · 30 s sample | 10% | 100% |
| Qwen3-TTS · 5 s sample | 55% | 85% |
| Qwen3-TTS · 30 s sample | 30% | 70% |
| OmniVoice · 30 s sample | 15% | 75% |
Weak points reported with it: the PoC fingerprint models have ~300M parameters each, far larger than AASIST; the best prosody measure was picked per condition; 10 English speakers, no phone-call audio yet. Next: a lightweight on-device encoder with per-contact profiles, and evaluation on Korean call audio.
PyTorch Lightning · Hydra · W&B · AASIST · WavLM · Seed-VC