On-Device Deepfake Image Detection
An MLOps pipeline that tracks every stage, from training data to the phone
This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.
A lightweight vision model that tells deepfaked faces from real ones on an Android phone, deployed in an education-office service. After the first release, accuracy could drop at training, conversion, quantization or in the app, and nothing recorded which. I proposed an MLOps pipeline and built it: Ray and DVC for data, W&B for experiments, and Airflow to evaluate every new data batch in five environments. An experiment now takes eight hours instead of three days, and a regression is traced to its stage within an hour instead of four days.
MLOps · Ray + DVC + Airflow + W&B
Before and after the MLOps pipeline
Time per task (hours, log scale)
On this base: test AUC 64.1% → 85.6%, Korean-subset accuracy 54.7% → 88.4%, and 84.1% AUC on a real phone through the app's own capture and crop path.
What the pipeline changed
Problem
A drop in accuracy took four days to trace
It could come from training, conversion, quantization or the app.
Each stage was checked by hand, one at a time.
Only about 30% of experiments reproduced.
Root cause
Nothing was recorded per stage
Six storage and label conventions mixed in the training data.
Backbone, input and preprocessing changed between runs, unlogged.
No fixed set measured every environment the same way.
Fix
Record every stage, compare automatically
Ray + DVC one schema for every dataset, versioned; only changed stages re-run.
W&B config, data version and metrics logged per run.
Airflow + adb every new data batch evaluated in five environments.
ResultExperiment 3 days → 8 h · 2 → 9 experiments a week · root cause 4 days → under 1 h
Airflow runs the same 20K-image regression set in five environments, from the GPU down to the app on a real phone. The drop from one environment to the next names the stage that broke.
Final model, test AUC per environment
| Environment | AUC | Δ | A drop here means |
|---|---|---|---|
| E0 · PyTorch on GPU | 85.6% | — | Reference |
| E1 · LiteRT FP32 on PC | 85.6% | 0.0 | Conversion error |
| E2 · LiteRT FP16 on PC | 85.3% | −0.3 | Quantization loss |
| E3 · Phone, PC-made tensors | 85.2% | −0.1 | Porting error |
| E4 · Phone, app capture + crop | 84.1% | −1.1 | Capture or crop error |
When a wider face crop dropped only E4, from 83.5% to 78.4%, the per-stage drop pointed to the phone's resize step in 52 minutes. A fix there brought E4 back to 84.1%.
Ray · DVC · Airflow · W&B · FiftyOne · PyTorch · LiteRT · Kotlin