On-Device Deepfake Image Detection

#mlops#ray#dvc#airflow#litert

An MLOps pipeline that tracks every stage, from training data to the phone

This is a company project, so this page covers only what can be shared: the goals, the system design, my contributions and the results.

A lightweight vision model that tells deepfaked faces from real ones on an Android phone, deployed in an education-office service. After the first release, accuracy could drop at training, conversion, quantization or in the app, and nothing recorded which. I proposed an MLOps pipeline and built it: Ray and DVC for data, W&B for experiments, and Airflow to evaluate every new data batch in five environments. An experiment now takes eight hours instead of three days, and a regression is traced to its stage within an hour instead of four days.

Period
Feb 2025 – Sep 2025 first release · Jun 2026 – present MLOps
Role
Proposed and built the MLOps pipeline; data, training, Android deployment
Scale
10M+ face images (50K Korean) · 20K-image regression set · 5 environments
Stack
Ray · DVC · Airflow · W&B · PyTorch · LiteRT · Kotlin

MLOps · Ray + DVC + Airflow + W&B

Before and after the MLOps pipeline

Time per task (hours, log scale)

100%Reproducible runsfrom about 30%
−41%GPU hours per experimentonly changed stages re-run
6Regressions stoppedbefore release

On this base: test AUC 64.1% → 85.6%, Korean-subset accuracy 54.7% → 88.4%, and 84.1% AUC on a real phone through the app's own capture and crop path.

What the pipeline changed

A drop in accuracy took four days to trace

It could come from training, conversion, quantization or the app.

Each stage was checked by hand, one at a time.

Only about 30% of experiments reproduced.

Nothing was recorded per stage

Six storage and label conventions mixed in the training data.

Backbone, input and preprocessing changed between runs, unlogged.

No fixed set measured every environment the same way.

Record every stage, compare automatically

Ray + DVC one schema for every dataset, versioned; only changed stages re-run.

W&B config, data version and metrics logged per run.

Airflow + adb every new data batch evaluated in five environments.

ResultExperiment 3 days → 8 h · 2 → 9 experiments a week · root cause 4 days → under 1 h

Airflow runs the same 20K-image regression set in five environments, from the GPU down to the app on a real phone. The drop from one environment to the next names the stage that broke.

Final model, test AUC per environment

EnvironmentAUCΔA drop here means
E0 · PyTorch on GPU85.6%—Reference
E1 · LiteRT FP32 on PC85.6%0.0Conversion error
E2 · LiteRT FP16 on PC85.3%−0.3Quantization loss
E3 · Phone, PC-made tensors85.2%−0.1Porting error
E4 · Phone, app capture + crop84.1%−1.1Capture or crop error

When a wider face crop dropped only E4, from 83.5% to 78.4%, the per-stage drop pointed to the phone's resize step in 52 minutes. A fix there brought E4 back to 84.1%.

Ray · DVC · Airflow · W&B · FiftyOne · PyTorch · LiteRT · Kotlin