Deepfake detection, measured after the repost
The detectors work in the lab. This is where they break in the wild.
Every deepfake detector reports near-perfect accuracy. This is the benchmark that measures what happens after the file gets compressed, re-encoded, and reposted, across video, audio, and image. Open, reproducible, and honest about the gap.
The gap table
The number a paper prints is the clean, in-distribution one. The number that matters is the one after the file has traveled. This is both, side by side. Larger gap, bigger surprise.
| Detector | Modality | Status | In-distribution | After it travels | The gap |
|---|---|---|---|---|---|
dct-linear | ▣ image | reference | 95.4% | 64.4% | -31.0% |
frame-frequency | ▶ video | reference | 100.0% | 49.9% | -50.1% |
radial-spectrum | ▣ image | reference | 94.2% | 61.5% | -32.7% |
spectral-audio | ≈ audio | reference | 100.0% | 81.3% | -18.8% |
Degradation curves
Accuracy does not fall off a cliff at one magic setting; it bleeds out across a whole range of ordinary insults. Here is each detector's AUC as compression, resolution loss, and mild perturbation get worse. Every point carries a bootstrap confidence interval; hover for exact values.
dct-linear
referencefrequency · synthetic-images · Frank et al., Leveraging Frequency Analysis for Deep Fake Recognition, ICML 2020
AUC vs degradation severity. The dashed line at 0.50 is a coin flip: a curve that reaches it has no signal left.
frame-frequency
referencefrequency · synthetic-video · Frame-level radial spectrum (cf. Durall et al., CVPR 2020), mean-pooled
AUC vs degradation severity. The dashed line at 0.50 is a coin flip: a curve that reaches it has no signal left.
radial-spectrum
referencefrequency · synthetic-images · Durall et al., Watch your Up-Convolution, CVPR 2020
AUC vs degradation severity. The dashed line at 0.50 is a coin flip: a curve that reaches it has no signal left.
spectral-audio
referenceaudio-antispoofing · synthetic-audio · Spectral front-end baseline (cf. AASIST, Jung et al., ICASSP 2022)
AUC vs degradation severity. The dashed line at 0.50 is a coin flip: a curve that reaches it has no signal left.
Pending: the real detectors
These are wired up and waiting on a GPU run against the real, gated datasets. They show here with no numbers, on purpose. A benchmark that invents results for models it never ran would be the exact dishonesty this project exists to call out.
aasist
audio · audio-antispoofing
Jung et al., AASIST, ICASSP 2022
clip-universalfakedetect
image · transformer
Ojha et al., Towards Universal Fake Image Detectors, CVPR 2023
efficientnet-b4
video · cnn
Bonettini et al., Video Face Manipulation Detection Through Ensemble of CNNs, ICPR 2020
xception-ffpp
video · cnn
Rossler et al., FaceForensics++, ICCV 2019