Frozen inputs
Every scored asset must have documented ground truth, license or consent status, a stable identifier and a cryptographic hash.
Public evaluation lab
Deepfake detector benchmark protocol.
A versioned protocol for measuring image, video and audio detector performance under realistic quality changes. Rankings remain unpublished until the labeled dataset, run log and publication gates are complete.
WHY THIS EXISTS
A working upload is not evidence of detector accuracy.
Deployment smoke tests confirm that a route accepts a file and returns a report. Accuracy claims require documented ground truth, representative samples, locked detector versions and error-aware metrics. This page keeps those claims separate.
Locked runs
Detector name, model version, threshold, settings, timestamp, failure and cost are recorded before comparison.
Visible trade-offs
False positives, false negatives, abstentions and subgroup results stay visible instead of collapsing into one headline percentage.
DATASET MATRIX
Three modalities, three primary truth classes.
Protocol v1 requires at least 200 eligible assets per primary class and modality before a public score. Text-authorship detection is excluded because it needs a separate dataset and ground-truth design.
| Modality | Authentic capture | Fully synthetic | Partial manipulation |
|---|---|---|---|
| Image | Camera and edited photographs | Fully generated images | Face swaps and localized edits |
| Video | Original and conventionally edited clips | Fully generated video | Face, lip and identity manipulation |
| Audio | Recorded and conventionally processed speech | Text-to-speech and voice generation | Voice conversion and partial replacement |
QUALITY CONDITIONS
- Highest-quality available original
- Platform recompression
- Resize or crop
- Screen capture or screen recording
REQUIRED METRICS
Measure the decision cost, not only the leaderboard.
Balanced accuracy
Keeps uneven class sizes from hiding a weak class.
False-positive rate
Shows how often authentic material is incorrectly flagged.
False-negative rate
Shows how often synthetic or manipulated material is missed.
Macro F1
Weights each primary class equally.
Calibration error
Tests whether reported confidence matches observed frequency.
Abstention coverage
Reports how often a detector declines or cannot process a file.
Latency and cost
Measures operational performance alongside accuracy.
PUBLICATION GATES
Results publish only when every gate passes.
- Dataset frozen before scoring
No post-result swapping of difficult assets.
- Rights and consent reviewed
Source, license and allowed evaluation use documented.
- Leakage review complete
Capture and generator families separated where the evaluation design requires it.
- Every stratum reaches minimum size
Small convenient slices cannot stand in for the full protocol.
- Failures retained in denominators
Timeouts, unsupported files and abstentions do not disappear.
- Locked run reproduced
Tables and confidence intervals regenerate from the published inputs and run log.
PUBLIC RESULT REGISTER
No detector ranking is available yet.
This is deliberate. Publishing a placeholder score or comparing a few hand-picked files would create a stronger claim than the evidence supports. The first results table will appear here only after all v1 gates pass.
- Protocol
- v1.0.0 · published
- Dataset manifest
- pending
- Run manifest
- pending
- Results and confidence intervals
- pending
REPRODUCIBILITY & CORRECTIONS
A benchmark must be possible to challenge.
Each future release will retain its protocol version, dataset hash manifest, detector versions, thresholds, timestamps, failed requests and correction history. Material corrections will update the version log rather than silently replacing an earlier result.
Versioned changes
Protocol, dataset and run changes receive distinct version identifiers.
Threshold disclosure
Default and adjusted decision thresholds are reported separately.
No authenticity warranty
A benchmark result describes performance on the benchmark, not certainty for every future file.
Found a protocol flaw or reproducible error? Include the protocol version, affected section and supporting evidence.
Submit a correction