What an accuracy percentage leaves out

A headline accuracy percentage does not make that decision for them.

There is no single accuracy figure that applies to every deepfake detector, file type and use case. The useful question is more specific: how often does this system miss the material we need to catch, and how often does it send an ordinary file for unnecessary review?

That matters whether you are checking one unfamiliar video or choosing an AI image detection API for a busy review queue.

What the new research adds

DFD-Lab, a preprint submitted on 20 September 2026, reports external-test AUROC scores of 0.504, 0.538 and 0.458 across three implementations on a Deepfake-Eval-2024 subset. JPEG-augmented training improved AUROC but reduced accuracy. These limited experiments do not assess DeepfakePolicy or establish the performance of all commercial detectors.

For a buyer, the important habit is to ask what a number measures before deciding whether it is good. The UK government's market study also identifies inconsistent metrics and datasets as obstacles to meaningful comparisons.

Four numbers that answer different questions

Define the target class first: fully generated photos and face-swapped videos are different tasks. These definitions assume a binary decision; unresolved files need separate reporting.

AUROC asks a different question: how well do scores separate the two classes across thresholds? A value of 0.5 corresponds to chance-level ranking. It is not “50% of files correctly classified”, and a high AUROC does not specify the false-alert rate at your chosen threshold.

If a report shows an 85% score, ask what that score means. Do not silently turn it into an 85% probability of fraud. A score, a model's measured test performance and the truth of a particular claim are three different things.

  • Accuracy — The share of all classifications that are correct.
  • Recall — The share of target AI-generated or manipulated files detected.
  • False-positive rate — The share of non-target files incorrectly flagged.
  • Precision — The share of flagged files that actually belong to the target class.

A detector can be mostly right and still fill your queue with false alerts

Consider this deliberately invented example. It is not a DeepfakePolicy benchmark or an estimate of fraud prevalence.

You receive 10,000 files: 100 belong to the target AI-generated class and 9,900 do not. Suppose a detector catches 90 of the 100 target files and incorrectly flags 2% of the others. All files receive a decision.

Overall accuracy is 97.92%: 9,792 correct decisions out of 10,000. Yet only 31.25% of the 288 alerts concern target files. The team still has to deal with 198 false alerts.

Both numbers are correct. They describe different parts of the same workload.

This does not mean the detector is useless. It means you should budget for review, decide what happens after an alert and avoid treating an alert as an accusation. A system that helps someone find suspicious material is doing a different job from one that automatically refuses a customer.

  • Target files correctly flagged — 90
  • Target files missed — 10
  • Non-target files wrongly flagged — 198
  • Non-target files correctly cleared — 9,702

Test the file your team receives, not just the original

An original camera file and a screenshot of that file should not be treated as interchangeable test inputs. NIST's forensic evaluation work specifically addresses generalisation and resistance to post-processing, including compression.

Our practical recommendation is to keep related versions together and compare their results. Try an original, the version exported by your upload form and a version sent through the channel customers actually use. Record which transformation produced each file.

For video, establish what the service examines. A still frame cannot answer every question about a clip. Ask about generated visuals, face manipulation and audio separately; do not treat a missing signal as a clean result. Our AI video detector page explains the scope of a video check, while the AI image detector covers photos.

What belongs beside an accuracy claim

Before relying on a result, we recommend asking for a short test record: the task, the labelled sample, the test date, the system version where available, the decision rule and the counts behind the percentages.

Keep an unanalysed file visible in that record. “Unsupported”, “failed” and “inconclusive” are not alternative spellings of “authentic”. A test with no observed false positives is encouraging, but a small sample does not establish a zero error rate.

We apply the same distinction to our own claims. DeepfakePolicy's published 98.8% image accuracy is product-reported, not independent certification, video accuracy or a probability for an individual file. See our methodology and limitations.

For the complete procurement workflow, use our deepfake detection service evaluation checklist. This article explains the numbers; that guide covers how to run and document the comparison.

Turn the result into a reviewable record

The next person should be able to tell which file was checked, what the system returned and what remains unresolved. A polished PDF cannot supply evidence that was missing from the original material.

For a small evaluation, start with a labelled set you are authorised to use. If you need integration, explore the photo and video API; inspect JSON and PDF report options before building your review process around them. A sandbox can test integration behaviour, not detection accuracy on your files.

DeepfakePolicy sells media checks and has a commercial interest in this topic. The useful standard is the same for us and for any other service: show what happens on relevant files, count the mistakes and keep the final business decision separate from the detector score.

FAQ

Frequently asked questions

Can a deepfake detector give a false positive?

Yes. A genuine photograph or clip can be flagged incorrectly. Ask for the original and review the available evidence before making a consequential decision.

Is the most accurate AI image detector also the best video detector?

Not necessarily. A photo benchmark does not establish performance on video, faces or audio. Compare results for the task you actually need.

Does a low score prove that a video is real?

No. It means the check did not return a strong target signal under its conditions. It does not verify the caption, identity or event shown.

Automated results require source and context review.

Continue with independent verification.

Check an image
Sources

Primary reading

We use original standards, regulators, public institutions and research papers wherever possible. Sources were last checked on 23 September 2026.