What a Detector Smoke Test Can - and Cannot - Prove
A smoke test checks whether uploading, analysis and reports work. Measuring detection accuracy requires a separate evaluation.
Why we bother with a smoke test
A live detector is more than a model. The browser has to accept the file, the server must validate and transfer it, an analysis provider has to respond, and the interface must turn that response into something a person can understand. A failure anywhere in that chain can make a good model look broken.
After a deployment, we send a known input through each route and check that a structured result comes back. The aim is deliberately modest: catch disconnected services, upload errors, response changes and silent failures before users do.
What happened in the launch checks
We exercised text, image, audio and video. A text sample came back as AI-assisted. A generated blue image frame was marked fake. A pure tone produced a not-applicable audio result because it contained no useful speech. A generated test video returned suspicious.
Those were useful outcomes for connectivity. The audio response was particularly important: a system should be allowed to say 'this is not applicable' instead of inventing confidence just because the interface expects a verdict.
The claim we are not making
Four samples are not a dataset. They say nothing reliable about false-positive rates, unfamiliar generators, demographic performance, compression resistance or accuracy at a chosen threshold. The inputs were selected to exercise the product, not to earn a flattering percentage.
An accuracy study needs documented ground truth, enough authentic and synthetic examples, separate test conditions and metrics that show the trade-off between missed fakes and falsely flagged real material. A smoke test is plumbing. Calling it a benchmark would be misleading.
What comes after connectivity
Broader testing should mix clean originals with social-media copies, include media from varied sources and cover tools the detectors may not have seen during development. Failures belong in the report alongside successes. Results should be separated by format and condition rather than compressed into one impressive headline number.
We also watch the product around the analysis: privacy handling, upload limits, timeouts, refunds, error messages and the words used to describe uncertainty. A technically correct score can still mislead if the page presents it as proof.
How to use DeepfakePolicy today
Treat the report as a second opinion. Combine it with the source, context and any available provenance. Keep original files when the outcome matters. Never let a single automated result decide somebody's rights, reputation or money.
Product transparency includes admitting what has not been measured yet. We will revise this guide when a broader evaluation is ready, including the inconvenient results rather than only the examples that make the detector look good.
Automated results require source and context review.
Continue with independent verification.
Try the live detectorPrimary reading
We use original standards, regulators, public institutions and research papers wherever possible. Sources were last checked on 12 August 2026.