# Reproducible detector evaluation accounting — utility v1

This offline Node.js 22+ utility makes no network requests and does not run a
detector, charge for checks, provide a media dataset or certify product accuracy.
Download `evaluate-detector-run-v1.mjs` from this directory. The published
benchmark protocol remains the governing specification; all publication gates
still apply. The DeepfakePolicy public ranking remains unavailable.

## Freeze before scoring

Prepare `source.json` with an `assets` array. Each asset needs:

| Field | Required value |
| --- | --- |
| id | Unique stable identifier |
| sha256 | SHA-256 of the exact evaluated bytes, 64 lowercase hexadecimal characters |
| truth | authentic_capture, fully_synthetic, or partial_identity_or_content_manipulation |
| modality | image, video, or audio, as required by the protocol |
| condition | highest_quality_available_original, platform_recompression, resize_or_crop, or screen_capture_or_screen_recording |
| source | Source and provenance record |
| rightsBasis | Documented permission or applicable licence; the script does not verify rights |
| groundTruthEvidence | Evidence supporting the label, independent of the detector prediction |
| family | Capture/source or generator family used for leakage review and clustered resampling |

Do not include credentials, personal information or private customer files in a
public manifest. Preserve the exact bytes in your own authorised evaluation
environment. Identical SHA-256 values cannot count as additional samples.

Run `node evaluate-detector-run-v1.mjs freeze source.json locked.json`.
The command refuses to overwrite an existing output. Archive the locked file
before the first detector request; the script records its timestamp and content
hash. A local timestamp alone is not independent proof of preregistration.

## Record all outcomes

Prepare `run.json` with `manifestSha256` (the locked `assetsSha256`), ISO
`startedAt`, a specific `detectorVersion`, a written `decisionRule`, and `results`.
Each result has an asset `id` and an `outcome`: one of the three truth-class names,
`abstained`, or `error`. Missing result rows remain `unreported`. Do not delete
failures or difficult files. Document any mapping from native model output;
do not turn authentic-class confidence or an AI signal into a calibrated fraud
probability. Audio in the research protocol is not a promise of live audio
support in the current image/video product.

Optional recorded fields are `latencyMs`, `cost` with an explicit ISO `currency`,
and `probabilities` containing three named class probabilities summing to one.
Unknown latency, costs or probabilities remain unavailable. Never enter zero
merely because the value was not recorded.

Run `node evaluate-detector-run-v1.mjs score locked.json run.json draft.json`.
Use a new output path for each attempt and preserve the original result log.

## Interpret the draft

Operational accuracy and balanced recall retain failed, abstained and missing
outcomes in their class denominators. The separately labelled answered-only
FPR/FNR, specificity, sensitivity and macro-F1 apply only to answered files;
always read coverage and unanswered counts alongside them. Missing classes
produce unavailable balanced metrics, not invented zero or perfect performance.

The output includes the confusion matrix, quality and modality breakdowns,
recorded latency, costs by currency, and a fixed-seed family-cluster percentile
bootstrap. Family labels must reflect the actual dependence between variants;
the script cannot establish independence. Small strata or few families do not
support general accuracy claims. Calibration requires documented probabilities.

The protocol requires at least 200 examples per primary class and modality.
The utility reports the counts but always sets `publicationEligible: false`:
rights, provenance, leakage review, dataset sufficiency and independent
reproduction require separate review. Software unit-test labels are fictional
fixtures and must never be published as measured detector performance.