A benchmark score tells you how a model performs on a public test set written by people you have never met. If you are buying an agent to screen invoices, review code or shortlist candidates, the question is different: does it decide the way your best person decides? That can be measured, and the method is cheap enough to run on a phone over a coffee.
The method in five steps
- Write synthetic cases. Twenty to fifty short cases of the decision in question. Synthetic means invented: no real customers, no real candidates, no real data. Include obvious yes/no cases, borderline ones, and a few with a planted flaw.
- Get the expert’s verdicts. The expert judges every case with a one-line reason. This is the only expensive step, and it is ten minutes on a phone.
- Hold out a golden set. Split the cases once and keep the split fixed. The held-out half is the golden set; the agent never sees its verdicts.
- Measure the noise floor. Run the same model twice on the golden set without any pack. If the two runs disagree with each other more than 10% of the time, stop: the measurement cannot distinguish the pack’s effect from randomness. We treat that as INDETERMINATE and never show a score.
- Measure before and after. Agreement with the expert without the pack, then with it. Publish both. A pack that does not beat the noise floor has not been shown to work.
Three rules that keep the number honest
- Minimum size. Below 30 golden cases the confidence interval is too wide to mean anything. INDETERMINATE, not a score.
- Fixed split. If cases move between train and golden, the number drifts upward for free. The split is content-hashed and never changes.
- Re-measure on model change. A pack verified on one model version says nothing about the next. The page shows the model and the date; when the model changes, the run is repeated.
What the number is not
It is not a certification and it has no legal weight. It is also not a quality score for the model; a model can be excellent and still disagree with your expert, because your expert has rules the model has never seen. That gap is exactly what a pack is supposed to close, and the before/after pair shows whether it did.
Running it yourself
The catalog’s verification runs come from an expert-knowledge engine that automates steps 1, 3, 4 and 5 and leaves steps 2 and the confirmation of extracted rules to the human. The free sample is ten cards. How verification works describes the thresholds; the submit page is where a run starts.