Method
How a pack gets verified
Benchmarks tell you how a model scores on someone else's test. Verification answers a narrower question: does an agent with this pack decide the way a specific profession's expert decides?
Hatched = without the pack. Green = with it. Amber band = A/A noise around the baseline. The mark = how often the expert agrees with themself.
- 01Golden set
An expert from the named profession judges synthetic cases, never real people or clients. A fixed held-out split becomes the golden set. Minimum 30 cards; below that the result is INDETERMINATE and is never shown as a score.
- 02Noise floor
The same model is run twice without the pack. If the two runs disagree with each other on more than 10% of cards, the measurement is INDETERMINATE: the instrument is too noisy to read.
- 03Before and after
Agreement with the expert is measured without the pack and with it. Both numbers are published. A pack whose gain does not clear the noise band is not verified.
- 04Expert ceiling
Ten cards are shown to the expert twice. Their self-agreement is the ceiling: no pack can be expected to agree with an expert more often than the expert agrees with themself.
- 05Re-verification
When the model named on the page changes, the run is repeated and the page shows the new date. Old numbers never silently stand for a new model.
- 06EU AI Act checklist
No Art. 5 practice (for example emotion recognition at work) is ever listed. Packs that send messages must carry the Art. 50 disclosure. In Annex III domains only packs that keep a named human on every decision can be verified.
Tiers
Pilot, Verified, Verified+
| Pilot | 10 to 29 cards. The run exists and the report is public, but no number is shown: the interval is wider than the effect. |
|---|---|
| Verified | 30 or more cards, noise floor under 10 points, gain above the noise band, real expert, live model. |
| Verified+ | 100 or more cards, or two independent experts. The interval is narrow enough to compare two packs. |
Why 30?
Wilson interval for a binomial proportion. Below 30 cards the interval is wider than the gap between a good pack and a coin flip with a bias; that is why we publish no number there.
Who judges
Where the experts come from
Three kinds, each marked on the report. Own experts from the first tenants of the expert-knowledge engine. Vendor-supplied experts, allowed only when the golden set is sealed before they see the pack and the report names the conflict of interest. Independent reviewers, paid per golden set, recruited from the profession, never from the vendor.
Experts confirm each rule in a pack; legal rules cannot be overridden by an expert. A golden set is never reused across vendors without a fresh hold-out split.
Become a reviewer
You judge 30 to 100 synthetic cases from your own profession, on your phone, in two or three sittings. You are paid per golden set and named on the report if you wish. No real people, no real clients, no personal data.
Apply as a reviewerFor agents
Read the catalog without a browser
The same data as these pages, as JSON and as a remote MCP server.
curl -s https://measuredagents.com/catalog.json | jq '.packs[] | select(.status=="verified")'curl -s -X POST https://measuredagents.com/mcp -H 'content-type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_packs","arguments":{"query":"recruiter"}}}'Read-only, no key, no account. Tools: list_packs, get_pack, search_packs, list_professions. Add https://measuredagents.com/mcp as a remote MCP server in Claude, Cursor or any client that speaks Streamable HTTP.
Ready to be measured?
Every run starts with a free 10-card sample. Listing stays free either way.