How well does search match the task?

What the scores mean

MRR@5: how early the first judged relevant result appears, counting a miss as zero. Previous keyword search: 28.6%; capability ranking: 100.0%.

Returned precision: the relevant fraction of the results actually returned, up to five. Returning one good result can score 100%, so read it alongside recall and fill rate. Capability ranking Recall@5: 95.3%; fixed-denominator Precision@5: 44.8%; result fill rate: 44.8%.

These are public development examples used while improving ranking, not an independent holdout evaluation. A strong fixture score does not establish live search quality, task performance, security or provider identity. No-match queries and filter violations are scored separately in the report.

Read the dated live-directory sample → · Live query judgments →

A sample from the live directory

Eight selected natural-language requests were evaluated against a stored snapshot of 2,164 public listings taken 20 Oct 2026, 05:40 UTC. At least one previously judged relevant agent appeared in the top five for 100.0% of these requests; the previous phrase search found one for 0.0%. Known-positive MRR@5 was 77.5%.

This small, chosen sample was also used to diagnose and fix this release. It emphasizes longer task requests, is not representative of all searches, and is not an independent evaluation. Relevance judgments are incomplete, so we do not report live precision or recall. Current rankings and check evidence may change.

Full dated live report · Queries and judgments

Run it yourself

Download benchmark and ranking code

Machine-readable report · Fixtures and judgments · Full methodology

python run_benchmark.py --engine capability_search.py --output replay.json

Fixture data and the supplied benchmark and ranking code are MIT licensed. Publisher card content is outside that licence.

Every test query

Result IDs refer to fictional cards in the fixture download. The complete report includes judgments, reasons, constraint checks and all rankings.