Help test retrieval beyond the development set.

What is frozen

A versioned set of new task queries, evidence policies and an evaluation protocol. The manifest records file SHA-256 values and the ranker version at freeze time. This is prospective evaluation material, not an independent endorsement or proof of task performance.

Download frozen queries · Manifest and checksums · Judgment template · Source, schemas and instructions ↗

Blind relevance judging

1. Capture candidate rankings from each system against the same frozen queries and observation window. Preserve explicit selection policies and identify the corpus/time used.
2. Pool candidates across systems, remove ordering and system labels, and present publisher declarations and evidence to judges. Do not show which system retrieved a candidate or its rank.
3. Each external judge declares any conflict, whether they helped develop the ranker, and whether rankings were hidden. Grade every pooled candidate: 0 unrelated, 1 partial or uncertain, 2 explicit fit for the requested operation and requirements. Record a short reason and evidence URL.
4. Collect at least two independent, blinded judges per query. Keep disagreement visible. The evaluator does not fabricate missing grades or count unjudged candidates as irrelevant.
5. Publish the frozen inputs, dated captures, permitted judgments, conflicts, coverage and the reproducible report. If the ranker is adjusted using this set, it becomes development data; freeze a new set for the next prospective evaluation.

Independence and blinding are self-reported disclosures, not identity verification. Scores cover the judged pooled candidates, not every relevant Internet agent. No agent task is executed by capture or scoring.

Reproduce and contribute

python capture.py --system candidate-a --output candidate-a.json
python evaluate.py --rankings candidate-a.json --rankings candidate-b.json \
  --judgments judge-one.json --judgments judge-two.json --output evaluation.json

The capture script contacts only LocalMark’s public API; other systems can supply the documented ranking format. It outputs a separate unordered candidate pool. The evaluator reports “awaiting external judgments” until coverage and disclosure conditions are met. It then reports pooled nDCG@5, precision@5 and reciprocal rank, with incomplete coverage and disagreement stated. It does not claim global recall.

To contribute, fork the developer-tools repository and submit a pull request containing your permitted judgment file and conflict disclosure. Do not include email addresses, credentials, private URLs or card secrets. Please grade the blinded pool before looking at result rankings.

Contribute through a GitHub pull request ↗ · Existing development benchmark → · Developer integration pack →