ERA Lab — LLM Evaluation Infrastructure & Benchmark Integrity
Problem
Language model evaluation benchmarks are worthless if training data has leaked into them. Tracking contamination at scale means fingerprinting hundreds of thousands of benchmark items, storing only hashes — never the text itself — and keeping fingerprint manifests provably consistent across the entire evaluation pipeline.
Approach
Contributed to fingerprinting infrastructure covering 17 benchmarks — MMLU, GSM8K, GPQA, AIME 2024/2025, Codeforces, LiveCodeBench — producing 8.4M SHA-256 n-gram hashes across 126 files for 127,696 items. Validated benchmark coverage across manifests and SHA256SUMS files, addressed missing registry entries surfaced during review, added cross-platform test fixes, and strengthened regression-test coverage (25 passed, 1 skipped).
Outcome
A shipped fingerprint set scanning ~225 documents per second on a single core with a 0.11% false positive rate across 19,000 test documents. Every benchmark is pinned to a commit hash and SHA-256, so contamination detection is both auditable and extensible — new benchmarks slot in through a TOML registry without touching the scanning logic. Contributed to an evaluation suite of 24 approved benchmarks for reproducible assessment of open-weight language models.