By asahifile 
When a frontier model ships, the world learns what it can do through a familiar ritual: a leaderboard, a handful of benchmark suites — coding, math, science, tool use — and a score to compare against rivals. A new piece of research from the staffing platform Mercor puts a hard number on how little of the American economy those scores actually describe. Drawing on the U.S. Department of Labor’s O*NET taxonomies, Mercor’s Benchmark Map — published October 15 — ties a curated set of 29 public AI benchmarks to the economy’s real task list and finds that only 208 of 923 occupations have a task with meaningful benchmark coverage. The company’s estimate: 90 percent of American working hours are spent on tasks with little or no coverage at all.
That gap matters well beyond academic scorekeeping. Enterprises comparing model vendors, policymakers forecasting labour-market effects, and workers weighing career bets are all currently relying on maps that, it turns out, cover roughly a tenth of the territory.
How the map was built
O*NET divides the U.S. workforce into 923 occupations grouped into 22 job families, with 18,797 discrete “tasks” catalogued across all of them. Mercor compiled 29 benchmarks selected for public availability, agentic character, and economic realism — including the company’s own APEX suite — and analysed them with an LLM judge that tags each benchmark prompt to the O*NET tasks it corresponds to, a methodology credit to Wang et al. That LLM tagging was validated against human expert review on a subset of the benchmarks.
A task counts as “covered” only if at least 5 prompts across the index map to it. Under that rule, 924 of 18,797 tasks are covered, and six of the 22 job families — Installation & Repair, Food Service, Protective Service, Transportation, Personal Care, and Production — contain no covered occupations whatsoever. To convert that into an hours-based figure, Mercor combined its coverage estimates with LLM-derived estimates of the working-time share of each task.
The coverage gap is not random
The research itself stress-tests whether the blind spots are accidental, and concludes they are structural. Benchmarks concentrate on tasks already rated as most “AI-exposed” — exposure-tagged tasks are about 6 times more likely to appear in benchmark coverage than unexposed ones, which is defensible when measuring model progress — but even after controlling for exposure, tasks from occupations with larger annual wage bills are significantly overrepresented. The study’s own illustration: heavily covered lawyers’ tasks sit alongside barely covered, similarly AI-exposed duties for legal secretaries and court clerks. Tasks rated most important to their occupation by O*NET domain experts are 6 to 10 times more likely to be benchmarked than marginal ones. The result is a scoreboard skewed toward high-wage, high-visibility knowledge work — software engineering and financial analysis — and near-silent on truck driving, gas plant operation, or food service.
Read the caveats before the headline
Mercor is unusually direct about its own limits, and the report deserves to be read with them attached. The 29-benchmark index is not a complete or representative sample of the universe of unsaturated, economically realistic agentic benchmarks, so any incomplete list biases the 90-percent estimate downward — and the bias is hard to bound. Per-family caps on benchmark count, meant to counter the overrepresentation of software tasks, deliberately understate coverage in the most-benchmarked families. Selection criteria such as “economic realism” introduce judgement calls that are hard to formalize. Independent coverage of the study, such as runtimewire’s, emphasizes that Mercor itself frames the numbers as provisional pending a larger index.
There is also a deeper measurement problem the report acknowledges: roughly 63 percent of U.S. jobs involve some physical-world component that today’s benchmarks — built on screens and APIs — cannot touch, and evals designed for automation say little about how much a model augments a human colleague doing the same job.
Why this matters more as agents spread
The practical risk is mispriced deployment. A procurement team that chooses between two models on leaderboard deltas is optimizing for the 10 percent of work that is measured; a policy office forecasting displacement from those same leaderboards will understate exposure in logistics, facilities, and food service while overstating it in elite software roles. Notably, the gaps sit in the mid-wage occupations where agent-based automation would be economically most consequential.
What to watch
- Whether independent labs reproduce or challenge the coverage methodology with a different benchmark index.
- Whether physical-task benchmarks and robotics evals emerge fast enough to close the 63-percent gap before agent deployments run ahead of measurement.
- Whether O*NET-weighted, occupation-specific model scores — which Mercor has begun prototyping — become the procurement standard.
Sources: Mercor Research — A benchmark map of the US economy; runtimewire.