Working Knowledge
Search⌘K
07 · Measuring Models

MMLU

Massive Multitask Language Understanding

InterOperators

~16,000 multiple-choice questions across 57 subjects (law, medicine, math…). A rough proxy for broad knowledge — often the first number labs cite. Now so widely topped that it's losing signal.

Concrete example

You'll see “MMLU: 9x%” in launch charts; treat it as a floor, not a verdict.

HallucinationConfident fabricationHumanEvalCoding ability benchmarkModel CardDocumentation for a model releasePerplexityHow “surprised” a model is

Evidence

Sources for this definition

  1. 1
    Measuring Massive Multitask Language Understanding

    Hendrycks et al. · Research · checked 2026-07-13