Common measurements for competing systems
MLCommons brings industry and academic researchers together to build AI benchmarks and datasets. Its work includes detailed test protocols, software harnesses, held-out data and evaluation models. Shared artifacts matter because two implementations of a written benchmark can otherwise produce numbers that are difficult to compare. [1]
The organization grew from the MLPerf benchmarking effort and launched as MLCommons in December 2020. MLPerf measures performance across AI workloads and systems; AILuminate extends the consortium's measurement work to model behavior and risk. These answer different questions: how efficiently a system runs and how it responds to potentially harmful requests. [1][3]
From performance tests to safety grades
MLCommons released AILuminate v1.0 on December 4, 2024. Its AI Risk and Reliability working group developed tests across twelve hazard categories, including privacy, violent crime and self-harm. The launch provided grades for general-purpose language models and began with English, with other languages planned. Those were the capabilities of the initial release, not a description of every later version. [3]
The accompanying research describes a five-level grading system and an evaluation process for judging responses to hazardous prompts. For developers and purchasers, that supplies a way to compare behavior under a shared test rather than accepting a separate safety claim from every vendor. It remains a benchmark result for a particular system and setup. [4]
What a safety grade leaves open
The technical report identifies uncertainty in automated evaluation and the limitations of single-turn interactions. A chatbot's response to one prompt does not cover the behavior of an agent acting over many steps, every deployment context or every language. The authors identify these as areas requiring further work. [4][5]
MLCommons' benchmarks are technical measurements, not a government approval or a guarantee that a deployment is safe. Their value comes from explicit tests and comparable results; decisions about a particular application still require understanding what the benchmark measures and what it leaves outside its scope. [1][4]
Sources
- About
MLCommons · retrieved Oct 23, 2026
- AILuminate
MLCommons · retrieved Oct 23, 2026
- MLCommons Launches AILuminate, First-of-its-Kind Benchmark to Measure the Safety of Large Language Models
MLCommons · Dec 4, 2024
- AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
arXiv · Feb 19, 2025
- Safety Resources
MLCommons · retrieved Oct 23, 2026