If you’ve ever aced a test only to bomb a retake with slightly reworded questions, you already understand the problem IBM just put a number on. The company’s research division has published a paper introducing BenchDrift, a tool designed to measure how much large language model benchmark scores shift when test prompts are rephrased without changing their meaning.
What BenchDrift actually measures
The core idea is deceptively simple. Take a benchmark question, rewrite it in a way that preserves its meaning and correct answer, then see if the model still gets it right. BenchDrift does this systematically across four distinct axes: linguistic, referential, pragmatic, and structural.
BenchDrift generates these variations at scale, then tracks what researchers call “bidirectional correctness flips,” instances where a model’s answer switches from right to wrong or wrong to right based solely on phrasing.
The research team, led by Shailja Thakur, Sungeun An, Chad DeLuca, and Hima Patel at IBM Research, tested eight models across three widely used benchmarks: GSM8K (grade-school math), MMLU (massive multitask language understanding), and MATH-Hard (advanced mathematical reasoning). Drift showed up everywhere.
Stronger models, bigger problems
Weaker models actually tended to benefit from rephrased questions, picking up correct answers they previously missed. Stronger models, meanwhile, showed increased vulnerability. They lost significantly more accuracy when questions were reworded, even when the underlying meaning stayed identical.
High confidence scores can deteriorate with minor changes in question length or style, meaning that even when a model appears certain about its answer, that certainty may be anchored to specific phrasing rather than actual comprehension.
Why benchmark fragility matters beyond the lab
BenchDrift joins a growing body of evidence suggesting this evaluation infrastructure has fundamental weaknesses. IBM’s own prior work includes projects like ITBench and BenchmarkCards, both aimed at improving transparency in AI assessment. BenchDrift extends that effort by providing a quantitative framework for what the researchers call the “wording effect,” the measurable gap between what a benchmark score claims and what it actually demonstrates.
The tool has been released as open source through a GitHub repository (IBM/BenchDrift), which includes the full pipeline code, Jupyter notebooks, and configuration files needed to generate question variations, validate that those variations preserve meaning, and detect correctness drift.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
