Why I stopped chasing benchmarks
Leaderboards reward the wrong kind of progress. Here is what I measure instead.
For three years, most of my working hours were organised around a single number. If the number went up, the week was good. If it stayed flat, I would find another trick, another hyperparameter, another few thousand training steps.
The number did go up. But somewhere along the way I noticed that the systems I was building were not getting much better at the things people actually needed them to do.
What a benchmark actually measures
A benchmark is a sample of a task, frozen at a moment in time. It is useful exactly as long as nobody optimises against it directly. Once a field starts competing on it, the score slowly stops meaning what it used to.
When a measure becomes a target, it ceases to be a good measure.
None of this is new. Economists call it Goodhart’s law. What surprised me was how quickly it happened, and how hard it was to notice from inside.
What I measure instead
These days I keep a short list of questions that no leaderboard answers:
- Where does it fail? I read fifty failures by hand before I look at any average.
- Who does it fail? Accuracy on the languages and dialects that are rarest in the training data.
- What does it cost? Memory, latency and energy, reported next to every result.
It is slower. The numbers are less impressive in a slide deck. But I trust them, and the work that came out of it is the best I have done.