How firm is each rank?
Each bar is the 95% bootstrap interval of a model's strength, on a scale where 400 points means 10-to-1 odds of winning a vote. Select two models to see how often their order flips when the votes are resampled.
Does a simple length check move the ranks?
Left: rank from the main fit. Right: rank after adding one term for how much longer one reply is than the other. Heavy lines are moves whose bootstrap interval excludes zero.
Simple length check, not Arena's style control. It uses the token count in the dataset's own metadata and ignores formatting, tone and prompt difficulty.
Who gets the votes?
A battle counts once for each of the two models in it, so shares add to 100% of appearances. Providers come from a fixed table in the repo.
Battles by provider
Battles by model
What cannot be audited from outside
Arena's newer Alignment Index is not built from this dataset, and its inputs are not open. What an outside reader can and cannot check:
- Session counts differ between sources. The announcement cites 90,000 agent sessions; a third-party analysis reports 72,509 on the live leaderboard. This is commentary from that analysis. I have not verified either number.
- Judge validation is not published. I found no published check of how well the index's AI judge agrees with human reviewers.
- Rubrics are not fully published. The methodology post describes failure modes, not every rubric.
- No data release is mentioned for the index. The open data Arena points to is the human-vote data audited above.
Arena's own posts: Alignment Index methodology, Series B announcement, Do LLM judges favor themselves?