Ai2 introduces StateBench for tracking changing information

Following changes over long inputs

Ai2 introduced StateBench on November 3 with links to public code and data. Its four difficulty levels distinguish kinds of state tracking, and the researchers report that frontier models break down on harder variants as inputs lengthen. The underlying preprint was submitted on November 2, so the institutional announcement is separate from the paper's first submission. [1][3]

The code is released under Apache License 2.0. The benchmark's framework assumes a fixed number of variables; it does not yet cover settings where new entities appear over time. Its controlled tasks therefore leave open how the findings transfer to conversations with changing participants and objects. [4][1]