Ai2 Introduces StateBench for Long Context State Tracking

Allen Institute for AI (Ai2) introduced StateBench in a blog post, a benchmark that tests how language models track changing information across long documents. It covers four levels of difficulty, from retrieving fixed values to following updates and dependencies between multiple variables.

Ai2 tested open pretrained models and four frontier models. Gemini 3.8 Flash, Claude Opus 5.5, GPT 6 Sol, and GPT 6 Luna performed well on the two easier levels, but accuracy dropped sharply on longer soluble and general finite tasks. Scores reached 13% or less on the longest general finite problems.

Researchers also trained transformer, hybrid, and Gated DeltaNet models with 275 million parameters directly on StateBench. Hybrid and Gated DeltaNet models maintained near perfect results across long sequences on the first three levels, while transformers declined as sequences grew and did not learn the hardest level. A Gated DeltaNet variant remained near perfect on the hardest tasks at about 10 times its maximum training length.