ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
Abstract
A diagnostic benchmark spanning eight domains that evaluates LLMs on six dimensions of reasoning about actions and change, including new ramification constraints for indirect effects. State-of-the-art models struggle across all dimensions, especially ramifications.
Leaderboard
Models evaluated on ActionReasoningBench. Scores are accuracy on the free-answer questions (zero-shot chain-of-thought, single-action sequences); Avg is the mean over the six dimensions. Accuracy falls further at longer action sequences and collapses once ramification constraints are added (see below).
| Model | Avg | Fluent | State | Exec. | Effects | Numerical | Composite | |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-4o | 57.8 | 88.5 | 73.3 | 27.1 | 82.5 | 11.1 | 64.5 |
| 2 | Fine-tuned Llama-3.1-8B | 54.3 | 76.9 | 75.6 | 31.3 | 60.5 | 8.9 | 72.4 |
| 3 | Llama-3.1-70B-Instruct | 47.4 | 71.2 | 64.4 | 33.3 | 67.5 | 4.4 | 43.4 |
| 4 | Llama-3.1-8B-Instruct | 19.9 | 30.8 | 28.9 | 8.3 | 20.0 | 6.7 | 24.6 |
o1-preview was additionally evaluated on the ramification subset — see below. Numbers are percentages.
Reasoning about actions and change
Reasoning about Actions and Change (RAC) is a foundational AI problem — track a world as actions transform it, and answer questions about the resulting state. It underpins planning, commonsense, and agents acting in dynamic environments, yet how well LLMs actually do it was underexplored. ActionReasoningBench is a diagnostic benchmark that stress-tests this across eight planning domains and six reasoning dimensions — and, critically, adds ramification constraints: the indirect effects of actions that a reasoner has to infer rather than read off.
unstack(A,B) has
direct effects the action states outright, and a ramification — an
indirect effect (B becomes clear) that follows only from a domain constraint. Models cope with the
direct effects but collapse on ramifications, which is exactly what ActionReasoningBench isolates.Six dimensions of RAC
Questions come in two answer formats — binary (true/false) and free-answer (a set of objects, actions, or fluents) — and probe six dimensions. The averages below (accuracy across the evaluated LLMs) show a clear cliff: the four dimensions long studied in RAC hold up, but the two the paper introduces collapse.
| Reasoning dimension | What it asks | Avg accuracy |
|---|---|---|
| Fluent Tracking | properties of an object after the actions | 73.6% |
| State Tracking | the complete set of fluents in the final state | 65.6% |
| Action Executability | which actions can (or can't) run | 58.7% |
| Effects of Actions | outcomes of performing a given action | 62.4% |
| Numerical RAC | counts & quantities over the state | 33.2% |
| Composite Questions | up to three dimensions combined, multi-step | 51.2% |
The two novel dimensions — Numerical RAC and Composite — drop average accuracy by 17.9%. Numerical reasoning over a symbolic world state is especially brittle.
The ramification challenge
The sharpest result comes from turning ramification constraints on. With indirect effects in play, frontier models fall off a cliff: evaluated with zero-shot chain-of-thought, GPT-4o solved none of the ramification questions, and o1-preview reached only 18.4% — despite being a reasoning-tuned model. Tracking what an action directly does is one thing; inferring everything that must then be true because of the world's rules is a different, much harder problem, and current LLMs largely can't do it.
How the benchmark is built
Every question is grounded in a real planner, not hand-authored, via a four-stage pipeline over eight IPC domains (Blocksworld, Depots, Driverlog, Gripper, Mystery, Satellite, Spanner, VisitAll):
- Plan & validate — a PDDL solver generates action sequences between initial/goal states; a validator checks them.
- Convert to ASP — domains, instances, and plans are translated to Answer Set Programming.
- Compute the state space — ASP derives the exact action–state space, so every answer is provably correct (including ramifications).
- Generate & rephrase — templated questions are produced from that ground truth, then rephrased into natural language by Llama-3.1-70B.
Can models be taught it?
Partly. Fine-tuning Llama-3.1-8B on ActionReasoningBench data lifts it dramatically — on Fluent Tracking it jumps from 30.8% to 76.9%, overtaking the untuned 70B model (71.2%). So the well-studied dimensions are learnable with targeted training. But the hardest cases — numerical reasoning and ramifications — remain stubborn even after fine-tuning, marking RAC as an open challenge rather than a solved one.
Why it matters
Agents that act in the world need to model how the world changes — including the consequences nobody spelled out. ActionReasoningBench gives that capability a rigorous, planner-grounded yardstick, and its ramification results draw a sharp line between reciting an action's stated effects and truly reasoning about change.
Resources
Paper: arXiv:2406.04046 · ICLR 2025 · Handa & Dolin, equal contribution