research

STEAM: Benchmarking Frontier LLMs in Real-Time Environments

We tested the System 2 intelligence of OpenAI's Astra and Anthropic's Fable models in dynamic tasks.

Brandon Samaroo

NOV 20, 2026
A vertical maze of bone platforms spiralling around a stone column, with a black ball partway up

Frontier LLMs can plan a multi-step task in text. Putting one in charge of a robot is harder, because the world keeps moving while the model thinks. These models reason in discrete bursts separated by seconds of latency, while the environment runs continuously. Real-time perception is a well-known challenge for motor control, but most tests of LLM planning avoid it.

We believe high-level reasoning faces the same bottleneck as motor control. Between observations, a model misses what changed in the scene and loses track of what it has already done. Given that, an LLM will struggle to run a robot on its own, however capable its reasoning.

Our view is that this pushes robotics toward a hybrid design, which some teams are already trying: a fast policy, often a vision-language-action model, handles reaching, grasping and moving, and an LLM plans the steps above it, fed the state of the world in time to use it.

STEAM (SpatioTemporal Evaluation of Action Models) is the benchmark we built to test that. It puts agents inside real-time 3D puzzle rooms, seen from an egocentric perspective. To beat a level, the agent has to work out the room's challenge and complete a sequence of steps to open a door, while platforms move, hazards swing and objects change state around it. We gave two frontier models, Astra and Fable, one run on each of 12 levels, with 100 actions per run. Each model planned and moved the agent itself, with no separate policy underneath.

Key Findings

  • Neither model solved a level. Both finished 0 of 24 runs.
  • Most of each run was spent standing still. The average run of 100 actions lasted about 11 minutes, and the models spent 56% of that time, about 6 minutes, waiting on their next action while the room kept moving.
  • They looped instead of making progress. About a quarter of all actions were full sweeps of a room that had not changed, and neither model kept track of which steps of a puzzle it had already done.
  • They did not recover when stuck. On Pause Platforms, Astra spent about 7 minutes sliding down the same ramp without trying anything different.

What STEAM Is

STEAM is a set of 30 puzzle rooms in Unity, plus more than 20 harder or altered versions. The agent sees each room through a first-person camera and has to do something in it: cross a hallway without hitting the obstacles swinging through it, climb a vertical maze, find a fuel cell and fit it into the right socket, or use the wiring on a wall of buttons to work out which ones do anything. Many rooms keep changing whether the agent acts or not: platforms move, carousels turn, hazards cycle on and off.

Paused

Each puzzle mainly tests one or two of five skills, so when an agent fails, we know where to look.

Skill What the agent has to do
Navigation Move through 3D spaces and get where it is going efficiently
Environmental understanding Read cues in the room, what objects can be used for, and what state things are in
Object interaction Pick up, move and use objects to change the room
Spatial reasoning Work out geometry, motion and how things will move
Planning and memory Keep track of what has happened and carry out long sequences of actions

Many of the designs are adapted from experiments in cognitive science and robotics research, where tasks are kept simple so one ability can be tested without the others getting in the way. STEAM tests reasoning about space and time, not fine manipulation. It does not test a specific robot body.

Eight STEAM puzzles: Obstacle Hallway Platform, Vertical Maze, Fuel Cell Hunt, Fuel Cell Carousel, Wall of Buttons, Spatial Reasoning Puzzles, Generator Room, Launchpad Puzzle.

How an Agent Is Scored

STEAM scores each run from the game engine's own log. Unity records the agent's position and heading, every collision, what is in view, and every attempt to pick up or use an object, including whether it changed anything in the room. The score is built from that log in four steps.

Engine record, then behavior metrics, then combined metrics, then skill scores.

Take path efficiency. It divides the length of the shortest route to the goal, which Unity's navigation system can compute, by the length of the route the agent actually walked. An agent that takes the shortest route scores 1.0. Every score in STEAM breaks down the same way, so a low score can always be followed back to what the agent did.

Each metric is weighted per level, and a weight can be positive or negative. In the Maze, an agent that walks straight to the right door without checking any other corridor has high path efficiency but has not shown that it understands the maze, so exploring counts in its favor. In Obstacle Hallway it is the reverse: path efficiency counts, and covering a lot of the floor usually means the agent kept failing the crossing. Whether a level is complete comes from the game itself: a goal trigger fires when the puzzle is solved.

Checking the Scores Against the Video

Most of the problems we found while building STEAM were in the scoring, not in the rooms. A score could be calculated correctly and still disagree with what we saw on screen. To find these cases, we recorded deliberately good and bad runs for every skill, watched them, and checked whether the scores agreed. When they did not, we fixed the signal, formula or weight and ran it again. We set the weights the same way, tuning them until the deliberately bad runs scored under 10% and the good runs over 90%. The main changes:

  1. Each puzzle has its own weights. A single set rewarded path efficiency even in rooms where the agent has to explore first.
  2. Pressing a switch and moving an object are scored separately, and the logs record whether each interaction worked and moved the puzzle forward.
  3. Rotation and position are scored separately in spatial puzzles, with more weight on precision and on how many tries it took.
  4. The logs record whether the puzzle actually progressed, separately from whether a mechanism fired. Some runs had been logged as successes when the video showed the action did nothing.
  5. Scores across repeated attempts measure how fast an agent improves, how much it remembers and how quickly it forgets.

We check that a higher score means a better agent in two ways. Each puzzle is based on a task already used in cognitive science or robotics research. We build runs that are bad at one skill on purpose and confirm that skill's score drops.

How We Ran the Test

We ran two frontier models, Astra (gpt-6-astra) and Fable (claude-fable-5-2), on 12 of STEAM's base levels. Each model got one run per level, 24 runs in all.

  • Each run allowed 100 actions. If the level was not solved by then, it counted as a failure.
  • Before each level, the agent got a briefing explaining how the level works and what the goal is. STEAM was designed to drop agents in with no instructions, but in early runs without a briefing the models did even worse, so every run here had that help.
  • The agent saw screenshots from the game and could send raw movement commands or use three tools we added: walk to a point, turn to a heading, and look around the room, without which navigation was poor.
  • Every one of the 2,400 actions was accepted by the game, and no run was cut short by an error.

Astra and Fable on 12 Levels

Neither model completed a level: in all 24 runs, the game's goal trigger never fired.

The scores show what happened before that. Exploration coverage was 1.00 in 21 of the 24 runs, which can mean careful exploration or an agent so lost that it went everywhere. On Freeze Cross and Launchpad, both models scored 0.87 or higher on collision quality, and Astra scored 1.00 on Obstacle Hallway and Pause Platforms. Interaction success was zero in most runs. Navigation and basic movement worked, but no run finished a multi-step task.

Paused
Astra moving along Pause Platforms in real time. It scored 1.00 on collision quality in this level and still did not finish it.

Per-level results for Astra and Fable on 12 STEAM levels: path efficiency, collision quality and interaction success. Neither model completed a level.

The individual scores are not zero. An agent can move efficiently and avoid every obstacle and still not solve the puzzle, which is why STEAM scores each skill separately.

Why the Models Failed

Twelve mechanically different puzzles failed at the same stage, which suggests a few shared causes rather than twelve separate weaknesses.

Response Time

Each model took 3.0 to 9.2 seconds per step, about 4 seconds on average, so a run of 100 actions lasted about 11 minutes and the agent was moving for only 44% of it. For a person playing a game, 100 inputs, most of them single key presses, take a minute or two.

In a turn-based game that costs nothing. In STEAM, as in the real world, the scene keeps changing while the agent waits: platforms move, carousels turn and hazards cycle. It is like playing blindfolded and being shown a photograph every few seconds. The agent decides what to do from a scene that has already changed.

Paused
Two runs where the agent stood still while the room kept moving, sped up until the agent moved again. Times are measured from the recordings.

Keeping Track of the Task

The levels that failed all require carrying task state from one step to the next: which plates are pressed, which generators are on, which cells are left. The clearest interaction loop either model completed was on Launchpad, where Fable pressed a switch, saw the arrow rotate, and pressed again. That loop needs no memory, because the result is visible in the next frame, and Launchpad is one of the few levels where nothing moves on its own.

The models managed when the next step was visible in the current frame, and failed when it depended on remembering an earlier one. The harness gave the model a fresh prompt at each step, without its earlier observations, so this is partly a property of our setup.

Briefings and Visible Cues

The briefing for Fuel Cells said the cells could be picked up. Neither model tried to pick one up in that room. Astra did pick things up on Carousel, so the action was available. The models acted on prompts they could see in the frame and ignored what the briefing told them.

Reading the Room

The models also missed cues a person reads without thinking. Generator Grid is a room of generators, and any of them can be interacted with, but only the broken ones need fixing, and those give off sparks and steam. A person would spot the broken ones, or at worst try every generator. The two models pressed interact 24 times in that room and fixed none.

Repeated Room Sweeps

Both models opened every run by looking around the room. They then kept doing it: about a quarter of all actions were full sweeps of the room, even when nothing had changed. Without memory of earlier steps, looking again is the only way to be sure. In Generator Grid, with 18 generators and a cue display, the sweeps stopped helping. Path efficiency fell to 0.01 to 0.05, an order of magnitude below the other rooms.

Two Control Styles

The two models played differently. Astra mostly sent raw movement commands and pressed interact often when it saw something door-like. It acted first and looked afterwards. Fable relied on the walk and turn tools, aimed carefully, and thought longest where the task was most stateful, but rarely committed to an interaction: in 6 of the 12 levels it never pressed interact at all.

What This Means for Robotics

Hybrid systems that pair a fast policy with a slow planner are still experimental in robotics, but our results make a clear case for them. In STEAM, each model both planned and moved the agent, so every move waited on a reply that took seconds. The models could navigate the rooms and avoid most hazards, but they spent 56% of each run standing still while the room kept moving. A fast policy underneath would keep the robot acting between plans.

A hybrid does not fix the planning on its own. Between observations, the models missed state changes and lost track of the steps they had already taken, so they never finished a sequence. A slow planner only works if it gets the current state of the world and a record of what it has already done.

If LLMs are going to plan in real environments, they need evaluations that measure continuous perception and working memory, and track both as models improve. STEAM is built to measure both.

A fair objection is that these models were not built to play 3D puzzle games. STEAM does not ask whether a model is good at games. It asks whether general reasoning holds up when the world does not wait, which is the constraint any robot in a warehouse or a kitchen works under.

AuthorShare

Follow what we're building.