Research · OCT 29, 2026Post-Training π0.5 on Egocentric Data
How Midcentury’s egocentric dataset outperformed EgoDex under unseen conditions on LIBERO-10.
Readresearch
We tested the System 2 intelligence of OpenAI's Astra and Anthropic's Fable models in dynamic tasks.

Frontier LLMs can plan a multi-step task in text. Putting one in charge of a robot is harder, because the world keeps moving while the model thinks. These models reason in discrete bursts separated by seconds of latency, while the environment runs continuously. Real-time perception is a well-known challenge for motor control, but most tests of LLM planning avoid it.
We believe high-level reasoning faces the same bottleneck as motor control. Between observations, a model misses what changed in the scene and loses track of what it has already done. Given that, an LLM will struggle to run a robot on its own, however capable its reasoning.
Our view is that this pushes robotics toward a hybrid design, which some teams are already trying: a fast policy, often a vision-language-action model, handles reaching, grasping and moving, and an LLM plans the steps above it, fed the state of the world in time to use it.
STEAM (SpatioTemporal Evaluation of Action Models) is the benchmark we built to test that. It puts agents inside real-time 3D puzzle rooms, seen from an egocentric perspective. To beat a level, the agent has to work out the room's challenge and complete a sequence of steps to open a door, while platforms move, hazards swing and objects change state around it. We gave two frontier models, Astra and Fable, one run on each of 12 levels, with 100 actions per run. Each model planned and moved the agent itself, with no separate policy underneath.
Key Findings
STEAM is a set of 30 puzzle rooms in Unity, plus more than 20 harder or altered versions. The agent sees each room through a first-person camera and has to do something in it: cross a hallway without hitting the obstacles swinging through it, climb a vertical maze, find a fuel cell and fit it into the right socket, or use the wiring on a wall of buttons to work out which ones do anything. Many rooms keep changing whether the agent acts or not: platforms move, carousels turn, hazards cycle on and off.
Each puzzle mainly tests one or two of five skills, so when an agent fails, we know where to look.
| Skill | What the agent has to do |
|---|---|
| Navigation | Move through 3D spaces and get where it is going efficiently |
| Environmental understanding | Read cues in the room, what objects can be used for, and what state things are in |
| Object interaction | Pick up, move and use objects to change the room |
| Spatial reasoning | Work out geometry, motion and how things will move |
| Planning and memory | Keep track of what has happened and carry out long sequences of actions |
Many of the designs are adapted from experiments in cognitive science and robotics research, where tasks are kept simple so one ability can be tested without the others getting in the way. STEAM tests reasoning about space and time, not fine manipulation. It does not test a specific robot body.
Eight STEAM puzzles: Obstacle Hallway Platform, Vertical Maze, Fuel Cell Hunt, Fuel Cell Carousel, Wall of Buttons, Spatial Reasoning Puzzles, Generator Room, Launchpad Puzzle.
STEAM scores each run from the game engine's own log. Unity records the agent's position and heading, every collision, what is in view, and every attempt to pick up or use an object, including whether it changed anything in the room. The score is built from that log in four steps.
Engine record, then behavior metrics, then combined metrics, then skill scores.
Take path efficiency. It divides the length of the shortest route to the goal, which Unity's navigation system can compute, by the length of the route the agent actually walked. An agent that takes the shortest route scores 1.0. Every score in STEAM breaks down the same way, so a low score can always be followed back to what the agent did.
Each metric is weighted per level, and a weight can be positive or negative. In the Maze, an agent that walks straight to the right door without checking any other corridor has high path efficiency but has not shown that it understands the maze, so exploring counts in its favor. In Obstacle Hallway it is the reverse: path efficiency counts, and covering a lot of the floor usually means the agent kept failing the crossing. Whether a level is complete comes from the game itself: a goal trigger fires when the puzzle is solved.
Most of the problems we found while building STEAM were in the scoring, not in the rooms. A score could be calculated correctly and still disagree with what we saw on screen. To find these cases, we recorded deliberately good and bad runs for every skill, watched them, and checked whether the scores agreed. When they did not, we fixed the signal, formula or weight and ran it again. We set the weights the same way, tuning them until the deliberately bad runs scored under 10% and the good runs over 90%. The main changes:
We check that a higher score means a better agent in two ways. Each puzzle is based on a task already used in cognitive science or robotics research. We build runs that are bad at one skill on purpose and confirm that skill's score drops.
We ran two frontier models, Astra (gpt-6-astra) and Fable (claude-fable-5-2), on 12 of STEAM's base levels. Each model got one run per level, 24 runs in all.
Neither model completed a level: in all 24 runs, the game's goal trigger never fired.
The scores show what happened before that. Exploration coverage was 1.00 in 21 of the 24 runs, which can mean careful exploration or an agent so lost that it went everywhere. On Freeze Cross and Launchpad, both models scored 0.87 or higher on collision quality, and Astra scored 1.00 on Obstacle Hallway and Pause Platforms. Interaction success was zero in most runs. Navigation and basic movement worked, but no run finished a multi-step task.
Per-level results for Astra and Fable on 12 STEAM levels: path efficiency, collision quality and interaction success. Neither model completed a level.
The individual scores are not zero. An agent can move efficiently and avoid every obstacle and still not solve the puzzle, which is why STEAM scores each skill separately.
Twelve mechanically different puzzles failed at the same stage, which suggests a few shared causes rather than twelve separate weaknesses.
Each model took 3.0 to 9.2 seconds per step, about 4 seconds on average, so a run of 100 actions lasted about 11 minutes and the agent was moving for only 44% of it. For a person playing a game, 100 inputs, most of them single key presses, take a minute or two.
In a turn-based game that costs nothing. In STEAM, as in the real world, the scene keeps changing while the agent waits: platforms move, carousels turn and hazards cycle. It is like playing blindfolded and being shown a photograph every few seconds. The agent decides what to do from a scene that has already changed.
The levels that failed all require carrying task state from one step to the next: which plates are pressed, which generators are on, which cells are left. The clearest interaction loop either model completed was on Launchpad, where Fable pressed a switch, saw the arrow rotate, and pressed again. That loop needs no memory, because the result is visible in the next frame, and Launchpad is one of the few levels where nothing moves on its own.
The models managed when the next step was visible in the current frame, and failed when it depended on remembering an earlier one. The harness gave the model a fresh prompt at each step, without its earlier observations, so this is partly a property of our setup.
The briefing for Fuel Cells said the cells could be picked up. Neither model tried to pick one up in that room. Astra did pick things up on Carousel, so the action was available. The models acted on prompts they could see in the frame and ignored what the briefing told them.
The models also missed cues a person reads without thinking. Generator Grid is a room of generators, and any of them can be interacted with, but only the broken ones need fixing, and those give off sparks and steam. A person would spot the broken ones, or at worst try every generator. The two models pressed interact 24 times in that room and fixed none.
Both models opened every run by looking around the room. They then kept doing it: about a quarter of all actions were full sweeps of the room, even when nothing had changed. Without memory of earlier steps, looking again is the only way to be sure. In Generator Grid, with 18 generators and a cue display, the sweeps stopped helping. Path efficiency fell to 0.01 to 0.05, an order of magnitude below the other rooms.
The two models played differently. Astra mostly sent raw movement commands and pressed interact often when it saw something door-like. It acted first and looked afterwards. Fable relied on the walk and turn tools, aimed carefully, and thought longest where the task was most stateful, but rarely committed to an interaction: in 6 of the 12 levels it never pressed interact at all.
Hybrid systems that pair a fast policy with a slow planner are still experimental in robotics, but our results make a clear case for them. In STEAM, each model both planned and moved the agent, so every move waited on a reply that took seconds. The models could navigate the rooms and avoid most hazards, but they spent 56% of each run standing still while the room kept moving. A fast policy underneath would keep the robot acting between plans.
A hybrid does not fix the planning on its own. Between observations, the models missed state changes and lost track of the steps they had already taken, so they never finished a sequence. A slow planner only works if it gets the current state of the world and a record of what it has already done.
If LLMs are going to plan in real environments, they need evaluations that measure continuous perception and working memory, and track both as models improve. STEAM is built to measure both.
A fair objection is that these models were not built to play 3D puzzle games. STEAM does not ask whether a model is good at games. It asks whether general reasoning holds up when the world does not wait, which is the constraint any robot in a warehouse or a kitchen works under.
Research · OCT 29, 2026How Midcentury’s egocentric dataset outperformed EgoDex under unseen conditions on LIBERO-10.
Read
Research · OCT 27, 2026Screening, annotating, and filtering the world’s largest egocentric dataset for specific training needs.
Read
Research · OCT 21, 2026Photorealistic rendering for classical physics engines, trained to close the sim-to-real gap.
Read