Akari: Learned Rendering for Robotics Simulation

The sim-to-real gap remains one of the biggest bottlenecks to robotics deployment. At Midcentury, we believe the path to closing it is to pair classical simulators with learned models that can capture what hand-written simulation cannot. Akari is the first of these models: a learned renderer trained to bring photorealistic rendering to classical physics engines and close the visual sim-to-real gap.

A robot that recognizes an object in a clean, evenly lit simulation still has to recognize it under glare, in shadow, or through a different camera. These differences contribute to the visual sim-to-real gap: the mismatch between what a robot sees during simulated training and what it encounters in deployment. Our goal is to learn more of that appearance from data while keeping the scene and task under simulator control.

For teams training and evaluating visual policies, this means pursuing more than cheaper frames. It means more useful robot rollouts for the same compute budget, with images that better represent the conditions the robot needs to handle.

We measured Akari across image quality, sequence length, and inference cost:

• Half-second image generation. Matched warm inference benchmarks average 0.46–0.47 seconds per frame across kitchen and warehouse environments on one RTX 5090, using four denoising steps.
• Close reconstruction of the reference scene. Two warehouse sequences achieve mean perceptual error, measured by LPIPS, of 0.046 and 0.052 against Unreal Engine renders. Lower is better; zero means identical under this metric.
• Long autoregressive sequences. Each warehouse evaluation spans 1k frames, with generated images feeding into subsequent frames while retaining the scene's overall appearance.

The results below break down the speed and sequence-quality measurements.

Paused

Why Learned Rendering

More Experiments Within the Same Budget

A simulation budget limits how many policy checkpoints, camera views, and task variations a team can test. If realistic camera observations consume less GPU time, that budget can cover more experiments. This is the economic reason we are working on learned rendering.

Path tracing computes how light interacts with a scene. More samples and more complex light transport can increase the work required for a clean image. Akari moves part of that work into training: the model learns a mapping from scene information to appearance, then reuses it during generation.

For a fixed model, resolution, and denoising schedule, the learned appearance pass has a fixed computation pattern. Adding another simulated light does not add another layer to the network. Learning to render that light correctly is still a data and generalization problem, and generating the input geometry still costs compute.

The product target is lower cost per usable rollout, not merely a convincing frame. That requires measuring the whole pipeline at comparable quality, including input generation and the cost of training. We have measured Akari's inference time; a matched path-tracing comparison and parallel rollout benchmark remain ahead.

Closer to What a Real Robot Sees

Visual fidelity matters when it preserves the cues a robot uses to act: the edge of a container, a reflection on a metal surface, or an object partly hidden in shadow. Making an image more attractive is not the same as making it more useful for robot learning. The target is agreement with real camera observations.

Learning appearance gives us a way to improve that agreement through data. With suitable real-image supervision, a renderer could learn surface, lighting, and camera characteristics that are difficult to reproduce by hand. This is the route we are pursuing toward a smaller visual sim-to-real gap. Today's experiments measure reconstruction of engine-rendered images; the next step is learning from real observations and testing the effect on robot behavior.

The Same Task Under Different Visual Conditions

Fidelity to one scene is only part of the problem. A warehouse looks different as daylight changes, overhead lights switch off, or camera exposure adjusts. A policy needs to handle those conditions without depending on one familiar appearance.

We want Matrix users to test the same task across controlled appearance variants: different illumination, surface finishes, and camera characteristics, with the task geometry and object positions held fixed. A team could then ask whether a grasping policy fails because of the grasp itself or because it cannot recognize the object under a particular lighting condition.

How It Works

The simulator keeps control of the scene: objects, geometry, camera position, and physical state. Akari learns the visual appearance of that scene.

Its inputs are structured render passes called G-buffers. They describe visible surface colors, orientations, depth, roughness, and metalness. Rather than asking a video model to invent a world, we give the renderer fresh evidence of the world at every step.

The renderer also uses the preceding image as visual context. After reference-image initialization, each generated frame becomes context for the next. This is what makes the renderer autoregressive.

Training includes generated history paired with reference targets, so the model learns to render from imperfect context. Fresh G-buffers give it an independent source of scene information instead of making it rely entirely on its previous prediction. These mechanisms are designed to resist accumulated error; they do not guarantee drift-free output.

For a robotics team, this separation is useful: the simulator remains the source of task state while the observation model can improve through training. It is also the basis for controlled appearance variation. Changes to lighting or materials must remain compatible with the scene inputs, and the generated image must preserve the task-relevant structure the policy acts on.

Measured Results

We evaluated three things separately: similarity to the engine reference, error over long sequences, and inference time. The reference images are Unreal Engine renders, not photographs or a matched path-tracing baseline.

Image Fidelity Across a Full Sequence

LPIPS measures differences in learned visual features; PSNR measures pixel-level reconstruction quality. Together with the comparisons below, these results show how closely Akari reproduces the warehouse's appearance, across a full sequence rather than a selection of stills.

What Happens Over 1k Frames

A renderer has to sustain appearance as the camera moves, not just produce one good image. Both sequences retain the warehouse's overall appearance across 1k frames. Error increases more on camera 2, so we describe this as long-sequence reconstruction rather than zero drift. The fitted trend uses all frames; it is not the difference between the first and last windows.

Inference Cost on One Consumer GPU

Each value averages 20 generation calls after five warm-up calls, using the same inference settings with inputs and weights already loaded. GPU execution is synchronized for timing. The measurement covers image generation, excluding scene capture, G-buffer preparation, model loading, and file output.

See the Output

Compare the unlit BaseColor input on the left with Akari’s output on the right across warehouse, office, and kitchen environments.

Loading…

Loading…

Loading…

Loading…

What Needs to Improve

Fine detail matters to a robot. A softened edge, small marking, or poorly resolved object can change a perception model's decision even when the room looks convincing overall. The crate markings below show a concrete remaining gap.

Temporal consistency also needs work. On sampled adjacent frame pairs from warehouse camera 1, the learned images change 7.5× more on average than the engine reference. This simple pixel-difference measure is not motion-compensated and does not isolate flicker, but it flags excess temporal variation that per-frame fidelity scores miss.

Akari is a proprietary Midcentury model and will be available exclusively through our simulation platform, Matrix. You can join the waitlist here.