Post-Training π0.5 on Egocentric Data

Egocentric data gives robot policies access to experience beyond robot demonstrations. Previous work from Physical Intelligence showed that post-training π0.5 on human demonstrations improved the policy’s performance on tasks in settings not covered by its robot demonstrations. As this approach develops, we also want to understand how the quality of that data affects the resulting policy.

A useful dataset needs more than recordings of the right activity. The actions need to be visible, the motion needs to be accurately captured, and the examples need to cover the conditions a policy will encounter. At Midcentury, we’re studying how these qualities affect performance to select and prepare data for specific training objectives. The question is practical: if additional human experience improves a policy trained with a small robot-demonstration budget, can higher-quality data amplify those gains?

Our bet is that the value of human experience depends on how it is captured, annotated, and integrated into training. As a first step toward understanding those differences, we post-trained separate π0.5 policies on Midcentury data and EgoDex, Apple’s public egocentric dataset, sampling equal amounts of data matched on environments and objects. We then evaluated the resulting policies on LIBERO-10.

Key Findings

• Midcentury data showed higher mean success than EgoDex in the three shifted camera views. The differences ranged from +0.3 to +5.8 percentage points, with equal means at the training view. The confidence intervals are wide and include zero in every view.
• The two strongest observed run pairs favored Midcentury across all four views. Ranked by their average advantage over their matched EgoDex runs, they led by 11.1 and 8.1 percentage points. These runs were selected based on their observed results.
• In a separate comparison, adding Midcentury data improved success over robot-only training at a shifted view. At a 12.2 cm sideways camera shift, mean success rose from 20.5% to 26.0% across 15 paired seeds.

These results suggest that the source and preparation of human data are worth investigating further. The experiments do not yet establish which qualities contributed to the observed differences.

Why Human Data

Robot demonstrations provide direct supervision for a particular embodiment. Human recordings offer a complementary source of experience: how objects are used, how actions fit together and how the same work looks as a person moves. For teams adapting a policy to a new use case, that is a reason to invest in human data alongside robot collection.

Video alone is not a robot action. In this study, estimated hand motion is converted into action targets and mixed with robot demonstrations during fine-tuning. The human and robot recordings are unpaired: the pipeline does not require a robot recording aligned with each human action. It still uses robot demonstrations of the benchmark tasks.

Human video and hand-tracking comparison.

Training Setup and Comparisons

We start from the π0.5 base checkpoint and update the full model, rather than training LoRA adapters. The recorded recipe uses 3,000 training steps, batch size 256 and a 300-step warmup. Each arm has the same nominal budget of 20 unique robot demonstrations: two per LIBERO-10 task.

Training and evaluation setup.

Success means completing all task goals at the final state. We compare results within matched training-seed pairs and give all ten tasks equal weight. The human-source comparison and robot-only comparison use different historical dataset builds, so their percentages should not be read as one three-arm experiment.

Matching Data and Training Budgets

The Midcentury and EgoDex clips used in this comparison were matched on environments and objects, with matching on some tasks. The research team verified this alignment for the clips used in the experiment.

The comparison also uses nearly equal amounts of converted human data and the same nominal robot-demonstration budget. Each arm includes two robot demonstrations for each of the ten LIBERO tasks. Human-frame counts are measured after conversion and filtering, so the volume comparison reflects the data supplied to training.

Results table.

The final human-frame totals differ by just six frames. Both arms use 10 Hz converted data and retargeted right-hand actions. The comparison also uses paired training seeds and common evaluation tasks.

Task matching is partial. Environment and object matching does not establish identical category frequencies or eliminate all sampling bias. The results apply to the selected samples and their processing recipes.

Selection and Processing Details

The recovered Midcentury clip list contains 73 source recordings, split into short training episodes. The EgoDex build contains 1,690 human episodes. Source recordings and converted episodes are different units; neither count establishes the number of independent people or environments.

The research team confirmed environment, object and partial task matching on October 26, 2026. This confirmation supplements the archived clip records. The retained records do not independently reconstruct that matching procedure; a complete reconstruction requires the original selection settings, source lists and labels used to establish the match.

The historical converter supports selecting eligible Midcentury recordings by list order, seeded shuffle or a pinned clip list. Its EgoDex task-balancing option alternates among EgoDex task folders. That within-source balancing operation is distinct from the cross-source matching confirmed by the team. Tracking-quality and action-content curation were also tested separately.

The human-source builds target approximately 120,000 human frames. The exported robot component repeats the 20 demonstrations 20 times, producing 110,560 robot frames. Repetition does not add unique demonstrations, and identical step counts do not guarantee equal robot-example exposure across mixtures.

The clip-list derivative totals 1,306 Midcentury episodes; the archived summary reports 1,307. The original sidecar is needed to resolve this one-episode discrepancy. The frame totals above come from the archived dataset summaries. Converted hours are frame count divided by 10 Hz, not a measurement of unique recording time.

Source-specific filtering, language, tracking and action normalization also differ. Later v3 manifests describe a revised pipeline and do not establish the selection used by these historical checkpoints.

Midcentury and EgoDex on LIBERO-10

The full paired comparison has equal mean success at the training view and higher Midcentury means in each of the three shifted views. The largest mean difference is +5.8 points at 6.1 cm forward. With five or six seed pairs per view, uncertainty is wide: every paired 95% interval includes zero.

The complete chart values and confidence intervals are in the following table.

Results table.

The estimates are compatible with both gains and losses relative to EgoDex. They do not establish general superiority or equivalence. They also do not measure the intrinsic quality of either complete source corpus independently of its conversion and training recipe.

The Strongest Runs Across Four Views

The two largest observed average advantages over EgoDex are +11.1 and +8.1 percentage points. Both runs have higher success than their matched EgoDex counterpart in all four camera views. We rank the five complete run pairs by their mean advantage across those views, weighting each view equally.

All run outcomes are in the tables below.

Results table.

The ranking uses observed evaluation outcomes and measures the advantage over EgoDex, not absolute Midcentury success. It does not estimate the expected gain for a new training run. One additional run pair has only three views and is shown as unranked; all of its available measurements remain in the full comparison.

Results table.

The simulation player requires JavaScript.

Why a Change in Camera Position Matters

A robot can face the same task after a camera is remounted or installed in a different position. The goal stays the same, but objects move within the image and their apparent shape or visibility can change. Evaluating a familiar task from an unfamiliar view tests whether the learned policy still completes it without retraining.

Camera views comparing the training position and a 12.2 cm sideways shift.

In the original nine-seed cohort, robot-only success falls from 39.3% at the training view to 20.3% at 12.2 cm sideways. This makes the perturbation useful as a controlled stress test. It does not reproduce every installation error or measure real-robot reliability.

Results at the Sideways View

Against robot-only training, the original cohort gains +5.3 points and the follow-up gains +5.8 points. Across both, success increases from 20.5% to 26.0%, with 12 of 15 seed pairs improving. The pooled gain is +5.5 points, with an unadjusted 95% interval of +2.6 to +8.5 points.

The pooled results are in the following table.

Results table.

Results table.

Intervals are unadjusted. After Holm correction across seven camera conditions, only 12.2 cm forward has p < 0.05 for full-task success. Under a broader fourteen-test family spanning success and partial credit, no full-task success comparison does. These are exploratory analyses, not preregistered confirmatory tests.

Human ego video includes changing viewpoints, making visual transfer a plausible explanation. This experiment does not isolate it from action targets, language, normalization or sampling. Higher shifted-view success also does not by itself demonstrate a smaller drop from the training view.

The simulation player requires JavaScript.

What This Means for Policy Training

This study tests whether human experience can help train robot policies. Adding Midcentury data is associated with higher shifted-view success than robot-only training, and some matched runs show substantial advantages over EgoDex.

The next step is to make those gains dependable. We would freeze the corrected processing pipeline, document source selection and effective sampling, and evaluate a prespecified set of checkpoints and seeds. Separate experiments should test viewpoint diversity and action supervision. Held-out tasks and real robots are needed to establish deployment value.

Our bet is on human data as a complement to robot data. The work ahead is to determine which experience helps, how to represent it and when the improvement carries into a new setting.

Methods and Sources

The reported success rates are historical LIBERO-10 simulation results. No new training runs or aggregate policy evaluations were conducted for this report. The simulation player contains new qualitative rollouts from historical checkpoints; these recordings are not the historical evaluation episodes and do not change the reported statistics.

Training, Scoring and Uncertainty

Recorded training settings: π0.5 base initialization, full-model fine-tuning, 3,000 steps, batch size 256, 300-step warmup, peak learning rate 5 × 10⁻⁵ and action horizon 10. A matched seed shares a numerical seed value; different mixtures can still produce different minibatches and optimizer trajectories.

Full-task success requires all recorded task predicates to be true at the final state. Each checkpoint and view has 20 evaluations per task, or 200 across ten tasks. Final predicates alone do not establish an action-ordering requirement. This report uses full success rather than partial-credit scores.

We compute differences within training-seed pairs, then average them. The 95% interval is the mean difference ± the Student t critical value × the standard error of paired differences. Training seed is the unit of uncertainty; evaluation episodes and the four views of a checkpoint are not independent training replications. All displayed intervals are unadjusted.

Displayed values are rounded to one decimal; gaps use unrounded measurements. The original analysis recomputed scores from archived episode files. One known incorrect-build run was quarantined and excluded. No missing outcome is replaced with zero.

Comparison Identities and Processing Limits

The EgoDex comparison pairs Midcentury and EgoDex runs across six matched training seeds. Five run pairs have all four views; the unranked pair is missing the 6.1 cm forward evaluation. Rank labels refer to observed mean advantages; the source records retain the original seed identifiers. The original robot-only comparison pairs Midcentury and robot-only runs on seeds 42–50. The upward view omits seed 48 from both arms.

The original Midcentury build has 137,110 human frames; the six-seed follow-up has 135,093 after additional action filtering. Both differ from the approximately 120,000-frame EgoDex comparison. Follow-up seeds 51–56 pair this filtered build with a matching robot-only baseline.

Later internal audits identified issues in legacy head-motion compensation and action normalization, including coordinate-frame mixing and mixture-dependent action scales. Their contribution to the observed gaps is unknown. No corrected-v3 policy results are available. These results therefore characterize the historical processing recipes, rather than a validated corrected pipeline.

LIBERO-10 is a small simulated kitchen-task suite. These experiments do not establish factory-task performance, real-robot reliability, bimanual transfer or collection-cost savings.

Media Provenance

The human illustration shows a face-free excerpt of a worker gauging and sorting metal components, with supplied hand tracks overlaid. The 1920 × 960 video runs at 30 fps. This footage illustrates Midcentury data but has not been verified as part of the historical training subset. Only right-hand actions were used for the evaluated policies.

The camera illustrations are native 2048 × 2048 LIBERO renders of one fixed stove-task state. The lateral offset is 0.122474487139 metres; orientation and field of view remain fixed. No policy inference was run to produce these images.

1. Physical Intelligence: π0.5, a VLA with open-world generalization ↗
2. Apple: EgoDex dataset and method ↗
3. LIBERO: benchmark and evaluation code ↗
4. Midcentury internal study records: archived evaluation episodes, dataset summaries, training configurations and recovered clip lists. The underlying inputs and numerical checks are recorded with this report’s source.