Sortex: A Data Quality Platform for Egocentric Video

Robotics has spent years looking for a pretraining data source that actually scales. Two questions matter most: does scaling the data reliably improve downstream robot performance, and what modeling and objective choices are needed for that scaling law to hold?

Egocentric human video has emerged as the strongest answer so far. EgoScale showed that more human ego data improves downstream robot performance. Dyna-2 pushed that to 1M hours and showed the gains transfer across the human-to-robot gap.

Ego data is now scaling quickly. Early datasets like Meta’s Ego4D offered roughly 3K hours of egocentric video. Today, Midcentury’s corpus alone has grown to more than 2M hours across 50+ environments and 20,000+ tasks.

As more data becomes available, the bottleneck is shifting to data quality.

Hours of footage alone tell us little about the activities represented, whether the relevant actions are visible, how much of the collection repeats the same work, or whether the data is actually useful for a specific training objective.

Sortex is the platform we built to answer those questions across a corpus this large.

At a fundamental level, a robotics data quality platform should do three things:

1. Qualitative and quantitative categorization of the environments and tasks within the data. First with individual clips, and then with entire datasets.
2. Distill, among a medley of metrics, a deep understanding of usability for training; or a “Robotics Training Usability Score.”
3. Enable blazing fast search over such a corpus for retrieval.

Sortex does all three across our full corpus. This post explores the custom-trained model, evaluation harness, and retrieval pipeline we use to understand and curate our ego data.

Because our platform is so crucial to the service we provide our customers, our goal with this report is to provide a high-level behind-the-scenes look at Sortex.

Motivation

Different training objectives require different data. A researcher studying tool use may need footage of precise hand-object interactions. Another may need long sequences with multiple dependent steps, or examples of the same action across different workplaces.

Finding these examples requires more than a broad category such as “assembly manufacturing.” It requires descriptions of the actions, objects, and conditions within a recording, together with measurements of whether those interactions are observable.

Dataset composition matters as well. Individually usable clips can still produce a collection dominated by the same workstation, viewpoint, or activity. Sortex combines measurements of individual clips with comparisons across the corpus so we can select footage against specific requirements.

Full Narration and Categorization: How We Get to 50+ Environments and 20,000+ Tasks

There are three broad phases: ingest and analyze recordings, organize the resulting annotations and measurements, and use them to retrieve and curate datasets. We store the outputs of expensive video processing so that filtering and selection can be repeated without decoding and labeling every recording again.

Loading…

A broad label describes the general activity in a recording. A detailed narration captures the steps that make up that activity.

A clip labeled “sewing,” for example, may include aligning fabric, guiding it through a machine, adjusting a fitting, cutting thread, and inspecting the result. Those distinctions matter when searching for particular interactions or understanding the skills represented in a collection.

Sortex generates structured descriptions of environments and activities, alongside objects, tools, materials, and hand actions. Temporal annotations describe how the activity progresses through the recording.

The pipeline uses sampled frames for coarse classification and video input for judgments that require temporal context. Classification outputs are constrained to a defined taxonomy, while descriptive fields preserve details that a fixed category would otherwise lose.

Loading…

A Brief Exercise on Quality

Defining quality within robotics data requires exploring questions the field is still working through. There is no current consensus on what makes a recording useful for training, or how that usefulness should be measured across different tasks and models.

As an applied research lab, we train models to understand how the data we collect affects their performance. More importantly, we use this continuous process to question our own assumptions about quality: which signals matter, which are misleading, and which depend on the training objective.

Our pipeline combines an in-house model and evaluation harness with a series of deterministic indicators. We use model-based judgments where understanding the scene or activity matters, and direct measurements where the question can be answered more reliably in code.

Motion blur can obscure an interaction. A camera may point away from the work, hands may leave the frame, or a recording may contain long stretches of inactivity. Faces, license plates, and identifying text also require screening before delivery. These are different problems, and each requires its own detection method and criteria.

Human review is a substantial part of this process. From reviewing the data ourselves, we’ve found four questions useful for guiding decisions around usability:

• Capture quality: Can we see the relevant work clearly?
• Annotation quality: Do the descriptions match the recording, and are required fields present?
• Demonstration quality: Was the task executed correctly? If there were mistakes, did the operator recover?
• Dataset composition: Does the collection provide the intended coverage, variation, and difficulty?

Each question depends on what the data will be used for. A failed grasp followed by a successful recovery may be useful for one training objective. For another, we may want only clean demonstrations. Similarly, footage with one hand outside the frame may still clearly capture the interaction we care about.

There are now entire companies focused on robotics data quality. The challenge is making these judgments consistently across large collections without discarding useful footage.

A particularly important problem is false positives. A model that flags too aggressively can make a dataset look cleaner while removing valuable examples. Real yield is lost when unusual tools, gloves, partial occlusion, or unfamiliar work environments are mistaken for unusable footage.

For this reason, we evaluate both the problems our checks catch and the usable footage they incorrectly reject. We retain the underlying measurements so that thresholds can change with the requirements of a dataset.

For hand visibility, for example, Sortex asks a model to count the operator’s visible hands in sampled frames, then calculates the visibility rate in code. This gives us an inspectable measurement rather than relying on a model’s estimate of a percentage across an entire video. For faces and license plates, detectors identify candidate regions, and a vision-language model helps verify ambiguous detections.

Loading…

We also measure repetition across recordings. Many clips can show the same workstation and activity with only small changes. These clips may each be usable, but selecting all of them can concentrate a dataset around a narrow set of conditions.

Visual embeddings support similarity comparisons and clustering, while selection applies limits across sites, tasks, environments, and other requirements. This lets us examine what an additional recording contributes to the collection, alongside its individual quality.

Altogether, these measurements inform how we screen individual recordings and assemble a dataset. Our scoring approach is constantly evolving, but the basic process is to apply required eligibility checks, rank the remaining clips using quality and novelty signals, and select against the intended dataset mix.

Applying Sortex to Public Datasets

To see how Sortex performs on footage outside our own collection, we ran it on three of the most popular public egocentric datasets, which have very different capture setups:

• EPIC-KITCHENS-100: recorded on head-mounted cameras in home kitchens
• RekaDaily-10k: recorded on head-mounted phones during everyday activities
• Gen-HumanEgo: recorded on a multi-camera rig in industrial settings

From each we drew 240 random 150-second clips, about 10 hours per dataset. Every clip went through the same pipeline we use on our own data: screening, then deduplication and selection. Screening settings were adapted to each dataset’s capture setup.

Sortex flagged 15.8% of EPIC-KITCHENS clips and 17.9% of RekaDaily clips at the clip level, and the two datasets fail for different reasons. In EPIC-KITCHENS, the main issues are darkness (6.7%) and camera shake (5.8%), which tend to cluster in particular kitchens and recording sessions. In RekaDaily, they are blur (7.1%) and visible faces (3.8%). Gen-HumanEgo’s capture quality was strong: most of our quality metrics had negligible failure rates in our 240-clip sample.

Loading…

Clip-level quality is only part of the picture. The quality of a dataset depends just as much on its overall diversity: how many different environments, tasks and viewpoints it covers, and how much of the footage repeats the same work. Among clips that pass screening, 80% of Gen-HumanEgo clips share a similar scene with another clip in the sample, and 42% are near-duplicates. For EPIC-KITCHENS, the shares are 39% and 5%; for RekaDaily, about 1% and 0%. A collection can be clean clip by clip and still offer far less coverage than its hours suggest. Comparing recordings across the collection, rather than one at a time, is how Sortex catches that.

Loading…

These results are directional. A 240-clip sample can’t capture everything in a full release, and some issues are hard for any automated check. Slow lighting flicker, for example, still slips past our current detector. But measuring capture quality and collection-level diversity together gives a far fuller view of what a dataset offers than counting hours or screening clips in isolation.

Conclusion

Ultimately, wrangling a 2M+ hour dataset is an extremely ambitious endeavor. One must juggle not only considerations of clip-level quality, broadly fitting into categories of both operator/user error as well as environment and clip-level issues, but consider the higher-level issues around diversity, repetitiveness, and how well a collection fits a training need.

A recording can be clear and correctly annotated while contributing little additional coverage to a collection. A diverse collection can still contain footage in which the relevant interactions are difficult to observe.

Sortex brings these questions into the same workflow. Categorization describes the work, quality measurements characterize its observability, and retrieval and selection help assemble footage for a specific training need.

As egocentric collections grow, this ability to understand and select their contents becomes increasingly important. Our goal is to make the composition of a training dataset explicit, measurable, and open to inspection.