What we check, and how.

Tamper-evident from the moment our host receives the file. Not from the sensor, though. No GoPro does it, the Quest doesn't, and none of the wearable rigs we run supports it.

C2PA attestation 1.4 · Capture support

Catches re-encodes, re-uploads, and the same scene shot again from a slightly different angle. Paying per passing episode gives people a reason to produce episodes that look like they'll pass, so this runs on everything rather than on a sample we keep quiet about. For scale: on the benchmark built out of deliberately edited copies, the winning entry hit 87.2% μAP against a 60.5% baseline. A plain re-submission is a far easier case than that.

Video Similarity Challenge, CVPR 2023

This happens in three places. At the depot, on a cycle, we run a full target-based calibration; that's the only step that produces a certificate. At pickup, the app scores a guided board sweep and won't release a kit that fails it. During the session, we track the gap between the online estimate and the depot values.

Drift is bigger than most people assume. Project Aria measured something like 25 arcmin of deformation between two cameras on the same rig, just from a wearer taking it off. As far as we can tell, the only published numeric gate in this area is DROID's post-hoc audit: IoU at or above 0.7, reprojection under 20 px, outliers at 2.5 × MAD.

Project Aria MPS · DROID Appendix G

Camera-to-IMU gets estimated online from the recording itself. No target needed, and published implementations get sub-millisecond given enough motion. Slow careful work doesn't excite the estimator the way a benchmark sequence does, so we report how well it converged alongside the number.

Within a kit is not the same as across kits. Hardware timestamping inside one rig is microsecond-grade; two independent devices writing to their own cards are not, so anything multiview ships with a hardware trigger or an in-frame sync device. Rolling shutter is a separate and larger problem: left unmodelled it makes trajectory error worse by roughly an order of magnitude. Our head-mounted kit is global shutter, which is most of why we chose it. The handheld's GoPro is rolling. We publish shutter type and line readout per sensor either way.

Exposure is scored against an objective built for robot vision rather than for photography. Blur is variance of Laplacian, which is the usual proxy. Codec, bitrate, chroma subsampling, GOP and any lossy re-encode get written down per episode. Nobody has published a threshold for how blurry is too blurry to train on, so ours comes out of our own ablations and we label it as such.

Allan-variance characterisation is a depot job: fifteen to twenty-four hours sitting still at a stable temperature, per kit. These are consumer MEMS parts, and the published advice is to inflate the resulting parameters by an order of magnitude, so we do. Per episode we check the cheap things. Gravity is a rough sanity check rather than a reference, since local g across India runs somewhere around 9.787 to 9.79 m/s².

Worth being precise about what's being offset from what. On a handheld kit the end-effector pose comes from the same camera, so the quantity that matters is camera to gripper encoder: when the aperture reading was taken, relative to its frame. On teleop it's observation frame to controller command.

This one goes first because timing error of this kind doesn't show up in any other check on this page, and there's no fixing it afterwards. The clearest published demonstration we know of is a control-latency study on a rolling-ball pick: a delay-aware policy reaches 0.96 at 50 ms and 0.72 at 100 ms, while a standard diffusion policy at the same delays reaches 0.20 and 0.01. That's a deployment result on a fast task, which is why we quote it. It isn't our error budget.

Delay-Aware Diffusion Policy

People pause without meaning to. At 30 to 60 Hz a two-second pause turns into a hundred near-identical action points, so the dull moments end up dominating the gradient while the interesting ones stay thin. The worse problem is that two identical frames labelled stay and move are a flat contradiction the model has to absorb somehow. AgiBot World only added idle removal after deployment turned it up.

We report this as a gap, never as an absolute number. At 30 to 60 Hz any smooth trajectory predicts itself, so an absolute score tells you the control rate rather than the copycat risk. If the margin is narrow, the observations aren't earning their place. The copycat problem itself is well established in the literature; turning it into an acceptance gap is our own construction, and we'd rather say that than let it pass as standard.

Smoothness feeds into this but never decides it. An audit on early gripper release put a state-trajectory detector at 0.863 AUROC and a smoothness scorer at 0.244, which is worse than guessing. Curating that dataset by smoothness produced a worse policy than not curating it at all: 27.3% against 36.0%. Early release isn't the kind of noise that averages out.

Auditing Demonstration Curation Metrics

Read this next to ACT-04 where you have it, rather than in place of it. Smooth and correct are different things.

We say where the label came from — human, VLM or scripted — and give you an audited agreement rate against human adjudication. False successes get reported separately, because that's the direction the errors actually run. Benchmarking VLM judges across 2,197 attempts put the best of them at 0.77 mean balanced accuracy, close to chance on assembly tasks that need contact detection, and biased toward calling things successful.

FailBench

Current VLAs mostly aren't reading the language at all. Language-only ablations come in at 0.6 to 0.9% where the full model reaches 94 to 98%, and one model averaged 93.3% grasp success against 11.1% task success across six suites, with nearly every failure being the wrong object picked up.

We flag and sample here rather than certify. Checking whether the wording disambiguates needs the same class of model SEM-01 has just told you not to trust, working on a harder problem. So: automated screening across the corpus, and human adjudication on a published sample and on everything that gets flagged.

LIBERO-Plus · RoboSemanticBench

About 67 ms at 30 fps. Contact timing is usually what decides whether a manipulation worked, but that's only a measurement where aperture makes it one. On head-mounted capture we mark interaction segments with a confidence value and don't put a frame tolerance on them at all.

A batch can hit every individual target and still contain nothing for overhead work in a tailor's shop. That empty cell is where the policy will fail. So when a cell comes up short we re-task it, rather than topping the total up from whichever neighbouring cell is easy to fill.

There's no robotics-specific study of train/test leakage we can point at, so this taxonomy is borrowed from general ML and from clinical machine learning, where the mechanism is well documented. Across 62 papers, random record-wise cross-validation gave a median error of 5.60% where subject-wise gave 13.00%. A contributor's hands, pace, kit and room are the same kind of fingerprint.

Kapoor & Narayanan · Saeb et al.

For a sense of scale: 3.25% of CIFAR-10's test set and 10% of CIFAR-100's turned out to be near-duplicates of training images, and fixing it moved reported error by 2.73 points on average and 3.52 at worst. A robot corpus built out of repeated takes starts out far more duplicated than CIFAR ever was.

So you can tell whether ten thousand episodes came from fifty people. On a hard benchmark, demonstrations grouped by operator skill produced 66.0%, 45.3% and 39.3%, and pooling all three (78.0%, from 300 demos) did worse than the proficient operator alone (84.0%, from 200).

robomimic, CoRL 2021

Attached to the episode, not to the batch. If either one is missing, the episode is rejected no matter how good the recording is, and it never gets delivered.

We're not going to tell you the automated pass is enough. The best purpose-built egocentric face detector reports 93.8% recall on RGB and 89.9% on greyscale, which is somewhere between one face in sixteen and one in ten missed, and it does worse on faces that are partly out of frame. Ego4D spent 780 hours of human labour de-identifying 500 hours of video and published no residual error rate at the end of it. If a vendor offers you clean automated redaction, ask them for that number.

EgoBlur · Ego4D privacy

How much redaction you need is decided by where you record, not by how good your blurring is. Ego-Exo4D released almost all of its 1,286 hours without de-identification, because the scenarios were closed environments. A workshop with five people in it who have all agreed is a different proposition from a shopfront open to the street.

Fewer than fifty rollouts is common, and whether that's enough depends entirely on how you report it. Fine-grained task progress separates policies in far fewer trials than binary success does. The thing to avoid is a bare percentage with no interval attached.

Here's the honest limit. Every published evaluation method — real rollouts, automated cells, simulation with visual matching — ranks policies against each other. None of them isolates what a dataset contributed. We'll run the comparison and report the interval. We won't dress up an overlapping interval as a result.

Beyond Binary Success

This is the layer that overrules the rest of the register when they disagree. Curating with a classifier trained on the policy's own rollouts reports a 15 to 35 point absolute gain in success across five real bimanual tasks. Separately, influence-function curation matches state of the art on under a third of the data in single-arm work. Two different methods and two different results, and they get conflated often enough that it's worth keeping them apart.

Demo-SCORE · CUPID

This is the comparison that would settle whether the register is worth what it costs. We went looking for a clean published version and couldn't find one we'd stand behind, which is worth knowing in itself: the case for verification currently rests on curation results and on first principles, not on a head-to-head that anyone has run and published.

No comparable dataset publishes one: not DROID, not Ego4D, not AgiBot World, not RoboMIND. It costs nothing to disclose, which makes its absence everywhere else more interesting than it should be.

It's also the number that tells you whether the rest of this page is working. A rate near zero means the register isn't being enforced; a very high one means the coaching isn't doing its job.

That's a real guarantee and you can check it. It's also a different guarantee from the one "signed at the sensor" implies, and we don't want the two confused.

Contact-state estimation from egocentric video runs around 64% AP, and worse when a hand is near an object, precisely the moment of interest. If you need certified contact timing, take a gripper kit.

Laplacian is the standard proxy, but nobody states what value makes a frame unfit to train on. And we found no study at all on what codecs, bitrates or rolling shutter do to policy learning. So instead of making a claim, we disclose the settings and leave the call to you.

There's no robotics literature on train/test leakage. DST-02 borrows its taxonomy from general ML methodology and from clinical machine learning. That's an argument by analogy, and we'd rather say so than let it pass as robotics evidence.

We can't tell good multimodality from bad consistency. Two people doing a task two valid ways is healthy data. Two people doing it inconsistently is a defect. In the statistics they look the same, and no published metric separates them. We report action entropy per cell so you can look for yourself.

Automated de-identification misses about one face in ten. Human review and a standing channel for reporting misses bring the risk down. They don't take it to zero, and we're not going to say they do.