FLA · Experiment 1 dataset · 16 Nov 2026 · all reports · summary

A dataset with eyes

To teach a camera from the sole, every footstep needs two things: what the robot saw before the step and what the sole felt after it. The step-1 walking data had a sole but no camera. I replayed a third of it exactly and rendered a torso camera on top, with honest floors and with floors that lie.

Simulation only. Booster K1 in MuJoCo, rendered camera, simulated sole.

In short

  1. 1,572 episodes, 54,010 footsteps. About 30 % of the step-1 walking data, replayed bit for bit. Every replay matched the stored run exactly.
  2. For each step: a camera frame before it, the sole after it, and the simulator's truth. The camera is RGB-D, on the torso, pitched down. The sole is stored as coarse features under three sensor models.
  3. Textures are split like the episodes. Test episodes show only floor textures that training never sees, but there are only three test textures per class.
  4. 26.6 % of the steps are deceptive. The floor looks like one class and behaves like another, or a sheet hides the gaps between planks.
  5. One known gap in the labels. The step-1 contact flag misses a foot that is loaded on its front edge. It touches up to 3.0 % of the heel-to-toe gait's steps in a wide window, and none inside the step windows that are used.
1

Why a new dataset

Step 1 of the project trained a simulated sensing sole on a Booster K1 humanoid to name the floor after each step. Its episodes store the joints, the IMU, the sole signal and the true floor, but no images. The question of experiment 1 is whether the sole can teach a camera: a vision model sees the floor before the step, and the sole supplies the label after it. For that, every footstep needs a frame from before and a sole reading from after, on the same trajectory.

Six floor classes: grippy hard floor (hard_grip), slippery hard floor (hard_slip), soft, rough, slope, and edge (planks and steps, where a foot can land partly over a gap). Three gaits: two ordinary locomotion policies that step with a flat foot (G113 and G80), and SONIC, human-like heel-to-toe walking.

2

Exact replay

The step-1 episodes do not store the full state of the simulation: no base pose and no terrain geometry. Re-running a stored task turned out to reproduce it bit for bit, so the frames come from a deterministic re-run. Each episode is re-simulated from its stored task, and every stored array is compared bit for bit with the original. An episode that differs would be dropped. None did: all 1,572 replays are identical.

The frames are rendered afterwards from the saved state. So the honest and the deceptive view of an episode are the same trajectory with different textures, and the sole signal is the same in both.

The whole step-1 set (6,139 episodes) would have taken about 9 hours to replay, so I took about 30 % of it, stratified by class, gait and split. Test strata are sampled about twice as densely as training and validation, so that at least 20 % of the steps can be deceptive and an honest test set is still left.

3

What is in it

SplitStepsFloor textures
Train29,105training textures only
Validation4,267validation textures only
Test (honest)6,269test textures only
Test (deceptive)14,369test textures only
All, 1,572 episodes54,01014.1 GB

Reading the table. An episode keeps the split it had in step 1, and no episode is in two splits. Deceptive episodes are step-1 test episodes, and 14,369 steps are 26.6 % of the whole set. Of the deceptive steps, 726 come from the hidden-gap kind (no_support, below).

4

Camera and frames

Contact sheet of twenty torso-camera frames over all floor classes and gaits, each with a green crop box, a red landing footprint, the depth image below and the step's labels
The final contact sheet: one step per cell over classes, gaits and looks. Green: the context crop. Red: where the foot will land. Below each frame: its depth image and the step's labels (episode, class, gait, look, distance).
5

Honest and deceptive floors

Every deceptive episode is rendered in both looks, so each deceptive step has an honest twin with the same motion and the same sole signal.

Twelve pilot frames from the torso camera: six honest floors and six deceptive ones, each with the red landing footprint and the green context crop
Pilot frames, the best frame before touchdown. The first six are honest floors of each class; the last six are deceptive, labelled with what they look like and what they are.
6

Steps, frames and crops

The crop is placed from where the foot really lands, which the simulator knows and a robot would have to predict. The independent re-check found that this geometry alone carries some signal; see the vision head.
7

Sole sanity check

The plan was a simple rule for the sole: while standing, it must read the robot's weight within 2 %, and in swing it must read zero.

The cause is the gap described next. The swing rule was then checked against the simulator's foot contacts, and the gap was documented with its numbers instead of re-running the replays.

8

Known gap: the edge-loaded foot

Step 1 sets its contact flag only when the force along the sole's normal, averaged over a frame, exceeds 2 N. At toe-off the box-shaped foot pitches 55-65° and loads its front edge, or it stubs the riser of a step. The ground then pushes hard, but almost along the plane of the sole. The contact flag stays off, the support state is "none", and the sole reads about zero. A sole touching the other foot also counts as contact.

Kept steps that overlap an edge-loaded frameG113G80SONIC
Inside the step window (touchdown to lift-off)0.0 %0.0 %0.0 %
Inside a fixed 30-frame window0.0 %0.4 %3.0 %

Reading the table. From the 60-episode noise-free re-run, 2,110 kept steps. The step windows start at touchdown, and none of them overlaps such a frame. In a fixed 30-frame window the share reaches 3.0 % of SONIC steps. The sole model and the contact labels miss toe-off and stub loads, mostly for SONIC.

The independent re-check could not verify these percentages: they come from the noise-free re-run, which it did not repeat.
9

Checks

The full re-check is on its own page: independent re-check of experiment 1.

10

Limits

Scores in this experiment are macro-F1 per step against simulator truth, on held-out textures. Next: the two teachers, the vision head, support state from vision, the VLM on real photos, the VLM tuned on the sole's labels, independent re-check. Experiment 2 uses its own dataset: dataset of experiment 2. Textures: ambientCG, CC0.