FLA · Experiment 2: dataset · 16 Nov 2026 · all reports · summary

A dataset for seeing the floor before the step

The Booster K1 walks over hard and soft floors in MuJoCo. A torso camera looks steeply down at one fixed patch of floor just ahead of the feet on every frame; the sole and the joints feel the floor after each step. This page describes how the camera and the patch were chosen, why the robot's shadow was switched off, which textures were used, and how the data are split.

Simulation only. Rendered camera, simulated sole, one gait.

In short

  1. 400 episodes, 16,918 steps, one gait. G113, an ordinary flat-foot walking policy, on two floor classes: hard and soft. 241 episodes were replayed from the step-1 data and all 241 came out bit-identical.
  2. One fixed patch per frame. The image rectangle [171, 66, 253, 140], about 0.3 × 0.3 m of floor centred 0.45 m ahead of the trunk. The robot's own body enters it in 0.68 % of frames.
  3. No shadows. With shadows on, the robot's shadow covered more than a tenth of the patch in 31.2 % of pilot frames, so they were switched off.
  4. 72 CC0 textures, 25 held out for test. No texture image is in two splits, and every episode keeps the split it had in the step-1 data.
  5. Deceptive looks. 85 test episodes are also rendered with the texture of the other class: hard that looks soft, soft that looks hard. They are 21.0 % of all steps and never used in training.
1

What the data are for

The question of this experiment is whether a vision model can learn the floor class from the sole's labels and name it before the foot lands. That needs, for every footstep, what the camera saw before the step and what the sole and the joints felt during it. To keep the task narrow, there is one gait, two floor classes and one fixed patch of floor that a model classifies on every frame. The results are on the fixed-patch results page.

2

Camera and patch: the pilot

The camera is fixed to the trunk at the mount of the first vision experiment: 0.09 m forward and 0.18 m up from the trunk origin, about 0.72 m above the floor while walking. Its intrinsics are like those of a RealSense D455: 424 × 240 px, vertical field of view 58°, fx = fy = 216.5 px, 10 frames per second.

A pilot on 5 training rollouts (two hard, two soft, one strip; commanded 0.26 to 0.42 m/s) tried 7 pitches from 55° to 72.5° and patch distances from 0.20 to 0.80 m in steps of 0.0125 m, with a segmentation pass on every frame to find robot pixels. The rule: the leading toe should be at the bottom of the image in at least half of the frames, and the median floor visible ahead of the leading toe should be 0.6 to 0.85 m; then take the nearest patch.

Six torso camera frames from the pilot on asphalt, plaster, red carpet, grass and a strip floor, each with a green box for the candidate patch; the robot's dark shadow falls on the floor in several of them
Pilot frames at pitch 67.5° with the pilot patch. The robot's shadow is still on: in two frames it reaches the patch.
Patch distance from the trunkRobot pixels in the patch, all speedsAt 0.40 m/s or faster
0.4375 m (what the pilot rule picked)3.2 %8.2 %
0.45 m (used)0.45 %2.1 %
0.45 m, final dataset, 80,000 frames0.68 %2.25 %

Reading the table. The first two rows come from a robustness check on 50 other saved rollouts. At the distance the pilot rule chose, the robot's own leg entered the patch in 8.2 % of frames at higher speed. Moving the patch slightly further out cut that to 2.1 %, so 0.45 m was used. In the final dataset the share is 0.05 % below 0.30 m/s, 0.38 % at 0.30 to 0.40 m/s and 2.25 % at 0.40 m/s or more. Nothing is masked; the robot share is stored per frame.

The chosen camera and patch

Ground footprint of the patch, 80,000 frames5th pctMedian95th pct
Centre, ahead of the trunk origin (m)0.4290.4520.496
Centre, sideways (m)−0.050−0.007+0.036
Near edge, ahead (m)0.2830.3000.334
Near edge, width (m)0.2760.2830.290
Far edge, ahead (m)0.5750.6030.657
Far edge, width (m)0.3140.3250.341
Area (m²)0.0860.0920.102

Reading the table. The patch stays put on the floor to within a few centimetres as the body sways: its centre moves 1.6 cm closer per degree the camera pitches down. In the pilot, the patch centre lay about 0.29 m ahead of the leading toe (0.23 to 0.36 m). This is why vision in this experiment sees the next floor one or two steps before the foot reaches it.

3

Shadows removed

The pilot frames showed the robot's shadow falling on the patch for some walking directions, because the sun is fixed in the world. Measured on the 5 pilot episodes with the light of the first vision experiment:

A shadow that follows the robot is a cue that has nothing to do with the floor, and it would differ between walking directions. The scene light was therefore set to cast no shadows, so no shadow can fall on the patch and no shadow flag is needed.

A real camera will see the robot's shadow. Models trained on these renders have never met one.
4

Textures and looks

72 CC0 photo textures from ambientCG (1K colour maps, 512 × 512): 36 hard (concrete, asphalt, tiles, marble, terrazzo) and 36 soft (carpet, grass, moss, and fabric limited to quilted, fleece, felt and terry cloth). The class follows the material: what a person would call that floor. A visual review then dropped foam (crumpled packaging, not a floor), plain flat-colour fabrics (they look like painted concrete) and one board-formed concrete that looks like planks.

Grid of 72 floor textures labelled by split and class: hard test, hard train, hard validation, soft test, soft train and soft validation
All 72 textures. Prefix: h or s for hard or soft, then te, tr or va for test, train or validation.
TexturesTestValidationTrain
Hard13419
Soft12420

Reading the table. The split is per class and stratified by material, with validation taken out of the training textures. Training episodes use training textures only, validation episodes validation textures only, test episodes test textures only. No image is in two splits (checked by hash). Test textures are new images of material families also seen in training: new textures, not new materials.

The independent re-check notes that validation has no asphalt, fabric, marble or tiles texture, and only 8 textures in all.

Looks

Twelve torso camera frames in two columns, each with the fixed patch as a green box and the stored crop beside it: honest hard, soft and strip floors at the top, deceptive floors below
Final camera and patch, no shadows. Top three rows honest, bottom three deceptive (for example, hard floor textured as carpet or moss, soft floor textured as tiles or terrazzo).
5

Episodes, steps and splits

400 G113 walking episodes from the step-1 data. 159 were saved rollouts and were copied; 241 were replayed deterministically, and every stored array was compared bit for bit: 241 of 241 identical, none dropped. Every episode keeps its step-1 split. All validation and test episodes of these classes are included; training is a fixed random subset.

EpisodesTrainValidationTestTest, deceptive
Hard98241434
Soft98241434
Strip, hard to soft306717
Steps, hard / soft5,174 / 4,4041,265 / 1,046789 / 6811,872 / 1,687

Reading the table. The deceptive column holds test episodes shown with the other class's look; their steps are counted by the real floor, so 1,872 are hard floors that look soft and 1,687 soft floors that look hard, 3,559 in all, 21.0 % of the 16,918 steps. The fixed-patch results use the 35 honest test episodes plus the honest look of the 85 deceptive ones as the main test set.

6

What is stored

7

Checks and limits

This page describes data, not scores; the results pages report macro-F1 per step on held-out textures. Related: fixed-patch results, teachers of experiment 2, re-check of experiment 2, dataset of the six-class experiment. Textures: ambientCG, CC0.