FLA · Experiment 1 vision head · 16 Nov 2026 · all reports · summary

Does the sole teach a camera better?

A frozen ViT looks at the floor before each step, and a small head on top learns to name the floor class. The head is trained three times, changing only who supplies the labels: the sole teacher, the joints and IMU teacher, or the simulator's truth. Six floor classes, three gaits.

Simulation only. Rendered camera, simulated sole. Few test textures: three per class.

In short

  1. On gaits the teachers saw, the two teachers tie. Vision taught by the sole scores 0.816, taught by joints and IMU 0.824. Taught by truth it scores 0.857; the re-check calls that gap inconclusive, not zero.
  2. On a gait the teachers never saw, only the sole teaches. Labels from SONIC steps: sole 0.488, joints and IMU 0.097. Truth would give 0.867, so the sole's labels still lose most of the way.
  3. This repeats the teachers' own gap. The re-check notes that the vision result restates the gap in the labels themselves (0.40 against 0.09 macro-F1), on one held-out gait.
  4. Vision reads the look, not the physics. On deceptive floors vision before the step matches the real floor on 0.07-0.12 of steps and the look on 0.66-0.78. The sole after the step matches the real floor on 0.820.
  5. The uncertainty is larger than the seed spread. With three test textures per class, resampling textures gives the sole-taught head a 95 % interval of 0.71 to 0.89.
1

What was done

2

Who teaches better

LabelsSettingAll gaitsG113G80SONICLabels right on training steps
Sole teacher(a) seen gaits0.816 ± 0.0280.7980.8330.8090.82
Joints + IMU teacher(a) seen gaits0.824 ± 0.0300.8260.8200.8190.82
Simulator truth(a) seen gaits0.857 ± 0.0090.8390.8940.8201.00
Sole teacher, shifted readout(a) seen gaits0.827 ± 0.0140.8250.8300.8150.81
Sole teacher(b) SONIC0.488 ± 0.0260.4830.4980.4800.43
Joints + IMU teacher(b) SONIC0.097 ± 0.0480.1040.0920.0910.13
Simulator truth(b) SONIC0.867 ± 0.0230.8520.8970.8351.00
Sole teacher, shifted readout(b) SONIC0.498 ± 0.0440.5090.4870.4960.42
Shuffled labels (control)0.197 ± 0.0320.2060.1940.183–
Three bar and line charts: test macro-F1 of the vision head by label source when labels come from the seen gaits, when they come from the unseen SONIC gait, and as a function of labelled steps per texture
(a) Labels from G113 and G80. (b) Labels from SONIC. (c) Fewer labelled steps per texture. Error bars: seed sd.

Reading the table. In (a) the two teachers give the same vision score, and in (b) the sole wins by 39 points. The last column explains both: in (a) the two label sets are equally right, agreeing with truth on 0.82 of training steps, in (b) the sole's labels are right far more often than the joints and IMU labels. Vision learns about as well as its labels allow; the sole-taught head in (b) slightly exceeds its own labels (0.40 macro-F1 against truth). The shifted readout of the sole changes nothing, so both conclusions survive that sensor change. The joints and IMU head in (b) is at or below the shuffled control.

The independent re-check reproduced every number here and adds the following. (a) sole against joints and IMU: a bootstrap over textures gives a difference of −0.009 [−0.052, 0.035]; the tie is expected, since the two label sets agree equally with truth and share the same wrong label on 8 % of training steps. (a) truth against sole: 0.041 [−0.016, 0.091]. Inconclusive, not "equal". (b) sole against joints and IMU: 0.396 [0.31, 0.48]. Real, but it restates the teachers' gap and rests on one held-out gait. The joints and IMU head answers "edge" on 63-99 % of steps. Shifted readout: the same teacher weights run on another readout of the same simulation; the sole is not built differently. Shuffled control: a random guesser matched to the test class mix scores 0.167. Once the clustering of predictions by texture is taken into account, the control is at chance.

No pad-sole data exists for the vision dataset, so the sensor-change test rests on the shifted readout only. For context, the sole teacher itself falls on the pad sole from 0.806 to 0.586 on seen gaits and from 0.437 to 0.325 on SONIC.

3

Fewer labels

The same comparison when only a few labelled steps per texture are used for training.

Labels, setting5 per texture20100All
Sole, (a)0.7430.7970.8040.816
Joints + IMU, (a)0.7700.7760.7880.824
Truth, (a)0.8080.8420.8550.857
Sole, (b)0.4590.4880.5020.488
Joints + IMU, (b)0.0870.0670.0980.097
Truth, (b)0.8380.8630.8760.867

Reading the table. Means over 3 seeds. In (a), sole and joints and IMU labels differ at no budget. In (b), the sole's labels beat the joints and IMU labels at every budget (0.459-0.502 against 0.067-0.098). Even 5 labelled steps per texture get most of the way.

4

Floors that lie

In the look_swap scenes every region shows a test texture of another class: a deceptive floor looks like one class and behaves like another. Vision answers before the step, the sole teacher after it.

HeadVision vs real floorVision vs lookSole vs real floorSole vs look
(a) sole labels0.120 ± 0.0100.656 ± 0.0290.820 ± 0.0010.035 ± 0.001
(a) joints + IMU labels0.108 ± 0.0100.700 ± 0.0210.820 ± 0.0010.035 ± 0.001
(a) truth labels0.071 ± 0.0110.784 ± 0.0040.820 ± 0.0010.035 ± 0.001
(b) sole labels0.118 ± 0.0120.473 ± 0.0660.820 ± 0.0010.035 ± 0.001
(b) joints + IMU labels0.074 ± 0.0180.217 ± 0.0140.820 ± 0.0010.035 ± 0.001
(b) truth labels0.081 ± 0.0060.799 ± 0.0180.820 ± 0.0010.035 ± 0.001
Bar chart for deceptive scenes: the three vision heads score about 0.1 against the physical floor and 0.66 to 0.78 against the look, while the sole teacher after the step scores 0.82 against the physical floor and near zero against the look
Deceptive scenes, heads from setting (a). Accuracy against the physical floor (left) and against the look (right).

Reading the table. Accuracy on 9,795 look_swap steps. Vision before the step reports what the floor looks like, whoever taught it; the head taught by truth follows the look most closely. The sole after the step reports what the floor is and almost never what it looks like. The sole's row is the same for every head because it is the same teacher on the same steps.

From the independent re-check: 3,848 look_swap steps are left out (no class, or a hard_mid look). The (b) heads range from 0.22 to 0.80 against the look. On SONIC steps the "sole teacher" here is the teacher trained with G113 held out, which did see SONIC. The second deceptive kind, a sheet hiding gaps between planks, keeps its class, so it is scored for support state instead: see support state from vision.
5

Four test episodes

Four whole test episodes, picked at random among two-floor episodes without a fall: two honest and two deceptive. The vision row is the (a) head taught by the sole (seed 0), the same head for the SONIC episode, which it never saw. Vision's frame is taken 0.7-1.8 s before touchdown; the sole's answer is shown at touchdown plus 0.1 s.

Third-person view, torso camera with the next footprint in red, and per step: vision before the step, the sole teacher after it, and the true floor. Green is right, red is wrong. 84 s.
EpisodeFloorsVision rightSole right
1, honest, G113grippy, then soft31 / 3736 / 37
2, honest, SONICgrippy, then soft7 / 2827 / 28
3, deceptive, G80grippy looks slippery, rough looks hard_mid8 / 3624 / 36
4, deceptive, G113grippy and slippery, both look sloped0 / 3836 / 38

Reading the clip. Episode 1 has 6 steps with no usable frame, counted as vision wrong. In episode 2 vision calls every soft step slippery: the held-out soft texture there, a checkerboard carpet, is never recognised as soft by this head (2 of 295 honest steps on it across gaits are right). That is a real texture failure, not a bug. In episode 3 the sole is only 24 of 36: on rough floor with the G80 gait it says grippy on 8 steps and edge on 3. In episode 4 vision is wrong everywhere because both floors look like a slope.

One episode each, so the counts illustrate; the measured numbers are in the tables above.

6

Robustness checks

7

Independent re-check

Full list: independent re-check of experiment 1.

8

Limits

Scores: macro-F1 per step against simulator truth on held-out textures, mean ± sd over 3 seeds; deceptive scenes as accuracy. Related: the dataset, the two teachers, support state from vision, the VLM on real photos, the VLM tuned on the sole's labels, independent re-check. Experiment 2, with a fixed patch and two classes: fixed patch. Model: DINOv3 ViT-B/16. Textures: ambientCG, CC0.