FLA · Experiment 2: fixed patch, hard against soft · 16 Nov 2026 · all reports · summary

Fixed patch: can the sole teach a camera hard from soft?

A torso camera on the Booster K1 looks at one fixed patch of floor ahead of the feet on every frame, and a small head names it hard or soft before the step. The sole names the floor after the step. Five comparisons, A to E: who teaches vision best, whether fusing vision with joints helps, which vision model, floors that look wrong, and floor changes.

Simulation only. Rendered camera, simulated sole, one gait.

In short

  1. A camera taught by the sole is as good as one taught by the simulator. 0.985 against 0.985 on 25 textures never seen in training. This is a tie by construction: every teacher is right on about 99 % of steps, so the test cannot rank teachers.
  2. Fusing vision with joints helps only because it answers later. Vision plus joints and IMU scores 0.997, but after the step, and it adds nothing over the sole alone. On deceptive floors it drops to 0.514.
  3. A small ViT is enough; the 7B VLM is not. DINOv3 ViT-S and ViT-B heads both score 0.985; SigLIP with no training 0.976; Qwen2.5-VL-7B with no training 0.668.
  4. New materials break the camera. With a whole material family held out of training, the head falls to 0.726.
  5. Vision reads the look, not the physics. On deceptive floors (they look like one class and behave like the other) vision scores 0.028; the sole after the step scores 0.997.
  6. At a floor change vision is early, the sole is on time. Vision already reports the new floor about 1.33 steps before the boundary, because the patch lies ahead of the feet.
1

What was done

The question: does a vision head trained on the sole's labels name the floor before the step as well as one trained on simulator truth? And how do other teachers, fusion with joints, other vision models, deceptive looks and floor changes compare?

Torso camera frames on several floors, each with the fixed patch marked by a green box and the stored crop beside it; the lower rows are deceptive floors
Torso camera, pitched down. The green box is the fixed patch; the stored crop is next to each frame.
2

One run on a test course

A new course, straight ahead: eight patches of 2.0 × 2.5 m, hard and soft in both orders, two of them deceptive. Patch 5 is soft but textured as terrazzo, patch 7 is hard but textured as carpet. All textures are test textures. G113 walks at 0.35 m/s. Nothing was trained on this course; the models are the ones from the comparisons below, one seed each.

Rows: the true floor, the ViT head before the step, the VLM with no training before the step, the sole and the joints after the step. Green is right, red is wrong. Shown at 0.6× speed.
Steps rightViT headVLM, zero-shotSoleJoints + IMU
All 126 steps91100125126
Honest patches, 103 steps9189102103
Deceptive patches against the real floor, 23 steps0112323
Deceptive patches against the look, 23 steps231200

Reading the table. All 12 honest errors of the ViT head are the last one to three steps before a change of floor: the patch lies about 0.3 m ahead of the leading toe, so the head names the next floor early. The sole's single error is a foot straddling a soft-to-hard boundary. The VLM's 11 right deceptive steps are not insight: it calls carpet hard, so it is right on the hard floor that looks like carpet (11 of 12) and wrong on all 11 steps of the soft floor that looks hard. On the honest soft carpet it gets 4 of 11, against 9 of 11 for the ViT head.

A single run with one seed of each model, so these counts are a demonstration, not an estimate. The torso view is rendered without shadows, as in the dataset.

3 · comparison A

Who teaches vision better?

The same head on DINOv3 ViT-B/16 features, trained on the same training frames. Only the source of the labels changes. Honest test looks, 5,029 steps.

Labels for the headMacro-F1 (± sd, 3 seeds)95 % interval, textures95 % interval, episodesAccuracy
Simulator truth0.985 ± 0.000[0.972, 0.993][0.979, 0.991]0.986
Sole teacher0.985 ± 0.003[0.974, 0.992][0.978, 0.990]0.985
Joints + IMU teacher0.984 ± 0.002[0.968, 0.993][0.977, 0.990]0.984
Both teachers together0.985 ± 0.002[0.971, 0.994][0.978, 0.991]0.985
Shuffled labels (control)0.565 ± 0.088[0.490, 0.629][0.533, 0.596]0.586

Reading the table. It is a tie, as expected. Every teacher-minus-truth and teacher-minus-teacher difference is within ±0.001 and every texture interval contains 0; sole minus truth is −0.001 [−0.004, +0.002]. The tie is built into the task: the teachers' labels agree with truth on 98.8 to 99.7 % of training steps (independent re-check), so the label sets differ on about 1 % of frames. The table therefore says the sole can replace simulator truth here, not that it is a better teacher.

4 · comparison B

Vision alone, or vision with joints and IMU?

One classifier was given both the mean ViT feature of the same five frames and the joints and IMU window of the step itself (touchdown to lift-off, at most 0.6 s). It therefore answers after the step. It was never trained on a deceptive look.

ClassifierAnswersHonest looks, 5,029 stepsDeceptive looks, real floor, 3,559 steps
Vision alone, sole labelsbefore the step0.985 ± 0.0030.027 ± 0.008
Vision alone, truth labelsbefore the step0.985 ± 0.0000.028 ± 0.003
Vision + joints + IMU, sole labelsafter the step0.997 ± 0.0010.514 ± 0.104
Vision + joints + IMU, truth labelsafter the step0.997 ± 0.0020.287 ± 0.003
Joints + IMU teacher aloneafter the step0.992 ± 0.0010.992 ± 0.000
Sole teacher aloneafter the step0.997 ± 0.0010.997 ± 0.001

Reading the table. On honest floors the fused classifier beats vision alone by +0.012 [+0.006, +0.022] and the joints teacher by +0.005 [+0.002, +0.007], both real. But it answers after the step, so the first comparison is not like for like, and it adds nothing over the sole alone (+0.000 [−0.003, +0.002] in the independent re-check). On deceptive floors adding vision makes the after-step answer far worse: trained where look and physics always agree, the classifier learned to lean on the look. The verdict is "do not fuse naively", not "fusion helps".

The re-check finds "learned to trust the look" only partly true: the three sole-label seeds score 0.42, 0.50 and 0.63 against the real floor, and 0.443 against the look. Two of the truth-label checkpoints were picked at the first epoch.
5 · comparison C

Which vision model?

Four models on the same patch and the same honest test steps. The two zero-shot models get no training: SigLIP compares the image with a few written class descriptions, and Qwen2.5-VL-7B is asked for a one-word answer, hard or soft, read from the probability of its first token.

ModelMacro-F195 % interval, texturesAccuracyFrames per second
DINOv3 ViT-B/16 + head, truth labels0.985 ± 0.000[0.972, 0.993]0.9861,770 ± 375
DINOv3 ViT-S/16 + head, truth labels0.985 ± 0.003[0.975, 0.992]0.9854,006 ± 801
DINOv3 ViT-S/16 + head, sole labels0.984 ± 0.004[0.974, 0.991]0.984–
SigLIP-B/16, zero-shot0.976 (1 run)[0.952, 0.989]0.9762,588 ± 131
Qwen2.5-VL-7B, zero-shot0.668 (1 run)[0.494, 0.834]0.70747 ± 3
DINOv3 ViT-B + head, whole material family held out0.726 ± 0.047[0.557, 0.890]0.736–

Reading the table. The small ViT loses nothing to the large one (−0.001 [−0.004, +0.004]) at more than twice the speed. SigLIP is 0.010 below the ViT-B head (interval [−0.021, −0.002]; in the re-check the upper end touches 0). The VLM is 0.318 below the ViT-B head, real, and most of its errors are soft floors called hard. Holding out a whole material family costs 0.259, also real: the "held-out textures" of the other rows are new textures of known families, not new materials.

Speeds: one H100, model alone, features not cached, batch 64 (VLM batch 16, preprocessing included), three timed repeats. Another job held the GPU at 62 to 100 % just before each repeat, so these are lower bounds with large spread, and the re-check could not re-time them.

6 · comparison D

Floors that lie

Hard floor that looks soft and soft floor that looks hard: 3,559 steps, scored once against the real floor and once against the class the floor looks like.

SourceAnswersAgainst the real floorAgainst the look
ViT-B head, truth labelsbefore the step0.028 ± 0.0030.972 ± 0.003
ViT-B head, sole labelsbefore the step0.027 ± 0.0080.973 ± 0.008
SigLIP, zero-shotbefore the step0.0330.966
Qwen2.5-VL-7B, zero-shotbefore the step0.2100.721
Sole teacherafter the step0.997 ± 0.0010.003 ± 0.001
Joints + IMU teacherafter the step0.992 ± 0.0000.008 ± 0.000
Sole + joints + IMU teacherafter the step0.996 ± 0.0010.004 ± 0.001
Vision + joints + IMU, truth labelsafter the step0.287 ± 0.0030.550 ± 0.010

Reading the table. Vision before the step reports the look, not the physics. On the very same steps with the honest look the ViT head scores 0.987. The sensors after the step report the physics: sole minus vision is +0.969 [+0.947, +0.984]. Sole minus joints is +0.005 [+0.002, +0.008], real but small. A head taught by the sole is fooled just as much (0.027) as one taught by truth (0.028): the sole teaches only what a floor usually looks like, it cannot make vision see stiffness.

The teachers read no image, so their deceptive score equals their honest score by construction. The deceptive looks reuse the 25 test textures, swapped between classes; the result shows that vision reads appearance, not what a model would do on a real ambiguous floor.

7 · comparison E

When the floor changes

Strip episodes cross from one floor to another. For each boundary: how many steps from the boundary step until a source names the new floor (0 = on the boundary step). "Never" means not within the new segment. "Early" counts steps just before the boundary that already gave the new class. Pooled over three seeds; 24 boundaries, all hard to soft.

SourceStripsBoundaries × seedsRight at onceMedian90th pctMeanNeverEarly: mean steps; share
Vision, truth labelshonest72690.00.00.1001.31; 0.71
Vision, sole labelshonest72710.00.00.0401.33; 0.75
VLM, zero-shothonest24130.08.01.6760.67; 0.33
Sole teacherhonest72690.00.00.0600.08; 0.07
Joints + IMU teacherhonest72660.00.00.1700.06; 0.03
Vision + joints + IMU, truthhonest72660.00.00.1100.04; 0.04
Vision, truth labelsdeceptive510–––510.00; 0.00
Vision, sole labelsdeceptive510–––510.00; 0.00
VLM, zero-shotdeceptive170–––170.00; 0.00
Sole teacherdeceptive51480.00.00.0800.00; 0.00
Joints + IMU teacherdeceptive51450.01.00.2400.00; 0.00
Vision + joints + IMU, truthdeceptive51010.025.014.20460.00; 0.00

Reading the table. At an honest boundary vision is ahead of the foot, not behind it: in 75 % of boundaries (sole labels) it already answers "soft" on the last hard steps, 1.33 steps early on average, because the patch lies about 0.3 to 0.5 m ahead of the landing foot. Against the floor under the foot these early answers count as errors; for a controller they might be a warning in time, but that was not tested. The sole and the joints switch on the boundary step itself. On deceptive strips vision never reaches the real floor, and the fused classifier fails in 46 of 51 cases. The VLM is slow even on honest strips: 13 of 24 at once, 6 never.

Only 24 boundaries, all hard to soft, one gait and one patch distance. Pooling three seeds triples the count without adding independent cases. The early rate depends on the 0.45 m patch distance and the walking speed. The deceptive strips use 17 of the test episodes, the honest strips 7 test episodes plus the honest look of those 17.

8

What it means and its limits

Scores are macro-F1 per step on held-out textures (honest looks: 5,029 steps on 25 textures; deceptive looks: 3,559 steps), mean ± sd over three training seeds unless marked "1 run". Every number in tables A to E was reproduced to three decimals by an independent re-check: re-check of experiment 2. Related: dataset of experiment 2, its teachers, the six-class experiment dataset, teachers, vision head, VLM with LoRA, re-check. Textures: ambientCG, CC0. Models: DINOv3, SigLIP, Qwen2.5-VL-7B.