Fixed patch: can the sole teach a camera hard from soft?
A torso camera on the Booster K1 looks at one fixed patch of floor ahead of the feet on every frame, and a small head names it hard or soft before the step. The sole names the floor after the step. Five comparisons, A to E: who teaches vision best, whether fusing vision with joints helps, which vision model, floors that look wrong, and floor changes.
In short
- A camera taught by the sole is as good as one taught by the simulator. 0.985 against 0.985 on 25 textures never seen in training. This is a tie by construction: every teacher is right on about 99 % of steps, so the test cannot rank teachers.
- Fusing vision with joints helps only because it answers later. Vision plus joints and IMU scores 0.997, but after the step, and it adds nothing over the sole alone. On deceptive floors it drops to 0.514.
- A small ViT is enough; the 7B VLM is not. DINOv3 ViT-S and ViT-B heads both score 0.985; SigLIP with no training 0.976; Qwen2.5-VL-7B with no training 0.668.
- New materials break the camera. With a whole material family held out of training, the head falls to 0.726.
- Vision reads the look, not the physics. On deceptive floors (they look like one class and behave like the other) vision scores 0.028; the sole after the step scores 0.997.
- At a floor change vision is early, the sole is on time. Vision already reports the new floor about 1.33 steps before the boundary, because the patch lies ahead of the feet.
What was done
The question: does a vision head trained on the sole's labels name the floor before the step as well as one trained on simulator truth? And how do other teachers, fusion with joints, other vision models, deceptive looks and floor changes compare?
- Data. 400 episodes of G113, an ordinary flat-foot walking policy, on hard and soft floors in MuJoCo: 16,918 steps. The floors carry 72 CC0 photo textures; 25 are held out for testing (13 hard, 12 soft). Details on the dataset page.
- The patch. About 0.3 × 0.3 m of floor, centred about 0.45 m ahead of the trunk, cut from the same image rectangle on every frame.
- The vision head. A small MLP on frozen DINOv3 features of the patch. A step's answer is the mean probability over the last 5 frames (0.5 s at 10 Hz) up to the last frame before touchdown. It is scored against the floor the foot actually touches.
- Training labels for a frame. The label of the step that later lands inside that frame's patch. Frames that no step lands in are not used for training. The 9 % of steps that no earlier patch covered give no training frame, but they are scored at test.
- Teachers. A classifier on the sole alone (coarse sole features: four pressure blocks, four shear vectors, centre of pressure, total force, contact area), one on joints and IMU alone, and one on both. They are cross-fitted so that none ever labelled an episode it trained on, and none trained on a test episode. See the teachers page.
- Test sets. Honest looks: 35 test episodes plus the honest look of 85 more, 5,029 steps on 25 textures. Deceptive looks: the same 85 episodes shown with the texture of the other class, 3,559 steps, never used in training.
- Uncertainty. Three training seeds, and 95 % intervals from resampling the test textures within each class and from resampling episodes. A difference counts as real only if its texture-resampled interval excludes 0.
- Chance. A random guess at the class prior scores 0.500, a constant majority answer 0.346. Soft floors make up 0.471 of the honest test steps.

One run on a test course
A new course, straight ahead: eight patches of 2.0 × 2.5 m, hard and soft in both orders, two of them deceptive. Patch 5 is soft but textured as terrazzo, patch 7 is hard but textured as carpet. All textures are test textures. G113 walks at 0.35 m/s. Nothing was trained on this course; the models are the ones from the comparisons below, one seed each.
| Steps right | ViT head | VLM, zero-shot | Sole | Joints + IMU |
|---|---|---|---|---|
| All 126 steps | 91 | 100 | 125 | 126 |
| Honest patches, 103 steps | 91 | 89 | 102 | 103 |
| Deceptive patches against the real floor, 23 steps | 0 | 11 | 23 | 23 |
| Deceptive patches against the look, 23 steps | 23 | 12 | 0 | 0 |
Reading the table. All 12 honest errors of the ViT head are the last one to three steps before a change of floor: the patch lies about 0.3 m ahead of the leading toe, so the head names the next floor early. The sole's single error is a foot straddling a soft-to-hard boundary. The VLM's 11 right deceptive steps are not insight: it calls carpet hard, so it is right on the hard floor that looks like carpet (11 of 12) and wrong on all 11 steps of the soft floor that looks hard. On the honest soft carpet it gets 4 of 11, against 9 of 11 for the ViT head.
A single run with one seed of each model, so these counts are a demonstration, not an estimate. The torso view is rendered without shadows, as in the dataset.
Who teaches vision better?
The same head on DINOv3 ViT-B/16 features, trained on the same training frames. Only the source of the labels changes. Honest test looks, 5,029 steps.
| Labels for the head | Macro-F1 (± sd, 3 seeds) | 95 % interval, textures | 95 % interval, episodes | Accuracy |
|---|---|---|---|---|
| Simulator truth | 0.985 ± 0.000 | [0.972, 0.993] | [0.979, 0.991] | 0.986 |
| Sole teacher | 0.985 ± 0.003 | [0.974, 0.992] | [0.978, 0.990] | 0.985 |
| Joints + IMU teacher | 0.984 ± 0.002 | [0.968, 0.993] | [0.977, 0.990] | 0.984 |
| Both teachers together | 0.985 ± 0.002 | [0.971, 0.994] | [0.978, 0.991] | 0.985 |
| Shuffled labels (control) | 0.565 ± 0.088 | [0.490, 0.629] | [0.533, 0.596] | 0.586 |
Reading the table. It is a tie, as expected. Every teacher-minus-truth and teacher-minus-teacher difference is within ±0.001 and every texture interval contains 0; sole minus truth is −0.001 [−0.004, +0.002]. The tie is built into the task: the teachers' labels agree with truth on 98.8 to 99.7 % of training steps (independent re-check), so the label sets differ on about 1 % of frames. The table therefore says the sole can replace simulator truth here, not that it is a better teacher.
- Shuffled control. 0.565 is 0.420 below the truth head and near chance once whole textures are allowed to flip together. The independent re-check puts each seed at chance but the mean of the three at the 96.3rd percentile of a texture-aware null: borderline, not cleanly at chance.
- Where the teachers do differ. Training only on stand and stop steps (commanded speed below 0.05 m/s), the joints teacher is right on 88.5 % of training steps and the sole on 97.2 %. The heads drop to 0.914 to 0.935 for every label source, with only 169 training frames, and sole minus joints is −0.013 [−0.038, +0.005]: no separation. The re-check finds truth minus joints at −0.009 [−0.020, −0.000], so call that borderline, and the claim that no subset separates the teachers rests on this one small subset.
- Which frames get which label. The rule above (the step that lands inside the patch) beats "the next step to touch down": 0.985 against 0.959 ± 0.013 with truth labels and 0.973 ± 0.020 with sole labels. The re-check finds the gain real for truth labels (+0.027 [+0.002, +0.072]) but not for sole labels (+0.012 [−0.002, +0.042]).
Vision alone, or vision with joints and IMU?
One classifier was given both the mean ViT feature of the same five frames and the joints and IMU window of the step itself (touchdown to lift-off, at most 0.6 s). It therefore answers after the step. It was never trained on a deceptive look.
| Classifier | Answers | Honest looks, 5,029 steps | Deceptive looks, real floor, 3,559 steps |
|---|---|---|---|
| Vision alone, sole labels | before the step | 0.985 ± 0.003 | 0.027 ± 0.008 |
| Vision alone, truth labels | before the step | 0.985 ± 0.000 | 0.028 ± 0.003 |
| Vision + joints + IMU, sole labels | after the step | 0.997 ± 0.001 | 0.514 ± 0.104 |
| Vision + joints + IMU, truth labels | after the step | 0.997 ± 0.002 | 0.287 ± 0.003 |
| Joints + IMU teacher alone | after the step | 0.992 ± 0.001 | 0.992 ± 0.000 |
| Sole teacher alone | after the step | 0.997 ± 0.001 | 0.997 ± 0.001 |
Reading the table. On honest floors the fused classifier beats vision alone by +0.012 [+0.006, +0.022] and the joints teacher by +0.005 [+0.002, +0.007], both real. But it answers after the step, so the first comparison is not like for like, and it adds nothing over the sole alone (+0.000 [−0.003, +0.002] in the independent re-check). On deceptive floors adding vision makes the after-step answer far worse: trained where look and physics always agree, the classifier learned to lean on the look. The verdict is "do not fuse naively", not "fusion helps".
Which vision model?
Four models on the same patch and the same honest test steps. The two zero-shot models get no training: SigLIP compares the image with a few written class descriptions, and Qwen2.5-VL-7B is asked for a one-word answer, hard or soft, read from the probability of its first token.
| Model | Macro-F1 | 95 % interval, textures | Accuracy | Frames per second |
|---|---|---|---|---|
| DINOv3 ViT-B/16 + head, truth labels | 0.985 ± 0.000 | [0.972, 0.993] | 0.986 | 1,770 ± 375 |
| DINOv3 ViT-S/16 + head, truth labels | 0.985 ± 0.003 | [0.975, 0.992] | 0.985 | 4,006 ± 801 |
| DINOv3 ViT-S/16 + head, sole labels | 0.984 ± 0.004 | [0.974, 0.991] | 0.984 | – |
| SigLIP-B/16, zero-shot | 0.976 (1 run) | [0.952, 0.989] | 0.976 | 2,588 ± 131 |
| Qwen2.5-VL-7B, zero-shot | 0.668 (1 run) | [0.494, 0.834] | 0.707 | 47 ± 3 |
| DINOv3 ViT-B + head, whole material family held out | 0.726 ± 0.047 | [0.557, 0.890] | 0.736 | – |
Reading the table. The small ViT loses nothing to the large one (−0.001 [−0.004, +0.004]) at more than twice the speed. SigLIP is 0.010 below the ViT-B head (interval [−0.021, −0.002]; in the re-check the upper end touches 0). The VLM is 0.318 below the ViT-B head, real, and most of its errors are soft floors called hard. Holding out a whole material family costs 0.259, also real: the "held-out textures" of the other rows are new textures of known families, not new materials.
- SigLIP needs the right words. Only one of three wordings works, the one that names materials, five per class (concrete, stone, tile, asphalt, wood; carpet, grass, moss, foam mat, cloth rug). These overlap the texture families, so this is zero-shot informed by the categories. The two generic sets ("hard / soft floor", "rigid / deformable surface") scored 0.37 and 0.36 on validation and 0.39 and 0.41 on test (re-check): below chance.
- The VLM. 99.9 % of the first-token probability falls on the two answer words. Two of three wordings tied on validation at 0.874; the score falls to 0.668 on test, because validation has only 8 textures.
- Held-out family, by family (accuracy, since each family is a single class): asphalt 0.207, carpet 0.473, grass 0.714, moss 0.810, fabric 0.868, tiles 0.983, concrete, marble and terrazzo 1.000. The family is that of the patch texture at the last frame, which differs from the touched floor on 67 steps; by touched floor the main head's carpet accuracy is 0.999 rather than 0.982. Between 19 and 174 training frames per held-out family straddle a boundary, so a little of that family was visible. Each family has only 1 to 5 test textures.
Speeds: one H100, model alone, features not cached, batch 64 (VLM batch 16, preprocessing included), three timed repeats. Another job held the GPU at 62 to 100 % just before each repeat, so these are lower bounds with large spread, and the re-check could not re-time them.
Floors that lie
Hard floor that looks soft and soft floor that looks hard: 3,559 steps, scored once against the real floor and once against the class the floor looks like.
| Source | Answers | Against the real floor | Against the look |
|---|---|---|---|
| ViT-B head, truth labels | before the step | 0.028 ± 0.003 | 0.972 ± 0.003 |
| ViT-B head, sole labels | before the step | 0.027 ± 0.008 | 0.973 ± 0.008 |
| SigLIP, zero-shot | before the step | 0.033 | 0.966 |
| Qwen2.5-VL-7B, zero-shot | before the step | 0.210 | 0.721 |
| Sole teacher | after the step | 0.997 ± 0.001 | 0.003 ± 0.001 |
| Joints + IMU teacher | after the step | 0.992 ± 0.000 | 0.008 ± 0.000 |
| Sole + joints + IMU teacher | after the step | 0.996 ± 0.001 | 0.004 ± 0.001 |
| Vision + joints + IMU, truth labels | after the step | 0.287 ± 0.003 | 0.550 ± 0.010 |
Reading the table. Vision before the step reports the look, not the physics. On the very same steps with the honest look the ViT head scores 0.987. The sensors after the step report the physics: sole minus vision is +0.969 [+0.947, +0.984]. Sole minus joints is +0.005 [+0.002, +0.008], real but small. A head taught by the sole is fooled just as much (0.027) as one taught by truth (0.028): the sole teaches only what a floor usually looks like, it cannot make vision see stiffness.
The teachers read no image, so their deceptive score equals their honest score by construction. The deceptive looks reuse the 25 test textures, swapped between classes; the result shows that vision reads appearance, not what a model would do on a real ambiguous floor.
When the floor changes
Strip episodes cross from one floor to another. For each boundary: how many steps from the boundary step until a source names the new floor (0 = on the boundary step). "Never" means not within the new segment. "Early" counts steps just before the boundary that already gave the new class. Pooled over three seeds; 24 boundaries, all hard to soft.
| Source | Strips | Boundaries × seeds | Right at once | Median | 90th pct | Mean | Never | Early: mean steps; share |
|---|---|---|---|---|---|---|---|---|
| Vision, truth labels | honest | 72 | 69 | 0.0 | 0.0 | 0.10 | 0 | 1.31; 0.71 |
| Vision, sole labels | honest | 72 | 71 | 0.0 | 0.0 | 0.04 | 0 | 1.33; 0.75 |
| VLM, zero-shot | honest | 24 | 13 | 0.0 | 8.0 | 1.67 | 6 | 0.67; 0.33 |
| Sole teacher | honest | 72 | 69 | 0.0 | 0.0 | 0.06 | 0 | 0.08; 0.07 |
| Joints + IMU teacher | honest | 72 | 66 | 0.0 | 0.0 | 0.17 | 0 | 0.06; 0.03 |
| Vision + joints + IMU, truth | honest | 72 | 66 | 0.0 | 0.0 | 0.11 | 0 | 0.04; 0.04 |
| Vision, truth labels | deceptive | 51 | 0 | – | – | – | 51 | 0.00; 0.00 |
| Vision, sole labels | deceptive | 51 | 0 | – | – | – | 51 | 0.00; 0.00 |
| VLM, zero-shot | deceptive | 17 | 0 | – | – | – | 17 | 0.00; 0.00 |
| Sole teacher | deceptive | 51 | 48 | 0.0 | 0.0 | 0.08 | 0 | 0.00; 0.00 |
| Joints + IMU teacher | deceptive | 51 | 45 | 0.0 | 1.0 | 0.24 | 0 | 0.00; 0.00 |
| Vision + joints + IMU, truth | deceptive | 51 | 0 | 10.0 | 25.0 | 14.20 | 46 | 0.00; 0.00 |
Reading the table. At an honest boundary vision is ahead of the foot, not behind it: in 75 % of boundaries (sole labels) it already answers "soft" on the last hard steps, 1.33 steps early on average, because the patch lies about 0.3 to 0.5 m ahead of the landing foot. Against the floor under the foot these early answers count as errors; for a controller they might be a warning in time, but that was not tested. The sole and the joints switch on the boundary step itself. On deceptive strips vision never reaches the real floor, and the fused classifier fails in 46 of 51 cases. The VLM is slow even on honest strips: 13 of 24 at once, 6 never.
Only 24 boundaries, all hard to soft, one gait and one patch distance. Pooling three seeds triples the count without adding independent cases. The early rate depends on the 0.45 m patch distance and the walking speed. The deceptive strips use 17 of the test episodes, the honest strips 7 test episodes plus the honest look of those 17.
What it means and its limits
- The sole can stand in for simulator truth as a teacher of vision: 0.985 against 0.985. The test is weak, because every teacher is about 99 to 99.7 % right on this two-class task.
- What vision learns is the look of known material families. A new family drops it to 0.726, a deceptive look to 0.03, and no teacher changes that. Only the sensors after the step know the physics: sole 0.997, joints and IMU 0.992.
- A cue without pixels. The independent re-check found that body state before the step (mainly trunk height: the robot stands about 6 mm lower on soft floor) predicts the floor at 0.957 and is not fooled by the look. This report never compares vision with such a baseline. Details on the re-check page.
- Easy task. Hard and soft differ by about four orders of magnitude in stiffness, so the tie in A says nothing about which teacher would be better on a harder task.
- Rendering. No shadows, no lighting change, one camera pose, textures on a flat plane (possibly seen in DINOv3 pretraining), no real images.
- Labels. Truth at the foot is the class of the touched region; the teachers' own scores use the earlier labelling, which differs on 36 of 16,918 steps (0.2 %). Interval widths are dominated by the 25 test textures. The VLM was used zero-shot only.
Scores are macro-F1 per step on held-out textures (honest looks: 5,029 steps on 25 textures; deceptive looks: 3,559 steps), mean ± sd over three training seeds unless marked "1 run". Every number in tables A to E was reproduced to three decimals by an independent re-check: re-check of experiment 2. Related: dataset of experiment 2, its teachers, the six-class experiment dataset, teachers, vision head, VLM with LoRA, re-check. Textures: ambientCG, CC0. Models: DINOv3, SigLIP, Qwen2.5-VL-7B.