A VLM taught by the sole
Qwen2.5-VL-7B looks at the torso-camera image of the floor just before a step and names one of six floor classes. Tested as it comes, and after a LoRA fine-tune on labels from the sole teacher, which only feels the step and never sees the image. Then on deceptive floors, and back on real photos.
In short
- LoRA on the sole's labels helps a lot in simulation. On held-out textures macro-F1 rises from 0.343 zero-shot to 0.721.
- It learns what floors look like, not what they are. On deceptive floors it follows the look more after tuning (0.347 to 0.524) and matches the real floor on only 0.124 of steps.
- It loses its skill on real ground. On the real photos macro-F1 falls from 0.679 to 0.375; it answers "grippy" to most of them.
- It is still behind a ViT with a small head on the same crops, 0.721 against 0.821, a borderline difference in a comparison that is not fully fair.
- The claim that did not hold: that tuning makes the model really worse against the real floor on deceptive scenes. The difference is inside the noise.
What was done
- Image. The same rows as the ViT head (the vision head): the best torso-camera frame before contact, cropped around the place where the foot will land and resized to 224 × 224. Steps with no class (a test-only floor type and boundary steps) are dropped.
- Question. The same wording as on the real photos (previous page), unchanged: answer in JSON with one of hard_grip (grippy hard floor), hard_slip (slippery hard floor), soft, rough, slope or edge (planks and steps).
- Training. 12,000 crops per seed, one epoch, drawn from the 22,421 eligible steps of training episodes with training textures, on G113 and G80 (two ordinary flat-foot walking policies). Labels come from the cross-fitted sole teacher: it reads only coarse sole features (four pressure blocks, four shear vectors, centre of pressure, total force, contact area), and no teacher labels an episode it trained on. LoRA seed k uses teacher seed k.
- Settings fixed before the pilot and never tuned: rank 16, alpha 32, learning rate 1e-4 with cosine decay, batch 16, language model only.
- Control. A LoRA trained on the seed-0 crops with shuffled labels.
- Evaluation. All 5,248 eligible test steps (held-out textures, honest look); a class-stratified sample of 4,000 of the 11,653 deceptive look_swap steps (every region shows a texture of another class); and the 1,350 real photos of the previous page.
- Statistics. Spread over 3 LoRA seeds is the sd; zero-shot is one deterministic run, with a bootstrap sd. A difference counts only if it is larger than twice the larger spread.
Results
| Test | Zero-shot | LoRA, sole labels | LoRA, shuffled |
|---|---|---|---|
| Held-out textures, macro-F1 (6 classes) | 0.343 ± 0.004 | 0.721 ± 0.040 | 0.061 |
| Held-out textures, accuracy | 0.416 | 0.726 ± 0.041 | 0.224 |
| Deceptive, accuracy against the real floor | 0.149 | 0.124 ± 0.006 | 0.167 |
| Deceptive, accuracy against the look | 0.347 | 0.524 ± 0.041 | 0.155 |
| Real photos, macro-F1 (3 classes) | 0.679 ± 0.012 | 0.375 ± 0.024 | 0.167 |
| Real photos, accuracy | 0.626 | 0.433 ± 0.017 | 0.333 |

Reading the table. Tuning on the sole's labels about doubles the score on new textures, a real gain of +0.378. The model answers in valid JSON every time, in every run. The shuffled control answers hard_grip for every crop, so the gain comes from the labels, not from learning the answer format or the crops.
Speeds on a shared H100: training 12.7 crops per second, 34.7 answers per second on simulated crops, 20.2 on real photos; peak memory 23.8 GB. The whole run took 1 h 47 min.
Per class
| Right on held-out textures | hard_grip | hard_slip | soft | rough | slope | edge |
|---|---|---|---|---|---|---|
| Zero-shot | 0.179 | 0.861 | 0.549 | 0.812 | 0.001 | 0.000 |
| LoRA, sole labels | 0.928 ± 0.024 | 0.923 ± 0.010 | 0.402 ± 0.011 | 0.560 ± 0.185 | 0.592 ± 0.040 | 0.840 ± 0.064 |
Reading the table. Zero-shot almost never answers slope or edge and calls most grippy floors slippery (856 of 1,173 steps): it cannot read slopes or planks from a crop. After tuning, slope and edge are mostly right. Tuning leaves two weak classes: soft (often called grippy or slippery) and rough, the least stable across seeds (often called grippy).
Floors that lie
On a deceptive floor the look belongs to one class and the physics to another. The camera sees only the look.
- Against the look: 0.347 zero-shot, 0.524 ± 0.041 after tuning, a real increase. After tuning the model follows the look even more.
- Against the real floor: 0.149 zero-shot, 0.124 ± 0.006 after tuning. Both are below the 1/6 of a uniform guess: neither reads the physics through a false look.
The ViT head does the same on the same 4,000 deceptive rows: 0.127 against the real floor, 0.667 against the look. It follows the look even more than the tuned VLM.
Back on real photos
| Right on real photos | hard_grip | rough | soft | Macro-F1 |
|---|---|---|---|---|
| Zero-shot | 0.39 | 0.68 | 0.80 | 0.679 ± 0.012 |
| LoRA, sole labels | 0.94 | 0.04 | 0.31 | 0.375 ± 0.024 |
Reading the table. After tuning on simulated crops the model answers hard_grip to most real photos (1,019-1,053 of 1,350 across seeds), and rough photos become hard_grip in 91-100 % of cases. It learned the simulator's textures, not floors. The drop is real, and it is not a shift toward the three classes that have no real photos: those get 8-13 % of the tuned model's answers, against 17 % zero-shot.
Per-class values and answer counts from the independent re-check. Real photos are GTOS-Mobile (numbers only, not shown).
A six-class course
One run of G113 at 0.35 m/s over a row of seven patches with held-out textures: grippy, slippery, soft, rough, a sloped ridge, then two deceptive patches (looks grippy but is slippery; looks slippery but is soft). Unlike the test set, the crop here is causal: the landing spot is forecast from past footsteps, not taken from the future. Rows: truth, the ViT head (taught by the sole, seed 0), the tuned VLM (seed 0), the sole teacher and the joints and IMU teacher.
| Patch (steps) | ViT head | VLM (LoRA) | Sole | Joints + IMU |
|---|---|---|---|---|
| Start (9) | 9 | 9 | 7 | 6 |
| Grippy (10) | 10 | 10 | 10 | 10 |
| Slippery (12) | 10 | 12 | 12 | 12 |
| Soft (11) | 11 | 11 | 11 | 11 |
| Rough (11) | 7 | 4 | 9 | 11 |
| Slope (13) | 4 | 0 | 12 | 9 |
| Deceptive: looks grippy, is slippery (16) | 0 | 1 | 15 | 15 |
| Deceptive: looks slippery, is soft (10) | 0 | 0 | 10 | 10 |
| End (15) | 14 | 14 | 15 | 14 |
| All scored steps (107) | 65 | 61 | 101 | 98 |
Reading the clip. On the five honest patches the ViT head is right on 42 of 57 steps and the VLM on 37. On the two deceptive patches both follow the look: against the look class each is right on 24 of 26 steps, against the real floor 0 and 1. The sole and the joints after the step get 25 of 26 each.
Independent re-check
- Every number reproduces to 3 decimals from the saved predictions, with the re-check's own answer parser.
- Rows. The 5,248 rows are the ViT head's test rows that have a best frame; the ViT head's 0.816 also counts 34 frameless steps. The re-check rebuilt 25 random crops: all byte-identical.
- Labels and leaks. All training lists are training split, training textures, G113 and G80, with 0 test episodes and 0 test textures. Every label equals the cross-fitted sole teacher's (which agrees with truth on only 0.81 of them). The shuffled list has the same rows and the same label counts.
- The question wording is the same in all 15 runs and was last changed before the real-photo test.
- Nothing was selected on test, with a caveat: the settings are identical in the pilot and the full run, with no checkpoint choice (one epoch, last weights), but the pilot did score 300 test-split steps before the run sizes were fixed.
- The ViT head against the VLM, on the same 5,248 rows: 0.821 ± 0.028 against 0.721 ± 0.040, a texture-bootstrap difference of +0.109 [0.001, 0.240], borderline. Not a fair fight: the VLM trains on 12,000 crops for one epoch with no validation; the ViT head uses 22,421 rows, stops early on validation labels and uses features weighted by the footprint, so it knows where the foot lands. The VLM sees only the crop.
- Not tested: the VLM was tuned only on the sole's labels from seen gaits, so there is no VLM comparison of sole against joints and IMU labels. The tuning itself was not re-run.
Full list: independent re-check of experiment 1.
What it means and limits
- The sole can teach a VLM in simulation: a large, real gain on new textures, from labels that never saw an image.
- Vision alone, before the step, cannot replace the sole when appearance lies. Tuning makes the model follow the look more.
- A simulation-only LoRA does not transfer to real ground. It costs most of the zero-shot skill on real photos.
- For this task the VLM is not worth its size. A 7B model, it is weaker than a ViT with a small head on the same crops (borderline).
- Limits: simulation, three test textures per class, one setting of labels (seen gaits, sole), and real photos for only three of the six classes.
Scores: macro-F1 per step against simulator truth on held-out textures; deceptive scenes as accuracy; real photos as macro-F1 over three classes. Related: the VLM on real photos, the vision head, the two teachers, the dataset, independent re-check. Experiment 2, a zero-shot VLM on a fixed patch: fixed patch. Model: Qwen2.5-VL-7B-Instruct, Apache-2.0. Textures: ambientCG, CC0.