FLA · Experiment 1 VLM with LoRA · 16 Nov 2026 · all reports · summary

A VLM taught by the sole

Qwen2.5-VL-7B looks at the torso-camera image of the floor just before a step and names one of six floor classes. Tested as it comes, and after a LoRA fine-tune on labels from the sole teacher, which only feels the step and never sees the image. Then on deceptive floors, and back on real photos.

Simulation, plus the real photos of the previous page (numbers only). One claim of the original report did not hold up.

In short

  1. LoRA on the sole's labels helps a lot in simulation. On held-out textures macro-F1 rises from 0.343 zero-shot to 0.721.
  2. It learns what floors look like, not what they are. On deceptive floors it follows the look more after tuning (0.347 to 0.524) and matches the real floor on only 0.124 of steps.
  3. It loses its skill on real ground. On the real photos macro-F1 falls from 0.679 to 0.375; it answers "grippy" to most of them.
  4. It is still behind a ViT with a small head on the same crops, 0.721 against 0.821, a borderline difference in a comparison that is not fully fair.
  5. The claim that did not hold: that tuning makes the model really worse against the real floor on deceptive scenes. The difference is inside the noise.
1

What was done

2

Results

TestZero-shotLoRA, sole labelsLoRA, shuffled
Held-out textures, macro-F1 (6 classes)0.343 ± 0.0040.721 ± 0.0400.061
Held-out textures, accuracy0.4160.726 ± 0.0410.224
Deceptive, accuracy against the real floor0.1490.124 ± 0.0060.167
Deceptive, accuracy against the look0.3470.524 ± 0.0410.155
Real photos, macro-F1 (3 classes)0.679 ± 0.0120.375 ± 0.0240.167
Real photos, accuracy0.6260.433 ± 0.0170.333
Four bar charts comparing the VLM zero-shot, after LoRA on sole labels and after LoRA on shuffled labels: held-out texture macro-F1, deceptive scenes against the physical floor and against the look, and real-photo macro-F1
Zero-shot (grey), LoRA on the sole's labels (blue), LoRA on shuffled labels (orange). LoRA: mean ± sd over seeds; zero-shot macro-F1: bootstrap sd.

Reading the table. Tuning on the sole's labels about doubles the score on new textures, a real gain of +0.378. The model answers in valid JSON every time, in every run. The shuffled control answers hard_grip for every crop, so the gain comes from the labels, not from learning the answer format or the crops.

From the independent re-check. The zero-shot bootstrap sd of 0.004 understates its uncertainty: resampling the 18 test textures gives sd 0.020 for zero-shot and 0.074 for LoRA. The gain survives: +0.378 [0.197, 0.526]. The shuffled control's 0.061 is exactly what a constant hard_grip answer scores, while random guessing at this class mix scores 0.167: it is a collapse floor, not "chance". Zero-shot (0.343) is above a texture-aware null (95th percentile 0.290).

Speeds on a shared H100: training 12.7 crops per second, 34.7 answers per second on simulated crops, 20.2 on real photos; peak memory 23.8 GB. The whole run took 1 h 47 min.

3

Per class

Right on held-out textureshard_griphard_slipsoftroughslopeedge
Zero-shot0.1790.8610.5490.8120.0010.000
LoRA, sole labels0.928 ± 0.0240.923 ± 0.0100.402 ± 0.0110.560 ± 0.1850.592 ± 0.0400.840 ± 0.064

Reading the table. Zero-shot almost never answers slope or edge and calls most grippy floors slippery (856 of 1,173 steps): it cannot read slopes or planks from a crop. After tuning, slope and edge are mostly right. Tuning leaves two weak classes: soft (often called grippy or slippery) and rough, the least stable across seeds (often called grippy).

4

Floors that lie

On a deceptive floor the look belongs to one class and the physics to another. The camera sees only the look.

This claim did not hold. The original report called the drop from 0.149 to 0.124 "real, but small". Its rule used only the LoRA seed sd, and zero-shot had no spread. A bootstrap over the 344 episodes gives −0.025 [−0.062, 0.010]: not a real difference. "Both below 1/6" holds. Also, the look accuracy uses 3,315 rows, not 3,990 as the report says, because hard_mid looks are left out too; the reported numbers match 3,315 rows. Bootstrap for the look increase: +0.176 [0.110, 0.246].

The ViT head does the same on the same 4,000 deceptive rows: 0.127 against the real floor, 0.667 against the look. It follows the look even more than the tuned VLM.

5

Back on real photos

Right on real photoshard_griproughsoftMacro-F1
Zero-shot0.390.680.800.679 ± 0.012
LoRA, sole labels0.940.040.310.375 ± 0.024

Reading the table. After tuning on simulated crops the model answers hard_grip to most real photos (1,019-1,053 of 1,350 across seeds), and rough photos become hard_grip in 91-100 % of cases. It learned the simulator's textures, not floors. The drop is real, and it is not a shift toward the three classes that have no real photos: those get 8-13 % of the tuned model's answers, against 17 % zero-shot.

Per-class values and answer counts from the independent re-check. Real photos are GTOS-Mobile (numbers only, not shown).

6

A six-class course

One run of G113 at 0.35 m/s over a row of seven patches with held-out textures: grippy, slippery, soft, rough, a sloped ridge, then two deceptive patches (looks grippy but is slippery; looks slippery but is soft). Unlike the test set, the crop here is causal: the landing spot is forecast from past footsteps, not taken from the future. Rows: truth, the ViT head (taught by the sole, seed 0), the tuned VLM (seed 0), the sole teacher and the joints and IMU teacher.

Booster K1 in MuJoCo, one run with no fall, 0.6× speed. The red box in the torso view is the forecast footprint, not the real one. Green is right, red is wrong; a yellow corner marks a step whose crop came from one of the two weakest fallback rules. 97 s.
Patch (steps)ViT headVLM (LoRA)SoleJoints + IMU
Start (9)9976
Grippy (10)10101010
Slippery (12)10121212
Soft (11)11111111
Rough (11)74911
Slope (13)40129
Deceptive: looks grippy, is slippery (16)011515
Deceptive: looks slippery, is soft (10)001010
End (15)14141514
All scored steps (107)656110198

Reading the clip. On the five honest patches the ViT head is right on 42 of 57 steps and the VLM on 37. On the two deceptive patches both follow the look: against the look class each is right on 24 of 26 steps, against the real floor 0 and 1. The sole and the joints after the step get 25 of 26 each.

One run, so the counts illustrate. 22 of the 107 steps had no usable crop under the main rule and use fallback crops; their forecast error is large (median 35.1 cm, against 8.8 cm under the main rule). Vision got 8 of those 22 right, the VLM 10.
7

Independent re-check

Full list: independent re-check of experiment 1.

8

What it means and limits

Scores: macro-F1 per step against simulator truth on held-out textures; deceptive scenes as accuracy; real photos as macro-F1 over three classes. Related: the VLM on real photos, the vision head, the two teachers, the dataset, independent re-check. Experiment 2, a zero-shot VLM on a fixed patch: fixed patch. Model: Qwen2.5-VL-7B-Instruct, Apache-2.0. Textures: ambientCG, CC0.