FLA step 2 · Booster K1 · 16 Nov 2026 · step 1: Terrain from Touch · all reports

Vision from Touch

Step 1 showed that a sensing sole can tell the K1 what it is standing on, after the step. Here the sole becomes the teacher: the robot looks, steps, and the sole says what the floor was. A vision model learns to predict that before the step. Tested with a ViT and with a vision-language model.

Simulation only. Rendered camera, simulated sole. One part uses real floor photos, and says so.

In short

  1. A camera can learn the floor from the sole's labels. A small head on a frozen ViT, taught only by the sole, scores 0.985 on hard against soft floor with textures it never saw. Taught by simulator truth it scores the same.
  2. The sole is the better teacher when the gait is new. With labels from a gait the teachers never saw, vision taught by the sole reaches 0.49, vision taught by joints and IMU 0.10, which is chance. On known gaits the two teachers tie.
  3. Vision reads the look, not the physics. On floors that look like one thing and behave like another, vision before the step scores 0.03. The sole after the step scores 0.997.
  4. A small ViT is enough. ViT-S matches ViT-B (0.985) at more than twice the speed.
  5. The 7B vision-language model is worse and far slower. Zero-shot it scores 0.668 at 47 frames per second. Tuned on the sole's labels it improves in simulation and gets worse on real photos.
  6. New materials are the weak point. With a whole material family held out of training, the ViT head drops from 0.985 to 0.726.
  7. The two-class task is easy, and it is still simulation. Hard and soft floors differ so much that it cannot rank the teachers. A real recording is still the missing piece.
1

What was done

Torso camera frames with the fixed patch of floor marked in green and the stored crop next to each
Torso camera, pitched down. The green box is the fixed patch the models classify; the crop is on the right of each frame. Bottom rows are deceptive floors.
Seventy-two floor textures grouped into hard and soft, training, validation and test
Hard and soft floor textures, 36 each. About a third of each class is held out for testing.
2

The idea in one clip

The K1 walks over hard and soft patches in both orders, then over two deceptive ones. The box in the camera view is the patch being classified. The panel shows the true floor, the two vision models before each step, and the sole and the joints after it. Green is right, red is wrong.

One run, gait G113, test textures. Shown at 0.6× speed.
Steps rightViT headVLM, zero-shotSoleJoints + IMU
All 126 steps91100125126
Honest patches, 103 steps9189102103
Deceptive patches, 23 steps0112323

Reading the clip. All twelve honest mistakes of the ViT head come one to three steps before a floor change: the patch lies ahead of the foot, so vision already reports the next floor. On the deceptive patches it follows the look on every step. The VLM's eleven is not insight. It calls carpet hard, so it is right on the hard floor that looks like carpet and wrong on all eleven steps of the soft floor that looks hard.

This is one run, so the counts illustrate the result. The measured numbers are in the sections below.

3

Who teaches vision better

The same ViT head, trained four times. Only the source of the labels changes.

Labels for the vision headVision scoreTeacher's own score
Simulator truth (ceiling)0.985–
Sole teacher0.9850.997
Joints + IMU teacher0.9840.990
Both together0.9850.995
Shuffled labels (control)0.565chance 0.500

Reading the table. A camera taught by the sole is as good as a camera taught by the simulator. But this experiment cannot say which teacher is better: with only hard and soft floors every teacher is right on about 99 % of steps, so all the label sets are nearly the same. The six-class experiment below is where the teachers come apart.

Hard against soft, gait G113, 5,029 test steps on 25 held-out textures. One classifier that both sees the patch and feels the step scores 0.997, but it answers after the step and adds nothing over the sole alone.

4

ViT, SigLIP and the VLM

Four vision models on the same patch and the same test steps.

ModelTrainingScoreFrames per second
DINOv3 ViT-B + headhead on sole labels0.9851,770
DINOv3 ViT-S + headhead on sole labels0.9854,006
SigLIP with text promptsnone0.9762,588
Qwen2.5-VL-7Bnone0.66847
DINOv3 ViT-B + head, a whole material family held outhead on sole labels0.726–

Reading the table. The small ViT loses nothing to the large one. SigLIP comes close with no training, but only when the prompts name the materials in the dataset; generic prompts such as "a hard floor" were below chance. The VLM mostly calls soft floors hard and is about forty times slower than the ViT.

The last row is the honest warning. The test textures are new, but they come from material families the head has already seen, such as other grass or other concrete. Hold a whole family out and the score falls to 0.726: asphalt 0.21, carpet 0.47, grass 0.71.

Speeds are for one H100 that another job was also using, so they are lower bounds.

5

Floors that lie

A hard floor textured as carpet, a soft floor textured as tiles. This is the case a camera cannot solve, and the reason to keep the sole.

SourceAgainst the real floorAgainst the look
ViT head, before the step0.030.97
Sole, after the step0.997–
Joints + IMU, after the step0.992–

Reading the table. Vision is confidently wrong: it reports what the floor looks like. A head taught by the sole is fooled just as much as one taught by truth, because both learned the same thing from honest floors, that a look goes with a class. The sole reads no image, so it is not fooled.

3,559 deceptive steps, kept out of training entirely.

6

When the floor changes

Only 24 boundaries in the test data, all from hard to soft.

7

Six classes and three gaits

The broad experiment: grippy, slippery, soft, rough, sloped and plank floors, three gaits, and an image crop around the place where the foot will land. Here the teachers differ.

Labels for the vision headGaits the teachers sawA gait they never saw
Sole teacher0.820.49
Joints + IMU teacher0.820.10
Simulator truth (ceiling)0.860.87
Sole teacher, shifted sensor model0.830.50
Bar charts of the vision head's score by label source, for gaits the teachers saw and for an unseen gait, and a line chart by number of labelled steps per texture
Left: labels from gaits the teachers saw. Middle: labels from the human-like gait they never saw. Right: the same with fewer labelled steps per texture.

Reading the table. This repeats the step-1 finding one level up. On a gait it never saw, the joints teacher labels almost every step as planks, and vision taught by it learns nothing. The sole teacher still gives usable labels. The gap rests on one held-out gait and on few test textures, so its size is uncertain; its direction is not.

Bar chart: on deceptive scenes vision before the step scores near zero against the real floor while the sole after the step scores high
Deceptive floors with six classes: vision before the step 0.07 to 0.12, sole after the step 0.82.
Twelve camera frames with the landing footprint in red and the crop in green
The first experiment cropped the image around the future footprint (red).
8

The VLM in detail

Bar charts for the VLM zero-shot, after LoRA and with shuffled labels: held-out textures, deceptive scenes against the real floor and against the look, and real photos
Zero-shot, LoRA on sole labels, and a shuffled-label control. The small drop in the second panel is not a real difference.

For this task the VLM is not worth its size. It is slower by a factor of forty, weaker than a ViT with a small head, and tuning it on simulation harms what it knew about real floors. Its use is elsewhere: questions a classifier cannot be asked.

9

Independent check

After each experiment a separate checker that did none of the work recomputed the numbers with its own code, searched for leaks between training and test, and looked at samples.

ExperimentClaimsHoldHold with caveatsDo not holdNot verifiable
Six classes, three gaits59332213
Fixed patch, hard against soft39201702
10

Limits

11

More clips

Six-class course: grippy, slippery, soft, rough, slope and two deceptive patches. Rows: truth, ViT head, VLM with LoRA, sole, joints.
Four test episodes, two honest and two deceptive: vision before each step, the sole after it, and the truth.

Macro-F1 per step on held-out textures; means over three training runs. Step 1 of the project: Terrain from Touch. Textures: ambientCG, CC0. Models: DINOv3 (ViT-B/16, ViT-S/16), SigLIP, Qwen2.5-VL-7B.