A sole that feels the floor, and a camera that learns from it
A humanoid robot with sensors under its feet can tell what it is standing on, but only once it has stepped there. This project asks whether that sense of touch can teach the robot's camera to recognise the floor one step earlier.
Read the whole project, step by step →
What can the sole feel?
More than the robot's joints can. It names the kind of floor a little better, and it is much better at knowing whether the foot is properly supported.
- 0.89 vs 0.82Naming the kind of floor, with the sole against joint sensors alone.
- 0.93 vs 0.76Knowing whether the foot is in the air, on an edge, or fully down.
- 0.1 s vs 0.3 sHow quickly a robot lifted off the ground is noticed.
- 4 + 4Four pressure zones and four sideways-force arrows per foot hold up when the sensor changes. A dense pressure map does not.
Can the sole teach the camera?
Yes. A camera that learned only from the sole's answers recognises hard and soft floor as well as one trained on perfect labels. It struggles when a whole kind of material is new.
- 0.985Hard against soft, on floor pictures it never saw. Trained on perfect labels: also 0.985.
- 0.49 vs 0.10Taught by the sole against taught by joint sensors, when the robot walks in a way the teachers never saw.
- 4,000 fpsA small vision model is enough, and fast. The large one scores the same at less than half the speed.
- 0.726When a whole material, such as all carpets, is missing from training.

Would a large language-and-vision model do better?
No. Asked to name the floor from the same picture, a 7-billion-parameter model is less accurate and about forty times slower. Tuning it on simulated floors makes it worse on real photos.
- 0.668Hard against soft with no training. It mostly calls soft floors hard.
- 47 fpsPictures per second. The large vision model manages about 1,800 and the small one about 4,000.
- 0.34 → 0.72Six floor types in simulation, before and after tuning on the sole's labels.
- 0.68 → 0.38The same model on real floor photos, before and after that tuning.

What if the floor looks like one thing and is another?
Then the camera is wrong and the sole is right. A hard floor that looks like carpet fools every vision model tested. This is the reason to keep the sole even after the camera has learned.
- 0.03The camera before the step, on floors made to deceive.
- 0.997The sole after the step, on the same floors.
- 0.97How often the camera reports what the floor looks like. It is confident, and wrong.
- 1.3 stepsOn honest floors the camera sees a change of floor this early, because it looks ahead of the feet.
Did it change how the robot walks?
Not yet. Two attempts to train gentler, touch-aware walking did not reach their goal. They are here because a negative result is still a result.
- 782 vs 680 NLanding force of the touch-aware policy on a Unitree G1 against the policy it started from, at equal speed. It lands harder.
- 10 of 24Falls in a second simulator for the best K1 policy, which walks without falling in the simulator it was trained in.
- 3Attempts at the K1. Two stood still; the third walked and then got slower with more training.
How much of this can be trusted?
The numbers are solid for the simulation they came from. Each one was recomputed by someone who did none of the work. What they say about a real robot is still open.
- 98Claims re-checked across the two experiments.
- 53 + 39Hold outright, and hold with a caveat attached.
- 1Does not hold: a small difference that turned out to be noise.
- 5Could not be verified.

- The sole and the camera are simulated. No real recording exists yet.
- Hard against soft is an easy pair: the two differ enormously in stiffness.
- The floors are flat pictures, with no depth, gloss or shadows from the robot.
- The simulated foot is a rigid box, and it misses forces that push along the sole.
All pages
- Summaries
- The project, step by step
- Vision from Touch (step 2)
- Terrain from Touch (step 1)
- The sole
- The sole, close up
- How many sensing points?
- Step 1, second run
- Step 1, first run
- The camera
- The five comparisons
- Sole or joints as the teacher
- Teachers, two floor types
- Teachers, six floor types
- Dataset and camera
- The larger dataset
- Foot support from vision
- The language model
- On real floor photos
- Tuning on the sole's labels
- Walking
- Softer landings on the G1
- A hiking recipe on the K1
- Re-checks
- Two floor types
- Six floor types
Words used on these pages
- The sole
- A sensor under each foot. In simulation it reads pressure at 19 by 8 points and sideways force (shear) as four arrows per foot.
- Teacher
- A program that names the floor after each step, from the sole or from the joints. Its answers are used to train the camera model.
- Joint sensors
- What every robot already has: joint angles and speeds, and the body's tilt and acceleration. Written "joints + IMU" on the detail pages.
- Vision model (ViT)
- A vision transformer, here DINOv3. It turns a picture into numbers; a small trained part turns those into "hard" or "soft".
- Language model (VLM)
- A vision-language model, here Qwen2.5-VL. It is shown the picture and asked for the floor type in words.
- No training (zero-shot)
- Using a model as it comes.
- Tuning (LoRA)
- A cheap way to adapt a large model by training a small add-on.
- Deceptive floor
- A floor that looks like one type and behaves like another, such as a hard floor with a carpet picture on it.
- Floor pictures never seen
- Textures kept out of training. All scores are on these. Written "held-out textures" on the detail pages.
- Gait
- A way of walking. G113 and G80 step with a flat foot; SONIC walks heel to toe, like a person.
- The score
- A number from 0 to 1, where 1 means every step right. With two floor types, guessing gives 0.5. Its technical name is macro-F1.
- Holds / holds with caveats
- The verdict of the independent re-check on each claim.