Terrain from Touch
Can a sensing sole tell a humanoid what it is standing on? A classifier trained and tested in simulation on the K1, with flat-foot walking and human-like heel-to-toe walking.
In short
- It works in simulation, and more sensor data helps a bit. Ground type: 0.89 with the sole, 0.82 from joints and IMU alone.
- The sole alone is almost as good as sole plus joints. Pressure map and shear with no joints at all: 0.88.
- The larger gain is support, not ground type. Knowing if a foot is in the air, on an edge or fully down: 0.93 against 0.76. A lifted robot is noticed in 0.1 s instead of 0.3 s.
- A sole-only model still works on a gait it never saw. A model that also reads joints does not: 0.53 against 0.07 on the human-like gait.
- Shear adds little on a known gait and more on a new one. +1 to +4 points on a known gait, +6 to +12 on an unseen gait. The second number is still being re-checked.
- The sole helps more on heel-to-toe walking than on flat-foot walking. Earlier papers mostly used flat-foot walking, based on the reviewed work.
- This will not transfer to the real robot as it is. The pressure simulation is roughly right, the shear simulation is weak, and sand, mud or grass cannot be simulated. A real recording is needed.
Experimental setup
- Robot and simulator. Booster K1 in MuJoCo.
- Walking. Three gaits. Two are ordinary locomotion policies that step with a flat foot (called G113 and G80 here). The third is human-like heel-to-toe walking, played through my SONIC-K1 policy from human motion clips.
- The sole. Simulated to be approximate the PaXini sole without its documentation. The foot is thin and rigid. Front and back carry the load, the middle does not touch. Pressure is read at 19 × 8 points per foot. Shear is read as four vectors per foot: front and back, each split into left and right halves.
- Surfaces. Six kinds: grippy, slippery, soft, rough, sloped, and planks with gaps where the foot can land on an edge. Plus a case where the robot is lifted off the ground.
- Data. My own dataset: 6139 episodes, 33 hours. Every moment is labelled with the ground type and whether each foot is in the air, partly supported or fully supported.
- The model. A small network that looks at the last second of signals and names the ground and the state of each foot. It only reads. It does not change how the robot walks.
What was compared
The same model was trained on different inputs, with and without the robot's own joints and IMU:
- joints and IMU only (no sole);
- four pressure blocks per foot (a very coarse sole);
- the full pressure map;
- each of these with and without the four shear vectors;
- shear only.
Results by input
Walking, on surfaces and gaits the model was trained on. Scores are F1 from 0 to 1, higher is better.
| Input | Ground type | Foot support |
|---|---|---|
| Joints + IMU only | 0.82 | 0.76 |
| Joints + IMU + 4 pressure blocks | 0.83 | 0.81 |
| Joints + IMU + 4 pressure blocks + shear | 0.84 | 0.81 |
| Joints + IMU + pressure map | 0.88 | 0.93 |
| Joints + IMU + pressure map + shear | 0.89 | 0.94 |
| Sole only: 4 pressure blocks | 0.78 | 0.79 |
| Sole only: 4 pressure blocks + shear | 0.82 | 0.80 |
| Sole only: pressure map | 0.86 | 0.92 |
| Sole only: pressure map + shear | 0.88 | 0.93 |
| Sole only: shear, no pressure | 0.79 | 0.75 |
Reading the table. Adding the sole helps, and the pressure map helps more than coarse blocks. On ground type the gain over joints and IMU is modest: 7 points at best. The sole without any joints (0.88) is almost as good as everything together (0.89). Shear alone is about as good as joints alone.
The first version of the sole had eight zones per foot and gave a similar picture: 0.80 without the sole, 0.86 with pressure, 0.86 with pressure and shear.
Support, lifting and edges
This is where the sole matters most, and it matches the two failure videos: stepping off the stage, and being lifted while the policy keeps "walking".
| What is measured | Joints + IMU | + 4 blocks | + pressure map |
|---|---|---|---|
| Foot support state (F1) | 0.76 | 0.81 | 0.93 |
| "Foot is only partly supported" (F1) | 0.36 | 0.47 | 0.80 |
| Robot lifted: time to notice | 0.30 s | 0.10 s | |
| Foot on an edge for 0.3 s or longer: missed | 37 % | 27 % | 2 to 3 % |
| Foot on an edge for less than 0.3 s: missed | 74 to 85 % for every input | ||
A gait the model has not seen
Each model was trained on two gaits and tested on the third. Models that read joints learn the gait as much as the ground, so they break. A model that reads only the sole keeps part of its skill.
| Ground type F1 on an unseen gait | Sole only | Sole + joints |
|---|---|---|
| Pressure map + shear, average of the three gaits | 0.63 | 0.53 |
| Pressure map, average | 0.57 | 0.51 |
| 4 blocks + shear, average | 0.52 | 0.37 |
| 4 blocks, average | 0.40 | 0.35 |
| Pressure map + shear, tested on the heel-to-toe gait | 0.53 | 0.07 |
What follows. The same models score 0.78 to 0.89 on a gait they were trained on, so a new gait costs a lot in every case. The classifier has to see the controller that will run on the real robot, or it should read the sole only.
Flat-foot against heel-to-toe
The robot walks about a minute across a course with all six surfaces in random order. The score is the share of time the ground type is named correctly.
| Gait | Joints + IMU | With the sole | Gain |
|---|---|---|---|
| Flat-foot policy G113 | 0.85 | 0.88 | +3 |
| Flat-foot policy G80 | 0.80 | 0.84 | +4 |
| Heel-to-toe (SONIC) | 0.70 | 0.81 | +11 |
| All three | 0.79 | 0.84 | +6 |
What follows. On the human-like gait joints alone say less about the ground, so the sole adds more. This is one result from one set of runs, not yet a separate confirmed test.
In the videos, the strips under the picture fill in as the robot walks: the true ground, then the model with the sole, then joints and IMU only, then a hand-written detector with no learning (0.31, shown only for reference). The panel on the right is the simulated sole.
Shear
| Sole-only models, ground type F1 | Without shear | With shear | Gain |
|---|---|---|---|
| Pressure map, known gait | 0.86 | 0.88 | +2.5 |
| 4 blocks, known gait | 0.78 | 0.82 | +4 |
| Pressure map, unseen gait (average) | 0.57 | 0.63 | +6 |
| 4 blocks, unseen gait (average) | 0.40 | 0.52 | +12 |
| Pressure map, unseen heel-to-toe gait | 0.42 | 0.53 | +11 |
| 4 blocks, unseen heel-to-toe gait | 0.24 | 0.36 | +12 |
- With joints in the input, shear adds about 1 point. In steady walking the horizontal force under the foot is set by how the robot moves, not by the floor, and the joints already carry that.
- A slippery floor is visible without the sole. Grippy against slippery from joints and IMU alone: 0.95. With the sole: 0.97.
- Shear matters more when the gait is new. See the lower rows of the table. This was an exploratory finding and needs a dedicated re-test.
- Shear clearly shows the floor when the foot slides on purpose, the way a person tests ice with a toe.
- The shear simulation is the weakest part. See section 8.
The close-ups are from the flat-foot gait. Close-ups for the heel-to-toe gait are not rendered yet.
How many points are needed
- So far only two layouts were compared: the full map (152 pressure points per foot) and four blocks per foot. The map is clearly better on a known setup: 0.88 against 0.83 on ground type, 0.93 against 0.81 on support.
- But the map's advantage depends on the simulation being right. When the same models are tested on a sole that is simulated a little differently, the map models fall apart and the block models hold up (table below).
- Running now: a sweep over 4, 8, 18 and 152 pressure points, each with and without shear, on known and unseen gaits, plus a version where the sensor is modelled as separate points with gaps between them, like the real one.
| Trained on the standard sole, tested on | Pressure map | Map + shear | 4 blocks | 4 blocks + shear |
|---|---|---|---|---|
| the same sole | 0.86 | 0.88 | 0.78 | 0.81 |
| shifted sensor settings (noise, gain, range) | 0.53 | 0.60 | 0.69 | 0.74 |
| a different contact model for the sole | 0.21 | 0.23 | 0.59 | 0.59 |
| a foot-shaped point layout instead of a rectangle | 0.24 | 0.21 | 0.76 | 0.74 |
Ground type F1, models with joints and IMU. Values in the lower three rows are read from the chart and rounded. Chart: v2_fig5_sensitivity.png.
Limits of the simulation
- Materials. Sand, mud, gravel, grass and carpet cannot be simulated here. The simulator has properties (slippery, soft, uneven), not materials. A model for real ground has to be trained on real recordings.
- Pressure: roughly right. The total force on each foot and where its centre lies are correct. How the force spreads across the sole is not: the simulated foot is a rigid box and while walking it loads its corners, which the real sole does not show.
- Shear: weak. The simulator gives one horizontal force per foot. Splitting it into four blocks or into points is my own construction, in proportion to pressure. It also has no partial slip, where part of the sole grips and part slides, and that is exactly what a real shear sensor would pick up. So the shear results may not be accurate in either direction.
- The sensor itself. Number of points, range per point, noise and whether the real sole reports shear per point are guesses until there is documentation.
- Transfer. Because of all this the trained model will not work on the real sole as it is. What should carry over is the method, the comparison between inputs, and the coarse signals (totals, centre of pressure, blocks).
Real-world validation requirements
- PaXini documentation, SDK or simulation examples: pressure and shear channels, point layout, ranges, and adapter requirements.
- A 15 to 30 minute K1 recording: standing, manual loading, and walking on two or three surfaces.
- The walking controller used during recording, to account for gait dependence.
Next
- Now: finish the sweep over the number of pressure and shear points, and the re-test of shear on an unseen gait.
- With a real recording: compare the simulated sole with the real signal, and train a small model that makes the simulated sole read like the real one. Then repeat the shear and layout questions on something trustworthy.
- Done since, in simulation: the sole as a teacher for vision. The robot looks, steps, and the sole says what the ground was. A vision model learns to predict that before the step. Results with a ViT and a VLM: Vision from Touch.
- Not done on purpose: changing the walk in response to the detected ground. For now this is detection only.
All other clips
Numbers are from the second simulation run unless stated: test split, walking, mean of three training runs. Full reports: REPORT_V2.md, REPORT.md. All tables: tables_v2.md. Charts: inputs, partial support, unseen gait, course.