The two teachers
Before a camera can learn from the sole, the sole's labels have to be good. Two step-level classifiers act as teachers: one reads only the sole, one reads only joints and IMU. Here is how well each names the floor, on gaits it was trained on, on a gait it never saw, and under other sensor models.
In short
- On gaits they saw, the two teachers tie. Sole 0.804, joints and IMU 0.805, macro-F1 per step over six floor classes.
- On a gait they never saw, only the sole still works. With SONIC held out: sole 0.399, joints and IMU 0.095. The other two hold-outs point the same way.
- The joints and IMU teacher does worse than guessing on the new gait. It calls almost every SONIC step "edge". The independent re-check notes that this is below a random guesser (about 0.17), so "chance" undersells the failure.
- Coarse sole features survive new sensor parameters, not a new sole. Shifted parameters cost 2 points on seen gaits; a differently built sole or a foot-outline mask costs 19-22 points.
- For support state the sole is clearly better. Sole 0.719, joints and IMU 0.445 on seen gaits.
What was done
- The question. For one footstep (touchdown to lift-off, at most 0.6 s), how well does each teacher name the floor class?
- Classes. Grippy hard floor (hard_grip), slippery hard floor (hard_slip), soft, rough, slope, and edge (planks and steps). Truth is the simulator's class 0.1 s after touchdown. Steps on hard_mid (an extra hard-floor type that exists only in the test data) and steps that land on a boundary between floors have no class: they are labelled but not scored.
- Gaits. G113 and G80 are two ordinary locomotion policies that step with a flat foot. SONIC is human-like heel-to-toe walking. The main setting trains on G113 and G80 and keeps SONIC unseen.
- The sole teacher reads only the coarse sole features of the stance foot: four pressure blocks, four shear vectors, centre of pressure, total force and contact area. Step 1 showed that the fine pressure map does not survive a change of sensor model; these coarse features do. The right foot is mirrored.
- The joints and IMU teacher reads the joints and the IMU over the same step window.
- Same network and recipe for both. The step-1 temporal network (a causal dilated CNN). Settings were chosen on validation steps of the seen gaits only: 18 settings × 2 seeds, then reused for every hold-out. Each model trains in 10-20 s on a GPU.
- Data. 215,393 walking steps from the step-1 data (train 141,307, validation 21,220, test 52,866). Scores are macro-F1 per step on the test split, mean ± sd over 3 seeds.
Seen and unseen gait
| Teacher | Sensor and test set | Seen gaits (G113, G80) | Unseen gait (SONIC) |
|---|---|---|---|
| Sole (coarse) | step-1 test, stored sensor | 0.804 ± 0.003 | 0.399 ± 0.004 |
| Joints + IMU | step-1 test, stored sensor | 0.805 ± 0.002 | 0.095 ± 0.015 |
| Sole, shuffled labels (control, one seed) | step-1 test | 0.057 | |
Reading the table. On gaits both teachers saw in training they are equally good, about 0.80. On SONIC the sole teacher drops to 0.40 but still carries information; the joints and IMU teacher collapses. It calls almost every SONIC step "edge". The gap is at least ten times the seed spread. The sole's SONIC labels are biased too: it almost never says hard_grip (F1 0.05) and says slope too often.
Per class
| Teacher | Gaits | hard_grip | hard_slip | soft | rough | slope | edge |
|---|---|---|---|---|---|---|---|
| Sole | seen | 0.76 | 0.96 | 1.00 | 0.57 | 0.94 | 0.61 |
| Joints + IMU | seen | 0.76 | 0.96 | 0.99 | 0.61 | 0.97 | 0.54 |
| Sole | unseen | 0.05 | 0.68 | 0.73 | 0.28 | 0.38 | 0.27 |
| Joints + IMU | unseen | 0.01 | 0.16 | 0.08 | 0.00 | 0.17 | 0.16 |
Reading the table. F1 per class, step-1 test set, mean of 3 seeds. On seen gaits both teachers are near perfect on soft, slippery and sloped floors and weak on rough, edge and grippy floors, which they confuse with each other. Part of that error is in the label itself: an "edge" step often never touches an edge. On SONIC the sole keeps slippery and soft floors partly, and the joints and IMU teacher keeps nothing.
Other held-out gaits
The same experiment with each gait held out in turn: train on the other two, test on all three.
| Held-out gait | Teacher | Seen gaits | Held-out gait |
|---|---|---|---|
| SONIC | Sole | 0.804 ± 0.003 | 0.399 ± 0.004 |
| SONIC | Joints + IMU | 0.805 ± 0.002 | 0.095 ± 0.015 |
| G113 | Sole | 0.747 ± 0.001 | 0.627 ± 0.024 |
| G113 | Joints + IMU | 0.730 ± 0.007 | 0.279 ± 0.029 |
| G80 | Sole | 0.779 ± 0.006 | 0.485 ± 0.010 |
| G80 | Joints + IMU | 0.762 ± 0.001 | 0.264 ± 0.021 |
Reading the table. Every hold-out points the same way: on the gait it never saw, the sole teacher is clearly ahead (0.40 to 0.63 against 0.095 to 0.279). The sole labels on a new gait are noisy but informative.
Other sensor models
A real sole will not match the simulated one. So the same sole teacher (trained on the stored sensor, SONIC held out) is scored on 288 separate test simulations (96 per gait), read out in different ways. The pad sole is a differently built sole that changes the dynamics, so it needs its own simulation of the same tasks (11,552 steps against 11,556).
| Teacher | Sensor | Seen gaits | SONIC |
|---|---|---|---|
| Sole | nominal sensor | 0.806 ± 0.004 | 0.437 ± 0.017 |
| Sole | shifted parameters | 0.784 ± 0.011 | 0.429 ± 0.009 |
| Sole | pad sole (own simulation) | 0.586 ± 0.012 | 0.325 ± 0.003 |
| Sole | foot-outline mask | 0.621 ± 0.004 | 0.419 ± 0.019 |
| Joints + IMU | nominal sensor run | 0.805 ± 0.004 | 0.112 ± 0.020 |
| Joints + IMU | pad sole run | 0.789 ± 0.005 | 0.105 ± 0.020 |
Reading the table. Shifted sensor parameters (blocks moved, twice the spread, noise and compliance, crosstalk) cost the sole 2 points on seen gaits (0.806 to 0.784, about twice the spread, borderline) and nothing on SONIC (0.437 to 0.429). The pad sole and the foot-outline mask cost 19 to 22 points. Coarse-feature labels survive a change of sensor parameters, not a change in how the sole is built. The joints and IMU teacher does not read the sole; its rows are there for reference.
On the replayed episodes of the vision dataset (617 of them labelled for this check), the same sole teacher on the stored / nominal / shifted readout scores 0.778 / 0.781 / 0.761 on seen gaits and 0.394 / 0.402 / 0.395 on SONIC. Its shifted labels agree with its stored-sensor labels on 88 % of steps.
Support-state teachers
A second question for the same teachers: is the foot fully supported, partly supported (over a gap or a step edge), or not in contact? Truth is the simulator's support state 0.1 s after touchdown. None means no contact (force along the sole normal of 2 N or less). Partial means contact with fewer than 90 % of the sole's 24 probe points over supporting ground; tilt and bumps of 2 cm or less do not count. Full is the rest. Same steps, inputs and recipe, with a three-way output weighted 1 / 3 / 1.
| Split | G113 | G80 | SONIC |
|---|---|---|---|
| Train | 245 / 3,590 / 49,927 | 692 / 3,361 / 48,649 | 295 / 1,791 / 32,757 |
| Validation | 43 / 513 / 7,606 | 156 / 455 / 7,413 | 41 / 271 / 4,722 |
| Test | 85 / 998 / 18,478 | 203 / 959 / 17,961 | 134 / 434 / 13,614 |
Reading the table. Steps per support state, none / partial / full. "None" is rare (0.9 % of steps, mostly on slippery floor). Partial support occurs almost only on edge terrain: 12,365 of 12,372 partial steps. The source report says 12,364; the independent re-check counted 12,365, and its count is used here.
| Held-out gait | Teacher | Seen gaits | Held-out gait | Partial F1, seen | Partial F1, held out |
|---|---|---|---|---|---|
| SONIC | Sole | 0.719 ± 0.015 | 0.486 ± 0.012 | 0.695 ± 0.021 | 0.254 ± 0.046 |
| SONIC | Joints + IMU | 0.445 ± 0.003 | 0.305 ± 0.035 | 0.363 ± 0.016 | 0.042 ± 0.023 |
| G113 | Sole | 0.693 ± 0.008 | 0.581 ± 0.040 | 0.608 ± 0.009 | 0.442 ± 0.096 |
| G113 | Joints + IMU | 0.417 ± 0.017 | 0.346 ± 0.008 | 0.265 ± 0.031 | 0.120 ± 0.007 |
| G80 | Sole | 0.597 ± 0.031 | 0.513 ± 0.007 | 0.626 ± 0.005 | 0.509 ± 0.032 |
| G80 | Joints + IMU | 0.418 ± 0.013 | 0.357 ± 0.012 | 0.281 ± 0.041 | 0.112 ± 0.028 |
Reading the table. Macro-F1 over the support states present in the test split, mean ± sd over 3 seeds. Unlike the floor class, here the sole is clearly ahead even on gaits both teachers saw. With three states, the chance reference is 0.333, and the shuffled-label control scores 0.324. The joints and IMU teacher is close to that on held-out gaits.
How vision learns support state from these labels: support state from vision.
Independent re-check
- Every headline number re-computed and confirmed: seen gaits, SONIC, the other hold-outs, all sensor variants (from the saved predictions), and the support teachers.
- The sole teacher really reads only the coarse features. All 44 sole models take exactly the 16 coarse features plus a validity flag as input. Holds with a caveat: for both teachers the step window ends at the simulator's noise-free contact flag, not at a sensed one.
- No leaks. The held-out gait is absent from the training and validation lists of every teacher, and no teacher list contains a test episode. When teachers label training steps for the vision head, they are cross-fitted: no teacher labels an episode it trained on (0 violations among 54,010 rows).
- Not verified by the re-check: how the 288 separate test simulations and the pad-sole simulation were built. It only re-scored their saved predictions.
Full list: independent re-check of experiment 1.
Limits
- Simulation. The sole is a simulated sensor on a rigid box foot.
- Three gaits. The unseen-gait result rests on three hold-outs, and the vision experiments use only one of them (SONIC).
- Labels with limits. An "edge" step often never touches an edge, and the step-1 contact flag misses a foot loaded on its front edge (see the dataset page).
- A sole built differently costs 19-22 points. A real sole will differ from the simulated one in more than its parameters.
Scores: macro-F1 per step against simulator truth on the step-1 test split, mean ± sd over 3 seeds. Related: the dataset, the vision head these teachers train, support state from vision, independent re-check. Teachers of experiment 2 (two classes, one gait): teachers of experiment 2. Step 1: Terrain from Touch.