FLA · Experiment 1 independent re-check · 16 Nov 2026 · all reports · summary

Independent re-check of experiment 1

After experiment 1 (six floor classes, three gaits, an image crop around the landing footprint), an independent re-check that did none of the work recomputed the reported numbers with its own code, searched for leaks between training and test, and looked at samples. This page lists what it found, claim by claim.

59 claims: 33 hold, 22 hold with caveats, 1 does not hold, 3 could not be verified.

In short

  1. The numbers are right. Every reported number the re-check could recompute reproduced, apart from two small count errors in the text, and every check program of the project passed when run again.
  2. No leaks. No test episode or test texture in any training set, and no teacher labelled an episode it trained on.
  3. One claim fails: that LoRA makes the VLM really worse against the real floor on deceptive scenes. The difference is inside the noise.
  4. Many caveats are about uncertainty. With three test textures per class, the seed spread understates how uncertain the scores are, and several "ties" are inconclusive rather than equal.
  5. Trivial baselines. The support-state results are about as good as always answering the most common state.
  6. Not ruled out: the crop's placement leaks some geometry, only one gait was held out for vision, the ViT may have seen the textures in pretraining, and nothing was tested on a real robot.
1

How it was done

2

Tally

PartClaimsHoldHold with caveatsDo not holdNot verifiable
Part 1: dataset, teachers, vision head, support state, VLM on real photos44261503
All, with part 2 (the VLM tuned on the sole's labels)59332213

Reading the table. "Hold with caveats" means the number reproduces but its reading needs a qualification: a wider interval, a trivial baseline, or a condition the report did not state. The tables below give each claim with its verdict and the caveat next to it.

3

Dataset

Details: the dataset.

ClaimVerdictNote
Test-list fingerprints the same on both machines and in the vision runholdsfor the honest and the deceptive test list
1,572 episodes, 54,010 steps (29,105 / 4,267 / 6,269 / 14,369 = 26.6 %), 726 hidden-gap stepsholdsexact; the step table is identical on both machines
Dataset check 17 pass / 0 failholds17 / 0 when run again
All replays bit-identicalholds with caveatsonly 20 episodes checked independently; there joints, IMU, labels and contact flags equal the step-1 files
Episode and texture splits disjointholds with caveats0 episodes in two splits; 42 / 12 / 26 textures disjoint; only 3 test textures per class (18 scored); the deceptive looks reuse the 26 test textures
4

Teachers

Details: the two teachers.

ClaimVerdictNote
Seen gaits: sole 0.804 ± 0.003, joints + IMU 0.805 ± 0.002holds
Unseen SONIC: sole 0.399 ± 0.004, joints + IMU 0.095 ± 0.015holds with caveatsjoints + IMU is a degenerate "edge" predictor, below a random guesser (about 0.17), so "chance" undersells how badly it fails
Other hold-outs: G113 0.627 / 0.279, G80 0.485 / 0.264holds
Other sensors, seen / SONIC: shifted 0.784 / 0.429, pad sole 0.586 / 0.325, outline 0.621 / 0.419holdsre-scored from saved predictions
Shuffled sole teacher 0.057holds0.052 seen, 0.070 unseen; collapses to nearly one class
The sole teacher reads only the 16 coarse featuresholds with caveatstrue for all 44 sole models; for both teachers the step window ends at the simulator's noise-free contact flag
5

Vision head

Details: the vision head. (a): labels from gaits the teachers saw; (b): labels from SONIC, which they never saw. Test: 5,282 steps with a class.

ClaimVerdictNote
(a) sole 0.816 ± 0.028 vs joints + IMU 0.824 ± 0.030, no real differenceholds with caveatstexture bootstrap −0.009 [−0.052, 0.035]; the tie is expected: both label sets agree equally with truth (0.821 / 0.825) and share the same wrong label on 8 % of training steps
(a) truth 0.857 ± 0.009, the 3-4 point gap not realholds with caveatstruth minus sole 0.041 [−0.016, 0.091]: inconclusive, not "equal"
(b) sole 0.488 ± 0.026 vs joints + IMU 0.097 ± 0.048, realholds with caveatsdifference 0.396 [0.31, 0.48]; restates the teachers' gap (labels 0.40 vs 0.09) and rests on one held-out gait; the joints + IMU head answers "edge" on 63-99 % of steps
(b) truth 0.867 ± 0.023; sole labels 0.40holdslabels 0.404 ± 0.016
Label budgets 5 / 20 / 100 / allholdsall values the same
(b) sole beats joints + IMU at every budgetholds
Shifted readout 0.827 ± 0.014 / 0.498 ± 0.044holds with caveatsthe same teacher weights on another readout of the same simulation; the sole is not built differently
No pad-sole data; teacher pad numbersholds
Deceptive scenes (9,795 steps): vision 0.07-0.12 vs real floor, 0.66-0.78 vs lookholds with caveats3,848 look_swap steps are excluded (no class or a hard_mid look); the (b) heads give 0.22-0.80 vs look
Sole teacher on the same steps: 0.820 vs real floor, 0.035 vs lookholds with caveatson SONIC steps this is the teacher trained with G113 held out, which did see SONIC
Hidden-gap steps left out of the class metricholdsa design statement
Dropping 614 crops that show the robot changes nothing by more than 0.008holdslargest change (b) sole 0.488 to 0.480
Crop without paddingholdsevery row reproduces
The pilot's SONIC oddity is gone (0.867 vs 0.857)holdsthe pilot numbers themselves were not verified
The viewing-distance explanation of a small remaining gapcould not verifyneeds retraining
Edge-loaded steps 0.0 / 0.4 / 3.0 %could not verifyfrom the dataset's noise-free re-run
6

Support state

Details: support state from vision.

ClaimVerdictNote
Truth definitionholdsthe prediction files' truth equals the dataset's support state
Counts 54 / 303 / 5,912; 5 / 303 / 381; 2 / 497 / 227holdsexact; the partial count in the step-1 data is 12,365 of 12,372, not 12,364 of 12,371 (off by one)
Support teachers: sole 0.719 / 0.486, joints + IMU 0.445 / 0.305holds
Vision (a): truth 0.529, sole 0.548, joints + IMU 0.411; shuffled 0.341holds with caveatsvision accuracy (0.94-0.95) equals the "always full" baseline of 0.943
Sole labels beat joints + IMU labels (0.548 vs 0.411; (b) 0.504 vs 0.264)holds
Sole labels as good as truthholds"none" has only 54 test steps
Sole teacher after the step 0.736 ± 0.017holds with caveatson SONIC steps it is the teacher that saw SONIC, as above
Edge steps: vision 0.462 vs sole 0.693; partial recall 0.864 vs 0.673holds
Hidden gaps: vision 0.423 / 0.652 vs sole 0.466 / 0.715holds with caveatsanswering "partial" every time scores 0.685 accuracy; vision is below that trivial baseline and the sole only 3 points above it
Partial recall falls to 0.642 (truth labels) and 0.525 (sole labels); sole 0.702holds
The edge-loaded gap sits inside "none"could not verifyfrom the dataset's noise-free re-run
7

VLM

Zero-shot on real photos (details)

ClaimVerdictNote
Macro-F1 0.678 ± 0.017, accuracy 0.624 ± 0.016, 0 invalid; per class 0.391 / 0.684 / 0.796holdsthe re-check's own answer parser agrees
239 of 1,350 answers are hard_slipholds
Question wording chosen on validation only (0.644 > 0.626 > 0.611)holds with caveatsthree validation runs, then one test run; the differences are within noise
Validation and test disjointholds0 shared files and 0 shared image bytes; different sources
Verdict "partly"holds with caveatsonly 3 of 6 classes tested; the ± comes from subsamples that share videos

Tuned on the sole's labels (details)

ClaimVerdictNote
Test and deceptive rows are exactly those of the vision headholds with caveatsthe 5,248 rows are the vision head's test rows that have a best frame; its 0.816 also counts 34 frameless steps; 25 of 25 rebuilt crops byte-identical
The real-photo list is the sameholdssame 1,350 files, order and truth
LoRA trains on the cross-fitted sole labelsholdstraining split and textures only, 0 test episodes or textures; label agreement with truth only 0.81
Question wording unchangedholdsthe same in all 15 runs
Nothing selected on testholds with caveatsno checkpoint choice, but the pilot scored 300 test-split steps before the run sizes were fixed
Held-out textures: LoRA 0.721 ± 0.040 vs zero-shot 0.343 ± 0.004, realholds with caveatsresampling textures gives sd 0.020 (zero-shot) and 0.074 (LoRA); the difference survives: +0.378 [0.197, 0.526]
Zero-shot almost never answers slope or edge; 856 of 1,173 hard_grip steps called hard_slipholds
LoRA weak on soft (0.40) and rough (0.56)holds
Deceptive vs real floor: LoRA 0.124 ± 0.006 vs zero-shot 0.149, "real, but small"does not holdthe numbers reproduce, but the 2× rule used only the LoRA seed sd and zero-shot had no spread; an episode bootstrap (344 episodes) gives −0.025 [−0.062, 0.010]; "both below 1/6" holds
Deceptive vs look: 0.524 ± 0.041 vs 0.347, realholds with caveats+0.176 [0.110, 0.246]; the look accuracy uses 3,315 rows, not 3,990 as reported
Real photos: 0.375 ± 0.024 vs 0.679 ± 0.012, LoRA destroys the skillholds with caveatsa real drop, not a move to classes without photos; per class hard_grip 0.39 to 0.94, rough 0.68 to 0.04, soft 0.80 to 0.31; LoRA answers hard_grip on 1,019-1,053 of 1,350
Shuffled LoRA 0.061 is at chanceholds with caveats0.061 is exactly a constant hard_grip answer; random guessing scores 0.167, so it is a collapse floor, not chance
0 invalid answersholds
The amended sanity line of the checkholdsthe right invariant; dropping the 12 affected rows moves the accuracies by at most 0.0002
Verdict: LoRA learns the look and loses real-photo skillholds with caveatsthe ViT head scores 0.821 ± 0.028 against 0.721 ± 0.040 on the same rows, +0.109 [0.001, 0.240], borderline and not a fair comparison; no VLM comparison of sole against joints + IMU labels
8

Leaks and controls

9

Twenty samples

Sheet of twenty test steps: for each, the square crop with grey padding below and the camera frame with the landing footprint in red and the crop box, with truth, look and the answers of every head and teacher printed above
Twelve honest test steps (two per class over three gaits), six look_swap and two hidden-gap steps, picked at random. For each: the square crop the heads see, and the camera frame with the landing footprint.
10

Threats and what could not be checked

Threats to the main conclusion

Could not check

Scores are macro-F1 per step against simulator truth on held-out textures unless a row says accuracy; intervals in brackets are bootstrap intervals over textures or episodes. Experiment 1 pages: dataset, teachers, vision head, support state, VLM on real photos, VLM with LoRA. The re-check of experiment 2: independent re-check of experiment 2.