FLA · Experiment 1 VLM on real photos · 16 Nov 2026 · all reports · summary

Can an open VLM name the floor from a real photo?

First a check on real ground: given one real photo of the ground and the project's six floor classes, does Qwen2.5-VL-7B pick the right class with no training at all?

Real photos, but only three of the six classes have any. Photos are not shown here: their licence is unclear.

In short

  1. Partly. Zero-shot, the model separates soft, rough and hard ground well above chance: macro-F1 0.678 ± 0.017 over three classes, where chance is about 0.33.
  2. It is not usable as a floor sensor as it is. Hard paving is its weak class: right on only 0.391 of those photos.
  3. Its most common error is the one that matters most. It answers "slippery" on 239 of 1,350 photos, and none of them are slippery. Dry concrete is called slippery.
  4. Half the task cannot be tested on these photos. There are no slippery floors, slopes or edges in the data.
1

What was done

2

Which photos count as which class

The photo set has 31 ground materials. They were mapped to the project's classes by material alone, from a contact sheet, before any model run.

Project classMaterials from the photo set
hard_gripasphalt, cement, brick, stone-asphalt, stone-brick, stone-cement
softgrass, turf, moss, sand, soil, dry leaves, leaves, wood chips
roughpebbles, shale, small limestone
hard_slipnone: no material is reliably slippery (no ice, wet or polished floor). Not tested.
slopenone: slope is geometry, not visible in close-up material photos. Not tested.
edgenone: geometry. Not tested.

Reading the table. Only three of the six classes can be tested on real photos. Ambiguous materials were left out, among them painted asphalt, metals, plastics, glass, cloth, paper, roots and large limestone (flat slabs in one set, rock piles in the other).

3

Results

Test, 1,350 photosValue
Macro-F1 (3 classes)0.678 ± 0.017
Accuracy0.624 ± 0.016
Invalid answers0.000 ± 0.000
Right on hard_grip photos0.391 ± 0.034
Right on rough photos0.684 ± 0.014
Right on soft photos0.796 ± 0.021

Reading the table. Well above chance (about 0.33), and the JSON format never failed. But even soft, the best class, is right on only 0.796 of its photos, and hard paving is wrong on most.

Confusion matrix of the zero-shot VLM on 1,350 real test photos: rows hard_grip, rough and soft; columns the six class names and invalid; the model never answers slope, edge or invalid
Confusion matrix on the whole test set: rows are the true class of the photo, columns the model's answer. The figure's title names the wording used, P1.
Wording (validation, 360 photos)Macro-F1AccuracyInvalid
P1, shortest, no class definitions0.6440.6470.000
P20.6260.6610.000
P30.6110.6610.000

Reading the table. P1 was chosen on validation by macro-F1 and run once on test. The differences between wordings (0.611 to 0.644) are within noise.

Speed: 18.34 answers per second (batches of 16, 74 s for 1,350 photos) on an H100 that another job kept at about 90 % load, so it is a lower bound.

4

Where it fails

What it means. The error that matters most for walking, grippy against slippery, is the one the model gets wrong most often, and this data cannot test real slippery floors at all. That argues for tuning the model (the next page), and for real photos of slippery floors before any claim about slip.

5

Independent re-check

Full list: independent re-check of experiment 1.

6

Limits

Scores: macro-F1 over the three classes that have real photos, mean ± sd over three test subsamples. Photos: GTOS and GTOS-Mobile (Xue, Zhang, Nishino, Dana, CVPR 2017; Xue, Zhang, Dana, CVPR 2018), numbers only. Model: Qwen2.5-VL-7B-Instruct, Apache-2.0. Related: the VLM tuned on the sole's labels, the vision head, the dataset, independent re-check.