A room-mapping benchmark from real booking traffic (v5.1)

Abstract

Data sources

Source	What it contributes	Used for gold?
Plexe panel	Supplier room names and prices from real searches, May 4 to August 11 2026: 85,548 hotels, 33.6 million requests.	Names only
Hotels' own websites	Official room lists, scraped after checking robots.txt, one request every 2 seconds per site.	Yes: the reference rooms
Plexe production groupings	How the live system grouped names (July to August 2026).	No: a baseline only, never an input to gold
People	Three blind review sheets (see Human validation).	Validation, plus 59 adjudicated pairs

Construction pipeline

Every stage below removes hotels for a stated reason. Bar length uses a log scale so the small final sets stay visible; the count is exact.

Horizontal axis: hotels remaining (log scale).

How a pair gets a label

A supplier listing is placed on a room from the hotel's own website only when its normalised words equal exactly one published room's words. Two listings on the same room form a same-room pair; two listings on different rooms, where neither room's words contain the other's, form a different-room pair. Anything ambiguous is left out rather than guessed.

Datasheet

Files and shapes

Every released file, with its size, its columns and an example value.

Label distributions and splits

Split sizes by stratum

Human validation

Three blind checks, each answered without seeing the machine's answer. The first pass of each sheet is the record.

Baselines and results

Scores are counted per supplier name, so identical names already share a row; that is why "Identical names together" scores the same as "Every name its own row".

Why the trivial token-set baseline scores near-perfect. v5.1 gold only checks names worded the same as the hotel's own room list: a same-room pair exists only when both names have exactly the words of one published room. Two names with the same set of words are therefore always the same room by construction, so grouping by word set never misses a match this gold can see. The gold cannot see differently worded names for one room, so a low missed-match number here is a floor, not a verdict.

The internal evaluation set (aggregates only)

Limitations

Missed matches measure only easy splits. v5.1 split numbers see names that differ by word order or punctuation, not differently worded names for the same room.
Gold granularity. A hotel's website may list "Deluxe Room" once while suppliers sell king and twin versions; the gold treats them as one room.
Machine-made labels. Most pairs come from an exact-words rule, not from people; a person agreed with the related internal gold 80 of 100 times on first pass.
Chain hotels are missing. hotels could not be scraped because their chain's website blocks automated access; we did not work around it.
Third-party room content. Supplier catalogues that carry Expedia content are used only as internal inputs and are never published here.
Coverage. Gold hotels carry of panel traffic; results may not hold for the long tail.

Changelog

v5.1 (2026-09-26): every hotel of the internal evaluation set removed, and the build now refuses to read it. Surviving pairs are byte-identical to v5.0.
v5 (2026-09-24): identity gold rebuilt from hotels' own websites on the panel frame; fresh sealed dev/test draw (seed 20260924).
v2.0: retired ("burned"). Its test half was scored twice, and 3,200 of its gold rows are in another repository, so its test numbers can no longer be trusted.

Download

Public v5.1 files only. The sealed test and the internal set are not released.