The search box answered 19 of 20 queries and none of them correctly

On 21 November someone typed 18米高8米宽 into the search box on a site of construction-quantity calculators. It means “18 m high, 8 m wide”, a scaffolding question arriving from a job-site comment thread. The box answered with a rebar hook-length table and a chart of bricks per cubic metre of masonry, both served at a relevance score of 1.00. No noun anywhere in the query. Two dimensions and a matcher that called the result a perfect match.

The author of the site wrote up the path from that query to the fix. The formula is the part worth reading. The index is 238 cards, each carrying one searchable string: title, category, description, slug, a few pinyin and synonym aliases, and stage one of the matcher is a plain substring AND over that string. That works until someone types a question instead of a noun.

The denominator was the whole bug

Chinese has no spaces. 18米高8米宽 stayed one token, no index string contained it, and stage one returned nothing. Correct behaviour, and the reason stage two exists: split the query into overlapping CJK bigrams and alphanumeric runs, drop every segment that appears nowhere in the index, then score each card as hits divided by surviving segments.

That drop step is normally the clever part. It keeps a common question phrase like 怎么算 from capping every real match at 20%. On this one it did its job perfectly, which was the failure. 米高 and 米宽 exist on no card. What survived was the digits, and only the digits.

score = hits / valid.length;        // fraction of surviving grams found
if (max >= 0.5) showEverythingTiedAtMax();

Two grams survived, so the denominator was 2, so a single hit scored 0.5 and cleared the threshold, and required matches is ceil(n/2) at every length: four grams need two, two need one, and a query with a single surviving gram needs one hit for a flat 1.00. Read that back carefully. The less of the query the parser could keep, the lower the bar for answering it, so confidence and comprehension moved in opposite directions. That is the inversion of what a relevance score is supposed to mean, and the fault sits in the formula rather than the data.

The surviving grams were 18 and 8. Two cards contain both, because both index strings contain 180: one page indexes 180度弯钩, a 180-degree hook, and the other indexes 120 / 180 / 240, wall thicknesses in millimetres. 18 and 8 are substrings of 180. One page means degrees, the other means millimetres. The matcher found identical evidence in both, and it was right about the strings and wrong about the world.

A rare substring is not a rare term

The frequency table explains the rest. The index holds 4,204 distinct CJK bigrams, and it is brutally top-heavy: 算器 sits on 206 of 238 cards, 计算 on 187, because every card title ends in “calculator”. A query carrying any fragment of that word scores 0.5 on most of the index before the matcher has learned what the page is about.

The reflex fix is IDF: weight rare grams up, common grams down. It makes this failure worse. 18 appears on 2 of 238 cards, so a rarity weighting puts it near the ceiling and raises its influence on exactly the fragment that caused the wrong answer. Rarity in a substring index measures typography. To the same matcher, 8, 800 and 1800 look like three unrelated, highly specific facts.

Ties finish the job. The fallback renders everything at the maximum in build order, so the answer is stable, reproducible and arbitrary. 1.5米 produced seven cards at 1.00 for the same reason: grams 1 and 5, one hit each, seven winners.

A relevance score is a claim about the query. This one was computed entirely from the document, so a query with nothing in it could still score perfectly. Any score that divides by what preprocessing threw away is measuring the parser.

Twenty dimension-only queries ran against the same index before and after a fix. The old fallback answered 19 of 20 with zero relevant results. The gated version answered 3 of 20, also zero relevant. Nineteen answers and not one about the right thing. That is the shape of this bug class: a system that answers almost everything and is almost never right, rather than an occasional bad match.

The misses that never got logged

The expensive part wasn’t the wrong page. Every search on the site reports a result count, and zero-result queries feed a demand list of things people asked for that the site doesn’t have. While the fallback kept answering these queries, they never appeared as misses. Two weeks of demand data sat in the log scored 1.00. A false positive in search is also a false negative in your backlog, and only one of those two is visible to the person typing.

The fix went in front of the score instead of inside it. Strip units, dimension words, numerals and question particles, then require a CJK character to survive or a latin identifier of two letters or more, because hrb400 and c30 are real subjects and stripping them would break every steel and concrete query on the site. If nothing survives, the query is a parameter lookup, not a document lookup, and a substring index has no business answering it.

Zero results beats a plausible wrong one here.

Someone who gets nothing retypes the query or opens the category list. Someone who gets a rebar table instead concludes the site doesn’t have a scaffolding calculator and leaves for good. Only one of those failures is recoverable.

Three of the twenty still get through. 跨度 and 层高 are nouns even when the query is a dimension: 6米跨度 returns four cards, 4.5米层高 returns two, both about paint coverage and wallpaper that mention ceiling height in passing. 间距 survives the strip in 18米高8米宽间距1.5 and drags fourteen cards with it, because it is a real word that also names a parameter rather than a product. A vocabulary of parameter nouns would close those. That part isn’t built, and the author says so.

The test worth copying costs nothing: for any fuzzy fallback you keep, count how many cards tie at the maximum, and count how many surviving grams are digits or fragments of digits. Both non-zero means the box is answering questions it cannot read. I would add one counter that stays on the server, the mean fraction of each query dropped before scoring, and alert when that number is high on queries that returned winners. High drop rate plus a high score is the signature of this bug, and it is invisible on the result page. The full debugging trail, including the harness that replays the twenty queries, is in the write-up.

Press ⌘K to search & switch themes