The enrichment machine, part 2.

First, the verdict. The pilot put fifty of our worst records — products with names like “SCREW,” useless to search and to humans — through both research models and printed the results on paper review sheets, because our resident domain expert, who knows these parts better than any database does, reviews on paper. His scores settled it: where the models reported high confidence, they were 90% accurate with zero outright wrongs — across both models. The thing entry 3 staked the project on held: the machine knows when it knows. When it’s sure, it’s right; when it’s wrong, it had already told us it wasn’t sure.

The review also exposed the machine’s laziest trick. A third of the pilot rows were what we started calling bare names: the model, having learned nothing, hands back the old name with a style number stapled to it and calls it research. Not wrong, exactly — just worthless, and worse, it can ride in under decent confidence. So the pipeline gained a bare-name filter: a result that doesn’t substantively identify the part doesn’t ship on confidence alone, no matter how sure the model feels about its non-answer.

Around those findings the pipeline grew its governor. The cheaper research model gets two passes before anything escalates — a surprising share of items resolve on the second look. What survives both passes unresolved goes to the heavier model, which is better and meaningfully more expensive, so it works behind hard caps: a per-item dollar limit, a daily dollar limit, a daily item count, all tunable from a settings page rather than buried in code. Results land where their confidence earned them: three lists for human merge approval — including one flagged “took two tries,” so the reviewer knows — a held queue where medium-confidence findings wait for human judgment, and, when neither model finds anything, an honest fallback: the part type and the style number, with nothing invented. Everything fires from buttons, not schedules; a machine that spends money doesn’t get to run while nobody’s watching.

One writing rule from this phase deserves its own sentence, because it applies far beyond product data. The machine now writes two texts per item: internal research notes, where hedging is allowed and sources are discussed — and customer-facing copy, which may state only what the confidence supports. No hedging, no “appears to be,” no marketing filler. If the research can’t support a plain declarative sentence, the sentence doesn’t ship.

And then the machine drifted. A fresh sample batch came back almost entirely low-confidence, and the tempting read was “hard batch.” Instead we re-ran a batch with known-good history — and it collapsed identically. Same parts, same prompts, same pipeline: different results. The research model’s retrieval had wandered — its citations now pointed at a government vulnerability database and home-improvement retailers instead of sewing-machine sources, presumably after changes on the provider’s side that nobody announces to you. The fix was one line: a domain hint prepended to the research query, steering it back to industrial-sewing sources. The known-good batch recovered to its old numbers — and the hint ships behind an A/B flag, so we can keep proving it’s the hint doing the work and not wishful thinking. The lesson is the entry’s spine: when part of your machine is a service you don’t operate, it will change under you, silently — and only a control sample can tell you whether the problem is your data or their model.

Status, plainly: the pilot is proven, the pipeline design is settled, and the production module that wraps it all in the platform’s backend — lists, buttons, caps on a settings page — is the build ahead, not the build behind. The expert’s verdict bought the machine its future; the governor is what makes that future safe to fund. Confidence decides the path, a human decides the merge, and the caps decide the bill.

← All entries