The active-inference fetcher our own benchmark refused

A 91-SKU paired A/B refuted this fetch policy: cost fell 76%, grounded attributes fell 40%, and 55 of the 91 SKUs lost more than five attributes each. The mechanism — a ranked fetcher grounded in Friston’s active inference and Weitzman’s break-even rule — is in the tree at services/runtime/lib/active-inference.mjs, 108 lines with 82 lines of tests, off by default behind a runtime flag, its file head carrying a warning block that records the refutation.

If you had asked us early on what the smartest part of unbin’s runtime was, we would have said this one. It is a ranked-fetch mechanism built on Friston’s active inference and Weitzman’s break-even rule — the Pandora-box search result: for each candidate source URL, estimate the expected reduction in uncertainty against the fetch cost, fetch in priority order, and stop when the marginal gain stops paying for itself. It lives at services/runtime/lib/active-inference.mjs — 108 lines, plus 82 lines of tests — and was, on paper, the most interesting thing we built.

It did not survive contact with the benchmark.

An early pilot on 8 SKUs looked favourable — cost down nearly half, no visible regression. The full A/B said otherwise. We ran the benchmark’s 91 agent-verified SKUs paired, mechanism on against mechanism off. Fetch cost fell 76%. Grounded attributes fell by about 40% — 55 of the 91 SKUs lost more than five attributes, and 49 of the 91 ran zero escalations, the progressively more expensive search steps the runtime takes when a product record is still missing fields. The commit that records the verdict, 9141fbd5, says it plainly: do not ship. The root cause was a myopic stop: the break-even rule only valued candidate URLs it already knew about, and at the decision point that list was usually empty — so the expected gain read as zero and the mechanism stopped before the cheap escalations that discover new sources. Elegant mathematics, wrong model of where the value comes from.

The rework came the same day. Commit 3150dbe9 gated the stop rule to the single most expensive escalation rung only — the cheap, discovery-driving escalations now always run. A follow-up, 93b1e99a, added a guard so a sparsely grounded record is never refused that expensive escalation; the refusal fires only once a record is already well grounded. In a smoke test on the worst losers from the A/B the guard recovered one outright and improved the rest — but it was still not a clean win, because the residual failures traced to fetch-cost ceilings and to sources the fetch layer genuinely cannot read. So the verdict stood.

Run the same 91-SKU benchmark against the reworked variant — the fixture we published with the primitive — and 25 SKUs still lose more than five grounded attributes each. Better than 55, worse than nothing. The mechanism stays off by default until every one of those 25 is on the right side of the line.

What remains on disk is the honest shape of that verdict. The mechanism is in the tree, off by default behind a runtime flag, and the file opens with a warning block that records the refutation. The reordering half — rank candidate fetches by expected uncertainty reduction, which we still consider the mechanism’s real contribution — survived the rework unchanged, but it runs only when the same flag is armed. Turning any of it on requires the full 91-SKU benchmark to re-clear it: cost same or lower, grounded attributes within about five per cent, no systematic loss. Until then, off.

The lesson we took from it: theoretical elegance is orthogonal to whether a mechanism earns its place under measurement, and small pilots flatter. The 8-SKU pilot said yes; the 91-SKU benchmark said no; the benchmark is the gate. We publish the refusal with the same receipts we would publish a win.

This is why we would show this one first. Not because it was our best idea. Because the benchmark refusing it — and the refusal standing, in the code and in the record — is the clearest evidence of the discipline we work by: measure, decide, and hold a mechanism off when the measurement says off.

The mechanism, its tests, and the full 91-row A/B fixture are published for reading: packages/active-inference in bargo-lv/primitives — the refutation is checkable, not just narrated.

Correction (2026-08-20): an earlier version of this note said this mechanism was deleted — “about 483 lines” — and cited commits d5604d21 and 0063afe3; that was wrong. The file is 108 lines, live, and was never deleted, and both cited commits belong to different mechanisms. The account above is the verified record: refuted by the 91-SKU A/B in 9141fbd5, reworked in 3150dbe9 and 93b1e99a, and held off by default.