[ note · 2026-07-11 ]

Determinism alarm — regression + silent-seam liveness

Per-organisation, per-product completeness regression plus liveness streaks on the two seams that fail quiet. 162 lines at services/api/src/lib/determinism_alarm.ts. Catches the class of failure where the system stops doing its job without breaking anything visible.

The founding incident: one organisation's per-product completeness score fell 0.88 to 0.69 to 0.62 across three paid runs. Peak-vs-latest drop 0.26. Nothing alarmed.

One organisation’s per-product completeness score fell 0.88 → 0.69 → 0.62 across three paid runs, a peak-vs-latest drop of 0.26 that nothing alarmed, because the fleet-averaged dashboard hid it. The determinism alarm at services/api/src/lib/determinism_alarm.ts is the 162-line check we built after it: per-organisation, per-product peak-vs-latest completeness regression, plus liveness streak detectors on the two seams that had actually failed quiet in production.

The failure mode that eats production runtimes is not the crash — the crash is obvious. It is the silent regression. A seam that used to fire stops firing. A completeness score that was 0.88 three runs ago is 0.62 now, and nobody notices, because no exception fires and no queue backs up. That exact sequence — 0.88 → 0.69 → 0.62 across three paid runs of one organisation’s product — sat in our production history while the fleet-averaged dashboard aggregate hid it. The determinism alarm is the check we built after it.

The file is services/api/src/lib/determinism_alarm.ts, 162 lines. Two things run.

First, a per-organisation, per-product completeness watch. A completed run records a completeness score for the record it produced. For every product an organisation has run at least twice in the past 30 days, we take the scores those runs recorded and compare the most recent against the peak over that window; a drop of more than 0.1 raises the alarm with the organisation, the product, and the size of the drop. Peak-versus-latest, not run-versus-previous-run, on purpose: the incident above was a slow bleed, and each adjacent step looked tolerable while the total drop was 0.26.

Second, liveness streaks on the silent seams. We did not instrument “every” seam — we enumerated the two that had actually failed quiet in production and wrote a streak detector for each. The vision seam: a completed run that expected images but metered zero vision spend means the vision stage never ran and the card shipped imageless, with nothing thrown anywhere; three of those in a row fires the alarm. The crawl seam: a completed run that grounded zero attributes means the source layer is returning nothing; four in a row fires the alarm. A single live run breaks either streak.

We chose alarm over auto-remediate on purpose. The sweep runs nightly, sets an operator gauge, and writes warning lines to the operator panel — the runtime keeps serving, and a query hiccup inside the monitor is swallowed rather than allowed to crash the scheduler. Auto-remediation would hide the underlying cause, and the underlying cause is usually more important than the immediate symptom.

The 162 lines are boring on inspection: two pure streak detectors and a peak-drop scan, each unit-testable with no infrastructure, plus a thin query, a gauge, and a scheduler. The value is not in the code. It is in having named the seams that fail quiet — a general framework over “every silent seam” would have been the more impressive sentence and the less useful file.

The runtime is auditable because we know where it can fail quietly.