Calibration monitor with an ECE-drift alarm

Measured on the 94-SKU non-circular benchmark after the fit: expected calibration error 0.018, and the high-confidence bin’s correctness rose from 80% to 91%. Isotonic and Platt calibrators on the runtime’s confidence outputs, with an ECE-drift alarm that flags a scheduled offline refit when live ECE exceeds the fitted baseline by more than 0.05.

A runtime that returns a value plus a confidence is telling you two things. If the confidence is systematically wrong — say, 0.8 confidences are correct 55% of the time instead of 80% — the second thing is worse than useless: it is actively misleading. The calibration monitor exists to make sure that does not happen quietly.

The file is services/runtime/lib/calibration.mjs, 146 lines. It implements two calibrators: isotonic regression (non-parametric and monotone, fitted by pool-adjacent-violators — no shape assumption) and Platt scaling (a two-parameter logistic sigmoid that stays robust on small samples). Neither is right everywhere, which is why both live in the file. The fit itself is an offline step: verified pairs of raw confidence and actual correctness in, one fitted model out — either fitter, chosen at fit time — persisted as a small JSON blob the hot path applies. With no fitted model present, the apply step is an identity pass-through: an engine without a fitted model behaves exactly as before.

The load-bearing piece is the ECE-drift alarm. Expected calibration error is measured over ten equal-width confidence bins — the population-weighted gap between claimed confidence and observed accuracy. The monitor compares the ECE on fresh labelled evidence against the baseline recorded when the model was fitted, and the rule is a fixed threshold, not a statistical band: when live ECE exceeds the baseline by more than 0.05 (the default; the threshold is dialable), the monitor flags recalibration. What flagging means concretely: the drift check returns a recalibrate verdict, the monitoring run reports it loudly and exits non-zero, and the runtime keeps serving on the model it already has.

We do not recalibrate on the hot path. Drift flags a scheduled offline refit: the same labelled-evidence pipeline refits the calibrator, and the serving path picks the new model up by file timestamp, without a restart. One guard in the refit is worth naming: if the fresh fit measures worse than the constants it would replace, the script refuses to write it — the existing behaviour stands until a fit earns its place. A recalibration is a deliberate, inspectable event, never a silent live patch.

The reason for drift usually turns out to be one of three things. A new source came online with different reliability characteristics. A prompt was updated without re-baselining. A downstream consumer started asking a question the model was not calibrated for. All three are worth knowing about, and none of them should be papered over by an invisible fix — which is why the refit is a scheduled step you can inspect, not a background one you cannot.

The file does one job and does it visibly. We considered fancier calibrators — beta calibration, temperature scaling with per-class bins, spline-based — and the fancier ones did not measurably beat isotonic-plus-Platt on our traffic distribution. So we did not ship them. The rule we hold ourselves to is: if the fancier variant does not clearly earn its place, the simpler one stays.

Ninety percent of what makes a runtime auditable is boring composition of well-tested primitives with a drift alarm on top. This is one of them.

The code is published for reading: packages/calibration in bargo-lv/primitives — source, tests, and the standalone fitter, all rights reserved.

Correction (2026-08-20): an earlier version of this note claimed we deliberately do not recalibrate at all. The true posture is the file’s own: ECE drift flags a scheduled offline refit. The text above now matches the code.