How We Evaluate
The methodology behind ThumbGate's enforcement quality — published with the numbers that don't flatter us left in. Every figure on this page is reproducible from the open-source repo with one command.
The headline we retired
ThumbGate's learned risk model used to report one quality figure: trainingAccuracy: 0.820 — measured on the same rows it trained on, against a corpus where answering "risky" unconditionally scores 71.1%. That is an 11-point in-sample lift presented as an 82-point achievement, and it said nothing about generalization at all.
We replaced it with held-out evaluation under two different splits, and we publish both results — including the one that inverts.
The result, honestly
Accuracy lift over the majority-class baseline, in points. 1,791 production rows, 12 independent group-aware stratified splits, mean across resamples. IID: +9.9 ± 1.7, AUC 0.887, 12/12 folds beat baseline. Novel-context: −10.5 ± 9.0, 1/12 folds, calibration error 0.31 vs 0.06.
On familiar traffic the model genuinely helps. On action types it has never seen, it is worse than a constant classifier — and unfamiliar actions are precisely the case a firewall exists for. That asymmetry is why deterministic rules enforce and the model only triages. We consider publishing that inversion more valuable than hiding it: it is the reason ThumbGate's blocking behaviour does not degrade with the model.
The methodology
| Practice | What it prevents |
|---|---|
| Lift vs the majority baseline, reported beside every accuracy | 71% base-rate corpora making any model look strong |
| MCC + ROC-AUC + Brier + calibration error | accuracy hiding degenerate or miscalibrated behaviour |
| Dual holdouts: IID and whole-category (distribution shift) | quoting only the flattering generalization estimate |
| Group-aware splits keyed on the feature vector | duplicate inputs straddling folds and inflating scores |
| 12 resamples, mean ± sd, paired t on shared folds | single lucky splits shipping as "improvements" |
| Per-fold vocabulary rebuild | held-out rows choosing the model's features (transductive fit) |
The evaluation suites
- Model harness —
npm run eval:risk: the resampling harness;--jsonemits the machine-readable report backing this page. - Trace-mined drift benchmark — real production gate decisions, redacted, become 60 replayable cases across 12 gates; CI fails when any real command's verdict moves. Silent gate death is the signature of every bypass we have found.
- Evasion matrix — 14 dangerous commands × 9 canonicalization transforms; this is the suite that measured the 62 holes closed in v1.29.2.
- Published-artifact verification — the npm tarball itself is installed into a throwaway prefix and driven through its public hook contract twice daily. CI proving the source blocks something is a different claim from the artifact users receive blocking it.
- Decision canary — live decision-distribution drift: a gate going silent, a spike in blocks, or a gate appearing/disappearing.
The gates that block a release
- A synthetic-fixture quality gate that separates "the trainer broke" from "the data lacks signal" — different problems, different fixes.
- A pure-noise canary: a trainer that finds >15 points of lift in random labels has labels leaking into features, and CI fails.
- Vacuity guards: a drift test that compared fewer than 20 cases fails; a single-class test fold refuses to report rather than emitting a meaningless pass. A check that cannot fail is treated as no check.
- Non-significant improvements do not ship. Two candidate upgrades (a feature-diversity cap, Platt calibration) are implemented, tested, and off — paired t of −0.31 and −0.72.
Error analysis: the five bugs we found in our own evaluator
The uncomfortable finding of this work: the evaluator was wrong more often than the model, and its errors all flattered us.
| Bug | Effect before the fix |
|---|---|
| Tied-hash rows straddling folds | test fold contained memorized rows |
| Cross-class duplicate leakage under label noise | same input in both folds with different labels |
| Group quota overshoot (one 655-row category) | novel-context result read +0.016; truth is −0.105 |
| Vocabulary built before the split | held-out rows chose the model's features |
| Raw text in the split key | identical model inputs split across folds |
Three of the five were found by adversarial review after we had published numbers from the leaky evaluator. The corrected conclusion was stronger, not weaker — and the correction is documented in the repo, not overwritten.
Reproduce it
Honest scoping: the measured corpus is private operator feedback telemetry and does not ship in the public repo. The exact machine-readable report behind the figures on this page is checked in with provenance at evals/risk-model-report.json. A clean clone gives you that report, the methodology write-up, the CI quality gate (synthetic fixtures / no private data), and the same harness to run against your feedback log. The figures on this page are first-party operator measurements documented in docs/ML-EVALUATION.md — not a committed private corpus dump.
git clone https://github.com/IgorGanapolsky/ThumbGate cd ThumbGate && npm ci npm run test:risk-quality # CI quality gate — no private data needed # Optional: re-run the harness on your own feedback log node scripts/eval-risk-model.js --corpus <your-own-feedback-log> --resamples 12 --json
Full methodology: docs/ML-EVALUATION.md · case studies: /case-studies · gate scorecard: /eval-scorecard