Case studies

1. A guardrail you could walk past with sudo

Hard technical problem · shipped in v1.29.2

Problem. Catastrophic gates (rm -rf ~, force-push, hard reset) matched the happy path but missed wrappers: sudo rm -rf ~ produced no match on 1.29.1.

Action. Built an adversarial grid instead of hand-picking examples; canonicalized commands centrally; re-evaluated only when the original form missed.

rm -rf ~        → deny
sudo rm -rf ~   → deny (after fix)
thumbgate@1.29.1 → 34+ holes
thumbgate@1.29.2 → 0 holes

Artifact. tests/gate-evasion-matrix.test.js in CI + 6-hourly job against the published npm artifact.

2. A seven-defect report where most defects weren’t real

Ambiguous problem · investigation discipline

Problem. A confident multi-defect report with statistics would have forced a core rewrite if accepted wholesale.

Action. Treated each claim as a hypothesis: read the consumer’s config, recompute totals from source data, reproduce before believing.

Result. Strict-mode env explained “warn-by-default not working.” Filtered tables understated warns. The investigation also surfaced the sudo evasion class above — worse than anything in the original report.

3. Production failure: a firewall enforcing nothing

Incident · fail-open PreToolUse hook

Problem. Local state vanished and thumbgate-hook was missing. An erroring PreToolUse hook fails open — the product looked “fine” while blocking nothing.

Action. Restored via thumbgate init, then proved enforcement with the same dangerous commands that must deny.

Lesson. This product’s failure mode is silence. Uptime green is not enough; known-dangerous commands must still be denied on a schedule against the published artifact.

4. Monetization that almost sold off the safety floor

Stakeholder conflict · product policy

Problem. Free-tier daily block caps + warn-by-default posture could downgrade irreversible actions to warnings after the cap.

Action. Split how many blocks free users get from which actions can never be discounted. Named catastrophic IDs stay deny regardless of tier.

Result. Upgrade pressure remains on negotiable gates; irreversible classes are not for sale.

5. Built the measurement before claiming “fixed”

Instrument first · property tests

Problem. “We fixed the bypass class” had been claimed twice and been wrong twice.

Action. Built a property suite over commands × spellings, with sandboxing and a minimum number of plain-form denials so empty measurements cannot pass.

Result. “Is it fixed?” became a command with a number, including against the npm tarball users actually install.

6. Agent memory crowded by near-duplicate lessons

Retrieval quality · memory dedupe/compaction

Problem. Promoted lessons accumulated paraphrases of the same failure. One real lesson could occupy every retrieval slot and starve distinct ones; a July 2026 audit found ~86% of promoted lessons were near-duplicates.

Action. Dedupe the candidate pool before ranking, and add a compaction CLI that merges near-duplicates with summed occurrence counts — while never merging opposite-signal lessons on the same topic.

dupe-1 "Railway deploy verification timed out…"  ← 1 slot
dupe-2 same lesson, rephrased                   ← slot 2, useless
dupe-3 same lesson, rephrased                   ← slot 3, useless
→ after fix: one survivor + 2 distinct lessons  ← full slot budget

Artifact. tests/lesson-retrieval-dedupe-backfill.test.js + tests/memory-near-dupe.test.js in CI; npm run memory:compact applies with a timestamped backup.