How we know ThumbGate works

Evaluation white paper · Updated 2026-07-29 · Source: docs/HOW-WE-KNOW-IT-WORKS.md

Offline goldens + live monitors, not vibes. unsafeActionRate must stay 0.

ThumbGate is a local-first PreToolUse enforcement layer: feedback becomes lessons, repeated failures become gates, and risky tool calls are checked before execution. This page is the buyer-readable map from that claim to evidence.

ThumbGate system architecture diagram — System map: agents, hooks, gate engine, governance, security, feedback promotion. Full gallery on /architecture.

1. Golden evaluation dataset

Committed suites encode expected allow / deny / warn decisions and reference answers — not aspirational prose.

Pack	Path
ThumbGate Bench	`bench/thumbgate-bench.json`
Agent safety eval	`config/evals/agent-safety-eval.json`
Prompt eval	`bench/prompt-eval-suite.json`
Observability / grounding	`bench/observability-eval-suite.json`
ProgramBench smoke	`bench/programbench-smoke.json`
Shell golden tests	`verification/golden_tests/**`

Live failures feed offline goldens through scripts/llm-behavior-monitor.js promotion candidates.

2. Offline regression tests

npm test
npm run test:coverage
npm run prove:adapters
npm run prove:automation
npm run self-heal:check

Observed results land in docs/VERIFICATION_EVIDENCE.md. Releases also require Changesets, SemVer, and version sync.

3. Tool-call correctness

ThumbGate Bench measures whether the gate decides correctly on the actual tool call.

Metric	Intent
`taskSuccessRate`	Decision matches expected
`unsafeActionRate`	Must stay 0
`capabilityRate`	Safe work still allowed
`falseBlockRate`	No “block everything” cheat
`replayStability`	Same suite → same decisions

Open the live scorecard Bench methodology →

4. Latency

Pre-action gate loop — Enforcement is local: deterministic match first, optional local embeddings, no cloud model on the critical path.

No LLM on block/allow for the PreToolUse decision
Routing budgets: lowLatencyBudgetMs: 300 for the fast classical lane
High-risk ambiguity can stop for human review instead of spending tokens

5. Cost

A blocked tool call does not spend the model round-trip
npx thumbgate cost prints conservative $ saved from recorded blocks
Budget ledger + tokenomics guardrails prevent runaway spend
Methodology: /agents-cost-savings · inventory: /numbers

6. Human review

Classifier route human_review for credentials, customer data, regulated workflows, payments
Rubrics require verification evidence before “done”
Protected actions need scoped approval; break-glass is short-lived and narrow
Outbound social replies stay draft-only until a human publishes

7. Production monitoring

Feedback pipeline sequence diagram — Capture → promote → block the repeat. Production monitors exist to catch silence (fail-open hooks) as well as loud breakage.

Deploy health: Railway /health version + dashboard smoke
self-heal:check — budget, tests, prove lanes must be HEALTHY
Gate stats + LLM behavior rates (wrong-tool, malformed, drift)
Published-artifact evasion jobs and silent-gate canaries

20-minute buyer audit

git clone https://github.com/IgorGanapolsky/ThumbGate
cd ThumbGate && npm ci
npm test
npm run prove:adapters && npm run prove:automation
npm run self-heal:check
npm run thumbgate:bench -- --json
curl -s https://thumbgate-production.up.railway.app/health

What this paper is not

Not external customer revenue proof — see commercial truth in the repo
Not a claim that free installs hard-block every risky command by default
Not a substitute for human review on payments, credentials, or regulated actions

Case studies ML evaluations Diagram gallery