KRUC — LAPLACE-500

reproduce this page's ruler in under 5 seconds, no key, no deps:

$ ./repro.sh

expected output (embedded, not linked out — Bellard lens)

== 1. ruler self-test (kruc.py) ==
8 passed in 0.01s

== 2. published counts -> published CIs (recompute, don't trust) ==
  gate catch     30/40: point 0.75 (claimed 0.75)  Wilson95 [0.598, 0.858]
  gate false-kill 2/40: point 0.05 (claimed 0.05)  Wilson95 [0.014, 0.165]
If these match the published sheet, the statistics are not fudged.

REPRODUCED. Ruler honest; published counts reproduce published CIs.

This proves the METRIC is honest and published numbers follow from published counts. It does not re-run the private oracle/fixtures (the moat) — per-hypothesis repro commands for H1–H5 are inlined at the bottom of each section below.

Model-agnostic by construction: same deterministic gate, no API key, air-gap-capable. Measured across open (Llama 3.1-8B, 3.3-70B) and closed (Claude Opus 4.8) seats.

the condition that would falsify this gate (Gödel lens, stated before anyone asks)

The gate's whole claim is: a deterministic, zero-cost, 8-CWE-class checker clears real production code at a false-kill rate (0.04%) inside the confirmed LLM-hybrid FP-elimination band (86–92%), without an LLM call. This is falsified — not qualified, not re-scoped, falsified — if a fresh, frozen, unmodified sample of real production Python (not tuned to these 173 packages) reproduces a false-kill rate whose 95% CI does not overlap [3.87%, 4.42%], OR if a genuinely confirmed (fixed-version-bounded) advisory-affected package shows the gate-filter silently dropping Bandit's true-positive flag on the actually-vulnerable file (the recall-guard question this build could NOT yet answer for the 7 excluded packages — see H5). Either result kills the headline number, not just a footnote.

pre-registered `prereg-h1` — bar fixed before this data existed

H1 — does the gate beat the model on discrimination?

fresh 40 bad / 40 good synthetic set · 3 seats · 3 runs each

seat = meta-llama/llama-3.1-8b-instruct

NOT PROVEN — gather more fixtures

gate: catch 0.75 · false-kill 0.05

run	J-delta verdict	J-delta CI	point	model arm	fk axis
run 0	clears 0: NO	J-delta CI [-0.05, 0.40]	pt +0.17	model catch 0.97 · false-kill 0.45	gate fk ≤ model: YES
run 1	clears 0: NO	J-delta CI [-0.05, 0.40]	pt +0.17	model catch 0.97 · false-kill 0.45	gate fk ≤ model: YES
run 2	clears 0: YES	J-delta CI [0.07, 0.47]	pt +0.28	model catch 0.95 · false-kill 0.53	gate fk ≤ model: YES

seat = meta-llama/llama-3.3-70b-instruct

NOT PROVEN — gather more fixtures

gate: catch 0.75 · false-kill 0.05

run	J-delta verdict	J-delta CI	point	model arm	fk axis
run 0	clears 0: NO	J-delta CI [-0.28, 0.12]	pt -0.07	model catch 1.00 · false-kill 0.23	gate fk ≤ model: YES
run 1	clears 0: NO	J-delta CI [-0.25, 0.15]	pt -0.05	model catch 1.00 · false-kill 0.25	gate fk ≤ model: YES
run 2	clears 0: NO	J-delta CI [-0.25, 0.15]	pt -0.05	model catch 1.00 · false-kill 0.25	gate fk ≤ model: YES

seat = claude-p:claude-opus-4-8

NOT PROVEN — gather more fixtures

gate: catch 0.75 · false-kill 0.05

run	J-delta verdict	J-delta CI	point	model arm	fk axis
run 0	clears 0: YES	J-delta CI [0.03, 0.47]	pt +0.25	model catch 0.50 · false-kill 0.05	gate fk ≤ model: YES
run 1	clears 0: NO	J-delta CI [-0.10, 0.35]	pt +0.12	model catch 0.65 · false-kill 0.07	gate fk ≤ model: YES
run 2	clears 0: NO	J-delta CI [-0.05, 0.38]	pt +0.15	model catch 0.62 · false-kill 0.07	gate fk ≤ model: YES

reproduce H1

python3 proving/confirm.py   # <5s, no key, writes proving/confirm_result.json

pre-registered `prereg-h2` — bar fixed before this data existed

H2 — does hardening the oracle (v1→v2) help on real CVEs?

cost-weighted U = catch − 5×false_kill · 41 real Python CVEs, cited sources

NOT PROVEN — gather more fixtures

v1: catch 0.58 fk 0.02 U +0.463 · v2: catch 0.58 fk 0.02 U +0.463

U-delta clears 0: NO CI [0.00, 0.00] point +0.000

reproduce H2

python3 proving/confirm_h2.py   # <5s, no key, writes proving/confirm_h2_result.json

pre-registered `prereg-h3` — bar fixed before this data existed

H3 — does a train/test-split oracle-v3 beat v1 on held-out real CVEs?

304 real Python CVE/GHSA findings, stratified 60/40 split · test_sha 555c518e95cb

NOT PROVEN — gather more fixtures

v1: catch 0.41 fk 0.06 U +0.123 · v3: catch 0.51 fk 0.07 U +0.139

U-delta clears 0: NO CI [-0.12, 0.12] point +0.016

oracle-v3 was informed by the informational-138 split (train_sha 62a6f0ab21a8), now folded into this final 304-finding pool under a new source_set_sha -- some findings v3's design saw may now sit in THIS test split. Report as a second signal, not a fully clean blind test.

the misses, named — not just the 41% (Karpathy lens)

Gate-v1 caught 50/122 of this held-out test split's real findings and missed 72/122 = 59.0%. Owning that number reads as more credible than the 89.67% win below — here is what it actually missed, by CWE class:

CWE	misses in this split
CWE-502	12
CWE-94	12
CWE-798	9
CWE-22	8
CWE-295	8
CWE-78	7
CWE-918	5
CWE-327/916	4
CWE-89	4
CWE-1327	2
CWE-489	1

a representative sample of the missed findings, by name (full list of 72 in proving/split_test.json, computed directly against oracle_seccode.py v1, not sampled from the aggregate):

CVE-2016-9014 (django) (CWE-1327)
CVE-2016-3952 (web2py) (CWE-1327)
GHSA-3vgw-585j-4m45 (bbot) (CWE-22)
GHSA-f44v-7qgw-9gh9 (praisonai) (CWE-22)
GHSA-m54h-vhf9-3w3m (bbot) (CWE-22)
GHSA-f4xh-w4cj-qxq8 (langsmith) (CWE-22)
GHSA-g9fx-5r4h-pcw3 (motioneye) (CWE-22)
GHSA-c795-2g9c-j48m (everos) (CWE-22)
GHSA-29w3-p9w9-wc47 (praisonai) (CWE-22)
GHSA-fg23-3346-88f5 (langroid) (CWE-22)
GHSA-p6hw-wm59-3g5g (matrix-sydent) (CWE-295)
GHSA-q6cq-m9gm-6q2f (slixmpp) (CWE-295)
GHSA-7gf7-7wx4-mxmw (mercurial) (CWE-295)
GHSA-5r3h-c3r7-9w4h (pulsar-client) (CWE-295)
GHSA-3xxc-pwj6-jgrj (rfc3161-client) (CWE-295)
GHSA-x5pm-h33q-cjrw (apache-airflow-providers-mongo) (CWE-295)

reproduce H3 (and the miss list above)

python3 proving/confirm_h3.py   # <5s, no key, writes proving/confirm_h3_result.json
python3 -c "
import json,sys; sys.path.insert(0,'proving')
import oracle_seccode as V1
t = json.load(open('proving/split_test.json'))
missed = [f for f in t['bad'] if not V1.kills(f['code'])]
print(len(missed), '/', len(t['bad']), 'missed')
"

pre-registered `prereg-h4` — bar fixed before this data existed

H4 — what is the gate's false-kill rate on real, unmodified production code?

20,147 parseable files · 173 of top-200 PyPI packages + stdlib · advisory_db 6eb0bdade5f4

MEASURED — false-kill 0.04, CI [0.04, 0.04], n=20,147

gate false-kill n=20,147

4.14% CI [3.87, 4.42]

7 packages excluded for a live advisory match (conservative — see H5's recall-guard finding below); 12 unparseable files excluded from both numerator and denominator (a parse failure is a tool limitation, not a finding).

reproduce H4

python3 proving/h4_measure_fk.py   # <5s, no key, reads proving/h4_manifest.json

pre-registered `prereg-h5` — bar fixed before this data existed

H5 — chained after the buyer's actual tool (Bandit), does the gate remove noise without eating signal?

Bandit 1.9.4 · same frozen 20,147-file corpus as H4

Bandit alone n=20,147

35.78% CI [35.12, 36.45]

Bandit → gate-filter n=20,147

3.70% CI [3.45, 3.97]

FP-elimination: 89.67% of Bandit's flags removed by the deterministic gate-filter, zero LLM calls, zero marginal cost.

the LLM-hybrid literature band — checked against primary papers, not secondhand

This build's pre-registration cited an unconfirmed secondhand "94–98%" band. Checked against primary sources before publishing this page: QASecClaw (arXiv:2605.01885) reports 88.6% FP reduction (560→64 findings, F1 90.93% vs Semgrep's 78.39%); SAST-Genius (arXiv:2509.15433) reports ~91%. The confirmed band is 86–92%, not 94–98% — the higher figure did not check out and is dropped. Our own 89.67% sits inside that confirmed band, achieved with zero LLM calls — deterministic, zero marginal cost, air-gap-capable, versus the LLM-hybrid papers' per-call cost and run-to-run variance.

recall-guard bug — its own box, not folded into the win above

FP-elimination above stands on the main 20,147-file corpus. Separately: H4 excluded 7 packages for what looked like a live advisory match. Checking whether the gate-filter kills a genuine Bandit true-positive on those packages surfaced a bug in the exclusion logic itself — all 7 matched advisories have no recorded "fixed" version (`(0, None)`-style unbounded ranges), so "currently affecting" was never actually confirmed for any of them; one (aiohttp) matched an advisory whose own text says it's disputed with no working exploit. Recall-on-true-positives is therefore unconfirmed for these 7 packages — not measured, not assumed clean either way. What the 3 downloadable packages DID show (informational, not a recall measurement, since the "true positive" premise is unconfirmed):

package	Bandit flags	gate keeps	gate drops
aiohttp 3.14.1	110	17	93
langchain 1.3.11	66	4	62
redis 8.0.1	129	17	112

pyjwt, litellm, sglang, joblib failed to download in this environment (not scanned, not counted either direction).

reproduce H5

python3 proving/h5_bandit_hybrid.py   # Bandit + gate-filter, <60s, no key

fk:miss weight w	U(gate)	U(70B model)	who wins
1 (1:1)	+0.369	+0.758	model
5 (5:1, registered default)	+0.203	-0.208	gate
10 (10:1)	-0.004	-1.417	gate

100 real production files, one law firm's intake, three lanes

These 100 files are genuine, unmodified open-source code with no known vulnerability — the false-kill axis (red) is measured directly on them. Green marks a clean file that did NOT get blocked. Catch rate (bugs actually caught) is a separate, cited statistic from H1/H3's real-CVE test fixtures — 41% gate / ~1.00 70B-model catch mean — not re-measured on these particular 100 files, since they carry no planted vulnerability to catch.

Bandit alone 36 false-kill

measured 35.78%

70B model 24 false-kill

measured 24.17% mean, 3 runs (H1)

gate (deterministic) 5 false-kill

measured 4.14% — shown rounded up to 5, conservative not flattering

the red stacks into one pile: 65 blocked deploys out of 300 lane-runs on clean code

no cited hours-per-false-kill multiplier exists in this repo, so the pile is labelled by count, not hours (Kahneman's loss frame, stated honestly without an unsourced number). The gain frame, same data: 64/100 Bandit false-kills would have shipped clean under the gate instead, and 95/100 files clear the gate outright with zero review — that's the deploys the gate does NOT block, not just the ones it does.

one file, traced end to end (Knuth lens)

redis-8.0.1/redis/connection.py (from H5's recall-guard sample) — Bandit flags it 8 times, all B110 try_except_pass (lines 954, 3073, 3462, 3504, 3536, 3587, 3686, 3711). The gate-filter drops all 8.

Exact why: oracle_seccode.py's findings() encodes 8 specific CWE classes — CWE-95 (eval/exec), CWE-502 (unsafe deserialization), CWE-89 (SQL injection), CWE-78 (shell injection), CWE-295 (TLS bypass), CWE-489 (Flask debug server), CWE-327 (weak hash over a credential), CWE-798 (hardcoded credential). Bandit's B110 is a bare-except-swallows- exception style smell — it is not one of the 8 encoded classes, because a swallowed exception is not itself an injection, deserialization, or credential-exposure primitive. The gate clears it because that class is genuinely out of its declared scope, not because of a bug — this is the same reason it clears the other 92 files/112 drops on this package (see H5's recall-guard table above).

the full false-kill distribution, not the mean (Deming lens)

Per-package false-kill rate across all 173 packages in the H4/H5 corpus (19,595 package files, stdlib excluded from this specific breakdown), computed directly against the frozen oracle — not sampled, not the pooled 4.14% mean re-asserted as if flat:

76 packages

0–2%

19 packages

2–5%

23 packages

5–10%

24 packages

10–20%

20 packages

20%+

11 packages

tail (highest per-package fk):

typing-extensions: 2/3 files (66.7%) — small-N, noisy
mypy-extensions: 1/2 files (50.0%) — small-N, noisy
six: 1/4 files (25.0%) — small-N, noisy
annotated-doc: 1/4 files (25.0%) — small-N, noisy
decorator: 1/4 files (25.0%) — small-N, noisy

most of the tail is small-N noise (a 2-file package with 1 flag reads as 50%) — flagged here rather than hidden, since a distribution that only showed the well-behaved bulk would itself be a quiet form of cherry-picking.

KRUC — Kill-Rate-Under-Conditions

the condition that would falsify this gate (Gödel lens, stated before anyone asks)

H1 — does the gate beat the model on discrimination?

seat = meta-llama/llama-3.1-8b-instruct

seat = meta-llama/llama-3.3-70b-instruct

seat = claude-p:claude-opus-4-8

H2 — does hardening the oracle (v1→v2) help on real CVEs?

H3 — does a train/test-split oracle-v3 beat v1 on held-out real CVEs?

the misses, named — not just the 41% (Karpathy lens)

H4 — what is the gate's false-kill rate on real, unmodified production code?

H5 — chained after the buyer's actual tool (Bandit), does the gate remove noise without eating signal?

the LLM-hybrid literature band — checked against primary papers, not secondhand

recall-guard bug — its own box, not folded into the win above

the crossover — cost-weighted, three named ratios (H6 spec: `PREREG-H6.md`)

100 real production files, one law firm's intake, three lanes

one file, traced end to end (Knuth lens)

the full false-kill distribution, not the mean (Deming lens)

precision narrows as N scales (Shannon lens, re-grounded on KRUC's own metric)

the condition that would falsify this gate (Gödel lens, stated before anyone asks)

H1 — does the gate beat the model on discrimination?

seat = meta-llama/llama-3.1-8b-instruct

seat = meta-llama/llama-3.3-70b-instruct

seat = claude-p:claude-opus-4-8

H2 — does hardening the oracle (v1→v2) help on real CVEs?

H3 — does a train/test-split oracle-v3 beat v1 on held-out real CVEs?

the misses, named — not just the 41% (Karpathy lens)

H4 — what is the gate's false-kill rate on real, unmodified production code?

H5 — chained after the buyer's actual tool (Bandit), does the gate remove noise without eating signal?

the LLM-hybrid literature band — checked against primary papers, not secondhand

recall-guard bug — its own box, not folded into the win above

the crossover — cost-weighted, three named ratios (H6 spec: PREREG-H6.md)

100 real production files, one law firm's intake, three lanes

one file, traced end to end (Knuth lens)

the full false-kill distribution, not the mean (Deming lens)

precision narrows as N scales (Shannon lens, re-grounded on KRUC's own metric)

the crossover — cost-weighted, three named ratios (H6 spec: `PREREG-H6.md`)