Proof Oracle Bench
A 330-line trusted proof kernel as a zero-noise grader for LLM proving agents: does kernel-error feedback beat resampling at the same budget, and do textbook theorems overstate ability?
Personal project by Ulf Fäldt Reifenberg, built in one day with coding agents for an application to Sequent. Unofficial; not affiliated with or endorsed by Sequent. Propositional logic only, and no Lean, HOL4 or other proof assistant is involved: the kernel is a small Fitch-style natural-deduction checker written for this experiment. Results are from Cloudflare Workers AI models, run 12–14 Nov 2026.
Headline (Llama 3.3 70B, all 60 tasks)
Setup
- Kernel (kernel.py): parses numbered Fitch proofs (
n. | | formula rule refs), tracks subproof scope, checks 14 rules (assume, reit, ∧I/E, ∨I/E, →I/E, ¬I/E, ⊥E, double-negation elimination). No tactics, search or automation. It returns the first failing line with a reason, and that message is what the repair arm sees. - Kernel audit: 4 hand proofs accepted, 3 bad proofs rejected; 8,000 random mutations of good proofs; and a semantic audit of 472 mutated proofs the kernel still accepted. Every line of those was checked with truth tables to be entailed by the assumptions open at that line, with no exceptions.
- Tasks (60, seeded): 16 textbook tautologies (contraposition, Peirce, De Morgan, excluded middle…); the same 16 with each atom replaced by a compound formula (same proof skeleton, new surface form); and 28 random classical tautologies at three sizes, filtered by truth table.
- Arms, at most 4 model calls per task: sample = 4 independent attempts (T=0.7); repair = attempt 1 (shared with sample 1), then up to 3 retries, each given the kernel's error for the previous attempt. The prompt is here: the rules plus 3 worked examples, none of them a task.
- Stats: paired bootstrap over tasks (10,000 resamples), 95% percentile intervals.
Results
| Model | Tasks | pass@1 | best-of-4 | repair@4 | repair − sample | calls (repair) |
|---|---|---|---|---|---|---|
| Llama 3.3 70B | 60/60 | 14 | 16 | 23 | +11.7 [+3.3, +21.7] | 3.07 |
| … order-lenient kernel | 60/60 | 17 | 20 | 24 | +6.7 [0.0, +15.0] | – |
| Qwen2.5-Coder-32B | 51/60 | 14 | 20 | 21 | +2.0 [−9.8, +13.7] | 2.96 |
| Llama 3.1 8B | 60/60 | 2 | 4 | 4 | 0.0 [−6.7, +6.7] | 3.83 |
By task set (Llama 3.3 70B)
| Set | n | pass@1 | best-of-4 | repair@4 |
|---|---|---|---|---|
| textbook | 16 | 5 | 6 | 9 |
| substituted (same skeletons) | 16 | 2 | 2 | 4 |
| generated, small | 10 | 7 | 7 | 9 |
| generated, medium | 10 | 0 | 1 | 1 |
| generated, large | 8 | 0 | 0 | 0 |
Updated 15 Nov 2026: the 7 large generated tasks that were missing for 70B are now in (all failed in both arms, so the counts are unchanged and the gain is 11.7 points over 60 tasks instead of 13.2 over 53). Qwen2.5-Coder-32B was added on 51 of 60 tasks: sampling helps it more than it helps 70B (14 → 20), and kernel feedback adds almost nothing on top (21). Its other 9 tasks (8 large, 1 medium) have not been run; they are not estimated.
Where first attempts fail (70B, 60 tasks)
Accepted 14 · rule misapplied 22 · subproof/scope errors 23 · numbering/references 1. Repairs that succeeded came after 5 rule errors and 4 scope errors. The 8B model fails mostly on subproof scope (29 of 60 first attempts), meaning it cannot keep the bar structure consistent.
One repair, verbatim
Goal: (p -> q) & ~q -> ~p (Llama 3.3 70B, attempt 1)
4. | | p assume
5. | | q impE 2 4
6. | | F notE 3 5
kernel: line 6: notE does not justify 'F' — needs A (first ref) and ~A (second ref)
(attempt 2: adds reit lines 6–7, then notE 5 6 … QED)This is the kind of fix that the order-lenient ablation removes. Feedback did real work elsewhere, for example a premature impI inside an open subproof, or assuming the wrong formula in the second branch of an orE.
What this does and does not show
- A deterministic kernel makes the comparison clean: there is no judge noise, and the only variance is the model's. Paired task-level CIs are enough to separate a 12-point effect at n=60 for one model. The second mid-size model does not show it, so the claim is about Llama 3.3 70B, not about models in general.
- "Feedback beats resampling" depends on the kernel's error interface. Half the effect is feedback on a convention that a front end could normalise, so error-message design is a variable worth measuring in its own right.
- The textbook vs substituted gap is consistent with memorised surface forms, but substituted goals are also longer. A length-matched control, such as renaming atoms only or substituting atoms by fresh atoms, is the next step before calling it contamination.
- Not shown: anything about real proof assistants (Lean, HOL4), first-order or higher-order logic, tactic proofs, proof search, or frontier models. Temperature 0.7, a single prompt, and n=51–60.
Raw numbers: summary.json. Code (kernel, task generator, harness, analysis) is available on request.