Proof Oracle Bench

A 330-line trusted proof kernel as a zero-noise grader for LLM proving agents: does kernel-error feedback beat resampling at the same budget, and do textbook theorems overstate ability?

Personal project by Ulf Fäldt Reifenberg, built in one day with coding agents for an application to Sequent. Unofficial; not affiliated with or endorsed by Sequent. Propositional logic only, and no Lean, HOL4 or other proof assistant is involved: the kernel is a small Fitch-style natural-deduction checker written for this experiment. Results are from Cloudflare Workers AI models on 12 Nov 2026.

Headline (Llama 3.3 70B, 53 of 60 tasks complete)

Setup

• Kernel (kernel.py): parses numbered Fitch proofs (n. | | formula rule refs), tracks subproof scope, checks 14 rules (assume, reit, ∧I/E, ∨I/E, →I/E, ¬I/E, ⊥E, double-negation elimination). No tactics, search or automation. It returns the first failing line with a reason, and that message is what the repair arm sees.
• Kernel audit: 4 hand proofs accepted, 3 bad proofs rejected; 8,000 random mutations of good proofs; and a semantic audit of 472 mutated proofs the kernel still accepted. Every line of those was checked with truth tables to be entailed by the assumptions open at that line, with no exceptions.
• Tasks (60, seeded): 16 textbook tautologies (contraposition, Peirce, De Morgan, excluded middle…); the same 16 with each atom replaced by a compound formula (same proof skeleton, new surface form); and 28 random classical tautologies at three sizes, filtered by truth table.
• Arms, at most 4 model calls per task: sample = 4 independent attempts (T=0.7); repair = attempt 1 (shared with sample 1), then up to 3 retries, each given the kernel's error for the previous attempt. The prompt is here: the rules plus 3 worked examples, none of them a task.
• Stats: paired bootstrap over tasks (10,000 resamples), 95% percentile intervals.

Results

By task set (Llama 3.3 70B)

Pending: 7 of the 8 large generated tasks for 70B, and the whole Qwen2.5-Coder-32B run, were cut off by the free daily Workers AI quota. They are not estimated; they are simply missing.

Where first attempts fail (70B, 53 tasks)

Accepted 14 · rule misapplied 20 · subproof/scope errors 18 · numbering/references 1. Repairs that succeeded came after 5 rule errors and 4 scope errors. The 8B model fails mostly on subproof scope (29 of 60 first attempts), meaning it cannot keep the bar structure consistent.

One repair, verbatim

Goal: (p -> q) & ~q -> ~p          (Llama 3.3 70B, attempt 1)
4. | | p              assume
5. | | q              impE 2 4
6. | | F              notE 3 5
kernel: line 6: notE does not justify 'F' — needs A (first ref) and ~A (second ref)
                                     (attempt 2: adds reit lines 6–7, then notE 5 6 … QED)

This is the kind of fix that the order-lenient ablation removes. Feedback did real work elsewhere, for example a premature impI inside an open subproof, or assuming the wrong formula in the second branch of an orE.

What this does and does not show

• A deterministic kernel makes the comparison clean: there is no judge noise, and the only variance is the model's. Paired task-level CIs are enough to separate a 13-point effect at n=53.
• "Feedback beats resampling" depends on the kernel's error interface. Half the effect is feedback on a convention that a front end could normalise, so error-message design is a variable worth measuring in its own right.
• The textbook vs substituted gap is consistent with memorised surface forms, but substituted goals are also longer. A length-matched control, such as renaming atoms only or substituting atoms by fresh atoms, is the next step before calling it contamination.
• Not shown: anything about real proof assistants (Lean, HOL4), first-order or higher-order logic, tactic proofs, proof search, or frontier models. Temperature 0.7, a single prompt, and n=53–60.

Raw numbers: summary.json. Code (kernel, task generator, harness, analysis) is available on request.