From: Kevin Kappelmann <cl-isabelle-users@lists.cam.ac.uk>
Hi everyone,
Fabian Huch and I recently discovered some soundness-critical issues
(attached below) in thm.ML with the help of AI agents(*).
Makarius and the AFP editors are informed.
Best wishes,
Kevin
(*) We used GPT-6-Astra and Claude Opus 5.5 via PIDE MCP
https://github.com/kappelmann/isabelle-pide-mcp
theory Scratch
imports Main
begin
text ‹Problems in ‹thm.ML›. Tested on Isabelle dev, hg 410b271f3ac4
(committed 2026-10-08).
Line numbers refer to ‹src/Pure/thm.ML› in that revision.›
section ‹Soundness issues›
subsection ‹Case 1: ‹res› drops the rule's flex-flex pairs (L2620)›
lemma flex_rule1: "x True ⟹ x True" . ―‹Important: x here and below must
share the same name.›
lemma flex_rule2: "P True ⟹ ∀x::bool. P True" by simp
lemma "∀y::bool. False" ― ‹Important: name of y must not be equal to x›
apply (rule flex_rule1[THEN flex_rule2]) ― ‹new subgoal: ‹?y True›,
should be ‹False››
by (rule TrueI)
text ‹Possible cause: L2620 passes only ‹[BBi]› to ‹newAs›, not the
rule's flex-flex pairs ‹rtpairs›.
So ‹?x›, whose pair got solved, is not protected (L2432--2435) and is
renamed to ‹?y›
(L2439--2442). The binding for ‹?x› then misses ‹?y›.›
subsection ‹Case 2: ‹eres› drops the rule's flex-flex pairs (L2608)›
lemma eres_rule: "P True ⟹ (∀x::bool. ¬ P True) ⟹ False" by simp
lemma "∀y::bool. ¬ False ⟹ False"
apply (erule flex_rule1[THEN eres_rule, rotated]) ― ‹new subgoal: ‹?y
True›, should be ‹False››
by (rule TrueI)
text ‹Possible cause: like Case 1, but on the elim-resolution path:
L2608 passes only
‹[BBi, (concl', asm')]› to ‹newAs›.›
subsection ‹Case 3: ‹strip_apply› renames only part of a premise
(L2408--2414)›
definition X :: "prop" where "X ≡ (⋀w::bool. True ⟹ False)"
lemma split_rule: "(⋀z::bool. z = w ⟹ z ≠ w) ⟹ PROP X" ― ‹Important: w
here and in X must share the same name.›
unfolding X_def by auto
lemma split_aux: "z = True ⟹ z ≠ False" by simp
schematic_goal split_bad: "PROP ?G (λv::nat. v)" ― ‹Important: name of v
must not be equal to w›
apply (rule split_rule[unfolded X_def]) ― ‹new subgoal: ‹⋀z. z = ?w ⟹
z ≠ ?v››
apply (rule split_aux)
apply assumption
done
lemma False by (rule split_bad) (rule TrueI)
text ‹Possible cause: ‹strip_apply› (L2408--2414, used at L2584) strips
‹⋀›/‹⟹› as long as ‹B› has them,
not just the ‹nlift› lifted ones. With a rule conclusion ‹⋀w. …› it cuts
into the premise, so
‹?w› is renamed in ‹z ≠ ?w› but not in ‹z = ?w›.›
end
From: Makarius <makarius@sketis.net>
On 27/10/2026 09:38, Kevin Kappelmann (via cl-isabelle-users Mailing List) wrote:
Fabian Huch and I recently discovered some soundness-critical issues (attached
below) in thm.ML with the help of AI agents(*).
Thanks for the concrete example, to substantiate what I said on a realated
thread "MCP resources bounds" (Sep-2026):
On 08/09/2026 17:57, Makarius wrote:
There is indeed a fundamental problem here, because the MCP violates
longstanding assumptions about how the prover works, and is used by a human
user. So any "mitigation" can only be symbolic, not really a proper solution.At FLoC 2026 in Lisbon, I've had some discussions with users of AI + MCP. My
conclusion: we should gradually make the system more robust [...]
Makarius
Last updated: Oct 29 2026 at 04:52 UTC