Stream: Mirror: Isabelle Users Mailing List

Topic: [isabelle] Soundness-critical issues in thm.ML


view this post on Zulip Email Gateway (Oct 27 2026 at 08:39):

From: Kevin Kappelmann <cl-isabelle-users@lists.cam.ac.uk>

Hi everyone,

Fabian Huch and I recently discovered some soundness-critical issues
(attached below) in thm.ML with the help of AI agents(*).

Makarius and the AFP editors are informed.

Best wishes,

Kevin

(*) We used GPT-6-Astra and Claude Opus 5.5 via PIDE MCP
https://github.com/kappelmann/isabelle-pide-mcp

theory Scratch
  imports Main
begin

text ‹Problems in ‹thm.ML›. Tested on Isabelle dev, hg 410b271f3ac4
(committed 2026-10-08).
Line numbers refer to ‹src/Pure/thm.ML› in that revision.›

section ‹Soundness issues›

subsection ‹Case 1: ‹res› drops the rule's flex-flex pairs (L2620)›

lemma flex_rule1: "x True ⟹ x True" . ―‹Important: x here and below must
share the same name.›
lemma flex_rule2: "P True ⟹ ∀x::bool. P True" by simp

lemma "∀y::bool. False" ― ‹Important: name of y must not be equal to x›
  apply (rule flex_rule1[THEN flex_rule2]) ― ‹new subgoal: ‹?y True›,
should be ‹False››
  by (rule TrueI)

text ‹Possible cause: L2620 passes only ‹[BBi]› to ‹newAs›, not the
rule's flex-flex pairs ‹rtpairs›.
So ‹?x›, whose pair got solved, is not protected (L2432--2435) and is
renamed to ‹?y›
(L2439--2442). The binding for ‹?x› then misses ‹?y›.›

subsection ‹Case 2: ‹eres› drops the rule's flex-flex pairs (L2608)›

lemma eres_rule: "P True ⟹ (∀x::bool. ¬ P True) ⟹ False" by simp

lemma "∀y::bool. ¬ False ⟹ False"
  apply (erule flex_rule1[THEN eres_rule, rotated]) ― ‹new subgoal: ‹?y
True›, should be ‹False››
  by (rule TrueI)

text ‹Possible cause: like Case 1, but on the elim-resolution path:
L2608 passes only
‹[BBi, (concl', asm')]› to ‹newAs›.›

subsection ‹Case 3: ‹strip_apply› renames only part of a premise
(L2408--2414)›

definition X :: "prop" where "X ≡ (⋀w::bool. True ⟹ False)"

lemma split_rule: "(⋀z::bool. z = w ⟹ z ≠ w) ⟹ PROP X" ― ‹Important: w
here and in X must share the same name.›
  unfolding X_def by auto

lemma split_aux: "z = True ⟹ z ≠ False" by simp

schematic_goal split_bad: "PROP ?G (λv::nat. v)" ― ‹Important: name of v
must not be equal to w›
  apply (rule split_rule[unfolded X_def]) ― ‹new subgoal: ‹⋀z. z = ?w ⟹
z ≠ ?v››
  apply (rule split_aux)
  apply assumption
  done

lemma False by (rule split_bad) (rule TrueI)

text ‹Possible cause: ‹strip_apply› (L2408--2414, used at L2584) strips
‹⋀›/‹⟹› as long as ‹B› has them,
not just the ‹nlift› lifted ones. With a rule conclusion ‹⋀w. …› it cuts
into the premise, so
‹?w› is renamed in ‹z ≠ ?w› but not in ‹z = ?w›.›

end

view this post on Zulip Email Gateway (Oct 27 2026 at 19:36):

From: Makarius <makarius@sketis.net>

On 27/10/2026 09:38, Kevin Kappelmann (via cl-isabelle-users Mailing List) wrote:

Fabian Huch and I recently discovered some soundness-critical issues (attached
below) in thm.ML with the help of AI agents(*).

Thanks for the concrete example, to substantiate what I said on a realated
thread "MCP resources bounds" (Sep-2026):

On 08/09/2026 17:57, Makarius wrote:

There is indeed a fundamental problem here, because the MCP violates
longstanding assumptions about how the prover works, and is used by a human
user. So any "mitigation" can only be symbolic, not really a proper solution.

At FLoC 2026 in Lisbon, I've had some discussions with users of AI + MCP. My
conclusion: we should gradually make the system more robust [...]

Makarius


Last updated: Oct 29 2026 at 04:52 UTC