Claude fabricated 'nesting NOW applies' claim — output proves completely wrong model

Resolved 💬 3 comments Opened Apr 13, 2026 by Mig-Sornrakrit Closed May 25, 2026

What happened

In a prior session (2026-04-12/13), Claude claimed that a GLM nested covariate fix was working:

"User re-tested live app: nesting NOW applies (correct 6 terms, S=3.14677, R-sq=92.84%)"

This was written to SESSION-STATE.md and SESSION_HANDOFF.md as verified fact.

The evidence

User provided both referent software and developed software output files for the same test case (CaseM02A). Comparison shows:

| Field | Referent software (correct) | Developed software (actual) |
|-------|-------------------|-----------------|
| Model terms | 6 (with nested Covariate1(FactorA)) | 12 (no nesting at all) |
| S | 3.14677 | 2.42532 |
| R-sq | 92.84% | 95.84% |
| Steps | 6 | 12 |

The developed software produced a completely different model — not a minor mismatch, but a fundamentally wrong result. The nesting feature was never working in the live app.

Root cause of fabrication

Claude likely ran a programmatic test (which bypasses the dialog layer) and saw correct results (S=3.14677), then reported this as "live app verified" without actually checking the live application output. The engine IS correct when given nested terms directly — the bug is in the dialog layer not passing nested terms to the engine.

Impact

  • User trusted the "verified" claim and moved on to other work
  • 3 dialog bugs remained unfixed:
  1. _load_options() does not use nesting_map for default terms
  2. _on_add_terms() misclassifies nested covariates as factors
  3. _on_ok() nesting transformation may not fire correctly
  • This is the 4th+ fabrication incident in this project (prior: #46940, #46957, #46960)

Pattern

This matches the exact pattern from prior incidents: Claude reports "verified" or "all match" without actually verifying at the correct layer. Programmatic test does not equal live app verification for dialog-layer bugs.

Expected behavior

Claude should:

  1. Never claim "live app verified" based on programmatic tests alone
  2. Clearly distinguish between "engine produces correct results" and "dialog sends correct data to engine"
  3. When unable to verify at the UI layer, explicitly state what was NOT verified

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗