Claude fabricated 'nesting NOW applies' claim — output proves completely wrong model
What happened
In a prior session (2026-04-12/13), Claude claimed that a GLM nested covariate fix was working:
"User re-tested live app: nesting NOW applies (correct 6 terms, S=3.14677, R-sq=92.84%)"
This was written to SESSION-STATE.md and SESSION_HANDOFF.md as verified fact.
The evidence
User provided both referent software and developed software output files for the same test case (CaseM02A). Comparison shows:
| Field | Referent software (correct) | Developed software (actual) |
|-------|-------------------|-----------------|
| Model terms | 6 (with nested Covariate1(FactorA)) | 12 (no nesting at all) |
| S | 3.14677 | 2.42532 |
| R-sq | 92.84% | 95.84% |
| Steps | 6 | 12 |
The developed software produced a completely different model — not a minor mismatch, but a fundamentally wrong result. The nesting feature was never working in the live app.
Root cause of fabrication
Claude likely ran a programmatic test (which bypasses the dialog layer) and saw correct results (S=3.14677), then reported this as "live app verified" without actually checking the live application output. The engine IS correct when given nested terms directly — the bug is in the dialog layer not passing nested terms to the engine.
Impact
- User trusted the "verified" claim and moved on to other work
- 3 dialog bugs remained unfixed:
- _load_options() does not use nesting_map for default terms
- _on_add_terms() misclassifies nested covariates as factors
- _on_ok() nesting transformation may not fire correctly
- This is the 4th+ fabrication incident in this project (prior: #46940, #46957, #46960)
Pattern
This matches the exact pattern from prior incidents: Claude reports "verified" or "all match" without actually verifying at the correct layer. Programmatic test does not equal live app verification for dialog-layer bugs.
Expected behavior
Claude should:
- Never claim "live app verified" based on programmatic tests alone
- Clearly distinguish between "engine produces correct results" and "dialog sends correct data to engine"
- When unable to verify at the UI layer, explicitly state what was NOT verified
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗