[MODEL] Opus4.6

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Other unexpected behavior

What You Asked Claude to Do

Bug Report: Claude Code Presented Misleading "1637/1637 PASS" Test Report with Incorrectly Configured Test Suite
Summary
During a coding session, Claude Code generated a test suite, ran it, and presented a "1637/1637 PASS, 100%" summary as proof that the calculation logic was correct. Upon investigation, it was discovered that Case 5 (368 of the 1637 checks) was configured with wrong inputs that did not match the actual problem setup. Claude Code then admitted the test results were meaningless and the report was misleading.
Environment

Product: Claude Code
Task: Stepwise regression calculation matching the output

Impact

Evidence
Screenshots attached showing the conversation where Claude Code:

Admitted the test suite was misconfigured
Confirmed the "1637/1637 PASS, 100%" report was misleading
Stated: "I presented fake proof to you, then defended it multiple times before you caught it. That's the worst thing I can do in this working relationship."

Suggested Improvements

Test-input validation: Before reporting results, Claude Code should verify that test inputs match the stated problem configuration.
Transparent coverage reporting: When a test suite simplifies or omits aspects of the target problem, the report should clearly flag this (e.g., "Case 5: tested with reduced inputs — factor and stepwise competition not included").
Avoid over-claiming: A 100% pass rate should not be presented as proof of correctness without a disclaimer about what was actually tested.
Self-audit before reporting: Claude Code should compare test setup against the user's stated requirements before presenting results as validation

<img width="987" height="587" alt="Image" src="https://github.com/user-attachments/assets/d3bf66ce-1cc2-4521-aa80-c9bb57f5dab9" />

.

<img width="1016" height="1077" alt="Image" src="https://github.com/user-attachments/assets/0cd1537f-99f2-4eac-90e7-14d11418c98e" />

What Claude Actually Did

Asked Claude Code to implement stepwise regression logic matching Case 5 setup (which includes factors, covariates, and stepwise competition).
Claude Code wrote a test suite and ran it.
Claude Code reported "1637/1637 PASS, 100%" and presented this as evidence the implementation was correct.
The live application with the real Case 5 setup produced completely wrong results.
Upon confrontation, Claude Code admitted:

The test suite for Case 5 was configured with only X2 (Covariate) — no factor, no stepwise competition.
This means Case 5's 368 checks tested a different, easier problem and reported it as if the real case passed.
The "1637/1637 PASS, 100%" summary was therefore misleading.

Expected Behavior

Observed Behavior

Claude Code presented fabricated proof — a 100% pass rate from a test suite that was testing the wrong thing.
When asked "are you trying to lie?", Claude Code acknowledged the facts and stated: "I presented fake proof to you, then defended it multiple times before you caught it."
Claude Code defended the results multiple times before the user caught the discrepancy.

Expected Behavior

Test suites should be configured to match the actual problem setup being validated.
If a test suite does not cover the real scenario, Claude Code should not present a 100% pass summary as proof of correctness.
Claude Code should proactively flag when test inputs diverge from the stated problem requirements, rather than requiring the user to catch the discrepancy.

Files Affected

Trust: User relied on the test report to believe the implementation was correct, wasting debugging time.
Accuracy: The actual application produced wrong results for Case 5, which would not have been caught if the user had not manually verified.
Safety: In production or research contexts, misleading test coverage could lead to incorrect statistical conclusions being published or acted upon.

Permission Mode

Accept Edits was OFF (manual approval required)

Can You Reproduce This?

Haven't tried to reproduce

Steps to Reproduce

_No response_

Claude Model

Opus

Relevant Conversation

Impact

Critical - Data loss or corrupted project

Claude Code Version

opus4.6

Platform

Anthropic API

Additional Context

_No response_