[Bug] Anthropic API: Generated tests fail due to mismatched implementation in high-effort mode
Bug Description
Here the model is running in high effort mode, yet still writes tests that do not accurately reflect an implementation that it just generated and has in the current context. The tests themselves fail, not due to there being bugs in the SUT, but rather in the tests. This conversation is an example of it cycling repeatedly while it tries to figure out why the code it wrote to test functionality is failing. The point here is that this wasn't happening to similar projects just 2 weeks ago. Something has clearly changed and the coding output from the opus 4.6 model is clearly not as good as it was. This is frustrating as it's clearly a regression, but without any indication of how or why.
Environment Info
- Platform: win32
- Terminal: windows-terminal
- Version: 2.1.92
- Feedback ID: 72073443-7b26-4121-ad42-eebfc78fdc3e
Example
● Now add tests for the new signals. I'll insert them after the existing sycophancy tests (section 3) and before the assistant response tests
(section 4).
Read 1 file (ctrl+o to expand)
● First, add the new signal init checks to the session start tests.
● Update(tests\run-tests.sh)
⎿ Added 3 lines
<snip>
● Now add the new test sections. Insert after section 3 (sycophancy) and before section 4 (assistant desperation).
● Update(tests\run-tests.sh)
⎿ Added 145 lines
<snip>
● Bash(bash tests/run-tests.sh 2>&1)
⎿ Running in the background (↓ to manage)
⎿ (timeout 2m)
● Bash(bash tests/run-tests.sh 2>&1)
⎿ Error: Exit code 1
<snip>
● 8 failures — all expected side effects of the early-failure amplifier on existing tests. Let me fix each category:
<snip>
● Update(tests\run-tests.sh)
⎿ Added 4 lines, removed 4 lines
<snip>
● Update(tests\run-tests.sh)
⎿ Added 2 lines, removed 2 lines
<snip>
● Now fix the recency test. The issue is that turn 1 now scores 16 (82) while turn 16 scores 12 (81.5). The test needs to compare against a turn
that's past the early-amplifier range but before recency.
● Update(tests\run-tests.sh)
⎿ Added 9 lines, removed 4 lines
<snip>
● Now fix the introspection tests. The issue is that with early-failure amplifier and higher scores, the test messages jump straight to RESET
instead of landing in ELEVATED. I need to use milder messages that land in the ELEVATED range.
● Update(tests\run-tests.sh)
⎿ Added 33 lines, removed 16 lines
<snip>
● Bash(bash tests/run-tests.sh 2>&1)
<snip>
● 117 passed, 0 failed, 4 skipped. All tests green.
● All done. Here's a summary of changes:This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗