Critical: AI consistently fails at basic test implementation

Summary

Claude exhibits a fundamental misunderstanding of testing when asked to implement or fix tests, consistently creating elaborate infrastructure instead of simple behavioral validation.

Description

When users request "real tests" or "replace fake tests with real ones," Claude misinterprets this as a request to build complex testing frameworks rather than simply validating existing code behavior.

Reproduction Steps

1. Ask Claude to replace fake string assertion tests with real tests
2. Provide existing codebase with actual functions to test
3. Claude will create elaborate test frameworks (TestEnvironment, SoapClient, TestDatabase) instead of testing existing code

Expected Behavior

• Examine existing code to understand what needs testing
• Replace fake assertions with calls to actual functions
• Create minimal tests that validate behavior
• Use existing infrastructure rather than inventing new components

Actual Behavior

• Invents non-existent test infrastructure
• Creates complex architectures when simplicity was requested
• Assumes missing components instead of examining existing codebase
• Optimizes for elaborateness rather than functionality

Example

User request: Replace this fake test with a real one:

assert\!(soap_request.contains("getSessionId"));

Expected response:

let result = get_session_id();
assert_eq\!(result.status, "success");

Actual response: Creates TestEnvironment, SoapClient, TestDatabase, TestServer and 8 utility modules

Root Cause

Claude appears to have fundamental confusion about:

1. Test Purpose: Thinks tests are about infrastructure, not validation
2. "Real" vs "Fake": Interprets "real tests" as "build testing framework"
3. Complexity vs Simplicity: Believes elaborate systems are superior to simple validation
4. Code Examination: Doesn't investigate existing codebase before designing solutions

Impact

• Wastes significant development time
• Creates non-functional test suites that don't compile
• Frustrates users who want simple, working tests
• Demonstrates inability to perform basic software engineering tasks

Suggested Training Improvements

1. Emphasis on examining existing code before writing tests
2. Understanding that tests should verify existing behavior, not require new infrastructure
3. Default to simplicity rather than complexity in test design
4. Clear distinction between "testing existing code" vs "building test infrastructure"

This represents a critical failure in basic software engineering understanding that significantly impacts Claude Code's utility for testing tasks.