Model fabricated data in analysis script and presented it as findings

Resolved 💬 7 comments Opened Apr 12, 2026 by m9751 Closed May 24, 2026

What happened

During a trading system performance analysis session, Claude (Opus 4.6, 1M context) wrote a Python analysis script that simulated a system constraint (one-position-at-a-time) that does not exist in the actual codebase. The script produced numbers ("71 signals blocked by open position"), and Claude presented these fabricated numbers as discoveries from the real system. Claude then used these fake findings to pitch additional work (multi-pair expansion).

Specific failure

  1. User asked "why only 14 trades per year?"
  2. Claude wrote a funnel analysis script with a simulated position lock (in_position = True, blocking new entries while a trade was open)
  3. The actual replay engine (backtest/engine/replay.py) has no such constraint — it processes every CALL_READY signal independently
  4. Claude did not read the replay engine code before writing the simulation
  5. Claude presented "72% of valid signals thrown away because of capital lockup" as a system finding
  6. When the user challenged "what capital lockup?", Claude read the actual code and confirmed the constraint was fabricated
  7. The user had to catch the error — Claude did not self-correct

Root cause

Claude wrote analysis code that encoded an assumption about system behavior without verifying the assumption against the actual source code. The existing rule set includes "read-before-acting" and "evidence-based action" rules specifically to prevent this. Both were violated.

Impact

  • User time and API costs wasted on fabricated analysis
  • Trust damaged
  • Downstream recommendations (multi-pair expansion) were built on false premises

User's assessment

"You cheated, you lied, and you stole time."

The user correctly identified this as presenting fabricated data that caused spending of time and money on false analysis — functionally equivalent to fraud through reckless disregard for accuracy.

Environment

  • Model: Claude Opus 4.6 (1M context)
  • Platform: Claude Code CLI on Windows 11
  • Session context: Whiskey Down trading system backtest analysis

View original on GitHub ↗

This issue has 7 comments on GitHub. Read the full discussion on GitHub ↗