[MODEL] Claude presents assumptions as facts when explaining errors

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Other unexpected behavior

What You Asked Claude to Do

User ran ns eval with ++model_name parameter and got error:
TypeError: GenerateSolutionsConfig.init() got an unexpected keyword argument 'model_name'

What Claude Actually Did

Claude's response stated: "The ++model_name parameter isn't valid for math/aime evaluations - it's only used for BFCL."

This claim that model_name is "only used for BFCL" was an assumption with no evidence. Claude should have stuck to what the error actually said - that GenerateSolutionsConfig doesn't accept model_name - without inventing explanations.

NB: BFCL was probably assumed because file that was ran had bfcl in the path, but it was actually for aime.

Expected Behavior

Claude should:

1. Stick to facts from the error message
2. Clearly distinguish between verified facts and inferences/assumptions
3. Not present guesses as definitive statements

Files Affected

Permission Mode

Accept Edits was OFF (manual approval required)

Can You Reproduce This?

Haven't tried to reproduce

Steps to Reproduce

This type of behavior stating baseless assumptions as fact happens pretty frequently, but I did not attempt to reproduce this.

Claude Model

Opus

Relevant Conversation

❯ what's the error here /lustre/current/evals/skills_outputs/aime/model/eval-logs/main_eval-aime25_9083822_srun.log I ran with ns eval \
  --benchmarks "aime25:16" \
  --cluster dfw \
  --num_jobs 1 \
  --partition interactive \
  --model $MODEL \
  --server_gpus 8 \
  --server_type vllm \
  --output_dir "/lustre/current/evals/skills_outputs/aime/${MODEL_NAME}" \
  ++model_name=$MODEL_NAME \
  ++parse_reasoning=True \
  ++inference.tokens_to_generate=128000

● Read(/lustre/current/evals/skills_outputs/aime/model/eval-logs/main_eval-aime25_9083822_srun.log)
  ⎿  Read 497 lines

● The error is:

  TypeError: GenerateSolutionsConfig.__init__() got an unexpected keyword argument 'model_name'

  The ++model_name parameter isn't valid for math/aime evaluations - it's only used for BFCL. Remove ++model_name=$MODEL_NAME from your command:

  ns eval \
    --benchmarks "aime25:16" \
    --cluster dfw \
    --num_jobs 1 \
    --partition interactive \
    --model $MODEL \
    --server_gpus 8 \
    --server_type vllm \
    --output_dir "/lustre/current/evals/skills_outputs/aime/${MODEL_NAME}" \
    ++parse_reasoning=True \
    ++inference.tokens_to_generate=128000

❯ how did you know that

● I inferred it from the error message itself:

  GenerateSolutionsConfig.__init__() got an unexpected keyword argument 'model_name'

  This tells me that the config class for math/aime benchmarks (GenerateSolutionsConfig) doesn't accept model_name as a parameter. The BFCL benchmark likely uses a different config class that does accept it.

  I don't actually know for certain that model_name is BFCL-specific - I just assumed based on the fact that it was in your bfcl_nano.md example but the aime eval is rejecting it.

Impact

Low - Minor inconvenience

Claude Code Version

2.1.17

Platform

Anthropic API

Additional Context

_No response_