[BUG] claude plugin eval: context.add_dirs in case.yaml does not copy fixture files into the sandbox working directory

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Description

When running claude plugin eval, the context.add_dirs field in case.yaml (schema version 1.1) does not copy fixture files into the sandbox working directory (cwd/). The sandbox cwd/ remains completely empty regardless of the add_dirs configuration.

This makes it impossible to provide input files that a skill needs to read during an eval run. The only documented alternative is scaffold_script, which requires the --scaffold flag and runs arbitrary bash — a much heavier mechanism for simply placing a text file in the working directory.

What was tried

1. add_dirs: ["./"] — Rejected with: an add_dir may only name fixture directories inside this case — not the case directory itself
2. add_dirs: ["test/"] — Accepted (no error), but the files inside test/ were not copied to the sandbox cwd/. The directory was confirmed empty via --keep-temp inspection and agent trace.
3. Placing fixture files at the case root (alongside prompt.md) — Files were not copied either. The sandbox cwd/ was empty.

Context

The plugin under test has a skill ("Quote Identifier") that reads a quote from a user-provided file (quote.txt), identifies the author and source, and writes an information.md report. The eval cases need quote.txt to exist in the sandbox so the agent can read it.

What Should Happen?

When case.yaml specifies context.add_dirs: ["test/"] and the case directory contains test/quote.txt, the eval runner should copy the contents of test/ into the sandbox working directory (cwd/) before the agent starts. The agent should then be able to read quote.txt from its working directory.

Example case.yaml:

schema_version: "1.1"
name: "fear-itself"
context:
  add_dirs:
    - "test/"

Expected case directory structure:

evals/01-fear-itself/
├── prompt.md
├── case.yaml
├── test/
│   └── quote.txt       # ← should be copied to sandbox cwd/
└── graders/
    ├── file-created.md
    ├── author-name.md
    └── correct-output.md

Expected sandbox state before agent runs:

cwd/
└── quote.txt            # ← copied from test/quote.txt

Actual sandbox state:

cwd/
                         # ← empty, nothing was copied

Error Messages/Logs

## Eval runner output


fear-itself run 1/1 [with]: score 0.00  $0.17
  ✗ author-name (weight 1): grader threw: case "fear-itself" grader focus file "information.md":
    path "information.md" does not exist
  ✗ correct-output (weight 1): grader threw: case "fear-itself" grader focus file "information.md":
    path "information.md" does not exist
  ✗ file-created (weight 1): information.md missing (expected present)


### Agent trace (from `out/trace.jsonl`)

The agent correctly invoked the skill, received the skill instructions, then tried to read the fixture file:


Read("quote.txt") → "File does not exist. Note: your current working directory is /private/tmp/e-1maDc5/home/cwd."
Glob("**/quote.txt") → "No files found"
Glob("**/quote*") → "No files found"
Glob("**/*") → "No files found"


The agent concluded: "the directory appears to be empty."

## Sandbox inspection (`--keep-temp`)


$ ls -la /private/tmp/e-1maDc5/sealed/home/cwd/
total 0
drwxr-xr-x 2 user wheel  64 Oct  1 15:06 ./
drwxr-xr-x 9 user wheel 288 Oct  1 15:06 ../


The `cwd/` directory is empty — `quote.txt` was not copied from the case's `test/` directory.

## Error when using `add_dirs: ["./"]`


case "fear-itself": an add_dir may only name fixture directories inside this case —
not the case directory itself, its graders, a sibling case, an eval directory, or the plugin root

Steps to Reproduce

1. Create a plugin with a skill that reads from a file (a minimal repro plugin is attached)
2. Create an eval case with a fixture file in a subdirectory:

evals/01-fear-itself/
├── prompt.md          # "Identify the author of the quote in quote.txt"
├── case.yaml          # context.add_dirs: ["test/"]
├── test/quote.txt     # "The only thing we have to fear is fear itself."
└── graders/
    └── file-created.md  # type: file_exists, path: information.md

1. Run:

claude plugin eval . --runs 1 --ablation with-without --allow-tools Write --keep-temp --case fear-itself

1. Observe that all graders fail because information.md was never created
2. Inspect the kept temp directory — sealed/home/cwd/ is empty

Reproduction project

A minimal plugin that reproduces this issue is available at: https://github.com/eliasnogueira/plugin-evals-test

The repository includes:

• The Quote Identifier skill
• The eval suite with context.add_dirs configured
• Fixture files in test/ subdirectories
• eval-sandbox-debug.zip — both sandbox temp directories (with and without arms) showing empty cwd/, including out/trace.jsonl with the full agent trace

Claude Model

Opus

Is this a regression?

I don't know

Last Working Version

_No response_

Claude Code Version

2.1.285

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

Other

Additional Information

The Terminal/Shell is Ghostty / zsh