[BUG] claude plugin eval never loads a plugin's declared dependencies, so the plugin under test fails to load and cases score the bare model

Status Open
Reported on v2.1.288
Maintainer reply None cached
Activity 0 comments · opened Oct 3, 2026

claude plugin eval loads only the plugin under test, so a plugin that declares dependencies in plugin.json never loads inside the eval sandbox, and every case scores the bare model instead of the plugin.

Repro

  • Setup - plugin A declares "dependencies": ["B"]; both live in the same marketplace, and both are installed and enabled at user scope (claude plugin list shows them).
  • Run - claude plugin eval A@<marketplace> --case <any> --runs 1 --ablation none --keep-temp --json out.json, and the same with a path target.
  • Init event of the graded run (trace.jsonl):
  • plugin_errors: [{"plugin": "A@inline", "type": "dependency-unsatisfied", "message": "Dependency \"B\" is not installed ..."}]
  • skills: no A:* skill; plugins: only the builtins.
  • Workarounds tried - a global --plugin-dir for B and a user-scope install of B@<marketplace>; neither reaches the sandbox.

Expected

  • Dependencies load - the eval resolves the target's dependencies the way an interactive session does (from the target's marketplace, or the user's installed copy) and loads them beside it.
  • A run that cannot load the plugin under test fails - today the case is graded and can pass on the bare model's behaviour, so a skill regression goes unnoticed. The with/baseline delta is the only hint, and only when ablation is on.

Environment

  • Claude Code 2.1.288, Linux.
  • Related, other load paths: #77801, #72925 (--plugin-dir dependencies never resolve).

View original on GitHub ↗