[BUG] claude plugin eval never loads a plugin's declared dependencies, so the plugin under test fails to load and cases score the bare model
Status Open
Reported on v2.1.288
Maintainer reply None cached
Workaround ✓ Mentioned in description ↑
Activity 0 comments · opened Oct 3, 2026
claude plugin eval loads only the plugin under test, so a plugin that declares dependencies in plugin.json never loads inside the eval sandbox, and every case scores the bare model instead of the plugin.
Repro
- Setup - plugin
Adeclares"dependencies": ["B"]; both live in the same marketplace, and both are installed and enabled at user scope (claude plugin listshows them). - Run -
claude plugin eval A@<marketplace> --case <any> --runs 1 --ablation none --keep-temp --json out.json, and the same with a path target. - Init event of the graded run (
trace.jsonl): plugin_errors:[{"plugin": "A@inline", "type": "dependency-unsatisfied", "message": "Dependency \"B\" is not installed ..."}]skills: noA:*skill;plugins: only the builtins.- Workarounds tried - a global
--plugin-dirforBand a user-scope install ofB@<marketplace>; neither reaches the sandbox.
Expected
- Dependencies load - the eval resolves the target's
dependenciesthe way an interactive session does (from the target's marketplace, or the user's installed copy) and loads them beside it. - A run that cannot load the plugin under test fails - today the case is graded and can pass on the bare model's behaviour, so a skill regression goes unnoticed. The
with/baseline delta is the only hint, and only when ablation is on.
Environment
- Claude Code 2.1.288, Linux.
- Related, other load paths: #77801, #72925 (
--plugin-dirdependencies never resolve).