claude plugin eval: every Bash call fails on stock Ubuntu (apply-seccomp setgroups, #43454) and the sandbox settings workaround cannot be applied

What's wrong?

claude plugin eval cannot run any Bash tool call on a stock Ubuntu install. That is the apply-seccomp sandbox failure already tracked in #43454, and the user-level workaround found in #81799 ("sandbox": {"network": {"allowAllUnixSockets": true}}) cannot be applied inside eval runs. Eval runs use a sealed config, and claude plugin eval has no option to pass sandbox settings.

The failure is also scored instead of reported. Every case that needs a shell gets 0 in both arms (with and without the plugin), so the with/without delta for those cases is meaningless, and nothing in the report says the sandbox never started.

Every Bash call in the run's trace fails with:

apply-seccomp: write /proc/self/setgroups (nested userns is capability-restricted; caller must provide CAP_SYS_ADMIN): Permission denied

Cases that only use Read, Write, or Edit run normally.

Environment

• Claude Code 2.1.283, native install
• Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic
• bubblewrap 0.11.1, apparmor 5.0.2-0ubuntu1~26.04.1, stock profiles (bwrap-userns-restrict loaded)
• kernel.apparmor_restrict_unprivileged_userns = 1 (default)

Outside bubblewrap, nested user namespaces work on the same machine: unshare -Ur sh -c 'unshare -Ur true' succeeds. Children of /usr/bin/bwrap run under unpriv_bwrap, which has audit deny capability, and the helper's nested-namespace setgroups write needs CAP_SYS_ADMIN.

Steps to reproduce

1. Ubuntu 24.04 or later with the default AppArmor profiles.
2. Create an eval case whose prompt makes Claude run any shell command (for example ls), with Bash in allowed_tools.
3. claude plugin eval <plugin> --allow-tools Bash --runs 1 --keep-temp
4. Every Bash call in out/trace.jsonl has the apply-seccomp error, and the case scores 0 in both arms.

What should happen?

Any of these would fix it for plugin authors:

1. A fix for #43454, so the eval sandbox arms on stock Ubuntu.
2. A way to pass sandbox settings to eval runs, such as allowAllUnixSockets, so the #81799 workaround can be used.
3. At least, detect that the sandbox could not start and report the run as an error rather than a score of 0.

Workaround we use

Running the same cases with claude -p outside the eval runner and applying the graders ourselves. That loses the runner's isolation.

Related: #43454, #81799, #97171 (another plugin eval case where every Bash call fails, with a different bwrap error).