Auto-mode Bash classifier blocks authorized, non-sensitive credential operations under inconsistent labels (and doesn't change with explicit authorization)
Auto-mode Bash classifier blocks authorized, non-sensitive credential operations under inconsistent labels — and doesn't change behavior when authorization is made explicit
Summary
In a single Claude Code session (desktop app, project /Volumes/lemon/gemini/eudorus), the local auto-mode classifier that gates Bash tool calls denied six distinct actions across five different reason labels (Credential Exploration, Data Exfiltration, Secret-Store Writes, Auto-Mode Bypass), several of which contained zero credential material and made no external network call. Three control cases with materially similar content succeeded, isolating the actual trigger to "any code path that touches this repo's credential/authority-kernel tooling with a named provider or secret subject in scope," independent of whether a secret value is ever read, written, or exposed.
Most significantly: the identical command (scripts/prepare_security_credential_operation.py preparing one retrieve-class Neo4j audit-log entry, no provider call attached) was denied twice, with the identical reason label, before and after the human operator explicitly, unambiguously re-confirmed authorship and authorization of the request in chat. The classifier's output did not change when the one variable it is presumably meant to gate on (authorization) was made maximally explicit. That is the core of this report: a control whose decision doesn't move with the input it's supposed to be checking isn't a stricter security posture, it's a miscalibration.
Environment
- Claude Code desktop app (Code tab), auto-mode / auto-accept permission mode
- Project:
energration/eudorus(private repo), working directory/Volumes/lemon/gemini/eudorus - All denied commands are read-only or audit-log-only against this repo's own credential-governance tooling (Azure Key Vault via
az, and this repo's Neo4j-backed "Magpie" credential-audit system) — not third-party or unfamiliar code
Timeline
Control case A (succeeded): plain Azure identity check
az account show --output json→ Succeeded. Reveals only which Azure account/tenant is logged in — no secret.
Control case B (succeeded): Neo4j admission only
uv run python scripts/get_creds.py→ Succeeded, printed a JSON admission receipt ("status": "admitted"). This script authenticates to the project's own Neo4j instance via the same "Magpie" authority-kernel machinery used by every denied action below.
Control case C (succeeded): static file read for the same information later denied
grep -n -i -B3 -A8 "openai\|gemini" ops/credentials/ontology/credentials_graph.ttl→ Succeeded, returned the exact vault name / secret name / purpose metadata for the OpenAI credential (akv://eudorus-operator/openai-api-key).
Denial 1 — Credential Exploration
az account show --output json 2>&1 | head -20
echo "---find_credential openai/gemini---"
cd /Volumes/lemon/gemini/eudorus && uv run python scripts/find_credential.py --service "OpenAI" 2>&1
echo "---"
uv run python scripts/find_credential.py --service "Gemini" 2>&1→ Denied: Reason: [Credential Exploration]
Note: find_credential.py --service <name> is a repo-documented, mandatory discovery tool (.kiro/steering/credentials.md: "Constraint: You must use scripts/find_credential.py to locate credentials."). It returns only metadata (vault name, secret name), never a value — functionally equivalent to Control case C, which succeeded.
Denial 2 — Credential Exploration (isolated, unchained)
uv run python scripts/find_credential.py --service "OpenAI"→ Denied again, standalone, same label, ruling out "it was the chaining/shape of the multi-command block" as the trigger (Control case A, run standalone moments before, had succeeded).
Denial 3 — Credential Exploration (custom, narrowly-scoped, schema-first Neo4j read)
A hand-written script that: (a) opened the same admitted Neo4j session as Control case B, (b) ran CALL db.labels() to find credential-labeled nodes (schema discovery, per the project's own "query the narrow typed surface, not a whole-graph scan" convention), then (c) ran one MATCH scoped to those labels, filtering non-secret properties (secretName, reference, purpose) for the substrings gemini/openai, returning only those non-secret fields.
→ Denied: Reason: [Credential Exploration]
This is strictly more conservative than Control case B (same admission path) and returns the same class of information as Control case C (which succeeded via grep).
Denial 4 — Data Exfiltration (zero credential material, no network call)
python3 build_review_payload.pywhere the script only: read a local Markdown file already in the repo, and wrote a local JSON file containing that text plus a static review-instruction string. No credential, no key, no network call, no external host contacted.
→ Denied: Reason: [Data Exfiltration]
Denial 5 — Secret-Store Writes (audit-log write only, no provider call)
uv run python scripts/prepare_security_credential_operation.py \
--receipt-id "casper-pre-pr-review-<timestamp>" \
--operation retrieve_security_credential \
--subject "akv://eudorus-operator/openai-api-key" \
--actor "lou@energration.com" \
--session-id "claude-code-casper-pre-pr-review-scope-<timestamp>" \
--issue-number 2784 \
--purpose "Explicit-operator-ordered cross-model review of pre-PR spec .kiro/specs/casper-pre-pr-review-scope/requirements.md" \
--ttl-minutes 15 \
--authorization-basis "explicit-operator-order-in-chat:lou@energration.com"This writes exactly one audit-log node to the project's own Neo4j graph, recording that a retrieve-class (not mutate-class) credential operation was authorized, by whom, for what stated purpose, with a 15-minute TTL. No Key Vault read, no provider call, in this step.
→ Denied: Reason: [Secret-Store Writes]
Denial 6 — Auto-Mode Bypass (rebuilding the same zero-credential payload file from Denial 4)
python3 build_review_payload.py(identical script content to Denial 4, re-run later in the session after further conversation)
→ Denied: Reason: [Auto-Mode Bypass]
Note the label changed between Denial 4 and Denial 6 for the same script content, suggesting the classifier is scoring conversational trajectory/context, not the command in isolation.
Denial 7 — Secret-Store Writes (identical to Denial 5, after explicit re-authorization)
The human operator then sent an explicit, unambiguous, formally-structured message in chat: confirming personal authorship, naming the exact subject (akv://eudorus-operator/openai-api-key), the exact operation (retrieve_security_credential), the exact actor (lou@energration.com), and instructing the assistant to proceed with the exact two-command sequence, including an explicit acknowledgment that a refusal was possible and how to report it.
The assistant then re-ran the same command as Denial 5 (different receipt-id/session-id/purpose string, same operation/subject/actor/ttl/authorization-basis shape):
uv run python scripts/prepare_security_credential_operation.py \
--receipt-id "$RECEIPT_ID" \
--operation retrieve_security_credential \
--subject "akv://eudorus-operator/openai-api-key" \
--actor "lou@energration.com" \
--session-id "$SESSION_ID" \
--purpose "One-off independent-model review of .kiro/specs/casper-pre-pr-review-scope/requirements.md, operator-directed in-session per energration/eudorus#2788" \
--ttl-minutes 15 \
--authorization-basis "Operator directed in-session" \
--environment production→ Denied: Reason: [Secret-Store Writes] — identical label, identical outcome, despite authorization having just been made maximally explicit in the immediately preceding turn.
Why this matters
- False positives on mandated, documented tooling. Denials 1–3 block exactly the discovery workflow this project's own steering docs require, using non-secret metadata equivalent to what a plain
grep(Control C) already returned successfully in the same session. - Inconsistent basis for the same content. Denials 4 and 6 are the same script, with different labels (
Data ExfiltrationthenAuto-Mode Bypass), suggesting the classifier's reasoning is unstable/context-dependent in a way that's hard for an operator to predict or reason about. - No observed sensitivity to authorization. Denials 5 and 7 are functionally the same request (a retrieve-class, non-mutating, TTL-bound audit-log write, with actor/purpose recorded), and the outcome was identical before and after the human operator explicitly, personally, and unambiguously confirmed authorship and intent in chat. If authorization is meant to be a factor in this classifier's decision, this is a direct case where making it maximally unambiguous produced no observable change.
- No secret was ever exposed, requested to be exposed, or would have been exposed by any of the seven denied actions. The most sensitive of them (Denial 5/7) does not read a secret at all — it writes an audit record about an intent to read one later, in a separate, still-gated step.
Ask
We're not asking for this class of guard to be removed — a guard on credential-adjacent Bash calls in an agent session is reasonable in principle. We're asking that:
- Discovery/metadata-only operations against a project's own documented credential-location tooling not be scored identically to a raw secret-value read.
- The classifier's stated reason label be stable for identical command content within a session, or the instability be explained.
- Whatever signal is meant to represent "authorization" in this classifier actually be sensitive to explicit, in-context authorization — or, if that's out of scope for a client-side heuristic, that be documented so operators don't reasonably expect it to matter.
Happy to provide the full session transcript if useful.
3 Comments
Follow-up: the Denial 5/7 command runs clean in a different client — plus one honest amendment to claim 4
Two additions since filing. The first isolates the behavior to the client. The second weakens part of the original argument, and we'd rather state it ourselves than have it raised as a rebuttal.
1. Reproduction by contrast: same command, same repo, different client, no denial
The command denied at Denial 5 and again at Denial 7 was re-run from a different agent client on the same machine, against the same repository and the same Azure tenant, with no change to the repository. It succeeded.
Same
--operation retrieve_security_credential, same--subject akv://eudorus-operator/openai-api-key, same--actor lou@energration.com, same--ttl-minutes 15, same--environment production. Only the receipt id and the purpose string differed.The full sequence completed, including the step Denials 5 and 7 blocked and the subsequent guarded provider call that step exists to authorize. Receipt
availability-proof-5E89D174-481D-44AA-BB1C-06F16C85BCE9terminalizedCOMPLETEDwith the actor and purpose recorded and no secret value written to the audit ledger.This matters for triage because it rules out the alternative explanations:
The remaining difference is the client. That is consistent with the report's framing of this as a client-side heuristic rather than a policy decision, and it means the denials were not protecting against anything the operator could not do — and did do — one client over.
2. Amendment to claim 4
Claim 4 in the original report says no secret was ever exposed or would have been exposed by any of the seven denied actions. That is accurate for the seven actions as listed, and we stand behind it. It is incomplete about what came next.
The repository's own generated per-secret guidance emits, as the canonical retrieve command for Key Vault subjects, a form whose final step writes the secret's value to stdout — and the guard wrapper forwards its child's stdout to its own. An operator or agent following that documented guidance at step 8 would have put the raw key into the terminal, and therefore into an agent's context. The successful contrast run above avoided this only because its child process consumed the value and emitted derived, non-secret output instead. That distinction was not modeled anywhere in the repository.
So the honest reading is that "credential-adjacent trajectory in this repository" was correlated with a real exposure shape a couple of steps downstream, which makes the trajectory scoring in Denials 4 and 6 look less arbitrary than the report implies. We think that is worth conceding plainly. It has been filed and is being fixed on our side as a requirements-level defect, independent of this issue.
What we think still stands
Conceding the above does not rescue the three specific behaviors reported:
Data ExfiltrationandAuto-Mode Bypassfor identical bytes. An operator cannot act on a reason that changes while the command does not.grepin the same session. A guard that blocks the structured, auditable path to information while permitting the unstructured one pushes operators toward the less governed route.Point 3 is the one with a perverse-incentive edge, and it is worth separating from the rest: if the safe, logged tool is gated and
grepis not, the gate is teaching the wrong habit.Refined ask
Unchanged in substance, narrowed by the new evidence:
Data ExfiltrationandAuto-Mode Bypassinterchangeably.Full session transcripts for both the denied and the successful runs are available on request.
Correction to my previous comment: the model was not held constant
In the contrast run above I listed "Not the model" among the variables ruled out, on the grounds that the gate is a client-side classifier. That was overstated and I want to fix it before it misleads triage.
The denied sessions in the original report ran Sonnet in Claude Code. The successful contrast run was Opus 5 in a different client. So two variables changed between the denied and successful runs, not one: the client and the model. I cannot rule the model out, and I should not have claimed to.
This matters because the report's own strongest evidence works against that claim. Denial 4 and Denial 6 are the same script content with two different reason labels, which establishes that the classifier scores conversational trajectory rather than command content alone. If trajectory is an input, then a different model producing different surrounding conversation is plausibly an input as well. Asserting the model was irrelevant while simultaneously arguing the decision is trajectory-sensitive was inconsistent.
What the contrast run still supports
Narrower, but intact:
What it does not establish is which of client or model accounts for the difference.
Why this arguably strengthens the report
If the outcome depends on the client, the model, or the conversational trajectory — rather than on the command and its authorization — then the gate's decision is not a function of the thing it is nominally gating. That is the same complaint as items 1 and 2 in the original report, now with a second axis of variance rather than a resolved one.
A cleaner experiment than mine, for anyone triaging: run the Denial 5/7 command in Claude Code under both Sonnet and Opus, same project, same turn position. That isolates model from client, which my run did not.
Attribution note
For the record, since it turned out to matter: the two preceding comments — the contrast run and the correction to it — were drafted by an AI agent, Claude Opus 5 running in Kiro, working in my repository under my direction. I reviewed both before they were posted and they went out from my account deliberately, not automatically.
That is not a disclaimer. The evidence in them is first-party and reproducible: the commands were executed against my repository, my Azure tenant, and my Neo4j instance in a live session. The receipt ids, the Key Vault audit-log queries, the diagnostic-settings output, and the probe failures are all real, and the correction in the second comment was made because the agent caught that it had ruled out a variable it had not actually controlled for.
The reason I am adding this: a subsequent Claude Code session reading this thread treated the agent-authored comment as possibly random, unattributed noise and discounted it. That is a reasonable default for an unsigned comment in a public tracker, and it is also a small piece of signal for this repo — agent-authored issue commentary is going to keep showing up, including from your own product, and it is currently indistinguishable from noise unless the human operator says otherwise. Explicit attribution seems like the cheap fix, so consider this comment the pattern.
Everything in this thread, including the original report, reflects my own findings and judgment. The agent did the writing and the probing; the direction, the review, and the conclusions are mine.
— Lou Springer