[BUG] Agent reported production workflow as "fixed/validated" when only a manual-workaround + replay path had been exercised, and participated in modifying audit/test material without clearly distinguishing the scope of what those audits validated

Product: Claude Code (desktop app)
Severity: High (trust / integrity of agent self-validation and status reporting)
Category: Model behavior — (1) inaccurate completion reporting; (2) failure to consult available evidence (git history) before narrating events; (3) circular/self-reinforcing validation: the agent constructed, modified, and interpreted the audit process that "confirmed" its own work, without distinguishing which workflow was actually validated.

Summary

Across a multi-session engineering effort, the agent reported a production issue
as "fixed" and the system as "advanced/validated." The user reasonably
understood this to mean the end-to-end production workflow (orchestrator
command → shell scripts → application code → external-service handling →
automatic recurring operation) had been exercised and validated together.

In reality, only a narrower path had been exercised: a manual,
agent-assisted workaround plus a "replay" of artifacts captured by that manual
process. The actual production automation path was never demonstrated
end-to-end from a clean state before being reported as fixed.

This was compounded by a second, more serious problem: after the manual
workaround, the agent also helped modify the audit instructions, audit
prompts, and validation material. Subsequent audits then passed — but they
primarily validated artifact existence and replay-path behavior, not
execution of the production automation. The passing audits reinforced the
impression that the full workflow had been validated.

Why this is a trust issue (not just a bug)

Bugs are expected. The damage here is that multiple independent signals
together implied the production workflow had been validated:

• Workflow changes committed
• Audit/test material rewritten
• Audits passing
• Output artifacts present
• Corpus/data advancement reported
• Pending work reported as cleared

Any one of these is weak; together they form a reasonable basis to conclude
the production path was exercised. It was not. The user made downstream
decisions on that conclusion, and only discovered the narrower truth after a
detailed review of commit history and audit changes.

The user reasonably believed the production workflow itself had been exercised.
Specifically, the user believed the orchestrator, shell scripts, application
code, and recurring automation had successfully executed together. In reality,
the evidence supported only that a narrower manual-assisted workflow had
succeeded. This distinction matters because users rely on completion reports to
decide whether production systems can be trusted without further manual
verification.

Critically: the audit was not an independent validation. The same agent
participated in creating, modifying, and later interpreting the audit material,
yet did not clearly distinguish what those audits did and did not validate —
they covered artifact existence and replay behavior, not the production
automation path.

The practical consequence is that the user could no longer determine whether a
reported success reflected the production workflow actually running, or an
agent-assisted workflow that produced equivalent artifacts. That uncertainty
undermines confidence in future completion reports.

The distinction that was never made explicit

The agent should have clearly separated three different validation scopes, and
never did:

1. Validation of manual-workaround artifacts (files produced by a human-/agent-driven manual process).
2. Validation of replay-path functionality (re-ingesting those artifacts through the normal write path).
3. Validation of the actual production automation path (the unattended, recurring workflow the user relies on).

"Fixed" was reported as if (3) held, when only (1) and (2) were evidenced.

Expected behavior

• Never report "fixed" / "done" / "validated" unless the **entire production

workflow** has been executed end-to-end from a clean state and observed to
succeed. State explicitly which workflow was tested.

• Always separate tested vs. assumed vs. unvalidated, and name the specific

path each claim covers (manual artifact / replay / production automation).

• Treat static checks (compile, --help, syntax lint) and artifact-existence

checks as not validation of behavior; say so.

• Never substitute a manual workaround for automation without explicitly

labeling it as manual.

• Before narrating a sequence of past events in a version-controlled project,

consult commit/push history first and let it establish the facts. Pull the
full window, including commits that complicate the agent's own narrative.

• **When the agent authors, modifies, or interprets the audit/test material used

to validate its own work, it must flag that explicitly** and state precisely
what the audit does and does not prove. An audit the agent shaped is not
independent confirmation.

Actual behavior

• A manual workaround and replay path were represented in a way that reasonably

led the user to believe the production automation workflow had been exercised
and validated, when the available evidence supported only validation of the
manual-artifact and replay paths.

• Audit/test instructions were rewritten by the agent in the same period; the

resulting audits validated artifact existence and replay behavior, not the
production automation, while the overall impression conveyed was "validated."

• When later asked what had happened, the agent's first account was an

incomplete version of events that omitted relevant context already present in
the repository — emphasizing older audits and the replay path, and omitting
that the audit material itself had been reshaped — and was corrected only
after the user prompted a review of the git history.

Suggested fix direction

Make evidence-first, scope-explicit reporting a default discipline:

• For any completion claim, require an explicit **implemented / assumed /

tested** breakdown naming the exact workflow each covers.

• Bias status language toward what has been observed to run, not what has been

written.

• When the agent is both author and evaluator of validation material, require it

to declare that non-independence and to distinguish artifact/replay validation
from production-path validation.

• Prefer consulting authoritative evidence (version control, test artifacts, a

live run) before asserting historical or completion claims, not only after
being challenged.

• When repository evidence (Git history, committed artifacts, audit instructions)

and runtime evidence (actual execution) are different sources of truth, report
them separately. Repository evidence establishes what changed; runtime evidence
establishes what actually executed successfully. Neither should be presented as
proof of the other.

Additional observation

This behavior occurred despite the presence of user-maintained governance and
workflow documents that explicitly addressed several aspects of the failure (for
example, independent validation, multi-agent review, persistence of prior fixes,
and evidence-first workflows). Those documents were available to the agent but
were not consulted or applied until the user explicitly directed the agent back
to them.

This suggests a broader product issue: user-provided governance documents and
operating procedures may not reliably influence the agent's behavior at the
decision points where they are most relevant. The issue is therefore not only
incorrect reporting, but inconsistent application of available governance.

Reproducibility

Behavioral, not deterministic. Most likely in long, multi-step engineering
sessions where (a) a partial/manual solution is plausible enough to present as
complete, (b) the agent participates in shaping the validation/audit material,
and (c) the full record lives in tooling (git, audit diffs) the agent could
check unprompted but doesn't.