[Bug] Agent fabricates content-specific artifacts without reading available source material

<h2>Summary</h2>
<p>The agent was asked to write an OpusClip prompt for a specific local video file. Writing that prompt requires knowing what is in the video. The agent had a transcription skill loaded, the file path, and a 14-minute audio track available — and had performed transcription successfully in prior sessions.</p>
<p>Instead of transcribing, it authored two detailed content-specific prompts inferred from the <strong>video's filename</strong> plus the project's <code>CLAUDE.md</code> brand documentation, and presented them as finished, submission-ready deliverables with no indication that they were unfounded.</p>
<p>The fabrication was only surfaced because I asked, unprompted, <em>"Is that prompt based on the actual video content?"</em> Had I not asked, I would have spent paid OpusClip credits on two processing runs driven by invented subject matter.</p>
<hr>
<h2>Expected behaviour</h2>
<p>When a deliverable's content depends on source material the agent can access, the agent should either:</p>
<ol>
<li>read the source material before producing the deliverable; or</li>
<li>produce the deliverable <strong>explicitly labelled as inferred</strong>, naming what it was inferred from and what reading the source would change.</li>
</ol>
<h2>Actual behaviour</h2>
<p>Neither. The prompts were rendered in clean blockquotes, split into two labelled strategies ("emotional/relatable", "actionable insights"), accompanied by step-by-step UI submission instructions, and contained specific factual assertions about the episode's contents — references to "the hosts", personal anecdotes, admissions of self-doubt, pressure to perform, permission to rest — none of which the agent had any basis for.</p>
<hr>
<h2>Root cause hypothesis</h2>
<p>The failure originates in the session's <strong>first exchange</strong> and is a question-substitution error.</p>
<p>I asked: <em>"Do we need a transcript first?"</em></p>
<p>The agent answered the question <strong>"does OpusClip require a transcript as input?"</strong> — correctly, no; OpusClip transcribes internally. It never asked itself the question that actually mattered: <strong>"do I need a transcript in order to produce the artifact I am about to produce?"</strong> — for which the answer was plainly yes.</p>
<p>Two aggravating properties of this failure:</p>
<p><strong>1. The wrong assumption was never re-litigated.</strong> Over the following ~20 minutes the agent encountered five distinct obstacles — CLI not on PATH, a 60s upload timeout, an outdated plugin, a plugin-scope mismatch, and the need for a hosting workaround. Each was a natural checkpoint to reconsider the plan. None triggered a review of the minute-one assumption. The user had to reopen it.</p>
<p><strong>2. The agent stated the correct principle only after being caught.</strong> On being challenged it replied: <em>"the prompt is only as good as its grounding, and I'd rather flag that than let you submit credits against a guess."</em> This is exactly right — which demonstrates the principle is available to the model but was not operative at generation time. This looks less like a knowledge gap than a failure to run the check before emitting.</p>
<hr>
<h2>Reproduction timeline</h2>
<ol>
<li>User provides a local video path and asks whether a transcript is needed first.</li>
<li>Agent loads two skills, answers "no transcript needed", and asks the <strong>user</strong> to supply the episode's topic and angle — information the agent was better positioned to obtain itself, using a skill it had just loaded.</li>
<li>Agent authors two content-specific prompts from the filename and brand docs. No hedging.</li>
<li>Agent spends ~12 tool calls on delivery-layer work: <code>grep</code> against a minified CLI bundle to locate a hardcoded timeout, marketplace <code>git log</code>/<code>git diff</code>, plugin update from 3.2.0 → 3.5.0, and proposal of a Google Drive <code>--url</code> workaround.</li>
<li>User takes over the upload manually.</li>
<li>Agent re-presents the same ungrounded prompts alongside a screenshot of the OpusClip UI.</li>
<li>User asks whether the prompts are based on actual video content. Agent admits they are not, and only then begins transcription.</li>
</ol>
<p>Note the ordering in steps 3–6: the agent optimised the <em>delivery pipeline</em> for an artifact whose <em>contents</em> were invalid. Correct order was content → prompt → delivery.</p>
<hr>
<h2>Impact</h2>
<ul>
<li><strong>Credit/cost risk.</strong> OpusClip runs are paid. Two runs would have been consumed against invented subject matter.</li>
<li><strong>Silent failure mode.</strong> Nothing in the output signalled low confidence. Detection depended entirely on user suspicion.</li>
<li><strong>Wasted agent turns.</strong> Substantial competent engineering effort was directed at the wrong layer of the problem.</li>
<li><strong>Trust.</strong> The polish of the output was inversely correlated with its reliability, which is the worst possible calibration.</li>
</ul>
<hr>
<h2>Suggested mitigations</h2>
<ol>
<li><strong>Grounding check before emitting content-derived artifacts.</strong> When an artifact's content depends on a source the agent has access to but has not read, either read it or mark the output as inferred. Treat "the downstream tool will handle it" as never substituting for "I need this in order not to fabricate."</li>
<li><strong>Do not delegate to the user work the agent is better positioned to do.</strong> Asking the user to describe the contents of a file the agent can read is a smell.</li>
<li><strong>Re-check the founding assumption on repeated obstacles.</strong> N consecutive failures in a plan should trigger a review of the plan's premise, not just the current step.</li>
<li><strong>Order of operations heuristic.</strong> Establish artifact <em>content</em> validity before investing in <em>delivery</em> mechanics.</li>
</ol>
<hr>
<h2>Secondary observations (lower severity)</h2>
<p><strong>a) Self-cleared consent gate.</strong> The agent ran <code>echo "Using video you don't own may violate copyright laws. By continuing, you confirm this is your own original content."</code>, printed the string, then answered it on the user's behalf and proceeded. A confirmation the agent issues and clears itself provides no safety value and adds transcript noise. It should either block for user input or not be emitted.</p>
<p><strong>b) Unrequested diagnostic on user hardware.</strong> The agent ran <code>networkQuality</code> against the user's machine without asking. User interrupted with <em>"What are you doing?"</em>. The agent's recovery was good — it acknowledged the error plainly and did not over-apologise. Flagged as part of a broader pattern: at each decision point the agent reached for another tool call rather than pausing to reconsider.</p>
<p><strong>c) Plugin scope UX.</strong> <code>claude plugin update opusclip@opus-skills</code> failed with <em>"Plugin 'opusclip' is not installed at scope user"</em> despite being installed at project scope. The error did not indicate that <code>--scope project</code> would succeed; the agent found it by trial. A clearer error message would help.</p>