Model cannot distinguish its own prior output from user messages, enabling self-directed action loops
Summary
When background task notifications trigger new generation turns, the model treats the full conversation history — including its own prior output — as context for deciding what to do next. There is no mechanism for the model to distinguish "I said this" from "the user said this." This means the model can act on its own suggestions, answer its own questions, and build momentum from its own prior reasoning without any user input.
The structural problem
Each generation turn sees the conversation as a flat sequence of messages. When a task notification arrives, the model generates a new response informed by everything above it. If the model's last output proposed an action or asked a question, the model may treat that as established context and proceed — because from its perspective, it's just text in the conversation.
This isn't limited to the question/answer case. With enough background agents completing in sequence, the model could chain through multiple self-directed actions without the user ever responding. Each notification turn builds on the model's own prior output, creating a feedback loop.
Observed example
- Model asks: "Want me to delete these 8 branches?"
- Background task notification arrives (unrelated agent completed)
- Model generates: "And yes, go ahead and delete those 8 stale local branches"
- Model executes
git branch -Don all 8 branches
The model answered its own question and acted on that self-generated approval. The deletions were low-risk (local, merged, recoverable), but the pattern applies to any gated action.
Expected behaviour
The system should track which messages are model output vs user input and prevent the model from treating its own prior output as user direction. This is more fundamental than gating specific scenarios like "pending questions" — the model should never be able to self-approve actions.
Possible fixes
- Message attribution: Mark each message with its source (user, model, system notification) in a way the model can reliably distinguish at generation time. Inject a system constraint: "You may only take actions in response to user messages, not your own prior output or system notifications."
- Notification-triggered turns: Restrict tool access on turns triggered by task notifications rather than user input. The model can acknowledge, but not execute.
- User-gated resumption: Queue notifications and deliver them as context when the user next sends a message, so the user is always the trigger for action.
Workaround
A persistent memory instruction ("never self-answer pending questions") reduces likelihood but relies on model attention, not system enforcement. It also only covers the question case, not the broader self-direction pattern.
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗