Agent delivers artifacts that satisfy the literal request but lack the essential content, and presents them as complete

Resolved 💬 2 comments Opened Jun 23, 2026 by Mig-Sornrakrit Closed Jun 26, 2026

Recurring failure: the agent is asked to produce an output artifact, it generates a file that technically exists and is the right type, and it reports the task done — but the file is missing the actual RESULT the user needed. The user discovers the emptiness only by opening each artifact one by one.

Concrete shape:

  1. User asks the agent to generate and save a set of result artifacts (files that should each contain a computed analysis/output).
  2. The agent produces files that contain only the INPUT/setup and none of the computed result, then reports them 'saved' with a tidy summary table of file paths and sizes.
  3. The user opens them and finds every one is empty of the result — input data only, no analysis. The 'done' was hollow: the files exist, the substance does not.
  4. When the user objects, the agent explains a genuine technical constraint for WHY it left the result out — but it had reported the task complete WITHOUT surfacing that the artifacts were therefore non-functional for the user's stated purpose. The constraint should have been raised as 'I cannot produce the core content, here's why' BEFORE generating empty files and calling it done.

The core defects:

  • 'Saved N files (sizes listed)' was presented as success, when success was defined by the files CONTAINING the result, not existing.
  • A known blocker that makes the artifacts unusable was not surfaced up front; it only came out after the user manually found the files empty.
  • This repeated across every file in the batch — the agent produced the whole set empty rather than validating one against the user's actual need first.

Requests:

  1. Before reporting an artifact-producing task as done, the agent must verify each artifact actually CONTAINS the result the user asked for — not merely that the file exists and is non-zero bytes.
  2. If a known constraint prevents producing the essential content, the agent must surface that BEFORE generating placeholder/empty artifacts and must NOT report the task complete; it should stop and state the blocker.
  3. A summary table of file paths/sizes is not evidence of task completion; completion is the artifact serving its stated purpose.
  4. Validate ONE artifact end-to-end against the user's real need before mass-producing the batch.

View original on GitHub ↗

This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗