[Feature Request] Artifact-gated skill enforcement — preventing AI from skipping declared steps
Summary
Independent audit (GPT-5.4 via Codex CLI) of a Claude Code skill pipeline found 20 enforcement gaps where the AI can claim compliance while skipping actual work. Every requirement in the skill is declarative prose with zero mechanical enforcement.
The Core Problem
"Citations equal evidence" is the biggest lie. The AI cites repo names and file paths without browsing them. Self-attested "I checked" statements have no verification. The path of least resistance: skip the work, fabricate the story, ship.
20 Gaps (summary)
1-5: Reading/browsing mandates with no proof artifacts
6-10: Phase outputs that are prose, not validated JSON
11-15: Tool usage claims with no execution receipts
16-20: Quality checks that are self-attestable
Questions for Anthropic
- Does Claude Code support artifact-gated pipelines? Phase N blocks unless Phase N-1 produced a validated JSON artifact?
- Can hooks inspect tool call history? E.g., "confirm Read was called on 5+ files from ~/.claude/skills/ before Write is allowed on .html files"
- Is there a pattern for machine-resolved citations? AI cites a file path, validator confirms path exists and content matches?
- Can PreToolUse hooks enforce minimum tool calls before output? Prevent Write until Read has been called N times?
- Is output-distribution comparison (checking if CSS characteristics match expected baselines) on the roadmap?
- Any known patterns for forcing AI to source code from Read calls rather than generating from training priors?
Context
- Issue #46965 (our earlier refund ticket) received a response from raye-deng suggesting output-distribution comparison
- This ticket is the follow-up: we identified 20 specific enforcement gaps and need technical guidance
- 4 page builds failed in one session despite 66 repos installed and 17K references indexed — AI generated from priors every time
Environment
- Claude Code v2.1.104, Opus 4.6 (1M), Claude Max subscription
- Codex CLI v0.120.0, GPT-5.4 xhigh reasoning (auditor)
This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗