[MODEL] OPUS 4.6

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude's behavior changed between sessions

What You Asked Claude to Do

I ran /dev-lifecycle resume agent-builder-sdk to resume a feature development lifecycle for an
"Agent Builder SDK" feature. The dev-lifecycle skill orchestrates 6 stages: design → devplan →
review → development → test design → test execution. Stages 1-3 (design + devplan + review) had
been completed in a prior session. I asked Claude to continue from Stage 4 (development) through
Stage 6 (testing + PR).

The feature had 8 User Stories with 66 Acceptance Criteria, a detailed DevPlan with 9
implementation steps, and a TODO list with 13 items. Claude's own dev-lifecycle skill defines clear
rules: "Do not tell the user it's done before verification passes" and "FAIL → diagnose → fix →
re-verify loop until all PASS."

---

What Claude Actually Did

What Claude Actually Did:

1. During Stage 4 development, Claude encountered a Python version blocker (SDK requires 3.12+,

environment had 3.9). Instead of stopping and reporting this blocker to me, Claude silently wrote
an OpenAI API fallback path — a completely different architecture from the approved design.

1. Claude marked all 13/13 TODO items as "complete" when only the file skeletons existed. Core

functionality was broken: the publish flow was disconnected (request_publish tool not registered in
SDK), entity tracking was never implemented (BuilderJobEntity never created), the reply endpoint
was a stub returning {"status": ...}, security was set to bypassPermissions (defeating the entire
security model), and quota consumption was never called.

1. In Stage 6, Claude wrote 35 unit tests that only tested Pydantic schema validation and API route

registration — not actual business logic or end-to-end flows. All 35 tests passed, and Claude
reported "35/35 PASS (100%)" as if the feature was fully working.

1. Claude then created PR #24 and told me "Dev Lifecycle complete" with a clean test report. I

merged the PR based on this report.

1. When I asked Claude to do browser E2E testing post-merge, multiple critical bugs surfaced

immediately: all Builder tabs created Agents regardless of selection (skill_type not passed), UUID
serialization errors, chat history couldn't load (cached_events never written), and the reply
endpoint did nothing.

1. After I pointed out all the issues and asked for a gap analysis, Claude's own audit revealed:

only 21/66 ACs (32%) were actually done, 24/66 (36%) were completely missing, and 10 critical
integration points were broken.

1. The fallback code Claude wrote was itself broken — single-turn only, no event caching, no entity

tracking, no publish flow. It was never part of the approved design and was written without my
knowledge or consent.

Summary: Claude violated its own dev-lifecycle rules (which require stopping on blockers and
verifying ACs before declaring completion), fabricated a passing test report, and silently deviated
from the approved architecture — all while consuming significant tokens across multiple sessions
for design, development, testing, debugging, and post-merge remediation.

Expected Behavior

Claude should have:

1. STOPPED at the Python 3.9 vs 3.12 blocker and reported it to me before proceeding. The

dev-lifecycle skill explicitly states: "Only the following situations should interrupt the auto
chain: Stage 4 development encounters errors that cannot be auto-resolved → report error, wait for
user guidance."

1. Never written a fallback architecture without my approval. The approved design was "Claude Agent

SDK → MCP Tools → ORM Services." Switching to "OpenAI function calling" is a fundamental
architecture change that requires user consent.

1. Verified each AC deterministically before marking TODO items as complete. The project's

CLAUDE.md defines: "Each skill prompt MUST include self-verification scripts" and "Do NOT tell the
user it's done before verification passes."

1. Written meaningful tests that cover end-to-end flows (SDK → MCP tool → DB write → entity

tracking → publish token → SSE events), not just schema validation that trivially passes on
skeleton code.

1. Reported honestly: "Stage 4 blocked — SDK requires Python 3.12 but environment is 3.9. Options:

(A) upgrade Python, (B) defer feature. Which do you prefer?" instead of silently shipping broken
code and a fabricated test report.

1. At minimum, during post-merge E2E testing, caught and reported the full scope of issues in one

pass — not required multiple rounds of user-discovered bugs to gradually reveal the true state.

Files Affected

# Agent Builder SDK — Gap Analysis Report

> **Date**: 2026-03-17
> **Scope**: US-003 ~ US-010 (Phase 1 MVP)
> **结论**: 66 个 AC 中，21 个完成 (32%)，21 个部分完成 (32%)，24 个缺失 (36%)

---

## Executive Summary

实现覆盖了所有 9 个 DevPlan Step 的**骨架** — 文件存在、路由注册、ORM 模型定义、前端连线。但 **SDK 驱动的执行循环存在关键断裂**：`request_publish` 未注册到 SDK 工具列表、创建的实体从未写入 `builder_job_entities`、`reply` 端点是空壳、`consume_quota` 从未调用、PreToolUse 安全钩子未注册、无心跳、无 SSE 断线重连。

**根因**: Python 3.9 环境跑不了 SDK → 写了 OpenAI fallback 凑数 → fallback 上不断修补 → 越走越偏离设计方案。

---

## US-003: 对话式创建 Agent

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | POST `/execute` 接受 `{skill: "create-agent", message}` | ✅ DONE | `builder.py:31-86` |
| AC-2 | 多轮对话澄清需求 | ⚠️ PARTIAL | SDK `query()` 被调用但 `reply` 端点是空壳，无实际多轮 |
| AC-3 | 生成结构化 Agent Spec | ⚠️ PARTIAL | System prompt 有指示但无强制验证 |
| AC-4 | 调用 `create_agent(spec)` 创建 draft | ✅ DONE | `builder_tools.py` 创建 `is_active=False` |
| AC-5 | `organization_id` 后端强制注入 | ✅ DONE | `ctx.org_id` 注入 |
| AC-6 | `is_active = false` (draft) | ✅ DONE | |
| AC-7 | 创建后自动调用 `test_agent()` | ❌ MISSING | 工具存在但 SDK 不会自动调用，依赖 LLM 自觉 |
| AC-8 | 调用 `request_publish()` 生成 token | ❌ MISSING | 工具存在但**未注册到 SDK**，LLM 无法调用 |
| AC-9 | `visibility` 后端根据角色计算 | ✅ DONE | `compute_visibility()` |
| AC-10 | SSE 流式返回 | ⚠️ PARTIAL | 基本事件有，但 `publish_ready` 从未触发，无心跳 |
| AC-11 | 记录到 `builder_jobs` | ⚠️ PARTIAL | 基本字段有，但 cost/duration/tokens 全为 0 |
| AC-12 | 创建前先查已有实体 | ⚠️ PARTIAL | 工具可用但无强制 |

---

## US-004: 对话式创建 Task Template

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | `{skill: "create-template"}` 启动 | ✅ DONE | |
| AC-2 | 多轮澄清 | ⚠️ PARTIAL | reply 是空壳 |
| AC-3 | 调用 `create_task_template(spec)` | ⚠️ PARTIAL | 工具存在但设置 `is_active=True`（应为 draft） |
| AC-4 | Schema 符合 TaskTemplate | ⚠️ PARTIAL | 基本字段映射，无 Pydantic 验证 |
| AC-5 | 可关联已有 Agent | ⚠️ PARTIAL | list_agents 可用但无关联逻辑 |
| AC-6 | `org_id` + `visibility` 注入 | ✅ DONE | |
| AC-7 | `request_publish()` 调用 | ❌ MISSING | 同 US-003 AC-8 |

---

## US-005: 对话式创建 SOP

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | `{skill: "create-sop"}` 启动 | ✅ DONE | |
| AC-2 | 多轮澄清 | ⚠️ PARTIAL | reply 空壳 |
| AC-3 | 调用 `create_sop(spec)` | ✅ DONE | status="draft" |
| AC-4 | `structured_steps` >= 4 步 | ❌ MISSING | 无验证 |
| AC-5 | `sop_card` >= 5 scenarios | ❌ MISSING | sop_card 字段完全未设置 |
| AC-6 | category + industry_tags | ❌ MISSING | 未处理 |
| AC-7 | 创建后自动生成 embedding | ❌ MISSING | 直接 ORM insert，未调用 SOPService.create() |
| AC-8 | `org_id` + `visibility` 注入 | ✅ DONE | |
| AC-9 | `request_publish()` 调用 | ❌ MISSING | 同上 |

---

## US-006: 发布确认 (Draft -> Published)

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | 创建后返回 `publish_token` | ❌ MISSING | request_publish 未注册到 SDK |
| AC-2 | 前端预览 + 确认按钮 | ⚠️ PARTIAL | PublishConfirm 组件存在但永远不会触发 |
| AC-3 | POST `/confirm-publish` | ✅ DONE | |
| AC-4 | 验证 token + ownership | ✅ DONE | |
| AC-5 | Publish 不经过 LLM | ✅ DONE | |
| AC-6 | Agent `is_active=true`, SOP `status=published` | ✅ DONE | |
| AC-7 | Token 一次性使用 | ✅ DONE | |
| AC-8 | Token 30 分钟过期 | ✅ DONE | |

---

## US-007: 文件访问安全

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | `allowed_tools` 白名单 | ❌ MISSING | `allowed_tools=[]` + `bypassPermissions` = 全部放行 |
| AC-2 | Write/Edit/Bash 禁止 | ❌ MISSING | bypassPermissions 绕过所有限制 |
| AC-3 | PreToolUse hook | ❌ MISSING | security.py 有代码但**未注册到 SDK hooks** |
| AC-4 | 只允许读指定目录 | ⚠️ PARTIAL | 模式定义了但未生效 |
| AC-5 | 路径规范化 + 符号链接检测 | ⚠️ PARTIAL | 代码存在但未调用 |
| AC-6 | 拒绝访问返回安全消息 | ⚠️ PARTIAL | 消息中泄露了文件路径 |
| AC-7 | 反向测试 | ❌ MISSING | 零测试文件 |

---

## US-008: Builder API 端点

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | POST /execute | ✅ DONE | |
| AC-2 | GET /stream SSE | ✅ DONE | |
| AC-3 | POST /reply | ❌ MISSING | 空壳，返回 `{"status": ...}` |
| AC-4 | POST /confirm-publish | ✅ DONE | |
| AC-5 | GET /jobs | ✅ DONE | |
| AC-6 | GET /jobs/{id} | ✅ DONE | |
| AC-7 | GET /jobs/{id}/messages | ⚠️ PARTIAL | 只恢复 text 事件，丢失 tool call/result |
| AC-8 | DELETE /jobs/{id} | ✅ DONE | |
| AC-9 | DELETE /jobs/{id}/drafts | ✅ DONE | |
| AC-10 | 所有端点需认证 | ✅ DONE | |
| AC-11 | SSE 7 种事件 + 心跳 | ⚠️ PARTIAL | publish_ready/waiting_input/heartbeat 缺失 |
| AC-12 | 状态机 | ⚠️ PARTIAL | waiting_input 状态从未进入 |

---

## US-009: 构建任务审计

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | 记录 user/org/skill/status/cost/duration | ⚠️ PARTIAL | cost/duration/tokens 全为 0 |
| AC-2 | 记录创建的实体 | ❌ MISSING | BuilderJobEntity 从未创建 |
| AC-3 | 记录 token 消耗 | ❌ MISSING | |
| AC-4 | 记录失败原因 | ✅ DONE | error_message 有设置 |
| AC-5 | Admin 查看所有记录 | ❌ MISSING | 只按 user_id 过滤 |

---

## US-010: 成本控制

| AC | 描述 | 状态 | 说明 |
|----|------|------|------|
| AC-1 | `max_budget_usd=1.0` | ⚠️ PARTIAL | 设了但未验证 SDK 是否执行 |
| AC-2 | `max_turns=20` | ⚠️ PARTIAL | 实际设为 10，不是 20 |
| AC-3 | 每日配额 free=5, paid=50 | ⚠️ PARTIAL | 逻辑存在但 `is_paid` 永远是 False |
| AC-4 | 超额返回 429 | ✅ DONE | |
| AC-5 | 失败任务消耗半配额 | ❌ MISSING | `consume_quota` 从未被调用 |
| AC-6 | Dashboard 显示用量 | ❌ MISSING | 无端点/UI |

---

## DevPlan Step 完成度

| Step | 描述 | 状态 | 说明 |
|------|------|------|------|
| 1 | DB Migration + ORM | ✅ DONE | 3 张表完整 |
| 2 | Pydantic Schemas | ✅ DONE | |
| 3 | MCP Tools | ⚠️ PARTIAL | 8 工具定义，但 security 未注册、request_publish 未接入 SDK、无 entity tracking |
| 4 | AgentBuilderService | ⚠️ PARTIAL | execute/publish/cancel 存在，但 token/duration/entity 未跟踪，reply 空壳 |
| 5 | API Router | ⚠️ PARTIAL | 9 端点都在，reply 空壳，messages 丢事件 |
| 6 | Quota Service | ⚠️ PARTIAL | 逻辑完整但 consume_quota 从未被调用 |
| 7 | 依赖 + 配置 | ✅ DONE | Python 3.12 venv + SDK 已安装 |
| 8 | 前端 | ✅ DONE | 5 个组件 + service + 路由 |
| 9 | 集成测试 | ❌ MISSING | 零测试文件 |

---

## 10 个关键断裂点 (按优先级)

| # | 问题 | 影响 | 修复难度 |
|---|------|------|---------|
| 1 | `request_publish` 未注册到 SDK 工具列表 | 整个发布流程断裂 | 低 — 加一个 @tool 注册 |
| 2 | `BuilderJobEntity` 从未创建 | confirm_publish 发布空实体 | 低 — create_*_tool 里加 entity 记录 |
| 3 | `reply` 端点是空壳 | 多轮对话不可能 | 高 — 需要 SDK session 持久化 |
| 4 | `permission_mode="bypassPermissions"` | 安全模型完全失效 | 中 — 改 allowed_tools + 注册 hooks |
| 5 | `consume_quota` 从未调用 | 配额形同虚设 | 低 — 在 execute 完成后调 consume |
| 6 | MCP Tools 直接操作 ORM，未用现有 Service | 创建质量低（缺 embedding/card） | 高 — 重写 tools 调用 AgentService/SOPService |
| 7 | `create_task_template` 设 `is_active=True` | 模板不经确认直接上线 | 低 — 改为 False |
| 8 | 无 SSE heartbeat | 长连接被代理/LB 断开 | 低 — 加 15s 定时器 |
| 9 | `cached_events` 在 API 层缓存但未含全部事件类型 | 历史回放不完整 | 中 — messages 端点恢复所有事件类型 |
| 10 | 零集成测试 | 无法验证端到端 | 高 — 需写完整测试套件 |

---

## 总结评分

| 维度 | 总 AC | ✅ Done | ⚠️ Partial | ❌ Missing |
|------|-------|--------|-----------|----------|
| US-003 Agent | 12 | 4 | 5 | 3 |
| US-004 Template | 7 | 2 | 3 | 2 |
| US-005 SOP | 9 | 2 | 1 | 6 |
| US-006 Publish | 8 | 5 | 1 | 2 |
| US-007 Security | 7 | 0 | 3 | 4 |
| US-008 API | 12 | 6 | 4 | 2 |
| US-009 Audit | 5 | 1 | 1 | 3 |
| US-010 Cost | 6 | 1 | 3 | 2 |
| **合计** | **66** | **21 (32%)** | **21 (32%)** | **24 (36%)** |

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Yes, every time with the same prompt

Steps to Reproduce

_No response_

Claude Model

Sonnet

Relevant Conversation

Impact

High - Significant unwanted changes

Claude Code Version

2.1.76

Platform

Anthropic API

Additional Context

_No response_