[BUG] Are these model capable enough to create technology knowledge base?
Preflight Checklist
- [x] I have searched existing issues and this hasn't been reported yet
- [x] This is a single bug report (please file separate reports for different bugs)
- [x] I am using the latest version of Claude Code
What's Wrong?
I have been trying to create a technology knowledge base by parsing various programming languages and createa meaningful RAG pipeline which can help my team mates/newbies joining the project. I have scrapped the progress 2 times and I am in my third run right now. I don't even know whether these models have the capability to create. Even if it is not, i expect to align after going through the existing documentation. Below is the summary
SESSION SUMMARY — Claude Code / build-feature harness
TASK: Build schema.py + scanners/base.py (foundation batch 1/4) for a Python
project, using a custom multi-agent harness (system-analyst → reviewer →
creator → reviewer → senior-reviewer → tester loop, "high" rigor config).
OUTCOME: 5 files delivered (schema.py, scanners/base.py, scanners/__init__.py,
2 test files), 24 tests passing, ruff clean. Not yet fully terminal — an
independent reviewer pass stalled/timed out (600s) near the end and needed
redispatch; final tester phase had not completed as of last update.
SCALE: ~5-6 hours elapsed wall-clock time. ~30+ subagent calls, roughly
4-5 million tokens (subagent usage only, rough estimate, not exact).
Planning alone took 3 revision cycles before human approval. Construction
took ~9 creator fix iterations against repeated reviewer/senior-reviewer
findings on a single 169-line file.
ISSUES ENCOUNTERED (worth flagging as feedback):
- Async coordination fragility: background subagents repeatedly produced
false "task complete" signals (3 separate bad polling heuristics in the
orchestrator: idle-timeout, a grep matching its own prompt text, a grep
matching a stale prior iteration's output). Each was self-caught before
causing harm, but consumed significant back-and-forth.
- I (the assistant) relayed a stale/unconfirmed status to the user at one
point ("plan is CLEAN, ready for approval") before the real verdict had
landed — corrected once caught, but shouldn't have happened.
- I initially deferred ~16 engineering judgment calls to the user for
approval that I should have decided myself (only 2-3 were genuine
spec-defect/business decisions) — user had to push back before I
corrected course.
- Loop budget (MAX_LOOPS=7, CLEAN_REVIEWS_REQUIRED=3) was undersized for
the rigor level chosen; had to be raised mid-session (to 10 and 2
respectively) to avoid discarding already-fixed work on a technicality.
- The review loop found real, substantive defects repeatedly (a data
validation gap invisible to structural tests, a .gitignore negation
bug, contradictory docstrings, false spec citations) — the rigor was
NOT wasted, but the cost-per-defect-found was high for a small file.
USER FEEDBACK GIVEN DURING SESSION:
- Preferred synchronous/foreground agent execution over background async
once coordination bugs surfaced.
- Pushed back hard on the assistant deferring ordinary engineering
decisions to the user instead of owning them ("why are you finding a
human scapegoat... you are the one who planned it upfront").
What Should Happen?
create a simple python programs to extract source code meanifully using AST and do the RAG process. I have created a loop engineering harness and it is taking me to a spin.
using opus for reviews and sonnet for building
Error Messages/Logs
Steps to Reproduce
i have pasted a summary above
Claude Model
Opus
Is this a regression?
Yes, this worked in a previous version
Last Working Version
_No response_
Claude Code Version
Claude Code version 2.1.205.
Platform
Anthropic API
Operating System
Windows
Terminal/Shell
VS Code integrated terminal
Additional Information
_No response_
This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗