[FEATURE] Self-tuning Claude — agents that iteratively improve their own instructions with testing

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

There's no way to deterministically test whether a change to .claude/ instruction files improves or worsens agent behavior. You edit a rule, rerun the agent, and eyeball whether it did better. There's no baseline, no regression detection, no way to know if a fix for one task broke another.

The Ralph Wiggum Technique solved this for task execution — Claude doesn't quit when it thinks it's done, it quits when the work is verified. But that loop improves a single run. The instructions that shaped the behavior are unchanged at the end.

What's missing is a test harness for instruction files — the same verified loop applied one level up. Run fresh agents against diverse targets using the current instructions, score the results, update the instructions, and verify the update actually improved outcomes. Revert on regression, keep on improvement. Code has tests. Instructions should too.

Proposed Solution

There's no test harness for .claude/ instruction files. Code has tests — you change a function, run the suite, and know immediately if you broke something. Instruction files have nothing. You edit a rule, rerun the agent, and eyeball whether it did better. There's no baseline, no regression detection, no way to know if a fix for one task broke another.

The Ralph Wiggum Technique showed that Claude works best in a verified loop — it doesn't quit when it thinks it's done, it quits when the work passes. That same principle should apply to instruction improvement: don't stop editing instructions when you think they're better, stop when fresh agents demonstrably pass across diverse targets.

Here's how it works:

An orchestrator agent creates sub-agent clones with its instructions (.claude/) that run in isolated git worktrees without access to the parent's context — each performing the same type of task against a different target.

After each completes, the orchestrator analyzes the performance of every clone. It updates its own instructions — reverting if there's a regression, changing approach to better accomplish the goal, or optimizing to use fewer tool calls and tokens.

Then the orchestrator creates new sub-agent clones with the updated instructions, again in isolated worktrees, and the loop repeats.

Because the same instruction set runs against several different targets simultaneously, it naturally tests that changes satisfy the goal for each without breaking the others. This is the generalization mechanism — overfitting to one target is caught by regression on the others.

The loop converges when fresh clones (clean context, no hints) consistently accomplish the goal across all targets.

The .claude/ instructions aren't static — they're the product of iterative refinement. The the plugin should launch parallel agents against real targets, scores their results, diagnoses failures, and fix the instructions. The agents' code is throwaway. The instruction improvements are the product.

graph TD
    O[Orchestrator] -->|"launches 8 parallel agents<br/>in isolated worktrees"| A1[Agent 1]
    O --> A2[Agent 2]
    O --> AN[Agent N...]

    A1 --> S[Score against 18-check scorecard]
    A2 --> S
    AN --> S

    S --> D{All passed?}
    D -- Yes --> C["Instructions converged"]
    D -- No --> F["Diagnose: which rule<br/>was too soft or missing?"]
    F --> R["Fix the instruction<br/>(generalized, not site-specific)"]
    R --> CC["Consistency check<br/>across all .claude/ files"]
    CC --> O

Alternative Solutions

_No response_

Priority

Low - Nice to have

Feature Category

Developer tools/SDK

Use Case Example

I built a proof of concept of this pattern at github.com/adam-s/intercept. It's rough and experimental — I was learning Claude Code skills and agents as I went — but it demonstrates the core loop working in practice.

The project discovers website APIs via browser traffic interception and creates typed proxy routes. The .claude/ instructions define a 5-step discovery protocol that agents follow. The goal: a fresh agent, given only the instructions and a target website, produces a working API plugin in one shot.

Here's how the tuning loop worked:

1. I defined the goal and wrote initial instructions in .claude/rules/discovery.md — a protocol for discovering a website's API transport layer.
2. The orchestrator launched 6-8 sub-agents in parallel, each in an isolated worktree, each targeting a different website. Same instructions, different targets.
3. After each completed, the orchestrator scored them against an 18-check scorecard — did they follow the pipeline in order? Fill the transport elimination table? Build routes for every transport found? Stay within budget?
4. For each failure, the orchestrator traced it to a specific instruction gap. Example: agents kept retrying 429 responses because the "don't retry" rule was in workflow.md but not in discovery.md. Fix: make the rule consistent across both files.
5. Updated instructions, re-cloned, re-ran. Checked that the fix didn't regress on other targets.
6. Repeat.

Early on, this was highly collaborative — I was in the loop every iteration, contributing ideas, catching things Claude missed, learning the system alongside it. Over ~40 iterations, it shifted. Claude handled more of the diagnosis and fixing unsupervised. By the later iterations, I was mostly reviewing diffs and approving commits while Claude ran the loop.

The instructions went from producing agents that wandered aimlessly to agents that consistently follow the protocol and produce working APIs across diverse websites — XHR pagination, GraphQL, WebSocket, embedded JSON, logged in session-gated endpoints.

Additional Context

_No response_