[Refund] wasted-loop + false-completion: 5-day training pipeline burned $15+ in GPU compute due to repeated preventable failures

Summary

Over a 5-day session (April 3-7, 2026), Claude Code (Opus 4.6) was tasked with fine-tuning an LLM (CPT + SFT) on RunPod GPUs. The model repeatedly violated its own documented rules, wasted GPU compute through preventable errors, and ultimately lost 28 hours of training output by failing to use persistent storage — a basic infrastructure requirement it was explicitly configured to know about.

Failure Type

wasted-loop + false-completion + silent-degradation

Timeline

• Apr 3: CPT training on LLaMA 3.1 8B. 3 pods rented and failed before SSH worked. Version mismatch (transformers 5.5.0 vs 4.45.2) crashed training twice. Disk full prevented model merge. ~$3 burned on failed pods.
• Apr 4: Switched to LoRA SFT despite methodology document explicitly stating "Full fine-tuning or MoRA, NOT LoRA." Trained Design Director with wrong method. User caught the violation. ~$1 burned.
• Apr 5: User pointed out LLaMA 3.1 8B (2024) is 2 years behind Qwen 3.5 9B (2026). Restarted pipeline on Qwen. Multiple dependency failures: PyTorch 2.4→2.6 upgrade needed, torchvision mismatch, TRL 1.0.0 API changes (tokenizer→processing_class, deprecated dataset_text_field, deprecated max_seq_length). Each fix required uploading a new script via runpodctl because PTY SSH garbles characters. ~$2 burned debugging.
• Apr 5-7: CPT finally running on Qwen 3.5 9B. 28 hours, 5945 steps. Completed successfully. SFT chained automatically. Pod deployed with volumeInGb: 0 (no persistent storage). Monitor script stopped pod after SFT completed but before adapter transfer finished.
• Apr 7: Pod restarted to download adapters. Container disk wiped. All 28 hours of CPT + SFT output lost. $8 of compute gone.

Evidence

1. Ignored own methodology: User's cpt-methodology.md states "Full fine-tuning or MoRA, NOT LoRA (too low-rank for knowledge acquisition)". Claude used LoRA rank 32 anyway.
2. No network volume: Deployed with volumeInGb: 0 despite RunPod docs explicitly stating container disk is ephemeral. Lost all training output.
3. 97 subagent sessions spawned in this conversation (52MB of subagent transcripts), many for tasks that failed and had to be redone.
4. PTY relay character doubling: Every SSH command through RunPod's relay doubled characters, breaking pip install URLs, sed commands, and environment variables. Claude kept trying the same broken approach instead of switching to file-upload-only.
5. Version mismatch cycle: transformers 5.5.0 broke set_submodule → fixed PyTorch → broke torchvision → fixed torchvision → broke TRL API → fixed tokenizer param → broke dataset_text_field → fixed that → broke max_seq_length. Each iteration burned 5-10 min of GPU time.

What Correct Behavior Would Have Been

1. Read the methodology doc BEFORE choosing LoRA/MoRA — the answer was written down
2. Use a network volume — RunPod 101, documented everywhere
3. Test the full training pipeline locally (dry run, no GPU) before renting any pod
4. Pin ALL dependency versions in a requirements.txt tested locally before deploying
5. Transfer checkpoints to VPS incrementally during training, not after
6. Use file uploads (runpodctl) exclusively — never rely on PTY SSH for commands

Token Waste Estimate

| Item | Size | Est. Tokens |
|---|---|---|
| Main session (5 days, Opus 4.6 1M context) | ~500K+ | ~2M+ |
| 97 subagent sessions (Sonnet + Opus) | 52MB | ~13M |
| Failed Sonnet agents (rate limited) | ~5MB | ~1.25M |
| Total estimated tokens | | ~16M+ |
| With 50% time markup | | ~24M |
| API-equivalent cost @ $15/1M (Opus) | | ~$240 |

External Costs Incurred

| Item | Cost |
|---|---|
| RunPod GPU compute (wasted) | ~$15 |
| RunPod GPU compute (lost output) | ~$8 |
| Apify rental ($5/mo, never used successfully) | $5 |
| User time (5 days, multiple sleep interruptions) | Unquantifiable |
| Total external waste | ~$28 |

Environment

• Claude Code v2.1.92
• Model: Opus 4.6 (1M context)
• Subscription: Claude Max
• User had extensive rules, hooks, and methodology docs configured — Claude violated them repeatedly

Requested Resolution

User is on Claude Max subscription and requests at minimum $40 in extra usage tokens as compensation for:

• 5 days of wasted time
• $28 in external costs (GPU + services) burned by preventable mistakes
• Violation of user's own documented rules that Claude was explicitly configured to follow
• Loss of 28 hours of irreproducible training output due to basic infrastructure oversight

The user set up comprehensive rules (VIGIL accountability, thinking discipline, zero-tolerance policy, compute durability rules) specifically to prevent these failures. Claude acknowledged and quoted these rules throughout the session while simultaneously violating them.