Claude Code overwrote ML training checkpoint causing ~16h GPU compute loss
Summary
Claude Code caused significant loss of ML training work by using resume=False with exist_ok=True on an existing YOLO training run, which overwrote best.pt, last.pt, and results.csv — destroying a 20-epoch checkpoint (mAP50-95=0.885) that took ~20 hours to train on GPU.
What happened
- User had completed 20 epochs of YOLOv26x-seg training (mAP50=0.986, mAP50-95=0.885)
- User asked Claude to continue training from epoch 20 to epoch 30
- Claude attempted
resume=Truewhich failed (YOLO AssertionError: training finished) - Claude then used
resume=Falsewith the same run name andexist_ok=Trueas a workaround - This silently overwrote the existing best.pt, last.pt and results.csv — destroying the epoch 20 checkpoint permanently
- Disk was at 98% full with no snapshots, and models/ was in .gitignore so no backup existed
Impact
- ~20 hours of GPU training time lost (epoch 20 weights unrecoverable)
- ~16 additional hours of recovery training required to get back to baseline
- Total: ~36 hours of wasted GPU compute
Expected behavior
Before taking any action that overwrites existing model weights/checkpoints, Claude should warn the user and ask for confirmation. Overwriting trained ML model files is an irreversible destructive action.
Suggested fix
Claude should treat ML checkpoint files (.pt, .pth, results.csv in training run directories) as sensitive files requiring explicit confirmation before overwrite, similar to how it should confirm before rm or force-push.
This issue has 5 comments on GitHub. Read the full discussion on GitHub ↗