Claude Code overwrote ML training checkpoint causing ~16h GPU compute loss

Resolved 💬 5 comments Opened Mar 10, 2026 by warp143 Closed Mar 13, 2026

Summary

Claude Code caused significant loss of ML training work by using resume=False with exist_ok=True on an existing YOLO training run, which overwrote best.pt, last.pt, and results.csv — destroying a 20-epoch checkpoint (mAP50-95=0.885) that took ~20 hours to train on GPU.

What happened

  1. User had completed 20 epochs of YOLOv26x-seg training (mAP50=0.986, mAP50-95=0.885)
  2. User asked Claude to continue training from epoch 20 to epoch 30
  3. Claude attempted resume=True which failed (YOLO AssertionError: training finished)
  4. Claude then used resume=False with the same run name and exist_ok=True as a workaround
  5. This silently overwrote the existing best.pt, last.pt and results.csv — destroying the epoch 20 checkpoint permanently
  6. Disk was at 98% full with no snapshots, and models/ was in .gitignore so no backup existed

Impact

  • ~20 hours of GPU training time lost (epoch 20 weights unrecoverable)
  • ~16 additional hours of recovery training required to get back to baseline
  • Total: ~36 hours of wasted GPU compute

Expected behavior

Before taking any action that overwrites existing model weights/checkpoints, Claude should warn the user and ask for confirmation. Overwriting trained ML model files is an irreversible destructive action.

Suggested fix

Claude should treat ML checkpoint files (.pt, .pth, results.csv in training run directories) as sensitive files requiring explicit confirmation before overwrite, similar to how it should confirm before rm or force-push.

View original on GitHub ↗

This issue has 5 comments on GitHub. Read the full discussion on GitHub ↗