Backgrounded command sometimes runs as multiple concurrent copies, causing real corruption for non-idempotent operations

Summary

A long-running Bash command started via the run_in_background mechanism sometimes ends up executing as multiple concurrent copies instead of one. Observed 5+ times across one extended session, escalating from "wasted duplicate work" to real data corruption in the most recent occurrence.

Impact (most recent occurrence)

A background shell command was launched to apply a sequence of database migration files across several newly-created databases. The command was not idempotent (each migration file is meant to run exactly once). Instead of one process running the sequence, three concurrent copies of the same script ended up running against the same databases at the same time — confirmed by observing two of the processes independently reporting different "currently processing" file names for the same target.

This corrupted the schema of the affected databases (partial/duplicate application of a non-idempotent sequence). Recovery was only possible because the affected databases were newly-created, empty of real data, and not yet referenced by any deployed service — they were deleted and recreated from scratch. Had this happened against an already-populated resource, this would not have been cleanly recoverable.

Suspected mechanism

Directly and repeatedly observed in the same session: a backgrounded Bash command occasionally reports something like:

Command did not complete within its Ns timeout and was moved to the background (ID: ...)

If there is retry/resubmission logic that fires when a command appears to have stalled or timed out — without first confirming whether the original invocation is still alive and actually running in the background — that would produce exactly this signature: the original process keeps running (legitimately backgrounded, not dead), and one or more additional, independent processes start executing the identical command shortly after, because something decided the original needed to be retried.

Across the 5+ occurrences in this session, the number of concurrent duplicates has varied (2, then 3 in the most recent case) — worth checking whether the duplicate count scales with retry attempts, elapsed time, or system load, which would corroborate this hypothesis.

Why this is hard to reproduce reliably

• It appears tied to a command exceeding its timeout while still legitimately running, which depends on system load and command duration — not something a minimal repro can force deterministically.
• Every occurrence so far has involved a genuinely long-running shell command (multi-minute database operations), not a trivial one.

Suggested fix direction

Before resubmitting/retrying a backgrounded command that appears to have stalled, check whether a process matching the original invocation is still alive and actively producing output, and skip the retry if so.

Severity

This session treated it as low-priority for a while because every prior occurrence was caught and safely contained with no lasting effect. That is no longer true — this occurrence caused real, only-by-luck-recoverable data corruption. For any user running non-idempotent operations (database migrations, financial transactions, irreversible API calls, etc.) via a backgrounded command, this is a correctness risk, not just an efficiency one.