Dagster jobs causing ephemeral-storage evictions on eks-prod-legacy

Additional Finding: Stale Code Server Deployments

Investigation revealed that multiple old code server versions are still running but essentially idle, contributing to pod density and disk pressure.

Activity Analysis (Last 2 Hours)

| Code Server | Age | Log Lines | Status |
|-------------|-----|-----------|--------|
| claims-prod-97cc97 | 8 months (May 2025) | 2 | ❌ Idle (startup only) |
| claims-prod-92a9c3 | ~1 hour | 86 | ✅ Active |
| eld-management-5c2518 | 8 months (May 2025) | 23 | ⚠️ Low activity |
| eld-management-91b53a | 10 days | 63 | ✅ Active |

Evidence

=== Activity comparison - OLD vs NEW claims ===
--- OLD (97cc97) last 2 hours ---
       2
--- NEW (92a9c3) last 2 hours ---
      86

The OLD claims-prod-97cc97 has only 2 log lines (just startup messages) - it's not processing any work.

Root Cause

Dagster Cloud deployments create new code server versions on each deploy, but old versions are retained for rollback capability. There's no automatic cleanup of very old versions, resulting in:

• 10 code server deployments running when only 4 are needed (one per project)
• Each deployment requests 1Gi ephemeral storage
• Combined with dagster-run pods, this causes over-commitment on nodes

Recommended Cleanup

Safe to delete (Data Team confirmation recommended):

• claims-prod-97cc97 - Clearly idle, 8 months old
• eld-management-prod-5c2518 - Much less active than newer versions

Impact of cleanup:

• Free 2+ pods worth of resources
• Reduce ephemeral storage requests by 2Gi+
• Lower pod density on dagster nodes
• Reduce likelihood of eviction cascades

Action Items

1. Data Team: Review and confirm old deployments can be deleted
2. Data Team: Implement retention policy for code server versions (keep last N versions)
3. DevOps: Consider adding monitoring for stale deployments

---
🤖 Generated with Claude Code

Summary: Root Cause Analysis

| Factor | Finding |
|--------|---------|
| Trigger | Multiple jobs running concurrently exhaust disk |
| Root Cause 1 | Nodes only have 17Gi disk (default Bottlerocket, no blockDeviceMappings configured) |
| Root Cause 2 | 10 code servers running when 4 would suffice |
| Root Cause 3 | Stale 8-month-old deployments never cleaned up |
| Root Cause 4 | 1Gi ephemeral requests underestimate actual usage |

Recommended Actions

Quick Wins

1. Delete stale code servers (Data Team) - immediate relief, free 2Gi+ requests
2. Increase node disk via EC2NodeClass blockDeviceMappings (DevOps) - e.g., 100Gi root volume

Longer Term

1. Data Team implements code server retention policy (keep last N versions)
2. Profile actual disk usage per job and update ephemeral-storage requests
3. Add disk pressure monitoring/alerting for dagster nodes

---
🤖 Generated with Claude Code

Resolution

Fix has been applied to devops-app-of-apps-k8s repo. All EC2NodeClasses now have proper blockDeviceMappings for Bottlerocket:

| NodeClass | Boot (/dev/xvda) | Data (/dev/xvdb) | Status |
|-----------|------------------|------------------|--------|
| ai | 10Gi | 100Gi | ✅ |
| dagster | 10Gi | 100Gi | ✅ |
| default | 10Gi | 100Gi | ✅ |
| devops | 10Gi | 100Gi | ✅ |

Impact

• Before: ~17Gi allocatable ephemeral storage per node
• After: ~95Gi allocatable ephemeral storage per node

Verification

New nodes confirmed provisioning with larger disk:

ip-10-0-64-223.ec2.internal: 95Gi allocatable (was 17Gi)

Remaining Items (Data Team)

• Clean up stale code server deployments (8-month-old versions still running)
• Implement retention policy for code server versions
• Profile actual disk usage and update ephemeral-storage requests accordingly

---
🤖 Generated with Claude Code

Closing - infrastructure fix applied. Remaining items (stale code server cleanup) are Data Team responsibility.

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.