Dagster jobs causing ephemeral-storage evictions on eks-prod-legacy

Summary

Dagster jobs on eks-prod-legacy are being evicted due to ephemeral-storage exhaustion on Karpenter-managed nodes.

Evidence

Active evictions observed:

4m41s   Warning   Evicted   pod/claims-prod-97cc97-58b459896f-4nfnr   
        The node was low on resource: ephemeral-storage. 
        Threshold quantity: 3210844697, available: 2997940Ki.

Affected pods (Error state):

• ai-cw-virtual-agent-dagster-prod-* (multiple)
• claims-prod-97cc97-* (multiple)
• eld-management-prod-* (multiple)

Problem node: ip-10-0-77-203.ec2.internal (dagster nodepool, r5a.2xlarge)

Root Cause

1. Insufficient disk provisioning - EC2NodeClass default has no blockDeviceMappings, resulting in ~17GB allocatable ephemeral storage per node (Bottlerocket default)

1. Under-requested storage - All Dagster jobs request only 1Gi ephemeral-storage but appear to use significantly more

1. High pod density - Some dagster nodes have 19+ pods; concurrent job execution exhausts available disk

Impact

• Intermittent Dagster job failures
• Pods evicted mid-execution
• Jobs need manual re-triggering

Recommended Fixes

Infrastructure (DevOps)

• Add blockDeviceMappings to EC2NodeClass with larger root volume (e.g., 100Gi)
• Consider creating a dedicated EC2NodeClass for dagster with appropriate disk sizing

Application (Data Team)

• Profile actual disk usage of jobs
• Update ephemeral-storage requests to match actual usage
• Ensure jobs clean up temp files after processing
• Consider PVCs for large data processing workloads

Related Context

This aligns with observations about "intermittent failures happening to some dagster jobs" related to ephemeral disk availability when running multiple jobs concurrently.

---
🤖 Generated with Claude Code