CRASH: Staging API cold start failure - CORS errors due to scale-to-zero + 1hr cooldown

Incident Summary

Date: 2026-01-30 (reported) / 2026-01-31 01:42 UTC (investigated)
Environment: Staging (ca-plexor-stg)
Impact: Complete service unavailability - users unable to login or use playground
Duration: Unknown (container was at 0 replicas when discovered)

User-Reported Symptoms

1. "No response in playground"
2. Login taking "super long time"
3. CORS error blocking requests:

Access to fetch at 'https://staging.api.plexor.dev/v1/auth/login' from origin 
'https://staging.plexor.dev' has been blocked by CORS policy: Response to preflight 
request doesn't pass access control check: No 'Access-Control-Allow-Origin' header 
is present on the requested resource.

Root Cause Analysis

Primary Cause: Scale-to-Zero + Extended Cooldown Period

The container app was configured with:

• minReplicas: 0 (scale to zero when idle)
• cooldownPeriod: 3600 (1 hour) - changed from 5 minutes on 2026-01-28

This combination meant:

1. After 1 hour of low traffic, container scaled to 0 replicas
2. No backend available to respond to CORS preflight OPTIONS requests
3. Browser blocked all requests due to missing CORS headers
4. Cold start delay (15-77 seconds for ML model loading) exacerbated the issue

Why CORS Fails at 0 Replicas

Browser → OPTIONS /v1/auth/login (preflight)
       → Azure Container Apps Ingress
       → No running replicas (0/0)
       → 502/503 with NO CORS headers
       → Browser blocks request

The CORS middleware code is correct but never executes because there's no container to run it.

Forensic Timeline

| Timestamp (UTC) | Event |
|-----------------|-------|
| 2026-01-28 | Cooldown changed from 5min to 1hr (commit 4907b77) |
| 2026-01-30 01:00:02 | Last deployment (revision 0000456) |
| ~2026-01-30 02:00 | Container likely scaled to 0 after cooldown |
| 2026-01-30 ~evening | Users report playground issues |
| 2026-01-31 01:42:35 | Investigation begins |
| 2026-01-31 01:45:28 | Manual scale-up triggered (revision 0000457) |
| 2026-01-31 01:45:50-01:46:05 | 16 startup probe failures during cold start |
| 2026-01-31 01:47:07 | BERT ML model fully loaded |
| 2026-01-31 01:47:15+ | Service operational |

Revision Gap Analysis

• Revision 0000456 created: 2026-01-30 01:00:02 UTC
• Revision 0000457 created: 2026-01-31 01:45:28 UTC
• Gap: ~24 hours 45 minutes of potential unavailability

Technical Details

Container App Configuration (at time of incident)

{
  "minReplicas": 0,
  "maxReplicas": 2,
  "cooldownPeriod": 3600,
  "pollingInterval": 30,
  "rules": [{"http": {"concurrentRequests": "10"}}]
}

Cold Start Breakdown

| Phase | Duration |
|-------|----------|
| Image pull (1.63 GB) | ~15 seconds |
| Container creation | ~2 seconds |
| Startup probe failures | ~15 seconds |
| ML model loading (BERT) | ~60 seconds |
| Total cold start | ~77 seconds |

Supporting Infrastructure Status

| Component | Status |
|-----------|--------|
| Cosmos DB | ✅ 100% available, 1.4ms latency |
| Key Vault | ✅ Operational |
| Container Registry | ✅ Image pull successful |
| App Insights | ⚠️ Empty request telemetry (instrumentation gap) |

Code Analysis Findings

CORS Configuration: ✅ Correct

• staging.plexor.dev in allowed origins list
• CustomCORSMiddleware properly implemented
• Error response CORS headers handled

Gateway API: ⚠️ Potential Issues Found

1. Empty response handling could return incomplete SSE streams
2. Provider selection doesn't validate API keys until call time
3. print() statements instead of structured logging

Token Refresh Service: ❌ Failing

{
  "error_type": "invalid_response",
  "error_message": "Unexpected status: 400",
  "event": "token_refresh_permanent_failure"
}

Key Vault persistence disabled (missing Azure SDK).

Immediate Fix Applied

az containerapp update --name ca-plexor-stg --resource-group rg-plexor-alpha \
  --min-replicas 1 --max-replicas 3

Current status: Service operational with 1 replica running.

Recommended Actions

P0 - Immediate (Cost-Aware Fix)

• [ ] Revert cooldown from 1 hour back to 5 minutes
• [ ] Keep minReplicas: 0 for cost management
• [ ] This allows scale-to-zero but recovers faster from cold starts

P1 - Short Term

• [ ] Add Azure Container Apps ingress-level CORS policy (responds even at 0 replicas)
• [ ] Implement uptime monitoring that hits staging every 4 minutes (keeps it warm)
• [ ] Fix health endpoint inconsistency (/health vs /v1/health)

P2 - Medium Term

• [ ] Reduce container image size (currently 1.63 GB)
• [ ] Lazy-load ML models after startup probe passes
• [ ] Add proper OpenTelemetry instrumentation for App Insights
• [ ] Fix token refresh service (install azure-identity SDK)

P3 - Architecture

• [ ] Consider Azure Front Door for CORS at edge level
• [ ] Implement pre-warming script in CI/CD pipeline
• [ ] Add synthetic availability tests in App Insights

Infrastructure Change Required

File: infra/azure/container-app.bicep

- var cooldownPeriod = isProduction ? 300 : 3600  // Staging: 1hr
+ var cooldownPeriod = 300  // 5 minutes for all environments

Lessons Learned

1. Scale-to-zero + CORS = silent failures - Browser can't distinguish "no backend" from "CORS blocked"
2. 1 hour cooldown too aggressive for staging - Users hit cold starts frequently
3. 77 second cold start is too long - ML model loading dominates startup time
4. App Insights gaps - No request telemetry made forensics difficult

Related Issues

• Issue #1447: CORS trailing slash normalization
• Issue #1451: CORS headers on error responses
• Commit 4907b77: Cooldown period change

Attachments

• Container logs: Available in Azure Portal
• Scale configuration: az containerapp show --name ca-plexor-stg -g rg-plexor-alpha

---
Investigated by: Claude Code (Partner-level SDE analysis)
Resolution: Revert cooldown to 5 minutes, implement edge-level CORS