fix: Health check dashboard — resolve red status checks across environments

Current Status Analysis

Dashboard Overview:

• ✅ Infrastructure established: Lambda + DynamoDB + Next.js frontend
• ✅ 50 API endpoints monitored (model endpoints removed to reduce costs)
• ⚠️ Multiple red/unhealthy checks across dev, demo, prod environments
• ⚠️ Production ping initially returning 301 redirects (fixed via redirect handling)

Red Status Checks Identified

1. Prod Ping: HTTP 301 Redirect (FIXED)

• Issue: Endpoint returning 301 instead of 200 (redirect loop)
• Root Cause: Prod gateway enforcing HTTPS redirect or path normalization
• Fix Applied: Updated httpGet() to follow redirects (up to 5 hops)
• Status: ✅ Deployed in latest Lambda version

2. API-Level Checks: Degraded Status (401/422 Errors)

Many endpoints return degraded (non-200) due to missing authentication context:

• /spend/logs — requires active user session + spend visibility
• /report/optimize — requires organization/team context
• /guardrails/* endpoints — require team/org setup
• /audit/* — requires audit permissions
• /organization/* — requires org context
• /team/*, /user/* — require valid user_id in context

Root Cause: Health check uses master key, which lacks user/team/org context. Gateway returns 401/422 for endpoints requiring business context.

Solution Options:

1. Accept degraded status as normal — health checks verify endpoints exist, not that they work with data
2. Use default user_id — pass a universal test user UUID (currently using ab01ded1-a124-4baa-b65b-089bf80d6fce)
3. Create health-specific endpoints — add /health/api-check that returns 200 without auth (simple ping)
4. Exclude context-dependent endpoints — remove endpoints requiring team/org/user data from health checks

3. Infrastructure Checks: ECS/ALB Status

• ping=healthy ✅ — liveliness check passing
• ecs=healthy/unhealthy — depends on task count matching desired
• alb=healthy/degraded — depends on target group health
• waf=healthy/disabled — depends on WAF config in each environment

Requirements:

• ECS services must be running with desired task count = actual count
• ALB targets must be healthy (security group, routing, app health)
• WAF must be enabled and reachable (or explicitly disabled config)

Cost Impact (Now Fixed)

Before: Health checks cost ~$2.23/day (~$800/year)

• Reason: Model endpoints (/models, /v1/models) invoke Bedrock for live model data
• Impact: Every health check hit the LLM provider unnecessarily

After: Health checks cost ~$0.00/day

• Removed model endpoints from check list
• Only metadata/governance endpoints checked (no LLM invocation)
• ✅ Reduced from 60 endpoints to 50 endpoints

Implementation Tasks

To make all checks green:

1. Verify prod environment setup:

• Confirm ECS tasks running and healthy
• Verify ALB target health (security groups, routing rules)
• Check WAF is properly configured

1. Choose API check strategy:

• Option A: Accept yellow/degraded as normal (endpoints are alive, just need context) — simplest, no code change
• Option B: Add universal test user to health check requests — pass UUID header to all API calls
• Option C: Create lightweight health endpoints that skip auth — add /health/api-check that returns 200

1. Test each environment:

• Deploy updated Lambda
• Trigger health check via EventBridge or manual invoke
• Verify all checks reach green or expected status

Files Modified

• health-dashboard/lambda/check/index.ts — redirect handling, model endpoints removed
• health-dashboard/src/app/[environment]/api/ApiDetail.tsx — timeline visualization, search, day filters
• Makefile — added deploy-health-lambda target

Success Criteria

• [ ] Prod ping returns 200 (green) — redirect fix deployed
• [ ] Choose and implement API check strategy (A/B/C above)
• [ ] All environment health checks displaying expected status
• [ ] No LLM cost incurred by health checks
• [ ] Dashboard accessible and showing real-time health data