Production-Ready Timeout and Retry Strategy for LLM Services

Problem Statement

Currently, LLM API calls can hang indefinitely in production, causing blocked sessions and poor user experience. This was observed in session 8DT981bAmb7Vs9HY5amIq6XT_Gn54nWfkXklNzwsYfQ where a GCP Vertex AI call hung for 2+ minutes without timing out, blocking the entire orchestration pipeline.

Current State Analysis

AWS Bedrock (BEST) ✅

• Timeout: 60s read, 10s connect (boto3 Config)
• Retries: 10 attempts with adaptive mode
• Error Handling: Comprehensive exception mapping (ThrottlingException, ModelTimeoutException, etc.)
• Status: Production-ready

Azure OpenAI (GOOD) ⚠️

• Timeout: 30s (via OpenAI SDK timeout parameter)
• Retries: 3 attempts (via max_retries parameter)
• Error Handling: Uses OpenAI SDK's built-in error handling (APITimeoutError, APIError)
• Status: Adequate but could be improved

GCP Vertex AI (CRITICAL) ❌

• Timeout: None - chat_session.send_message() can hang indefinitely
• Retries: None at service level (relies on SDK defaults)
• Error Handling: Basic exception catching only
• Status: Production risk - MUST FIX

Observed Production Issue

# app/services/llm/vertex_ai_service.py:325
response = await chat_session.send_message(input_text)  # ⚠️ NO TIMEOUT

Impact: Sessions hang indefinitely, requiring container restarts. Fallback mechanism cannot activate because the call never fails - it just hangs.

---

Proposed Solution: Unified Timeout & Retry Strategy

Architecture Goals

1. Consistency: All providers should have comparable timeout/retry behavior
2. Observability: Clear logging and metrics for timeout/retry events
3. Configurability: Easy to tune per environment (dev vs prod)
4. Graceful Degradation: Timeout failures should trigger fallback, not crash
5. Production Hardened: Tested patterns from industry best practices

Technical Design

1. Configuration Layer (app/core/config.py)

Add unified LLM timeout/retry configuration:

class Settings(BaseSettings):
    # Unified LLM Service Timeouts
    LLM_DEFAULT_TIMEOUT: int = Field(
        default=60, 
        description="Default timeout for LLM API calls in seconds"
    )
    LLM_TRIAGE_TIMEOUT: int = Field(
        default=30, 
        description="Timeout for fast triage operations in seconds"
    )
    LLM_ORCHESTRATION_TIMEOUT: int = Field(
        default=90, 
        description="Timeout for complex orchestration reasoning in seconds"
    )
    LLM_GENERATION_TIMEOUT: int = Field(
        default=45, 
        description="Timeout for text generation operations in seconds"
    )
    
    # Retry Configuration
    LLM_MAX_RETRIES: int = Field(
        default=3, 
        description="Maximum retry attempts for transient failures"
    )
    LLM_RETRY_BASE_DELAY: float = Field(
        default=1.0, 
        description="Base delay for exponential backoff (seconds)"
    )
    LLM_RETRY_MAX_DELAY: float = Field(
        default=10.0, 
        description="Maximum delay between retries (seconds)"
    )
    LLM_RETRY_JITTER: bool = Field(
        default=True, 
        description="Add random jitter to retry delays"
    )
    
    # Circuit Breaker (existing but enhance)
    CIRCUIT_BREAKER_FAILURE_THRESHOLD: int = Field(
        default=3,
        description="Consecutive failures before opening circuit"
    )
    CIRCUIT_BREAKER_TIMEOUT_SECONDS: int = Field(
        default=300,
        description="Seconds to wait before attempting recovery (5 min)"
    )

Rationale:

• Operation-specific timeouts: Triage should be fast (<30s), orchestration can take longer (90s)
• Exponential backoff: Prevents thundering herd on retry
• Jitter: Distributes retry load across time
• Configurable per environment: Production vs development needs differ

---

2. Retry Decorator with Exponential Backoff

Create app/services/llm/retry_decorator.py:

import asyncio
import random
import logging
from functools import wraps
from typing import Callable, Tuple, Type
from app.core.config import settings

logger = logging.getLogger(__name__)

# Transient errors that should trigger retry
TRANSIENT_ERROR_TYPES = (
    asyncio.TimeoutError,
    ConnectionError,
    # Provider-specific transient errors
    "ThrottlingException",  # Bedrock
    "ServiceUnavailableException",  # Bedrock
    "APITimeoutError",  # Azure OpenAI SDK
    "InternalServerError",  # GCP
)

def with_retry_and_timeout(
    operation_name: str,
    timeout_seconds: int = None,
    max_retries: int = None,
    transient_errors: Tuple[Type[Exception], ...] = None
):
    """
    Decorator that adds timeout and exponential backoff retry logic.
    
    Args:
        operation_name: Name for logging (e.g., "triage", "orchestration")
        timeout_seconds: Override default timeout (uses settings if None)
        max_retries: Override max retries (uses settings if None)
        transient_errors: Tuple of exception types to retry on
    
    Usage:
        @with_retry_and_timeout("orchestration", timeout_seconds=90)
        async def generate_response(...):
            ...
    """
    if timeout_seconds is None:
        timeout_seconds = settings.LLM_DEFAULT_TIMEOUT
    if max_retries is None:
        max_retries = settings.LLM_MAX_RETRIES
    if transient_errors is None:
        transient_errors = (asyncio.TimeoutError, ConnectionError)
    
    def decorator(func: Callable):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            session_id = kwargs.get('session_id', 'unknown')
            
            for attempt in range(1, max_retries + 2):  # +1 for initial attempt + retries
                try:
                    # Apply timeout to the operation
                    result = await asyncio.wait_for(
                        func(*args, **kwargs),
                        timeout=timeout_seconds
                    )
                    
                    # Success - log and return
                    if attempt > 1:
                        logger.info(
                            f"[{session_id}] {operation_name} succeeded on attempt {attempt}/{max_retries + 1}"
                        )
                    return result
                
                except asyncio.TimeoutError as e:
                    logger.warning(
                        f"[{session_id}] {operation_name} timed out after {timeout_seconds}s "
                        f"(attempt {attempt}/{max_retries + 1})"
                    )
                    
                    if attempt > max_retries:
                        raise TimeoutError(
                            f"{operation_name} timed out after {timeout_seconds}s "
                            f"({max_retries} retries exhausted)"
                        ) from e
                    
                    # Calculate exponential backoff with jitter
                    delay = _calculate_backoff_delay(attempt)
                    logger.info(f"[{session_id}] Retrying {operation_name} in {delay:.2f}s...")
                    await asyncio.sleep(delay)
                
                except Exception as e:
                    # Check if this is a transient error worth retrying
                    is_transient = any(
                        isinstance(e, err_type) if isinstance(err_type, type) else err_type in str(type(e))
                        for err_type in transient_errors
                    )
                    
                    if is_transient and attempt <= max_retries:
                        logger.warning(
                            f"[{session_id}] {operation_name} failed with transient error: {e} "
                            f"(attempt {attempt}/{max_retries + 1})"
                        )
                        delay = _calculate_backoff_delay(attempt)
                        logger.info(f"[{session_id}] Retrying {operation_name} in {delay:.2f}s...")
                        await asyncio.sleep(delay)
                    else:
                        # Non-transient error or retries exhausted - propagate immediately
                        logger.error(
                            f"[{session_id}] {operation_name} failed with non-transient error "
                            f"or retries exhausted: {e}"
                        )
                        raise
            
        return wrapper
    return decorator


def _calculate_backoff_delay(attempt: int) -> float:
    """Calculate exponential backoff delay with jitter."""
    base_delay = settings.LLM_RETRY_BASE_DELAY
    max_delay = settings.LLM_RETRY_MAX_DELAY
    
    # Exponential backoff: base * 2^(attempt - 1)
    delay = base_delay * (2 ** (attempt - 1))
    delay = min(delay, max_delay)  # Cap at max_delay
    
    # Add jitter if enabled (±25% random variation)
    if settings.LLM_RETRY_JITTER:
        jitter_range = delay * 0.25
        delay = delay + random.uniform(-jitter_range, jitter_range)
    
    return max(0.1, delay)  # Minimum 100ms delay

Key Features:

• Timeout enforcement: Uses asyncio.wait_for to wrap any async operation
• Smart retry logic: Only retries transient errors (timeouts, throttling, service unavailable)
• Exponential backoff: Prevents overwhelming recovering services
• Jitter: Reduces retry storms (thundering herd problem)
• Observability: Logs every attempt with timing and outcome

---

3. Apply Decorator to Each Service

GCP Vertex AI (app/services/llm/vertex_ai_service.py):

from app.services.llm.retry_decorator import with_retry_and_timeout

@with_retry_and_timeout(
    operation_name="vertex_ai_generation",
    timeout_seconds=settings.LLM_DEFAULT_TIMEOUT,
    transient_errors=(asyncio.TimeoutError, ConnectionError)
)
async def generate_response_stateful(
    self,
    session_id: str,
    model: str,
    input_text: str,
    ...
) -> LLMResult:
    # Existing implementation - timeout now enforced by decorator
    ...

AWS Bedrock (app/services/llm/bedrock_llm_service.py):

# Bedrock already has good retries via boto3 Config, but we can add operation-level timeout
@with_retry_and_timeout(
    operation_name="bedrock_generation",
    timeout_seconds=settings.LLM_DEFAULT_TIMEOUT,
    max_retries=0  # Boto3 handles retries, we just add timeout layer
)
async def generate_response_stateful(...):
    ...

Azure OpenAI (app/services/azure_responses_service.py):

# Azure SDK has timeout/retry, but we can add operation-level timeout for consistency
@with_retry_and_timeout(
    operation_name="azure_responses_api",
    timeout_seconds=settings.LLM_DEFAULT_TIMEOUT,
    max_retries=0  # OpenAI SDK handles retries
)
async def create_response(...):
    ...

---

4. Operation-Specific Timeout Mapping

Update app/agent/nodes/ to use operation-specific timeouts:

Triage Node (app/agent/nodes/triage.py):

# Fast classification - shorter timeout
result = await llm_service.generate_response_stateful(
    session_id=session_id,
    model=model,
    input_text=input_text,
    timeout=settings.LLM_TRIAGE_TIMEOUT  # 30s
)

Orchestration Node (app/agent/nodes/orchestration.py):

# Complex reasoning - longer timeout
result = await llm_service.generate_response_stateful(
    session_id=session_id,
    model=model,
    input_text=input_text,
    timeout=settings.LLM_ORCHESTRATION_TIMEOUT  # 90s
)

---

5. Enhanced Circuit Breaker Integration

Update app/services/llm/provider_health.py to distinguish timeout failures from other errors:

async def record_failure(self, provider: str, error: Exception, is_timeout: bool = False):
    """
    Record a failed operation for a provider.
    
    Args:
        provider: Provider name
        error: The exception
        is_timeout: Whether this was a timeout (vs other error type)
    """
    async with self._lock:
        health = self._health[provider]
        health.last_failure = datetime.now()
        health.consecutive_failures += 1
        health.consecutive_successes = 0
        
        # Timeout failures count toward circuit breaker but with different weight
        failure_weight = 0.5 if is_timeout else 1.0
        health.weighted_failure_count += failure_weight
        
        if health.weighted_failure_count >= self.MAX_FAILURES_BEFORE_CIRCUIT_OPEN:
            logger.warning(
                f"Circuit breaker OPEN for '{provider}' after "
                f"{health.weighted_failure_count} weighted failures "
                f"(timeouts: {health.timeout_count}, errors: {health.error_count})"
            )
            health.is_healthy = False

Rationale: Timeouts might be network issues, not provider problems. Weight them lower to avoid premature circuit opening.

---

6. Observability & Metrics

Add structured logging and metrics:

# Log timeout events with context
logger.warning(
    "llm_timeout",
    extra={
        "provider": provider,
        "operation": operation_name,
        "timeout_seconds": timeout_seconds,
        "session_id": session_id,
        "model": model,
        "attempt": attempt,
    }
)

# Emit metrics (if using Prometheus/StatsD)
metrics.increment("llm.timeout", tags=[f"provider:{provider}", f"operation:{operation_name}"])
metrics.histogram("llm.retry_count", attempt, tags=[f"provider:{provider}"])
metrics.histogram("llm.latency", duration_ms, tags=[f"provider:{provider}", f"success:{success}"])

---

Implementation Plan

Phase 1: Critical Fix (Week 1) 🔴

• [ ] Add timeout wrapper to GCP Vertex AI service (immediate fix)
• [ ] Add basic timeout configuration to config.py
• [ ] Deploy to production with conservative timeouts (90s default)
• [ ] Monitor for timeout events in logs

Phase 2: Retry Strategy (Week 2) 🟡

• [ ] Implement retry_decorator.py with exponential backoff
• [ ] Apply decorator to all three providers
• [ ] Add operation-specific timeout configuration
• [ ] Test retry behavior under various failure scenarios

Phase 3: Enhanced Circuit Breaker (Week 3) 🟢

• [ ] Update ProviderHealthTracker with weighted failure tracking
• [ ] Distinguish timeout vs error failures
• [ ] Add automatic circuit breaker reset logic
• [ ] Add metrics for circuit breaker state transitions

Phase 4: Observability (Week 4) 🔵

• [ ] Add structured logging for all timeout/retry events
• [ ] Implement Prometheus metrics (if available)
• [ ] Create dashboard for LLM service health
• [ ] Add alerting for high timeout rates

---

Testing Strategy

Unit Tests

• Test timeout enforcement at various thresholds
• Test retry logic with mocked transient failures
• Test exponential backoff calculation
• Test jitter distribution

Integration Tests

• Simulate slow LLM responses (mock with asyncio.sleep)
• Test fallback behavior when primary provider times out
• Test circuit breaker opening after repeated timeouts
• Test session affinity preservation across retries

Load Tests

• Measure impact of retry overhead on throughput
• Test behavior under high concurrency (100+ sessions)
• Verify no thundering herd during retry storms

Production Canary

• Deploy to 10% of traffic first
• Monitor timeout rates, fallback rates, latency percentiles
• Gradually increase rollout to 100%

---

Success Metrics

Before (Current State):

• GCP: Indefinite hangs (observed: 2+ minutes)
• Azure: 30s timeout, 3 retries
• Bedrock: 60s timeout, 10 retries
• Inconsistent behavior across providers

After (Target State):

• All providers: Configurable timeouts (30-90s based on operation)
• All providers: 3 retries with exponential backoff
• All providers: Automatic fallback on timeout
• P99 latency: < 5s for triage, < 15s for orchestration
• Timeout rate: < 0.1% of requests
• Fallback success rate: > 95% (timeout → next provider succeeds)

---

Risk Analysis

Risks:

1. False positives: Timeouts that are too aggressive may trigger unnecessary fallbacks
2. Retry storms: Simultaneous retries from many sessions could overwhelm recovering services
3. Increased latency: Retry logic adds overhead (mitigated by exponential backoff)
4. Cost increase: Retries consume additional API quota

Mitigations:

1. Start with conservative timeouts (90s), tune down based on data
2. Exponential backoff + jitter prevents retry storms
3. Monitor P95/P99 latency - abort if degraded
4. Track retry rate metrics - alert if > 5%

---

Open Questions

1. Should we implement request-level timeout override? (e.g., urgent triage vs slow orchestration)
2. Should timeout failures count toward circuit breaker? (proposed: half weight)
3. What's the optimal timeout for each operation? (requires production data analysis)
4. Should we add a global "emergency mode" that disables retries? (for cascading failure scenarios)

---

Related Issues

• Session hang incident (2025-11-19): 8DT981bAmb7Vs9HY5amIq6XT_Gn54nWfkXklNzwsYfQ
• Circuit breaker implementation: [Link if exists]
• LLM fallback system: [Link if exists]

---

References

Industry Best Practices:

• AWS SDK Retry Strategy
• Google Cloud API Retry Strategy
• Circuit Breaker Pattern (Microsoft)
• Exponential Backoff and Jitter (AWS Blog)

Code References:

• app/services/llm/vertex_ai_service.py:325 - GCP timeout vulnerability
• app/services/llm/bedrock_llm_service.py:38-44 - Bedrock retry config (good example)
• app/services/llm/provider_health.py:38-41 - Circuit breaker thresholds
• app/core/config.py:486-491 - Azure timeout config

---

Acceptance Criteria

✅ Done when:

• [ ] All three LLM services have timeout enforcement
• [ ] Retry decorator implemented and tested
• [ ] Operation-specific timeouts configured
• [ ] Circuit breaker enhanced with timeout tracking
• [ ] Comprehensive logging for timeout/retry events
• [ ] Production deployment successful without incidents
• [ ] Documentation updated (CLAUDE.md, README)
• [ ] Runbook created for timeout troubleshooting

---

Priority: 🔴 P0 - Critical
Effort: ~2-3 weeks
Impact: Prevents production outages, improves user experience
Owner: TBD