Add retry logic for transient PostgreSQL connection failures

Problem

The system occasionally experiences transient PostgreSQL connection failures when saving state to Azure PostgreSQL. When these failures occur during critical operations (like task completion), the system does not retry, causing tasks to become stuck in an inconsistent state.

Example Error

2026-01-12 17:27:58,078 [ERROR] shared.storage.postgres_store: Failed to initialize PostgresStateStore: PostgreSQL connection failed for postgresql://bumblebee_admin:****@bumblebee-postgres.postgres.database.azure.com:5432/bumblebee?sslmode=require. Error: ConnectionDoesNotExistError: connection was closed in the middle of operation
2026-01-12 17:27:58,078 [ERROR] root: Failed to save task 714-002-update-orchestration

Impact

When a PostgreSQL connection error occurs during a save operation:

• Task completion is not persisted to the database
• Task becomes stuck in limbo (marked as completed in memory but still pending in DB)
• Manager cannot automatically recover without manual database intervention
• Orchestrations stall waiting for tasks that appear incomplete

In the specific case that triggered this issue:

• Task 714-002-update-orchestration was completed by Tech Lead at 17:27:53
• PostgreSQL connection failed at 17:27:58 when saving the completion
• Task remained stuck in dev_completed state and never progressed to QA
• Manual database reset was required to recover

Root Cause

The system treats PostgreSQL connection errors as non-retryable failures. Connection errors like "connection was closed in the middle of operation" are transient and should be retried with exponential backoff.

Proposed Solution

1. Implement retry logic in shared/storage/postgres_store.py for save operations:

• Detect transient connection errors (ConnectionDoesNotExistError, connection timeouts, etc.)
• Retry with exponential backoff (e.g., 3-5 retries with 1s, 2s, 4s delays)
• Use existing _is_retryable_error() function to identify retryable errors
• Apply to critical operations: save(), save_orchestration(), save_task()

1. Add circuit breaker pattern (optional enhancement):

• Track consecutive failures
• Temporarily stop retrying if too many consecutive failures occur
• Alert/log when circuit breaker trips

1. Improve error visibility:

• Log retry attempts with backoff timing
• Emit metrics for PostgreSQL connection failures
• Surface persistent connection issues in health checks

Related Code

• shared/storage/postgres_store.py - PostgresStateStore implementation
• agents/manager/tools.py - Task result handling
• agents/manager/orchestration.py - Orchestration state updates

Acceptance Criteria

• [ ] PostgreSQL save operations retry on transient connection errors
• [ ] Exponential backoff between retry attempts (configurable)
• [ ] Maximum retry count is configurable (default: 3-5)
• [ ] Retry attempts are logged with timing information
• [ ] Non-retryable errors (auth failures, invalid SQL) fail immediately
• [ ] Tests verify retry behavior for common connection errors
• [ ] System recovers automatically from transient Azure PostgreSQL connection issues