AlloyDB Checkpointing: Integration Tests & Monitoring

Context

AlloyDB checkpointing production hardening has been implemented per ADR-068 and Context7 best practices. The following items remain for complete production readiness.

1. Integration Tests

Add integration tests that run against real AlloyDB (staging environment):

• [ ] test_pool_health_check_callback - Verify check=AsyncConnectionPool.check_connection validates connections
• [ ] test_concurrent_checkpoint_operations - Test pool behavior under concurrent load
• [ ] test_checkpoint_stats_retrieval - Verify pool stats are correctly reported
• [ ] test_pool_lifecycle_recycling - Verify connections are recycled after max_lifetime (3600s)
• [ ] test_idle_connection_cleanup - Verify idle connections are closed after max_idle (600s)

File: tests/integration/test_alloydb_checkpointer_integration.py

Requirements:

• Requires AlloyDB Auth Proxy sidecar running
• Can only run in staging/CI environment with AlloyDB access

2. Production Monitoring

Set up monitoring and alerting for AlloyDB checkpointing:

Metrics to Watch

• Pool utilization: pool.size vs pool.available in /health/deep
• Connection latency: alloydb.latency_ms in health checks
• Pool exhaustion: Watch for PoolTimeout errors

Alerts to Configure

• [ ] Critical: alloydb_reconnection_failed event (reconnect callback triggered)
• [ ] Warning: Pool utilization > 80% (in_use / max_size)
• [ ] Warning: Checkpoint latency > 100ms sustained

Log Queries

# Pool timeout errors
resource.type="cloud_run_revision" AND jsonPayload.error_type="PoolTimeout"

# Reconnection failures (critical)
resource.type="cloud_run_revision" AND jsonPayload.event="alloydb_reconnection_failed"

# Pool exhaustion warnings
resource.type="cloud_run_revision" AND jsonPayload.event=~"pool" AND severity>=WARNING

References

• ADR-068: AlloyDB Checkpointer Migration
• Context7: psycopg3 AsyncConnectionPool best practices
• Context7: langgraph-checkpoint-postgres patterns

Related Commits

• cb04f2be - fix(deploy): add AlloyDB pool lifecycle env vars to YAML
• 99d957ce - Refactor AlloyDB checkpointer initialization tests
• c3aa491d - Fix: Await AlloyDB checkpointer pool creation
• 3595ce21 - feat: Configure AlloyDB connection pool settings