Create monitoring Google Sheet for hex_ai dataset

Problem

Business users rely on the hex_ai BigQuery dataset via their BI tool, but have no visibility into:

• What tables/views exist in the dataset
• Row counts and data freshness
• Whether tables have gone empty (data quality issue detection)

When new tables are added to services/ab-terraform/src/queries/hex_ai/, users should automatically see them without manual intervention.

Proposed Solution

Create a Google Sheet that automatically stays in sync with the hex_ai dataset using one of two approaches:

Approach A: Native Connected Sheets (Manual Setup)

Pros:

• Native Google Sheets feature, no code to maintain
• Automatic refresh scheduling built-in
• Users can interact with data directly in Sheets

Cons:

• Manual setup required for each new table tab
• 10,000 row display limit per Connected Sheet
• Requires BigQuery permissions for all viewers

Implementation:

1. Create Google Sheet with "Catalog" tab using Connected Sheets query:

SELECT 
  table_name,
  table_type,
  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), TIMESTAMP_MILLIS(creation_time), DAY) as days_old,
  row_count,
  size_bytes
FROM `assembled-wh.hex_ai.INFORMATION_SCHEMA.TABLES`
ORDER BY table_name

1. Add individual tabs per table using Connected Sheets (e.g., "borrower_monthly_financials_trailing_24m" showing all 9,337 rows)

1. Set refresh schedule (daily at 6am after ab-terraform job completes)

Approach B: Cloud Run Service (Programmatic)

Pros:

• Automatically creates new tabs when tables detected
• Can handle >10k rows per table
• Follows existing service patterns in this repo
• Auditable via Cloud Logging

Cons:

• New service to maintain
• Requires Google Sheets API integration
• More complex than native solution

Implementation:

• New service: services/ab-hex-monitoring-sheet/
• Scheduled via Cloud Scheduler (daily after ab-terraform)
• Queries INFORMATION_SCHEMA to discover tables
• Uses Google Sheets API to create/update tabs
• One catalog tab + auto-generated tabs per table

Current State

• 1 table in hex_ai: borrower_monthly_financials_trailing_24m (9,337 rows)
• SQL location: services/ab-terraform/src/queries/hex_ai/*.sql
• Refresh schedule: ab-terraform runs daily at 05:00 UTC

Decision Needed

Which approach should we use?

1. Native Connected Sheets (faster, manual tab creation)
2. Cloud Run service (automated, more maintainable)

Acceptance Criteria

• [ ] Google Sheet created and shared with business users
• [ ] Catalog tab shows all tables in hex_ai with row counts
• [ ] Individual tabs show actual data from each table
• [ ] Sheet refreshes automatically after ab-terraform job completes
• [ ] Documentation added to hex_ai readme.md explaining the monitoring sheet