LLM Insights showing placeholder text instead of AI-generated content

Problem Summary

The profile-generator service is returning placeholder text for LLM insights instead of actual AI-generated content. Users see generic messages like:

• "Strong overall performance"
• "Continue practicing"
• "Analysis pending - generating insights from your match data."

Environment

• Service: profile-generator-dev (Cloud Run)
• Project: mentoroid-app-dev
• Region: asia-southeast1

Investigation Checks Performed

1. Service Deployment Status

gcloud run services describe profile-generator-dev --region=asia-southeast1 --project=mentoroid-app-dev

✅ Service is deployed and running

2. Environment Variables Configuration

gcloud run services describe profile-generator-dev --format="yaml(spec.template.spec.containers[0].env)"

Result:

- name: OPENAI_API_KEY
  valueFrom:
    secretKeyRef:
      key: latest
      name: openai-api-key
- name: ANTHROPIC_API_KEY
  valueFrom:
    secretKeyRef:
      key: latest
      name: anthropic-api-key
- name: LLM_PROVIDER
  value: claude
- name: BUCKET_NAME
  value: mentoroid-pga-dev

✅ All env vars appear correctly configured

3. Secrets Verification

gcloud secrets list --project=mentoroid-app-dev | grep -E "openai|anthropic"

✅ Both openai-api-key and anthropic-api-key secrets exist

4. Cloud Run Revisions

gcloud run revisions list --service=profile-generator-dev --region=asia-southeast1 --limit=5

Result:

• profile-generator-dev-00046-dzf (latest, serving traffic)
• Image: sha256:dcc785c6f058be95fe006d3350c6aaea340e26b840ab45585489dbabb61b7eda

5. Error Logs Analysis

gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="profile-generator-dev" AND severity>=WARNING' --limit=30

Critical Error Found:

IndexError: string index out of range
  File "/app/shared/gcp_utils.py", line 45, in bucket
    self._bucket = self.client.bucket(self.bucket_name)
  File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/_helpers.py", line 134, in _validate_name
    if not all([name[0].isalnum(), name[-1].isalnum()]):
IndexError: string index out of range

Root Cause Analysis

Issue 1: Empty BUCKET_NAME in shared module

The shared/gcp_utils.py has an empty string default:

# Line 27 in shared/gcp_utils.py
BUCKET_NAME = os.environ.get('BUCKET_NAME', '')  # Empty default!

While profile_generator/main.py has a sensible default:

# Line 35 in profile_generator/main.py
BUCKET_NAME = os.environ.get('BUCKET_NAME', 'mentoroid-pga-dev')

The ManifestManager uses GCSStorageClient from shared/gcp_utils.py, which fails when BUCKET_NAME is empty.

Issue 2: LLM API Connection Errors (Previous Investigation)

Earlier logs showed both LLM providers failing:

ERROR:core.llm.claude_client:Claude API error: Connection error.
ERROR:core.llm.openai_client:OpenAI API error: Connection error.
WARNING:main:Could not generate performance summary via LLM: Failed to generate response: Connection error.

This may indicate:

• Network egress issues from Cloud Run
• VPC configuration blocking external API calls
• Or could be secondary to the BUCKET_NAME failure

Code Files Involved

1. /functions/gcp/shared/gcp_utils.py:27 - Empty BUCKET_NAME default
2. /functions/gcp/shared/gcp_utils.py:45 - Bucket property that fails
3. /functions/gcp/shared/manifest.py:68 - load_manifest uses storage
4. /functions/gcp/profile_generator/main.py:124 - ManifestManager initialization
5. /functions/llm_framework/core/llm/claude_client.py - Claude API client
6. /functions/llm_framework/core/llm/openai_client.py - OpenAI API client

Possible Fixes

Option 1: Fix shared/gcp_utils.py default

# Change from:
BUCKET_NAME = os.environ.get('BUCKET_NAME', '')

# To:
BUCKET_NAME = os.environ.get('BUCKET_NAME', 'mentoroid-pga-dev')

Option 2: Ensure BUCKET_NAME is passed explicitly

Update ManifestManager and GCSStorageClient initialization to always pass bucket_name explicitly from the calling code.

Option 3: Redeploy Docker image

The deployed Docker image may contain stale code. Trigger a full CI/CD rebuild:

gh workflow run deploy.yml -f environment=dev

Terraform Configuration Reference

The Terraform correctly sets BUCKET_NAME (terraform/gcp/modules/cloud-run/main.tf:292-294):

env {
  name  = "BUCKET_NAME"
  value = var.bucket_name
}

Next Steps

1. [ ] Fix the empty default in shared/gcp_utils.py
2. [ ] Trigger CI/CD pipeline to rebuild and redeploy Docker image
3. [ ] Verify LLM API connectivity after BUCKET_NAME fix
4. [ ] Test profile generation end-to-end
5. [ ] Monitor logs for any remaining errors

Related Files Changed Recently

• LLM_PROVIDER was manually changed from openai to claude via gcloud command
• This may have caused a revision without rebuilding the Docker image

Labels

• bug
• profile-generator
• llm
• gcp