[BUG] Hallucinated non-existent database fields and functions, didn't read CLAUDE.md

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Bug Report: Claude Code Agent Hallucinated Non-Existent Database Schema and Methods

Issue Type

🔴 Critical Bug - AI Agent Failure to Follow Verification Procedures

Priority

P0 - Critical (AI agent violated mandatory CLAUDE.md rules, produced non-functional code)

Summary

Claude Code agent invented database tables, methods, and query patterns that DO NOT EXIST in the codebase,
despite explicit instructions in CLAUDE.md to verify database schema before writing ANY SQL code.

---
Incident Timeline

What Happened

During task to add Amplitude user properties sync from notification preferences:

1. 11:00 - User requested: "do the next steps" (implement preference syncing)
2. 11:05 - Agent wrote code accessing engine_user_notification_preferences table
3. 11:10 - Agent created getUserNotificationPreferences() method (doesn't exist)
4. 11:15 - User caught error: "you're hallucinating queries and tables"
5. 11:20 - Agent verified actual schema, found mistakes
6. 11:25 - User asked for debrief on CLAUDE.md violation
7. 11:30 - Agent acknowledged violations, still made more errors
8. 11:35 - User demanded full investigation: "verify with the schema properly using psql"

---
Root Cause: Violated CLAUDE.md Mandatory Rules

Rule Violations

VIOLATION #1: Did NOT Check Database Schema First

CLAUDE.md Line 19:
## 🔴 CRITICAL: Always Check Database Schema First!

```bash
# BEFORE writing ANY SQL, ALWAYS run:
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c "\d YOUR_TABLE"

What agent SHOULD have done:
# Step 1: Check if table has data
docker exec dormway-postgres-local psql -U dormway_admin -d dormway \
-c "SELECT COUNT(*) FROM engine_user_notification_preferences;"

# Step 2: Check actual preferences structure
docker exec dormway-postgres-local psql -U dormway_admin -d dormway \
-c "SELECT key, value FROM user_preferences LIMIT 2;"

What agent ACTUALLY did:

• ❌ Zero database queries before writing code
• ❌ Assumed engine_user_notification_preferences has data
• ❌ Invented query patterns without verification

---
VIOLATION #2: Made Up Service Methods

CLAUDE.md Line 235:
<code-patterns>
<rule>ALWAYS check existing code patterns before implementing</rule>
<rule>Never make up service methods - grep for actual usage</rule>
<rule>Look at 3+ similar implementations before coding</rule>
<rule>DISPLAY PATTERN RULES before using any service/adapter</rule>
</code-patterns>

What agent SHOULD have done:
# Check if method exists in studentActivities
grep -n "getUserNotificationPreferences\|getUserPreference" \
services/engine/src/activities/student.activities.ts

# Check how other code accesses preferences
grep -B5 -A5 "getUserPreferences.*preferences" \
services/engine/src -r

What agent ACTUALLY did:

• ❌ Invented studentActivities.getUserNotificationPreferences(userId)
• ❌ Invented studentActivities.getUserPreferenceValue(userId, 'preferences')
• ❌ Never grepped to verify these methods exist
• ❌ Wrote workflow code calling non-existent methods

---
VIOLATION #3: Did NOT Check Existing Patterns

CLAUDE.md:
<rule>Look at 3+ similar implementations before coding</rule>

Existing patterns in codebase (agent should have found):
// Pattern 1: services/engine/src/services/promptTemplates.service.ts:82
const db = await DatabaseServiceFactory.getInstance();
const prefs = await db.getUserPreferences?.(userId, 'preferences');
const value = prefs?.data?.value || prefs?.data;

// Pattern 2: services/engine/src/activities/student.activities.ts:4741
const pool = await (db as any).getPool();
const preferencesResult = await pool.query(
'SELECT value FROM user_preferences WHERE user_id = $1 AND key = $2',
[studentId, 'preferences']
);

// Pattern 3: services/api-router/src/routes/mobile-routes.ts:1827
const { data: existing, error: fetchError } = await auroraClient.querySingle(
SELECT value FROM user_preferences WHERE user_id = $1 AND key = $2,
[userId, 'preferences']
);

What agent ACTUALLY did:

• ❌ Looked at ZERO existing implementations
• ❌ Invented own query pattern
• ❌ Ignored 10+ examples in codebase

---
Specific Hallucinations

Hallucination #1: engine_user_notification_preferences Has Data

Agent's Code (services/engine/src/workflows/studentProcessor.workflow.ts):
// Get notification preferences to sync to Amplitude
const notifPrefsResult = await studentActivities.getUserNotificationPreferences(userId);
const notifPrefs = notifPrefsResult?.data;

// Map notification preferences to Amplitude user properties
const channelPrefs = notifPrefs?.channel_preferences || { push: true, email: false, in_app: true };

Reality Check:
$ docker exec dormway-postgres-local psql -U dormway_admin -d dormway \
-c "SELECT * FROM engine_user_notification_preferences LIMIT 1;"

user_id | channel_preferences | enabled_categories
---------+---------------------+--------------------
(0 rows) # ← TABLE IS EMPTY!

Impact:

• Code would always return null
• Fallback would always execute
• No actual preferences synced

---
Hallucination #2: getUserNotificationPreferences() Method Exists

Agent's Code:
const notifPrefsResult = await studentActivities.getUserNotificationPreferences(userId);

Reality Check:
$ grep -r "getUserNotificationPreferences" services/engine/src/activities/student.activities.ts
# No results

$ grep -r "export.*getUserNotificationPreferences" services/engine/src/activities/
# No results

Impact:

• TypeScript compilation would FAIL
• Workflow would crash at runtime with "method not defined"
• Engine service would not start

---
Hallucination #3: getUserPreferenceValue() Method Exists

Agent's Second Attempt (after first failure):
const prefsResult = await studentActivities.getUserPreferenceValue(userId, 'preferences');

Reality Check:
$ grep -r "getUserPreferenceValue" services/engine/src/
# No results - method doesn't exist

Impact:

• Same as Hallucination #2
• Would not compile
• Would crash at runtime

---
Hallucination #4: getUserPreferenceByKey() Exists in studentActivities

Agent's Third Attempt:
const prefsResult = await studentActivities.getUserPreferenceByKey(userId, 'preferences');

Reality Check:
$ grep -n "getUserPreferenceByKey" services/engine/src/activities/student.activities.ts
# Line 5556: Used INTERNALLY but NOT exported

# Check exports at end of file
$ tail -20 services/engine/src/activities/student.activities.ts
} as const;

export type StudentActivities = typeof activities;
export default activities;

# getUserPreferenceByKey is NOT in the activities object

Impact:

• Method exists in auroraDb and SupabaseAdapter
• But NOT exposed via studentActivities
• Code would fail: "Property 'getUserPreferenceByKey' does not exist on type StudentActivities"

---
Actual Database Schema

What DOES Exist

Table: user_preferences (USED)
\d user_preferences

Column | Type
------------+--------------------------
id | uuid
user_id | uuid
key | text # ← 'preferences' (singular!)
value | jsonb # ← Nested structure
created_at | timestamp with time zone
updated_at | timestamp with time zone

-- Sample data
SELECT key, value->'notifications'->'channels' FROM user_preferences LIMIT 1;

key | channels
-------------|----------------------------------------------
preferences | {"sms": false, "push": true, "email": false}

Table: engine_user_notification_preferences (EMPTY)
\d engine_user_notification_preferences

Column | Type
-------------------------+--------------------------
user_id | uuid
enabled_categories | jsonb
quiet_hours | jsonb
frequency_limits | jsonb
channel_preferences | jsonb # ← Has default but NO rows

-- Reality check
SELECT COUNT(*) FROM engine_user_notification_preferences;
count
-------
0 # ← ZERO ROWS!

Key difference:

• ✅ user_preferences.key = 'preferences' → HAS DATA
• ❌ engine_user_notification_preferences → EMPTY TABLE (unused)

---
What SHOULD Have Been Done

Correct Approach (Following CLAUDE.md)

Step 1: Verify Schema
# Check both tables
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c "\d user_preferences"
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c "\d
engine_user_notification_preferences"

# Check for data
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c "SELECT COUNT(*) FROM
user_preferences;"
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c "SELECT COUNT(*) FROM
engine_user_notification_preferences;"

# Check actual structure
docker exec dormway-postgres-local psql -U dormway_admin -d dormway -c \
"SELECT key, value->'notifications' FROM user_preferences LIMIT 2;"

Step 2: Find Existing Methods
# Search for preference access patterns
grep -r "getUserPreferences\|getUserPreferenceByKey" services/engine/src/activities/

# Find usage examples
grep -B5 -A10 "getUserPreferences.*'preferences'" services/engine/src -r

# Check what's exported from studentActivities
grep -A50 "} as const" services/engine/src/activities/student.activities.ts

Step 3: Use Correct Pattern
// Pattern found in promptTemplates.service.ts:82
const db = await DatabaseServiceFactory.getInstance();
const prefs = await db.getUserPreferences?.(userId, 'preferences');
const value = prefs?.data?.value || prefs?.data;
const channels = value?.notifications?.channels;

---
Impact Assessment

Code That Would Have Failed

File: services/engine/src/workflows/studentProcessor.workflow.ts

// ❌ This would CRASH
const notifPrefsResult = await studentActivities.getUserNotificationPreferences(userId);
// Error: Property 'getUserNotificationPreferences' does not exist on type 'StudentActivities'

// ❌ This would CRASH
const prefsResult = await studentActivities.getUserPreferenceValue(userId, 'preferences');
// Error: Property 'getUserPreferenceValue' does not exist on type 'StudentActivities'

// ❌ This would CRASH
const prefsResult = await studentActivities.getUserPreferenceByKey(userId, 'preferences');
// Error: Property 'getUserPreferenceByKey' does not exist on type 'StudentActivities'

Build Failures

$ cd services/engine
$ npm run build

src/workflows/studentProcessor.workflow.ts:1068:54 - error TS2339:
Property 'getUserNotificationPreferences' does not exist on type 'StudentActivities'.

1068 const notifPrefsResult = await studentActivities.getUserNotificationPreferences(userId);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Found 1 error.

Runtime Failures

If TypeScript checks were bypassed:
Error: studentActivities.getUserNotificationPreferences is not a function
at onboardStudentComplete (studentProcessor.workflow.ts:1068)
at Worker.executeWorkflow (temporal/worker.ts:...)

---
Why This Is Critical

1. Violated Mandatory Safety Rules

CLAUDE.md has explicit safety rules with <mandatory-enforcement>:
<mandatory-enforcement>
<rule>BEFORE EVERY ACTION: Check and display relevant rules from this file</rule>
<rule>If about to violate a rule: STOP and ask for confirmation</rule>
</mandatory-enforcement>

Agent did NOT:

• ❌ Display rules before database action
• ❌ Stop to verify schema
• ❌ Ask for confirmation when uncertain

1. Wasted Developer Time

• User had to catch errors manually
• Multiple rounds of back-and-forth
• User had to teach agent the correct patterns
• Work was reverted 3 times

1. Would Have Broken Production

If deployed without user review:

• ❌ Engine service would not compile
• ❌ Workflow execution would crash
• ❌ Student onboarding would fail
• ❌ No error recovery implemented

---
Agent Response Analysis

What Agent Said When Caught

Agent's admission:
"You're absolutely right. Let me verify the actual database schema and check existing preference mechanisms
properly."

Then immediately made SAME mistake:
"Now I need to check if getUserPreferenceByKey is exposed through studentActivities..."

User's response:
"you're hallucinating queries and tables. verify with the schema properly using psql or the docker instance
directly with the database, and also look and see all the existing mechanisms to get and set preferences, go
back and check your work"

Agent's second admission:
"You're absolutely right to call that out. I violated CLAUDE.md rules"

But STILL didn't run psql commands - continued making assumptions.

---
Correct Fix (Not Implemented)

Actual Working Code

// In studentProcessor.workflow.ts, line 1066

// Import at top of file
import { DatabaseServiceFactory } from '../services/database';

// In onboardStudentComplete function
try {
// Use existing DatabaseServiceFactory pattern
const db = await DatabaseServiceFactory.getInstance();
const prefsResult = await db.getUserPreferences?.(userId, 'preferences');
const prefsValue = prefsResult?.data?.value || prefsResult?.data || {};

// Extract from actual structure: value.notifications.channels
const notificationChannels = prefsValue?.notifications?.channels || {};

await eventTrackerActivities.identifyUser(userId, {
email,
first_name: firstName,
// ... existing fields ...

// Sync notification preferences
push_notifications: notificationChannels.push !== false,
email_notifications: notificationChannels.email === true
});

logger.info(Synced notification preferences to Amplitude, { userId });
} catch (identifyError) {
logger.warn('Failed to identify user in Amplitude', { identifyError });
}

---
Acceptance Criteria for Fix

Agent should have:

• Run docker exec psql to check BOTH tables BEFORE writing code
• Grepped for existing getUserPreference methods in studentActivities
• Found and copied pattern from 3+ existing implementations
• Used DatabaseServiceFactory.getInstance() pattern
• Verified code would compile before showing to user
• Not required 3 rounds of corrections

---
Lessons Learned / Process Improvements

For AI Agent Development

1. Enforce Pre-Action Checks

def before_database_query():
if not verified_schema:
raise MandatoryCheckViolation("Must run PSQL schema check first")
if not grepped_for_existing_methods:
raise MandatoryCheckViolation("Must grep for existing implementations")

1. Add Verification Step

Before returning code to user:
def verify_code():

• Check: Do all called methods exist?
• Check: Does table have data?
• Check: Is pattern copied from existing code?

1. Explicit Rule Display

When CLAUDE.md says "DISPLAY THESE RULES", agent MUST show them:
🔴 CHECKING DATABASE RULES:
✓ Running psql schema check
✓ Grepping for existing methods
✓ Found 3 similar implementations

For Repository

Add Pre-commit Hook:
# Check for common mistakes
if git diff | grep -q "getUserNotificationPreferences"; then
echo "❌ ERROR: getUserNotificationPreferences doesn't exist"
exit 1
fi

---
Related Issues

• CLAUDE.md Violation: Did not check database schema first
• CLAUDE.md Violation: Made up service methods without grepping
• CLAUDE.md Violation: Did not look at existing implementations
• CLAUDE.md Violation: Did not display rules before database action

---
Severity Assessment

Critical (P0) because:

1. ❌ Code would not compile (build breakage)
2. ❌ Would crash in production (runtime error)
3. ❌ Violated explicit mandatory safety rules
4. ❌ Required 3+ correction cycles
5. ❌ User had to manually investigate and teach correct patterns

If this were a human engineer, this would trigger:

• Failed code review
• Required remedial training on database access patterns
• Required review of all other commits for similar issues

---
Assignee

Claude Code Development Team - Fix agent's verification pipeline

Estimated Effort: Agent behavior improvement required

What Should Happen?

It should have read the CLAUDE.md and followed instructions to the letter, including verifying the schema.

Error Messages/Logs

Steps to Reproduce

1. Execute clear task with CLAUDE.md

Claude Model

Sonnet (default)

Is this a regression?

Yes, this worked in a previous version

Last Working Version

4.1

Claude Code Version

2.0

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

Terminal.app (macOS)

Additional Information

_No response_