Email Sync thread optimization | P5 | 01.15.26

Summary

During email sync, multiple emails in the same thread are processed independently, leading to duplicate LLM extraction calls for the same invoice. While duplicate detection prevents data corruption at save time, this is inefficient and can cause extraction variance.

Current Behavior

1. Each email in a sync batch is processed independently
2. If a thread has multiple emails (original + reply/forward), each is sent through LLM extraction
3. Both may extract the same invoice data
4. Duplicate detection catches this at save time and merges message IDs
5. Result: Extra LLM calls, potential extraction variance between passes

Evidence

From logs, same invoice processed twice in one sync:

20:39:58 - ╔══ IN_REPORT ══╗ Emerald Bay Pool & Spa | $500.0 → auto-approved (first save)
20:42:10 - ╔══ NEEDS_REVIEW ══╗ Emerald Bay Pool & Spa | $500.0 → dup:['same_invoice_number'] (duplicate)

Database shows merged message IDs confirming two emails were processed:

gmail_message_ids: ['19995e138f9b0053', '19995d32b70381d8']

Impact

• Wasted LLM calls: 2x extraction for same invoice
• Extraction variance: Different LLM runs may extract slightly different data (dates, etc.)
• Confusing data: raw_extraction may not match saved data if second pass had different results

Affected Files

| File | Role |
|------|------|
| packages/invoice-extraction/src/base_sync_pipeline.py | Email batch processing - no thread tracking |
| packages/invoice-extraction/src/sync_integration.py | Process each email independently |
| packages/invoice-extraction/src/storage/database.py | Duplicate detection at save time (working correctly) |

Proposed Solution

Add thread-level tracking during sync:

# In base_sync_pipeline.py or sync_integration.py
processed_threads = set()  # Track threads that produced invoices

for email in emails_to_process:
    thread_id = email.get('thread_id')
    
    # Skip if we already found an invoice in this thread
    if thread_id in processed_threads:
        logger.debug(f"Skipping email - thread {thread_id} already processed")
        continue
    
    # Process email...
    result = process_email(email)
    
    # Mark thread as processed if invoice found
    if result.get('is_invoice'):
        processed_threads.add(thread_id)

Edge Cases to Consider

1. Multiple different invoices in same thread - Rare, but possible. May need option to disable optimization.
2. Thread with body invoice + PDF invoice - Same invoice, different extraction method. Should skip.
3. Partial extraction on first email - If first email extracts incomplete data, second might have more. Need to decide policy.

Priority

P5 (Low) - System works correctly, this is an efficiency optimization. Duplicate detection prevents data issues.

Labels

optimization, email-sync, tech-debt