GenAI text enrichment fails with 'Unterminated string' errors on large documents

Problem Description

The text enrichment function in data_ingestion_new.py fails with JSON parsing errors when processing large academic papers. The error occurs because Gemini's response gets truncated when the output exceeds the token limit, resulting in malformed JSON.

Affected Documents

The following documents were failing with 'Unterminated string starting at: line X column Y' errors:

1. Document ID: 1:0:KAGObJcBrVHYQy6YmIxX

• Paper: "Supplementation of Basal Substrate to Boost up Substrate Str..."
• Text size: 35,449 characters
• Error: Unterminated string starting at: line 88 column 15 (char 16875)
• Status: ✅ Now works (17 paragraphs, 2 tables extracted)

1. Document ID: 1:0:hwGLbJcBrVHYQy6YCGFM

• Paper: "Soil Contamination with Heavy Metals and Its Impact on Food..."
• Text size: 48,297 characters
• Error: Unterminated string starting at: line 76 column 15 (char 16418)
• Status: ✅ Now works (21 paragraphs, 0 tables extracted)

1. Document ID: 1:0:v8qJbJcBWy9IQJkl4ao1

• Paper: "Behavioral approach to food waste: an experiment..."
• Text size: 42,412 characters
• Error: Unterminated string starting at: line 112 column 15 (char 18614)
• Status: Should work now with the fix

Root Cause Analysis

Testing revealed that:

• With max_output_tokens=2000: JSON truncated at ~8,434 characters
• With max_output_tokens=4000: JSON truncated at ~17,482 characters
• With max_output_tokens=8000: JSON response completes successfully

The truncation occurs mid-string in the JSON response, causing parsing failures.

Temporary Fix Applied

Increased max_output_tokens from 4000 to 8000 in _process_text_with_gemini() function, which resolves the immediate issue for most documents.

Enhancement Needed: Implement Robust Chunking System

While increasing token limits helps, we need a more robust solution for very large documents:

Proposed Enhancement

1. Automatic Document Chunking

• Split documents >50k characters into smaller chunks
• Process each chunk independently with Gemini
• Merge results while maintaining paragraph/table indices

1. Robust JSON Parsing

• Detect truncated responses
• Extract partial data when possible
• Log warnings for incomplete processing

1. Adaptive Token Management

• Dynamically adjust chunk size based on content density
• Monitor token usage in responses
• Implement retry logic with smaller chunks if truncation detected

Benefits

• Handle documents of any size reliably
• Prevent data loss from truncation
• Improve system resilience
• Better resource utilization

Error Logs Sample

DEFAULT 2025-07-28T10:04:43.692143Z ERROR:data_ingestion_new:Error processing text with Gemini: Unterminated string starting at: line 88 column 15 (char 16875)
DEFAULT 2025-07-28T10:04:43.692155Z ERROR:data_ingestion_new:Error during text enrichment for document ID 1:0:KAGObJcBrVHYQy6YmIxX: Unterminated string starting at: line 88 column 15 (char 16875)

Next Steps

1. Implement the chunking system as described above
2. Add comprehensive error handling for partial JSON responses
3. Create unit tests for large document processing
4. Monitor production for similar failures
5. Consider implementing streaming responses for very large outputs