feat(kg): Improve LLM relationship extraction to match/exceed GLiNER quality

Summary

The LLM-based relationship extraction in hybrid mode currently contributes only 3% of relationships due to code limitations, not model quality. This issue tracks the work needed to make LLM extraction effective for both local (Ollama) and cloud (Groq, OpenAI) providers.

Problem Analysis

Current State (Hybrid Mode Test Results)

| Source | Relationships | % of Total |
|--------|---------------|------------|
| GLiNER (co-occurrence) | 763 | 77% |
| Rule-based (cross-batch) | 200 | 20% |
| LLM (qwen2.5:7b) | 26 | 3% |

Despite running for ~1 hour, the LLM only extracted 26 relationships.

Root Causes Identified

1. Limited Relationship Types (15 vs 80+)

LLM code (extract-relationships-node-llm.ts lines 83-98):

const KNOWN_RELATIONSHIP_TYPES = new Set([
  'WORKS_AT', 'LOCATED_IN', 'RELATED_TO', 'PART_OF', 'CREATED_BY',
  'OWNS', 'MANAGES', 'REPORTS_TO', 'BELONGS_TO', 'CONTAINS',
  'USES', 'PROVIDES', 'DEPENDS_ON', 'CONNECTED_TO', 'ASSOCIATED_WITH',
]);

// Any unknown type → RELATED_TO fallback!
type: z.string().transform((t) => {
  return KNOWN_RELATIONSHIP_TYPES.has(normalized) ? normalized : 'RELATED_TO';
})

GLiNER code (gliner_service.py lines 516-643) has 80+ relationship types:

• affiliated_with, authored_by, published_by, covers, documents
• built_with, produces, integrates_with, deployed_in
• measures, configures, implements, enables
• And 60+ more domain-specific types

Impact: LLM extracts rich types like authored_by, covers, built_with → all become RELATED_TO.

2. Exact Entity Name Matching

const sourceEntity = state.raw_entities.find(
  (e) => e.name?.toLowerCase() === rel.source.toLowerCase()
);

If LLM returns "Acumatica" but entity is "Acumatica ERP" → NO MATCH, relationship dropped!

3. Limited Prompt

The prompt (lines 215-249) only lists 15 relationship types. The LLM doesn't know about the rich types GLiNER uses.

Proposed Solution

Phase 1: Expand Relationship Types

1. Create shared RELATIONSHIP_TYPE_RULES constant (TypeScript version of GLiNER's 80+ rules)
2. Update KNOWN_RELATIONSHIP_TYPES to include all types from GLiNER
3. Remove the RELATED_TO fallback transformation or make it configurable

Phase 2: Add Fuzzy Entity Matching

1. Implement substring/fuzzy matching for entity names
2. Use Levenshtein distance or similar algorithm
3. Add configurable similarity threshold (e.g., 0.8)

Phase 3: Improve LLM Prompt

1. Include all 80+ relationship types in prompt (grouped by category)
2. Add examples for each relationship type
3. Consider using few-shot prompting with domain examples

Phase 4: Cloud Provider Support (Groq, OpenAI)

1. Add Groq provider to adaptive-llm-factory.ts
2. Test with Llama 3.3 70B for quality comparison
3. Add cost estimation and rate limiting

Phase 5: Quality Comparison Framework

1. Create benchmark script to compare extraction methods
2. Metrics: relationship count, type diversity, RELATED_TO %, isolated entities %
3. Document expected quality vs speed tradeoffs

Files to Modify

| File | Changes |
|------|---------|
| backend/src/workflows/ingestion/knowledge-graph-builder/nodes/extract-relationships-node-llm.ts | Expand types, fuzzy matching, better prompt |
| backend/src/services/llm/adaptive-llm-factory.ts | Add Groq provider |
| backend/src/config/relationship-types.config.ts | New shared config for 80+ types |
| backend/src/types/llm-config.types.ts | Add Groq to providers |

Acceptance Criteria

• [ ] LLM extraction contributes >20% of relationships in hybrid mode
• [ ] RELATED_TO fallback remains <5%
• [ ] Fuzzy entity matching catches 90%+ of near-matches
• [ ] Groq cloud provider works with Llama 3.3 70B
• [ ] Quality comparison documented with benchmarks

Related

• Issue #264: KG Quality Improvements
• Current test document: bfc56531-db8d-4f42-93af-f7f1758acc94

Labels

enhancement, knowledge-graph, llm