feat: Performance Audit PF-001 & PF-007 Implementation - Parallel Search & PostgreSQL Tuning

Summary

Implemented performance optimizations from the Performance & Scalability Audit Report (13-PERFORMANCE-SCALABILITY-AUDIT-REPORT-01-01-2026).

Completed Work

1. Parallel Search Node (PF-001) - 50-150ms Latency Reduction

Problem: Vector and keyword searches executed sequentially in hybrid retrieval, adding unnecessary latency.

Solution: Created a new parallel-search-node.ts that executes both searches concurrently using Promise.allSettled.

Files Changed:

• backend/src/workflows/rag/hybrid-retrieval/nodes/parallel-search-node.ts - New file
• backend/src/workflows/rag/hybrid-retrieval/workflow.ts - Updated routing

Before:

check-semantic-cache → vector-search (50-100ms) → keyword-search (50-100ms) → merge-rrf
Total: 100-200ms (additive)

After:

check-semantic-cache → parallel-search (both concurrent) → merge-rrf
Total: 50-100ms (max of both)

Key Features:

• Uses Promise.allSettled for graceful partial failure handling
• If one search fails, still returns results from the other
• Only affects hybrid strategy; semantic and keyword strategies unchanged
• Combines errors from both searches into state

2. PostgreSQL Performance Tuning (PF-007) - 20-30% Query Improvement

Problem: PostgreSQL running with default configuration, not optimized for workload.

Solution: Added performance tuning parameters to docker-compose.yml.

File Changed:

• docker-compose.yml - Added command args for postgres service

Configuration Added:
| Parameter | Value | Purpose |
|-----------|-------|---------|
| shared_buffers | 256MB | 25% of container memory (1GB) |
| effective_cache_size | 768MB | OS cache hint for query planner |
| work_mem | 16MB | Memory for sorts/hashes per operation |
| maintenance_work_mem | 128MB | Memory for VACUUM, CREATE INDEX |
| random_page_cost | 1.1 | Optimized for SSD storage |

---

Testing Required

Parallel Search Testing

1. Start the backend locally:

``bash
cd backend && npm run dev
``

1. Execute hybrid search queries:

• Send a query with search_strategy: 'hybrid'
• Verify both vector_results and keyword_results are populated
• Check logs for parallel-search-node entries

1. Verify semantic-only still works:

• Send a query with search_strategy: 'semantic'
• Verify only vector_results are populated

1. Verify keyword-only still works:

• Send a query with search_strategy: 'keyword'
• Verify only keyword_results are populated

1. Test error handling:

• Simulate Vespa being unavailable
• Verify graceful degradation (errors in state, no crash)

1. Performance comparison:

• Time hybrid queries before/after
• Expected: 50-150ms improvement

PostgreSQL Tuning Testing

1. Restart the database container:

``bash
docker-compose down database && docker-compose up -d database
``

1. Verify settings applied:

``bash
docker exec courdx-postgres psql -U postgres -c "SHOW shared_buffers;"
docker exec courdx-postgres psql -U postgres -c "SHOW effective_cache_size;"
docker exec courdx-postgres psql -U postgres -c "SHOW work_mem;"
docker exec courdx-postgres psql -U postgres -c "SHOW maintenance_work_mem;"
docker exec courdx-postgres psql -U postgres -c "SHOW random_page_cost;"
``

1. Expected output:

• shared_buffers: 256MB
• effective_cache_size: 768MB
• work_mem: 16MB
• maintenance_work_mem: 128MB
• random_page_cost: 1.1

---

Already Implemented (No Changes Needed)

These audit recommendations were found to already be implemented:

| Finding | Status | Location |
|---------|--------|----------|
| PF-011: Embedding Cache | ✅ Exists | embedding-cache.service.ts (7-day TTL) |
| PF-012: Query Result Cache | ✅ Exists | semantic-cache.service.ts (24hr TTL, similarity matching) |
| PF-004: Worker Concurrency 3-5 | ✅ Configured | document: 5, KG: 3, sync: 3 via queue_concurrency |
| PF-002: Comprehensive Indexing | ✅ Good | Indexes on all FKs and query fields |
| PF-005: Redis Connection Pool | ✅ Good | Factory pattern with health monitoring |
| PF-008: Appropriate Index Coverage | ✅ Good | Multi-column indexes for common patterns |
| PF-009: Model-Aware Chunk Size | ✅ Excellent | Prevents embedding truncation |
| PF-010: Comprehensive Model Support | ✅ Excellent | 50+ models with metadata |
| PF-014: Workers Bootstrap | ✅ Good | Independent operation with Vault |

---

Optional Future Enhancements

These items are not critical and can be implemented later:

Quick Wins (Low Priority)

| Item | Effort | Impact | Notes |
|------|--------|--------|-------|
| Enable pg_stat_statements | 30 min | Query analysis | Add to PostgreSQL config for diagnostics |

Medium-Term (Optional)

| Item | Effort | Impact | Notes |
|------|--------|--------|-------|
| Response Streaming | 2 days | Perceived latency | Would require workflow changes |
| Worker Auto-Scaling | 2 days | Dynamic resources | Based on queue depth |
| Document Metadata Cache (PF-013) | 1 day | Reduce DB reads | Cache frequently accessed docs |

Long-Term (Infrastructure)

| Item | Effort | Impact | Notes |
|------|--------|--------|-------|
| Vespa Distributed Deployment (PF-016) | 1 week | Horizontal scaling | Multiple content/container nodes |
| PostgreSQL Read Replica | 1 week | Query distribution | For high-read workloads |
| GPU Worker Pool | 2 weeks | Embedding throughput | For large-scale ingestion |
| Memgraph Clustering (PF-015) | 1+ week | Graph scaling | Consider Memgraph Enterprise |

---

Related

• Audit Report: docs/Generated/audit reports/13-PERFORMANCE-SCALABILITY-AUDIT-REPORT-01-01-2026.md
• Branch: staging

---

Do not close this issue until testing is complete.