agent-memory: Search accuracy validation concerns and Cognee Cloud limitations

Summary

During rigorous validation of the agent-memory system, several accuracy and reliability concerns were identified that affect the system's ability to return precise, verifiable search results.

Environment

• Repository: floe (devtools/agent-memory)
• Cognee Cloud API: api.cognee.ai
• Search Type: GRAPH_COMPLETION (default)
• Python: 3.11.11

Issues Identified

1. GRAPH_COMPLETION Returns AI-Synthesized Answers (Not Verbatim)

Problem: The default search type GRAPH_COMPLETION returns AI-generated summaries based on the knowledge graph, not verbatim excerpts from indexed documents.

Impact: Cannot retrieve exact quotes or verify precise wording from source documents.

Example:

# Source document (ARCHITECTURE-SUMMARY.md) contains:
# "Layer 1: FOUNDATION (Framework Code)"
# "    │ Owner: floe Maintainers"
# "    │ Distribution: PyPI, Helm"

# Search returns AI-paraphrased version:
./scripts/memory-search "four layer model"
# Result: "Foundation Layer: Contains the core framework code managed by floe maintainers."

The result is semantically correct but not verbatim - this makes precision validation difficult.

2. Result [2+] Often Contains Hallucinated/Generic Content

Problem: While Result [1] is typically accurate, subsequent results frequently contain generic or hallucinated content unrelated to the indexed documents.

Reproduction:

./scripts/memory-search "11 plugin types compute orchestrator catalog" --top-k 2

Expected: Both results should reference floe's 11 plugin types from documentation.

Actual:

[1] Floe has 11 distinct plugin types, which include Compute, Orchestrator, and Catalog...
    ✅ ACCURATE - matches source documentation

[2] The question seems to ask about the relationship and functionality of 11 different 
    types of plugins within a compute orchestrator catalog. However, the provided context 
    does not specify any nodes or connections...
    ❌ HALLUCINATED - generic response, not from indexed content

Pattern observed across multiple queries:

| Query | Result [1] | Result [2] |
|-------|------------|------------|
| "four layer model" | ✅ Accurate | ❌ Generic IT layer model |
| "11 plugin types" | ✅ Accurate | ❌ "context does not specify" |
| "CompiledArtifacts contract" | ✅ Accurate | ❌ Generic explanation |

3. CHUNKS and SUMMARIES Search Types Fail with 409 Errors

Problem: Alternative search types that might provide verbatim retrieval fail consistently.

Reproduction:

cd devtools/agent-memory
uv run agent-memory search "CompiledArtifacts" --type CHUNKS --top-k 3
uv run agent-memory search "CompiledArtifacts" --type SUMMARIES --top-k 3

Expected: Return document chunks or summaries.

Actual:

[warning] request_retryable_status  attempt=1 status_code=409
[warning] request_retryable_status  attempt=2 status_code=409
[warning] request_retryable_status  attempt=3 status_code=409
[warning] request_retryable_status  attempt=4 status_code=409
[warning] request_retryable_status  attempt=5 status_code=409
Error: Request failed after 5 attempts. Circuit breaker state: closed

Search Type Availability:

| Search Type | Status | HTTP Code |
|-------------|--------|-----------|
| GRAPH_COMPLETION | ✅ Works | 200 |
| CHUNKS | ❌ Fails | 409 |
| SUMMARIES | ❌ Fails | 409 |
| INSIGHTS | ❓ Untested | Likely 409 |

4. No Source File Attribution in Search Results

Problem: Search results don't include the source file path, making it impossible to trace answers back to source documents.

Evidence (from cognee_client.py:936-945):

items.append(
    SearchResultItem(
        content=content,
        source_path=item.get("source", item.get("source_path")),  # Often None
        relevance_score=float(item.get("score", 0.0)),
        ...
    )
)

Impact: Users cannot verify which document an answer came from without manual grep searches.

Recommendations

Short-term Mitigations

1. Document the accuracy pattern - Add to docs that Result [1] is most reliable
2. Add source cross-reference guide - Show users how to grep for verification
3. Implement quality validation tests - Already exists (agent-memory test)

Medium-term Improvements

1. Investigate 409 errors - Contact Cognee Cloud support about CHUNKS/SUMMARIES availability
2. Add accuracy scoring - Track which results match source documents
3. Implement source attribution - Parse source metadata from Cognee responses

Long-term Solutions

1. Hybrid search approach - Combine GRAPH_COMPLETION with local grep for precision
2. Result filtering - Detect and filter hallucinated "context does not specify" responses
3. Contract tests for accuracy - Validate specific facts (11 plugins, 4 layers) in CI

Validation Script

To reproduce all issues:

#!/bin/bash
# Save as: scripts/validate-search-accuracy.sh

echo "=== Test 1: GRAPH_COMPLETION accuracy ==="
./scripts/memory-search "four layer model" --top-k 2

echo -e "\n=== Test 2: Hallucination detection ==="
./scripts/memory-search "11 plugin types" --top-k 2

echo -e "\n=== Test 3: CHUNKS search type ==="
cd devtools/agent-memory
uv run agent-memory search "test" --type CHUNKS --top-k 1 2>&1 | head -10

echo -e "\n=== Test 4: SUMMARIES search type ==="
uv run agent-memory search "test" --type SUMMARIES --top-k 1 2>&1 | head -10

echo -e "\n=== Test 5: Quality validation ==="
uv run agent-memory test --threshold 80

Test Results Summary

| Test Category | Tests | Status |
|---------------|-------|--------|
| Unit Tests | 346 | ✅ All passed |
| Contract Tests | 10 | ✅ All passed |
| Quality Validation | 3 | ✅ All passed |
| Accuracy Validation | 5 | ⚠️ Concerns identified |

Related Files

• devtools/agent-memory/src/agent_memory/cognee_client.py - Search implementation
• devtools/agent-memory/tests/contract/test_cognee_api_contract.py - API field validation
• devtools/agent-memory/docs/verification-protocol.md - Verification documentation
• CLAUDE.md - Cognee Cloud API quirks section

Labels

• agent-memory
• quality
• documentation