Switch Big Book pipeline to Batch API for 50% cost savings on eligible calls

Summary

Switch Big Book pipeline Claude API calls from Message API to Batch API where eligible, saving ~$225/mo (23% reduction) with minimal effort. The Batch API gives a 50% per-token discount and the infrastructure already exists in anthropic_client.py from the Short Book batch switch.

Background

March API spend on the civly key was $2,241. The Big Book pipeline accounts for roughly $1,000/mo across ~52 effective runs (39 completed + 44 failed at ~30% token usage). Per-book cost is ~$19, broken down:

| Phase | Cost/Book | Model | Batch-eligible? |
|-------|-----------|-------|-----------------|
| Phase 3a — Independent Section Research (30 calls) | $6.75 | Sonnet | Yes |
| Phase 2.5 — Deep Research (agentic loop) | $8.10 | Opus | No (tool_use) |
| Phase 2 — Data Extraction (~30 calls) | $2.13 | Mixed | Yes (hard) |
| Phase 3b — Summary Sections (6 calls) | $1.98 | Sonnet | Yes |
| Phase 3c — Personality Scoring | $0.25 | Mixed | Yes |
| Phase 1 — Social Discovery | $0.03 | Sonnet | Not worth it |

Implementation Plan

Step 1: Batch Phase 3a — Independent Section Research ⭐ (biggest win)

File: backend/app/services/big_book_section_research.py — _analyze_section_data() (line ~1891)

Currently 25-35 independent Sonnet calls run via asyncio.gather. Zero inter-dependencies.

Change:

1. Collect all section prompts (system + user message) into a list
2. Submit as a single multi-request batch via create_message_batch()
3. Poll with poll_batch_until_done()
4. Distribute results back to section records via custom_id matching

Savings: ~$3.38/book → ~$175/mo
Latency: Minimal — batch of 30 requests typically completes in a few minutes, similar to current parallel execution.

Step 2: Batch Phase 3b — Summary Sections

File: backend/app/services/big_book_section_research.py — _research_summary_section() (line ~2004)

6 sequential Sonnet calls (executive_summary, path_to_victory, attack_lines, etc.).

Change: Use single-request batch for each, same pattern as Short Book's _run_batch_request().

Savings: ~$0.99/book → ~$51/mo
Latency: Adds ~2-3 min polling overhead per section (~12 min total extra). Acceptable for a background pipeline.

Step 3 (optional, lower priority): Batch Phase 2 — Data Extraction

~18 Sonnet + 3 Opus + 8 Haiku calls scattered across 15+ service files. Requires restructuring the Celery task flow (currently each data source task makes its own synchronous Claude call). Not recommended until Steps 1-2 are validated.

Savings: ~$1.07/book → ~$56/mo

What CANNOT be batched

Phase 2.5 — Deep Research ($8.10/book, the single most expensive phase) is an agentic loop using tool_use with web_search, fetch_page, record_finding, complete_research. Each iteration depends on tool results from the prior call. This is inherently incompatible with Batch API.

Expected Results

| Scenario | Per Book | Monthly | Savings |
|----------|---------|---------|---------|
| Current (all message) | ~$19 | ~$1,000 | — |
| Steps 1+2 (recommended) | ~$15 | ~$775 | ~$225/mo (23%) |
| Steps 1+2+3 | ~$14 | ~$715 | ~$290/mo (29%) |

Technical Notes

• Batch infra already exists: create_message_batch(), poll_batch_until_done(), get_message_batch_results() in anthropic_client.py
• Short Book already uses this pattern successfully (ShortBookService._run_batch_request())
• Phase 3a is the textbook batch workload: many independent, no-tool calls submitted at once
• Two legacy claude-3-haiku-20240307 calls in gov contracts + SEC should be migrated to HAIKU_MODEL constant while we're in there