Scaling Considerations: PreCog Sweep to 1,000 Drones

Current Performance Baseline (Single Node)

Sweep Timing (Cache Warm)

| Phase | Duration | Notes |
|-------|----------|-------|
| Fetch feeds (parallel) | 1.5s | Network bound |
| Dedup (Bloom filter) | <1ms | O(1) lookup |
| Novelty check | <1ms | 90% cache hit rate |
| SSL enrichment | <1ms | Cached, skip non-IPs |
| Azure File Share persist | 0.5s | Gzip compressed |
| Total | ~2-3s | With warm cache |

Pattern #57 Cache Stats

• URLhaus: 4,070 items cached
• Cache hit rate: 90%+
• API calls saved: 36,619+
• Storage: 74.3KB compressed (Azure File Share)

---

Rate Limits & External Constraints

| Service | Limit | Impact |
|---------|-------|--------|
| GitHub Search API | 30 req/min | Novelty checks |
| GreyNoise (free) | 50 req/day | RIOT filtering |
| OpenPhish | Updates 2x/day | Polling faster = waste |
| URLhaus | Updates every 5 min | Primary fresh feed |
| ThreatFox | Real-time | Requires API key |

---

Single Node Scaling (No Re-architecture)

| Interval | Sweeps/Day | Status |
|----------|------------|--------|
| 10 min (current) | 144 | ✅ Conservative, safe |
| 5 min | 288 | ✅ Safe with cache |
| 1 min | 1,440 | ✅ Aggressive but stable |
| 30 sec | 2,880 | ⚠️ Edge of rate limits |
| 15 sec | 5,760 | ❌ Will hit GitHub limits |

Recommended single-node max: Every 60 seconds

---

Multi-Drone Architecture (1,000 Drones)

Challenge: Shared State

• Bloom filter must be consistent across drones
• Can't have 1,000 drones all hitting GitHub API
• Need coordinator pattern

Proposed Architecture

┌─────────────────────────────────────────────────────────┐
│                    COORDINATOR (1x)                      │
│  - Fetches feeds every 60s                              │
│  - Maintains master Bloom filter                        │
│  - Publishes novel IOCs to queue                        │
│  - Persists to Azure File Share                         │
└─────────────────────┬───────────────────────────────────┘
                      │ Azure Service Bus / Redis Streams
                      ▼
┌─────────────────────────────────────────────────────────┐
│                 WORKER DRONES (1,000x)                  │
│  - Subscribe to novel IOC queue                         │
│  - Perform enrichment (SSL, PTR, brand detection)       │
│  - Report results back to coordinator                   │
│  - Stateless, horizontally scalable                     │
└─────────────────────────────────────────────────────────┘

Work Distribution

| Task | Coordinator | Drones |
|------|-------------|--------|
| Fetch feeds | ✅ | ❌ |
| Bloom dedup | ✅ | ❌ |
| GitHub novelty | ✅ | ❌ |
| SSL enrichment | ❌ | ✅ |
| PTR lookup | ❌ | ✅ |
| Brand detection | ❌ | ✅ |
| STIX generation | ✅ | ❌ |
| OTX pulse | ✅ | ❌ |

Throughput at Scale

With 1,000 drones doing SSL enrichment:

• 10 concurrent connections per drone = 10,000 parallel TLS handshakes
• 3 second timeout = ~3,000 IPs/second theoretical max
• ~10 million IPs/hour enrichment capacity

Infrastructure Requirements

| Component | Spec | Monthly Cost (Est) |
|-----------|------|-------------------|
| Coordinator | 1x B2s (2 vCPU, 4GB) | $30 |
| Drones | 1,000x B1s (1 vCPU, 1GB) | $7,500 |
| Azure Service Bus | Standard tier | $10 |
| Azure File Share | 10GB | $2 |
| Total | | ~$7,550/month |

Cheaper Alternative: Spot Instances

| Component | Spec | Monthly Cost (Est) |
|-----------|------|-------------------|
| Coordinator | 1x B2s | $30 |
| Drones | 1,000x Spot B1s (70% discount) | $2,250 |
| Messaging + Storage | | $12 |
| Total | | ~$2,300/month |

---

Implementation Phases

Phase 1: Single Node Optimization (Now)

• [x] Pattern #57 Bloom caching
• [x] Azure File Share persistence
• [ ] Reduce sweep interval to 60s
• [ ] Add metrics/observability

Phase 2: Multi-Node Ready (Future)

• [ ] Extract coordinator logic
• [ ] Add message queue for work distribution
• [ ] Stateless drone worker mode
• [ ] Redis for shared Bloom filter

Phase 3: Full Scale (1,000 Drones)

• [ ] Kubernetes deployment manifests
• [ ] Auto-scaling based on queue depth
• [ ] Geographic distribution (multi-region)
• [ ] Cost optimization with spot instances

---

Key Decisions Needed

1. Message Queue: Azure Service Bus vs Redis Streams vs Kafka?
2. Drone Packaging: Container Apps vs AKS vs VMs?
3. Bloom Sync: Redis vs periodic File Share pull?
4. Cost vs Speed: How much are we willing to spend for sub-minute detection?

---

References

• Pattern #57: Quantum-Inspired Probabilistic Verification
• Current implementation: scripts/precog-sweep/
• Cache: scripts/precog-sweep/cache/enrichment-cache.js
• Config: scripts/precog-sweep/config.js

---

"The cache is doing the heavy lifting. With 90% hit rate, you're basically doing a 2-second health check most runs."