perf: Parallelize signal generation strategies for faster execution

Resolved 💬 3 comments Opened Dec 4, 2025 by matthew-wills Closed Feb 3, 2026

Summary

The signal generation script (scripts/signals/signals_all.py) currently runs 10 trading strategies sequentially, despite them being functionally independent. This results in long execution times (often 2-3+ minutes total).

Current Behavior

  • 10 strategies execute one after another
  • Each strategy loads its own market data independently
  • No data sharing or caching between strategies
  • Console output interleaved with execution
  • Typical strategy times: 12-25 seconds each

Example from recent run:

MOC_LONG: 23.4s
MOC_SHORT: 18.1s
MR_LONG: 12.2s
TREND: 18.4s
... (total ~2-3 minutes)

Desired Behavior

  • Run strategies in parallel to reduce total execution time
  • Maintain orderly console output (strategies print in consistent order)
  • Maintain orderly log file output for debugging/verification
  • Handle errors gracefully without affecting other strategies

Technical Considerations

  1. Strategies are independent - Each receives read-only current_positions_df and usable_capital
  2. Norgate API - May have rate limits on concurrent requests
  3. Output ordering - Need to buffer/queue output to maintain readability
  4. Logging - Python's logging module is thread-safe but output interleaving is a concern

Proposed Options

  1. Multi-processing with separate modules - Each strategy in its own file, run as subprocess
  2. concurrent.futures ThreadPoolExecutor - Simpler but GIL-limited
  3. concurrent.futures ProcessPoolExecutor - True parallelism within single script
  4. Hybrid approach - Group strategies by market (US vs AU) and run groups in parallel

Requirements

  • [ ] Orderly console output (print strategies in consistent order after completion)
  • [ ] Orderly log output for verification
  • [ ] Error isolation (one strategy failure doesn't stop others)
  • [ ] Configurable parallelism (--parallel flag or config setting)

🤖 Generated with Claude Code

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗