perf: Parallelize signal generation strategies for faster execution
Resolved 💬 3 comments Opened Dec 4, 2025 by matthew-wills Closed Feb 3, 2026
Summary
The signal generation script (scripts/signals/signals_all.py) currently runs 10 trading strategies sequentially, despite them being functionally independent. This results in long execution times (often 2-3+ minutes total).
Current Behavior
- 10 strategies execute one after another
- Each strategy loads its own market data independently
- No data sharing or caching between strategies
- Console output interleaved with execution
- Typical strategy times: 12-25 seconds each
Example from recent run:
MOC_LONG: 23.4s
MOC_SHORT: 18.1s
MR_LONG: 12.2s
TREND: 18.4s
... (total ~2-3 minutes)
Desired Behavior
- Run strategies in parallel to reduce total execution time
- Maintain orderly console output (strategies print in consistent order)
- Maintain orderly log file output for debugging/verification
- Handle errors gracefully without affecting other strategies
Technical Considerations
- Strategies are independent - Each receives read-only
current_positions_dfandusable_capital - Norgate API - May have rate limits on concurrent requests
- Output ordering - Need to buffer/queue output to maintain readability
- Logging - Python's logging module is thread-safe but output interleaving is a concern
Proposed Options
- Multi-processing with separate modules - Each strategy in its own file, run as subprocess
- concurrent.futures ThreadPoolExecutor - Simpler but GIL-limited
- concurrent.futures ProcessPoolExecutor - True parallelism within single script
- Hybrid approach - Group strategies by market (US vs AU) and run groups in parallel
Requirements
- [ ] Orderly console output (print strategies in consistent order after completion)
- [ ] Orderly log output for verification
- [ ] Error isolation (one strategy failure doesn't stop others)
- [ ] Configurable parallelism (--parallel flag or config setting)
🤖 Generated with Claude Code
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗