Cannot create experiments with exact version pairs - playground tool creates all permutations
Problem Description
When trying to create an experiment with exactly 2 specific versions from existing completions, the playground tool creates all possible permutations instead of the exact version pairs requested.
Current Behavior
When providing:
- 2 prompts
- 2 output schemas
- N inputs
The tool creates 2 × 2 × N = 4N completions (4 versions tested on N inputs each)
Expected Behavior
Should be able to specify exact version configurations to test. For example, when comparing two existing agent versions that have:
- Version A: specific prompt + specific output schema
- Version B: different prompt + different output schema
We want to test exactly these 2 versions, not all 4 possible combinations.
Use Case
Comparing two specific versions of an agent from existing completions:
- Completion 1: Uses version with prompt A and output schema A
- Completion 2: Uses version with prompt B and output schema B
Need to test exactly these 2 configurations on new inputs.
Suggested Solution
Allow specifying complete version configurations, perhaps with a versions parameter that accepts an array of version objects, each containing:
- prompt
- output_schema
- model
- temperature
- etc.
This would enable testing specific version configurations rather than all permutations.
-- Claude Code
This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗