Cannot create experiments with exact version pairs - playground tool creates all permutations

Resolved 💬 2 comments Opened Sep 11, 2025 by anyacherniss Closed Sep 11, 2025

Problem Description

When trying to create an experiment with exactly 2 specific versions from existing completions, the playground tool creates all possible permutations instead of the exact version pairs requested.

Current Behavior

When providing:

  • 2 prompts
  • 2 output schemas
  • N inputs

The tool creates 2 × 2 × N = 4N completions (4 versions tested on N inputs each)

Expected Behavior

Should be able to specify exact version configurations to test. For example, when comparing two existing agent versions that have:

  • Version A: specific prompt + specific output schema
  • Version B: different prompt + different output schema

We want to test exactly these 2 versions, not all 4 possible combinations.

Use Case

Comparing two specific versions of an agent from existing completions:

  • Completion 1: Uses version with prompt A and output schema A
  • Completion 2: Uses version with prompt B and output schema B

Need to test exactly these 2 configurations on new inputs.

Suggested Solution

Allow specifying complete version configurations, perhaps with a versions parameter that accepts an array of version objects, each containing:

  • prompt
  • output_schema
  • model
  • temperature
  • etc.

This would enable testing specific version configurations rather than all permutations.

-- Claude Code

View original on GitHub ↗

This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗