Test Framework: E2E tests require hardcoded API keys

Resolved 💬 3 comments Opened Oct 15, 2025 by vishnujayvel Closed Jan 11, 2026

Issue Description

The end-to-end tests in tests/v2/test_progress_modal_ui.py and tests/v2/test_latency_optimizations.py require API keys to be hardcoded directly in the test files, which is not ideal for automated testing or CI/CD pipelines.

Current Behavior

Tests are currently marked with @pytest.mark.skip and include placeholder values:

configure_api_keys(
    popup_page,
    gemini_key='YOUR_GEMINI_KEY',  # Replace with real key
    splitwise_key='YOUR_SPLITWISE_KEY',  # Replace with real key
    group_id='YOUR_GROUP_ID'  # Replace with real group ID
)

Problems

  1. Security Risk: API keys would be committed to source control if hardcoded
  2. Not Automated: Tests cannot run in CI/CD without manual key insertion
  3. No Test Isolation: Using production group can pollute real data
  4. Manual Setup: Each developer needs to manually update test files

Affected Tests

Progress Modal Tests (4 tests):

  • test_progress_modal_appears_on_button_click
  • test_progress_modal_step_transitions
  • test_progress_modal_shows_error_state
  • test_progress_modal_persistence_during_workflow

Latency Optimization Tests (3 tests):

  • test_llm_cache_hit_on_duplicate_order
  • test_async_comment_does_not_block_success_message
  • test_end_to_end_latency_improvement

Total: 7 tests currently skipped

Proposed Solution

Option 1: Environment Variables

import os

GEMINI_KEY = os.getenv('TEST_GEMINI_API_KEY')
SPLITWISE_KEY = os.getenv('TEST_SPLITWISE_API_KEY')
GROUP_ID = os.getenv('TEST_SPLITWISE_GROUP_ID')

# Skip if not configured
pytestmark = pytest.mark.skipif(
    not all([GEMINI_KEY, SPLITWISE_KEY, GROUP_ID]),
    reason="API keys not configured in environment"
)

Option 2: Test Configuration File

# tests/v2/test_config.json (gitignored)
{
  "gemini_api_key": "...",
  "splitwise_api_key": "...",
  "test_group_id": "..."
}

Option 3: pytest-env Plugin

# pytest.ini
[pytest]
env =
    TEST_GEMINI_API_KEY=file:.env:GEMINI_API_KEY
    TEST_SPLITWISE_API_KEY=file:.env:SPLITWISE_API_KEY
    TEST_GROUP_ID=file:.env:SPLITWISE_GROUP_ID

Recommendation

Option 1 (Environment Variables) is recommended because:

  • ✅ Standard practice for CI/CD
  • ✅ No additional dependencies
  • ✅ Works with GitHub Actions secrets
  • ✅ Easy to configure locally (.env file)

Related Issue

See also: "Create dedicated test Splitwise group to avoid polluting production data" (separate issue)

Impact

Current: 7/12 tests (58%) are skipped
After Fix: All tests can run in CI/CD with proper secrets configuration

Files to Modify

  • tests/v2/test_progress_modal_ui.py
  • tests/v2/test_latency_optimizations.py
  • tests/v2/conftest.py (add fixture for API keys)
  • .env.example (template for developers)
  • .gitignore (ensure .env is ignored)
  • README.md or tests/v2/README_LATENCY_TESTS.md (document setup)

Priority

Medium - Tests work with manual setup, but automation would improve developer experience and enable CI/CD.

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗