On-demand debug logging for agent runtime diagnostics

What problem does this solve?

Scenario: Spent 2-3 hours debugging why ObjectStructure finding (confirmed in database) wasn't being detected at runtime, causing multipart/form-data serialization to fail.

The Issue:
Agent correctly sends flat fields:

[
  {"param_path": "body#metadata/api_name", "value": "Test API"},
  {"param_path": "body#metadata/version", "value": "1.0.0"},
  {"param_path": "body#metadata/base_url", "value": "https://example.com"}
]

System should:

1. Group flat fields by top-level (metadata)
2. Check body#metadata for ObjectStructure finding
3. Serialize as JSON object: {"api_name":"Test API", "version":"1.0.0", ...}
4. Send as multipart part with content-type: application/json

System actually did:

• ObjectStructure check returned FALSE (finding exists in DB but not detected at runtime)
• Failed to serialize, sent flat fields or errored

Root Cause Unknown Without Logs:

• Is finding not loaded from DB into CallTemplate?
• Is ParameterStructureEnforcer filtering it out?
• Is cache stale?
• Is finding attached to wrong parameter path?

Current State:

• No visibility into runtime decision points
• DEBUG logs disabled in production (performance)
• Can't reproduce agent behavior deterministically
• Spent hours reading code, checking DB, writing mock tests to guess what's wrong

The Gap:
When developers troubleshoot code, they enable debug logs. Agents need the same capability - but current logging is all-or-nothing (DEBUG everywhere kills performance, no DEBUG leaves us blind).

Proposed solution

On-Demand Debug Level via Request Header

Allow agent/customer to explicitly request verbose logs for specific requests, just like developers enable debug mode when troubleshooting.

Header: X-Debug-Level: <level>

Levels:

• minimal (default): Errors, exceptions, request ID
• decisions: Key decision points - which branch taken, why
• verbose: Full diagnostic context - findings, parameter trees, templates
• trace: Complete execution trace (nuclear option)

Two Activation Modes:

1. Agent/Customer Instruction (per-request)

Customer: "Agent, enable verbose logging and retry the upload"
Agent: Adds X-Debug-Level: verbose to next request
System: Generates detailed logs for that specific request
Developer: Analyzes logs to diagnose issue

2. Platform Knob (account-level)

Admin Panel: Enable debug_level=verbose for customer account
All agent requests include verbose logs
Developer: Diagnoses issue asynchronously
Developer: Fixes bug, disables verbose

Example Implementation:

// Request context
public class RequestContext {
    private static final ThreadLocal<DebugLevel> DEBUG_LEVEL = 
        ThreadLocal.withInitial(() -> DebugLevel.MINIMAL);
    
    public enum DebugLevel { MINIMAL, DECISIONS, VERBOSE, TRACE }
    
    public static void setFromHeaders(HttpHeaders headers) {
        String level = headers.getFirst("X-Debug-Level");
        if (level != null) {
            DEBUG_LEVEL.set(DebugLevel.valueOf(level.toUpperCase()));
        }
    }
    
    public static DebugLevel get() { return DEBUG_LEVEL.get(); }
    public static boolean atLeast(DebugLevel level) { 
        return get().ordinal() >= level.ordinal(); 
    }
}

// In runtime code (e.g., MultipartFormDataAssembler)
if (RequestContext.atLeast(DECISIONS)) {
    log.info("Part '{}': detected as {} (ObjectStructure={}, ArrayStructure={})", 
        name, type, hasObject, hasArray);
}

if (RequestContext.atLeast(VERBOSE)) {
    log.info("Part '{}': VERBOSE DIAGNOSTICS:\n" +
             "  rootParameter.path: {}\n" +
             "  findingSet.size: {}\n" +
             "  findingSet.keys: {}\n" +
             "  ObjectStructure present: {}\n" +
             "  template.bodyParameters: {}",
        name, param.path(), findings.size(), findings.keys(), 
        hasObject, template.bodyParameters());
}

Alternatives considered

Alternative 1: Always-on decision logging

• Log all decision points at INFO level always
• Downside: Performance cost on every request (millions of agent calls)
• Verdict: Wasteful - only need verbose when troubleshooting

Alternative 2: Post-mortem analysis

• Rely on error messages and exceptions
• Downside: Only shows "it failed here" not "why did it take this path?"
• Example: Error says "missing data type" but doesn't show ObjectStructure=false
• Verdict: Insufficient for diagnosing complex decision flows

Alternative 3: Sampling (1% of requests get verbose)

• Randomly enable verbose logs for small percentage
• Downside: Might miss the specific failing request
• Verdict: Unreliable for rare edge cases

Alternative 4: Agent self-diagnosis

• Agent reads logs and self-corrects
• Downside: Circular dependency - logs are for debugging agent behavior
• Verdict: Wrong abstraction - logs are for human troubleshooters

Additional context

Concrete Example - What We'd See With Verbose Logging:

Without (current state):

ERROR: Parameter 'body#metadata' is missing a data type

→ Diagnosis time: 2-3 hours of code tracing, DB queries, mock tests

With X-Debug-Level: verbose:

INFO: Building multipart/form-data body with 3 body parameters
INFO: Grouped into 2 multipart parts: [metadata, spec]
INFO: Part 'metadata': detected as primitive (ObjectStructure=false, ArrayStructure=false)
VERBOSE: Part 'metadata': DIAGNOSTICS:
  rootParameter.path: body#metadata
  findingSet.size: 2
  findingSet.keys: [param.doc.summary, param.constraint.required]
  ObjectStructure present: FALSE ← SMOKING GUN
  template.bodyParameters: [body#metadata, body#metadata/api_name, body#metadata/version, body#metadata/base_url, body#spec]
ERROR: Parameter 'body#metadata' is missing a data type

→ Diagnosis time: 5 minutes - immediately see ObjectStructure missing from findingSet

Next debugging step becomes obvious:
"Why isn't ObjectStructure in the findingSet when it exists in DB?"

• Check ApiTemplateService loading logic
• Check ParameterStructureEnforcer filtering
• Check cache invalidation

Benefits:
✅ Zero cost until needed - Only pay for verbosity when debugging
✅ Customer/agent controlled - Explicit opt-in via instruction
✅ Developer controlled - Platform can enable for troubled accounts
✅ Privacy-aware - Explicit consent for detailed logs
✅ Mirrors developer workflow - Same as enabling debug logs during development
✅ Scalable - Handles millions of unpredictable agent paths without performance hit

Initial Scope:
Start with critical runtime components where decisions are made:

• MultipartFormDataAssembler (object vs primitive decision)
• ApiCallCommandExecutor (content type selection)
• ApiTemplateService (template loading, caching)
• ParameterStructureEnforcer (finding filtering)
• ParameterResolver (value binding)

Expand to other components as patterns emerge.