Skip to main content

Intelligent Error Handling Guide

Welcome to intelligent error handling in TruvaG3! This guide will walk you through how tools report errors and how agents recover from them intelligently. Think of it as teaching your components to have a meaningful conversation when things go wrong.

Table of Contents​

The Problem We're Solving​

Imagine you're at a restaurant. You order a "cheeseburger with extra pickles" and the waiter comes back saying:

"Here's a grilled cheese sandwich."

That's not helpful! You wanted a cheeseburger. The waiter should have said:

"Sorry, we're out of beef patties today. Would you like a chicken burger instead, or we have great veggie options?"

This is exactly what happens with traditional API error handling. When a tool fails, it often:

  1. Returns mock/fake data (hiding the real problem)
  2. Returns cryptic error messages with no context
  3. Gives up without explaining what could be done differently

Intelligent error handling fixes this. Tools tell agents exactly what went wrong with enough context for the agent (or its AI) to try again with corrected input.

The Solution: A Clear Separation of Concerns​

TruvaG3 splits error handling into two clear roles:

ComponentResponsibilityDoes NOT
ToolReports what happened (error code, category, context)Suggest fixes, implement retry logic, return mock data
AgentAnalyzes errors, decides retry strategy, uses AI to correct payloadsHide errors from users

Think of it like a doctor and a patient:

  • Patient (Tool): "I have a headache, it's on the right side, started yesterday, pain level 7"
  • Doctor (Agent): "Based on your symptoms, let's try this treatment..."

The patient describes the problem clearly. The doctor decides how to fix it.

Part 1: The Protocol Types (in core)​

TruvaG3 provides three core types for error communication. These live in the core package because they're the shared vocabulary between tools and agents.

ErrorCategory: Classifying Errors​

Error categories help agents quickly decide how to handle errors:

import "github.com/truvaagents/truva-g3/core"

// Available categories
core.CategoryInputError   // Bad input format - might be fixable
core.CategoryNotFound     // Resource doesn't exist - might exist with different parameters
core.CategoryRateLimit    // Too many requests - wait and retry
core.CategoryAuthError    // Authentication failed - NOT fixable by retry
core.CategoryServiceError // Backend service down - transient, retry later

When to use each:

CategoryExampleTypically Retryable?
CategoryInputErrorMissing required fieldYes, with fixed input
CategoryNotFoundCity "NYC" not foundYes, try "New York, US"
CategoryRateLimitAPI quota exceededYes, after waiting
CategoryAuthErrorInvalid API keyNo, needs config fix
CategoryServiceErrorWeather API is downYes, same input later

ToolError: Structured Error Information​

ToolError provides all the context an agent needs to understand and potentially fix an error:

type ToolError struct {
    Code      string            `json:"code"`      // Machine-readable: "LOCATION_NOT_FOUND"
    Message   string            `json:"message"`   // Human-readable: "City 'NYC' not found"
    Category  ErrorCategory     `json:"category"`  // Classification for routing
    Retryable bool              `json:"retryable"` // Permission for eligible same-step retries
    Details   map[string]string `json:"details"`   // Context for AI analysis
}

Creating a ToolError:

err := &core.ToolError{
    Code:      "LOCATION_NOT_FOUND",
    Message:   "Location 'Flower Mound, TX' not found in weather database",
    Category:  core.CategoryNotFound,
    Retryable: true,  // Agent CAN try with corrected input
    Details: map[string]string{
        "original_location": "Flower Mound, TX",
        "hint":              "Try 'City, Country' format (e.g., 'London, UK')",
    },
}

ToolResponse: Standard Response Envelope​

All tool responses should use this envelope for consistency:

type ToolResponse struct {
    Success bool        `json:"success"`          // Did it work?
    Data    interface{} `json:"data,omitempty"`   // Result if successful
    Error   *ToolError  `json:"error,omitempty"`  // Error details if failed
}

Success response:

response := core.ToolResponse{
    Success: true,
    Data: map[string]interface{}{
        "temperature": 22.5,
        "condition":   "sunny",
        "location":    "London",
    },
}

Error response:

response := core.ToolResponse{
    Success: false,
    Error: &core.ToolError{
        Code:      "LOCATION_NOT_FOUND",
        Message:   "City 'XYZ123' not found",
        Category:  core.CategoryNotFound,
        Retryable: true,
    },
}

HTTPStatusForCategory: Consistent HTTP Status Codes​

Use this helper to return the right HTTP status code:

// Returns appropriate HTTP status for each category
status := core.HTTPStatusForCategory(err.Category)

// Mapping:
// CategoryInputError   → 400 Bad Request
// CategoryNotFound     → 404 Not Found
// CategoryAuthError    → 401 Unauthorized
// CategoryRateLimit    → 429 Too Many Requests
// CategoryServiceError → 503 Service Unavailable

Part 2: Tool Implementation​

Tools have one job: report errors clearly and honestly. Never return mock data. Never implement retry logic. Just describe what happened.

Step 1: Handle the Capability Request​

When an agent calls your tool, you need to handle the request in a specific order. Here's what happens:

  1. Parse the request - Read the JSON body sent by the agent
  2. Validate inputs - Check that required fields are present and valid
  3. Call the real API - Make the actual external API call
  4. Handle errors - If something fails, classify it and return a structured error
  5. Return success - If everything works, return the data

Notice how each error scenario returns a ToolError with appropriate category and whether it's retryable. The tool never guesses or returns fake data.

func (w *WeatherTool) handleCurrentWeather(rw http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    // Step 1: Parse the request
    var req struct {
        Location string `json:"location"`
        Units    string `json:"units"`
    }
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        // Bad JSON = not retryable (agent sent malformed data)
        w.sendError(rw, &core.ToolError{
            Code:      "INVALID_REQUEST",
            Message:   "Failed to parse request body",
            Category:  core.CategoryInputError,
            Retryable: false,
        })
        return
    }

    // Step 2: Validate inputs
    if req.Location == "" {
        // Missing field = retryable (agent can add the field)
        w.sendError(rw, &core.ToolError{
            Code:      "MISSING_LOCATION",
            Message:   "Location is required",
            Category:  core.CategoryInputError,
            Retryable: true,
            Details: map[string]string{
                "hint": "Provide a location like 'London, UK' or 'Tokyo, Japan'",
            },
        })
        return
    }

    // Step 3: Call the real API
    weather, err := w.fetchWeatherFromAPI(ctx, req.Location, req.Units)

    // Step 4: Handle errors
    if err != nil {
        toolErr := w.classifyAPIError(err, req.Location)
        w.sendError(rw, toolErr)
        return
    }

    // Step 5: Return success
    w.sendSuccess(rw, weather)
}

Step 2: Classify API Errors​

Recommended for upstream API calls: Use core.ClassifyUpstreamError(err) instead of writing per-tool classification logic. It extracts HTTP status codes from error messages (formats like "error 400", "status 429", "code: 502") and returns the correct (HTTPStatus, Category, Retryable) tuple.

  • Upstream 400/404/409/422 → Tool returns 400 with Retryable:false → stop. A tool-owned explicit opt-in for a known correctable condition can permit parameter correction.
  • Upstream 401/403 → Tool returns 401/403 → Fail immediately
  • Upstream 429 → Tool returns retryable 429 → executor's unchanged-input backoff
  • Upstream 5xx / unknown → Tool returns retryable 502 → executor's unchanged-input backoff

See: API Reference — ClassifyUpstreamError and Tool Development Guide — Upstream Error Handling

When an external API fails, your tool needs to translate that failure into a structured ToolError. This is where you add intelligence - examining the raw error and deciding:

  • What category? Is it a user input problem, authentication issue, or server error?
  • Is it retryable? Can the agent fix this by changing the input, or is it unfixable?
  • What hints can we provide? What information helps the agent (or AI) correct the request?

Here's the decision process:

  1. Check a known, safely correctable lookup condition → A spelling correction is permitted by this capability → Explicitly opt in; a generic 404 is not sufficient
  2. Check for authentication errors → API key is wrong → NOT retryable (agent can't fix credentials)
  3. Check for rate limit errors → Too many requests → Retryable after waiting
  4. Default to service error → Something unexpected → Retryable (might be transient)
func (w *WeatherTool) classifyAPIError(err error, location string) *core.ToolError {
    errStr := err.Error()

    // Check 1: Location not found (agent can retry with different spelling)
    if strings.Contains(errStr, "city not found") {
        return &core.ToolError{
            Code:      "LOCATION_NOT_FOUND",
            Message:   fmt.Sprintf("Location '%s' not found in weather database", location),
            Category:  core.CategoryNotFound,
            Retryable: true, // Agent CAN fix this with corrected location
            Details: map[string]string{
                "original_location": location,
                "hint":              "Try 'City, Country' format (e.g., 'Paris, France')",
            },
        }
    }

    // Check 2: Authentication failed (agent CANNOT fix this)
    if strings.Contains(errStr, "401") || strings.Contains(errStr, "invalid API key") {
        return &core.ToolError{
            Code:      "API_KEY_INVALID",
            Message:   "Weather API authentication failed",
            Category:  core.CategoryAuthError,
            Retryable: false, // Agent can't fix API keys - needs human intervention
        }
    }

    // Check 3: Rate limited (agent should wait and retry)
    if strings.Contains(errStr, "429") || strings.Contains(errStr, "rate limit") {
        return &core.ToolError{
            Code:      "RATE_LIMIT_EXCEEDED",
            Message:   "Weather API rate limit exceeded",
            Category:  core.CategoryRateLimit,
            Retryable: true, // Agent CAN retry after waiting
            Details: map[string]string{
                "retry_after": "60s", // Tell agent how long to wait
            },
        }
    }

    // Check 4: Default - treat as temporary service error
    return &core.ToolError{
        Code:      "SERVICE_UNAVAILABLE",
        Message:   "Weather service temporarily unavailable",
        Category:  core.CategoryServiceError,
        Retryable: true, // Agent CAN retry with same input later
        Details: map[string]string{
            "original_error": errStr, // Include raw error for debugging
        },
    }
}

Step 3: Send Responses with Correct HTTP Status​

The final piece is sending the response back to the agent. Two key points:

  1. Use HTTPStatusForCategory() - This ensures the HTTP status code matches the error category. The agent uses this status code to make quick routing decisions.
  2. Always wrap in ToolResponse - Whether success or failure, use the standard envelope so agents can parse it consistently.
func (w *WeatherTool) sendError(rw http.ResponseWriter, toolErr *core.ToolError) {
    // Set content type so agent knows to parse JSON
    rw.Header().Set("Content-Type", "application/json")

    // Use the helper to get the right HTTP status (400, 401, 404, 429, 503)
    rw.WriteHeader(core.HTTPStatusForCategory(toolErr.Category))

    // Wrap in standard envelope
    response := core.ToolResponse{
        Success: false,
        Error:   toolErr,
    }
    json.NewEncoder(rw).Encode(response)
}

func (w *WeatherTool) sendSuccess(rw http.ResponseWriter, data interface{}) {
    rw.Header().Set("Content-Type", "application/json")
    rw.WriteHeader(http.StatusOK)

    // Wrap in standard envelope
    response := core.ToolResponse{
        Success: true,
        Data:    data,
    }
    json.NewEncoder(rw).Encode(response)
}

Common Tool Error Codes​

Here are recommended error codes for common scenarios:

// Location/Resource errors
const (
    ErrCodeLocationNotFound = "LOCATION_NOT_FOUND"
    ErrCodeSymbolNotFound   = "SYMBOL_NOT_FOUND"
    ErrCodeResourceNotFound = "RESOURCE_NOT_FOUND"
)

// Input validation errors
const (
    ErrCodeInvalidRequest   = "INVALID_REQUEST"
    ErrCodeMissingField     = "MISSING_REQUIRED_FIELD"
    ErrCodeInvalidFormat    = "INVALID_FORMAT"
)

// Authentication errors
const (
    ErrCodeAPIKeyMissing  = "API_KEY_MISSING"
    ErrCodeAPIKeyInvalid  = "API_KEY_INVALID"
    ErrCodeUnauthorized   = "UNAUTHORIZED"
)

// Service errors
const (
    ErrCodeRateLimitExceeded  = "RATE_LIMIT_EXCEEDED"
    ErrCodeServiceUnavailable = "SERVICE_UNAVAILABLE"
    ErrCodeTimeout            = "REQUEST_TIMEOUT"
)

Part 3: Agent Implementation​

Agents receive errors from tools and decide how to handle them. With AI, agents can even analyze errors and generate corrected payloads.

Production Recommendation: The retry logic shown in this section is for educational purposes. For production agents, use the orchestration module which provides:

  • Layer 4 Semantic Retry: LLM-powered error analysis
  • Automatic Parameter Correction: AI fixes payloads based on error context
  • Workflow-Level Coordination: Retry logic integrated with multi-step workflows

See: orchestration/README.md

The Retry Decision Flow​

When a tool returns an error, the agent needs to answer two questions:

  1. Should I retry? (Based on HTTP status code)
  2. With what payload? (Same payload or AI-corrected payload)

Here's the decision tree:

For the framework executor, explicit retryable:false stops all same-step retries before this status routing. Its Go zero value is sent on the wire. The wire parser recognizes it only inside an object-valued error when the envelope also supplies success:false. Missing, null, non-boolean or misplaced flags mean unspecified permission, not false. See Component Retry Policy on the Wire for non-Go tool integration. Transient 408/429/5xx preserve admitted parameters; non-transient 4xx correction requires eligibility and budget. 401/403/405 stop immediately. HITL-approved and exact-bound values are protected, and delegated orchestrator calls are single dispatch. See the executor routing contract.

HTTP Status Code Decision:
2xx (Success) → No retry needed
401/403/405   → NOT retryable
408/429      → Retry unchanged after bounded backoff
5xx (Server) → Retry with SAME payload (transient error)
Other 4xx    → Check ToolError.Retryable and correction gates

Retry Strategy:
┌─────────────────────────────────────────────────────────────────────┐
│ 5xx (Server Error) → Retry with SAME input (server might recover)   │
│ 4xx (Client Error) → Retry with CORRECTED input (AI fixes payload)  │
└─────────────────────────────────────────────────────────────────────┘

Agent Retry Configuration​

Before implementing retry logic, define these configuration types. They live in your agent code, not in core:

The manual loop below is a teaching example for agents that do not use the orchestration executor, not its current implementation or a second retry layer to wrap around it. It omits the executor's policy-presence, immutable-binding, HITL and dispatch-evidence protections. Framework-based orchestrator agents configure the existing executor instead of copying this simplified loop.

// RetryConfig controls how the agent handles failed tool calls
type RetryConfig struct {
    MaxRetries      int           // How many times to retry (default: 3)
    UseAI           bool          // Should AI analyze errors and suggest fixes?
    BackoffDuration time.Duration // How long to wait between retries
}

// ErrorContext bundles everything AI needs to analyze an error
type ErrorContext struct {
    HTTPStatus      int                    // e.g., 404
    OriginalRequest map[string]interface{} // The payload that failed
    ToolError       *core.ToolError        // Structured error from tool
    ToolName        string                 // e.g., "weather-tool"
    Capability      string                 // e.g., "current_weather"
    AttemptNumber   int                    // Which retry attempt this is
}

// Default configuration - sensible starting point
func DefaultRetryConfig() RetryConfig {
    return RetryConfig{
        MaxRetries:      3,
        UseAI:           true,
        BackoffDuration: 1 * time.Second,
    }
}

Implementing Intelligent Retry​

This is the heart of intelligent error handling. Here's how the retry loop works:

Step-by-step process:

  1. Make the HTTP call to the tool
  2. Check for network errors - If the call didn't even reach the tool, retry with the same payload
  3. Check for success (2xx) - If successful, we're done!
  4. Parse the error response - Extract the ToolError from the response body
  5. Route based on HTTP status:
    • 401/403 (Auth) → Stop immediately. Agent can't fix credentials.
    • 429 (Rate limit) → Wait the specified time, then retry with SAME payload
    • 5xx (Server error) → Wait briefly, then retry with SAME payload (server might recover)
    • 4xx (Client error) → Check if Retryable is true. If so, use AI to fix the payload.
  6. If AI correction works, retry with the corrected payload
  7. If all retries fail, return the error to the caller
func (a *Agent) callToolWithRetry(
    ctx context.Context,
    tool *core.ServiceInfo,
    capability *core.Capability,
    initialPayload map[string]interface{},
    config RetryConfig,
) (*ToolResult, error) {

    currentPayload := initialPayload
    var lastError *core.ToolError
    var lastHTTPStatus int

    for attempt := 0; attempt <= config.MaxRetries; attempt++ {

        // ══════════════════════════════════════════════════════════════
        // STEP 1: Make the HTTP call to the tool
        // ══════════════════════════════════════════════════════════════
        resp, body, err := a.callToolHTTP(ctx, tool, capability, currentPayload)

        // ══════════════════════════════════════════════════════════════
        // STEP 2: Handle network errors (couldn't reach the tool at all)
        // ══════════════════════════════════════════════════════════════
        if err != nil {
            if attempt < config.MaxRetries {
                time.Sleep(config.BackoffDuration)
                continue // Retry with same payload - network might recover
            }
            return nil, fmt.Errorf("network error after %d attempts: %w", attempt+1, err)
        }

        lastHTTPStatus = resp.StatusCode

        // ══════════════════════════════════════════════════════════════
        // STEP 3: Check for success - we're done!
        // ══════════════════════════════════════════════════════════════
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return a.parseSuccessResponse(body), nil
        }

        // ══════════════════════════════════════════════════════════════
        // STEP 4: Parse the error response to get ToolError details
        // ══════════════════════════════════════════════════════════════
        toolResp := a.parseToolResponse(body)
        if toolResp.Error != nil {
            lastError = toolResp.Error
        }

        // ══════════════════════════════════════════════════════════════
        // STEP 5: Route based on HTTP status code
        // ══════════════════════════════════════════════════════════════

        // 5a: Auth errors (401/403) - STOP! Agent can't fix credentials
        if resp.StatusCode == 401 || resp.StatusCode == 403 {
            return nil, fmt.Errorf("auth error: %s", lastError.Message)
        }

        // 5b: Rate limited (429) - Wait and retry with SAME payload
        if resp.StatusCode == 429 {
            retryAfter := a.parseRetryAfter(lastError) // Check Details["retry_after"]
            time.Sleep(retryAfter)
            continue
        }

        // 5c: Server error (5xx) - Retry with SAME payload (transient error)
        if resp.StatusCode >= 500 {
            if attempt < config.MaxRetries {
                time.Sleep(config.BackoffDuration)
                continue
            }
        }

        // 5d: Client error (4xx) - Check if AI can fix the payload
        if resp.StatusCode >= 400 && resp.StatusCode < 500 {
            // First check: Is this error marked as retryable?
            if lastError == nil || !lastError.Retryable {
                return nil, fmt.Errorf("client error: %s", lastError.Message)
            }

            // ══════════════════════════════════════════════════════════
            // STEP 6: Use AI to analyze the error and correct the payload
            // ══════════════════════════════════════════════════════════
            if config.UseAI && a.aiClient != nil && attempt < config.MaxRetries {
                corrected, err := a.aiCorrectPayload(ctx, ErrorContext{
                    HTTPStatus:      resp.StatusCode,
                    OriginalRequest: currentPayload,
                    ToolError:       lastError,
                    ToolName:        tool.Name,
                    Capability:      capability.Name,
                    AttemptNumber:   attempt + 1,
                })

                if err == nil && corrected != nil {
                    currentPayload = corrected // Use the AI-fixed payload
                    continue                   // Try again with corrected payload
                }
            }
            break // AI couldn't fix it, stop retrying
        }
    }

    // ══════════════════════════════════════════════════════════════════
    // STEP 7: All retries exhausted - return the error
    // ══════════════════════════════════════════════════════════════════
    return nil, fmt.Errorf("failed after %d attempts: %s", config.MaxRetries+1, lastError.Message)
}

AI Error Correction​

This is where the "intelligent" part of intelligent error handling comes in. When a tool returns a retryable error, we ask AI to:

  1. Analyze the error - Understand why the request failed
  2. Decide if it's fixable - Some errors (like auth failures) can't be fixed by changing input
  3. Generate a corrected payload - If fixable, create a new payload that might work

How it works:

  • We build a prompt that gives AI all the context: the error details, original payload, and hints from the tool
  • AI returns a structured JSON response with its analysis and corrected payload
  • We use low temperature (0.1) for deterministic, consistent results
  • If AI says it can't fix the error, we return nil to signal "give up"
func (a *Agent) aiCorrectPayload(ctx context.Context, errCtx ErrorContext) (map[string]interface{}, error) {

    // ══════════════════════════════════════════════════════════════════
    // Step 1: Build the prompt with all context AI needs
    // ══════════════════════════════════════════════════════════════════
    prompt := fmt.Sprintf(`You are an API error analyzer. A tool call failed and you need to fix it.

## Error Information
HTTP Status: %d
Tool: %s
Capability: %s
Error Code: %s
Error Category: %s
Error Message: %s
Error Details: %v

## Original Request Payload
%s

## Your Task
1. Analyze why this request failed
2. Determine if it can be fixed by modifying the input
3. If fixable, generate a corrected JSON payload

## Response Format
Return ONLY valid JSON:
{
  "can_fix": true/false,
  "analysis": "Brief explanation",
  "corrected_payload": { ... }  // Only if can_fix is true
}

## Examples
- "Flower Mound, TX" failed? Try "Flower Mound, Texas, US"
- "MSFT Inc" failed? Try just "MSFT"
- API key invalid? can_fix: false (can't fix credentials)`,
        errCtx.HTTPStatus,
        errCtx.ToolName,
        errCtx.Capability,
        errCtx.ToolError.Code,
        errCtx.ToolError.Category,
        errCtx.ToolError.Message,
        errCtx.ToolError.Details,
        formatJSON(errCtx.OriginalRequest),
    )

    // ══════════════════════════════════════════════════════════════════
    // Step 2: Call AI with low temperature for consistent results
    // ══════════════════════════════════════════════════════════════════
    response, err := a.aiClient.GenerateResponse(ctx, prompt, &core.AIOptions{
        Temperature: 0.1, // Low = more deterministic (we want consistent fixes)
        MaxTokens:   300, // Keep it short - we just need JSON output
    })
    if err != nil {
        return nil, err
    }

    // ══════════════════════════════════════════════════════════════════
    // Step 3: Parse AI's response into a structured result
    // ══════════════════════════════════════════════════════════════════
    var result struct {
        CanFix           bool                   `json:"can_fix"`
        Analysis         string                 `json:"analysis"`
        CorrectedPayload map[string]interface{} `json:"corrected_payload"`
    }

    if err := json.Unmarshal([]byte(response.Content), &result); err != nil {
        return nil, err // AI returned invalid JSON - can't fix
    }

    // ══════════════════════════════════════════════════════════════════
    // Step 4: Return corrected payload or nil if unfixable
    // ══════════════════════════════════════════════════════════════════
    if !result.CanFix {
        return nil, nil // Signal to caller: "AI says this can't be fixed"
    }

    return result.CorrectedPayload, nil
}

Part 4: Real-World Example Flow​

Now let's see how all the pieces fit together. We'll trace through a real scenario where a user asks for weather in a city that the API doesn't recognize at first, but succeeds after AI-powered correction.

The scenario: User asks for "weather in Flower Mound, TX" - a small city in Texas. The weather API doesn't recognize "TX" as a country code, but after AI analyzes the error and reformats the location, the second attempt succeeds.

Step 1: Agent Generates Initial Payload​

User request: "What's the weather in Flower Mound, TX?"

Agent (via AI) generates payload:
{
  "location": "Flower Mound, TX",
  "units": "metric"
}

Step 2: Tool Returns Structured Error​

Tool calls OpenWeatherMap API
API returns: 404 city not found

Tool returns HTTP 404:
{
  "success": false,
  "error": {
    "code": "LOCATION_NOT_FOUND",
    "message": "Location 'Flower Mound, TX' not found in weather database",
    "category": "NOT_FOUND",
    "retryable": true,
    "details": {
      "original_location": "Flower Mound, TX",
      "hint": "Try 'City, Country' format"
    }
  }
}

Step 3: Agent Decision Flow​

Agent receives HTTP 404 response

Decision tree:
├── HTTP 404 is a 4xx error
├── Check ToolError.Retryable → true
├── AI correction enabled → yes
└── Use AI to analyze and fix

Step 4: AI Analyzes and Corrects​

AI input: Error context with original payload and error details

AI output:
{
  "can_fix": true,
  "analysis": "TX is a US state abbreviation. OpenWeatherMap prefers full format.",
  "corrected_payload": {
    "location": "Flower Mound, Texas, US",
    "units": "metric"
  }
}

Step 5: Retry Succeeds​

Agent retries with corrected payload:
{
  "location": "Flower Mound, Texas, US",
  "units": "metric"
}

Tool returns HTTP 200:
{
  "success": true,
  "data": {
    "location": "Flower Mound",
    "temperature": 25.3,
    "condition": "clear sky",
    "source": "OpenWeatherMap API"
  }
}

Step 6: Agent Returns Result​

Agent logs: "Tool call succeeded after AI-assisted retry (attempt 2)"
User receives: Weather data for Flower Mound, Texas

Configuration Options​

Configure retry behavior via environment variables:

# Maximum retry attempts (default: 3)
export TRUVAG3_AGENT_MAX_RETRIES=3

# Enable AI-powered error correction (default: true)
export TRUVAG3_AGENT_USE_AI_CORRECTION=true

# Backoff duration between retries (default: 1000ms)
export TRUVAG3_AGENT_RETRY_BACKOFF_MS=1000

Best Practices​

For Tool Developers​

  1. Always use ToolResponse envelope

    // Always wrap responses
    response := core.ToolResponse{Success: true, Data: result}

  2. Never return mock data

    // BAD: Hiding the error
    if err != nil {
        return mockWeatherData() // Don't do this!
    }

    // GOOD: Report the error
    if err != nil {
        return classifyAndReturnError(err)
    }

  3. Include helpful details

    Details: map[string]string{
        "original_input": userInput,
        "hint":           "Try this format instead...",
        "api_error":      err.Error(), // For debugging
    }

  4. Set Retryable correctly

    // Retryable: true → Permit policy-eligible retries, including correction
    // Example: "City not found" - maybe different spelling works

    // Retryable: false → Stop all same-step retries, including transient resends
    // Example: "Invalid API key" - needs configuration change

For Agent Developers​

  1. Respect explicit refusal before status-based retry

    // An explicit retryable:false stops same-step retries.
    // Otherwise HTTP status distinguishes transient retries from correction.

  2. Don't retry auth errors

    if status == 401 || status == 403 {
        return err // Agent can't fix credentials
    }

  3. Use exponential backoff for rate limits

    if status == 429 {
        retryAfter := parseRetryAfter(toolErr) // From Details["retry_after"]
        time.Sleep(retryAfter)
    }

  4. Log retry attempts for debugging

    a.Logger.Info("Retry attempt", map[string]interface{}{
        "tool":    toolName,
        "attempt": attemptNum,
        "reason":  toolErr.Code,
    })

Quick Reference​

Error Category to HTTP Status​

CategoryHTTP StatusRetryable?
CategoryInputError400Maybe (with corrected input)
CategoryNotFound404Maybe (with corrected input)
CategoryAuthError401No
CategoryRateLimit429Yes (after waiting)
CategoryServiceError503Yes (transient)

Retry Decision Matrix​

Subject to budgets and the protection rules above; explicit false overrides every retry row, and missing policy does not allow unchanged non-transient 4xx.

HTTP StatusSame Payload?AI Correction?
2xxN/A (success)N/A
401/403/405Don't retryDon't retry
408/429Yes (after backoff)No
5xxYes (after backoff)No
Other non-transient 4xx + permitted correctionNoIf eligible
4xx + Not RetryableDon't retryDon't retry

Complete Tool Error Example​

&core.ToolError{
    Code:      "LOCATION_NOT_FOUND",
    Message:   "Location 'Flower Mound, TX' not found",
    Category:  core.CategoryNotFound,
    Retryable: true,
    Details: map[string]string{
        "original_location": "Flower Mound, TX",
        "hint":              "Try 'City, Country' format",
        "api_error":         "404 city not found",
    },
}

Benefits of This Approach​

  1. Clean Separation: Tools report, agents decide
  2. AI-Powered Recovery: Agents can fix errors tools can't anticipate
  3. Transparency: No mock data, clear error reporting
  4. Flexibility: Agents choose their own retry strategies
  5. Observability: Clear logs showing retry attempts and corrections

Trade-offs​

ConsiderationImpactMitigation
AI token cost~300 tokens per retry analysisCap retries at 3, skip AI for non-retryable errors
Latency~500ms per AI analysisUse low temperature for fast responses
ComplexityAgent retry logic is more complexWell-documented patterns, copy-paste examples

Orchestration Module: Multi-Layer Type Safety​

When using the orchestration module's AI-driven natural language mode, an additional layer of error handling addresses a common LLM limitation: type mismatches in generated parameters.

The Problem​

LLMs sometimes generate JSON parameters with incorrect types:

// LLM generates (WRONG):
{"lat": "35.6897", "lon": "139.6917"}

// Tool expects (CORRECT):
{"lat": 35.6897, "lon": 139.6917}

This causes Go's json.Unmarshal to fail with errors like:

json: cannot unmarshal string into Go struct field .lat of type float64

The Solution: Multi-Layer Defense​

The orchestration module implements four layers of defense:

Layer 1: Prompt Guidance​

The LLM planning prompt includes explicit type rules and examples. While helpful, LLMs don't always follow instructions perfectly.

Layer 2: Schema-Based Coercion​

Before calling a tool, the executor automatically coerces parameters to match the capability schema:

// Automatic coercion using capability schema
// Input:  {"lat": "35.6897", "lon": "139.6917", "count": "5"}
// Output: {"lat": 35.6897, "lon": 139.6917, "count": 5}

This layer is:

  • Deterministic: Uses strconv.ParseFloat, strconv.ParseInt, strconv.ParseBool
  • Schema-driven: Only coerces when the capability defines expected types
  • Safe: If coercion fails, the original value is preserved

Layer 3: Validation Feedback Retry​

For eligible, correction-permitted errors that slip through Layers 1 and 2, the orchestrator can follow the sequence below. The shipped currency-tool's "Amount must be greater than 0" response has retryable:false and stops instead; using it in this walkthrough requires an explicit, condition-specific retryable:true opt-in plus the usual executor correction gates.

  1. Detects the type-related or validation error
  2. Sends the error context back to the LLM
  3. Requests corrected parameters
  4. Retries with the corrected payload

Error patterns detected:

// Type errors
"cannot unmarshal string into"
"json: cannot unmarshal"
"type mismatch"

// Validation errors (business logic)
"must be greater than"
"must be positive"
"is required"
"cannot be empty"

Layer 4: Contextual Re-Resolution (Semantic Retry)​

When Layer 3's analysis concludes that an error cannot be fixed by adjusting parameters alone — but the source data needed to compute the correct value is present in the execution trajectory — Layer 4 takes over. It re-runs parameter resolution with the full prior step results in context, letting the LLM compute corrected parameters from upstream data rather than guessing.

Implemented in orchestration/contextual_re_resolver.go; driven by orchestration/error_analyzer.go decisions. See Error Handling Guide §6 "Contextual Re-Resolution (Layer 4)" for the full flow, including when Layer 4 fires versus when Layer 3 alone is sufficient.

Configuration​

config := orchestration.DefaultConfig()

// Layer 3 is enabled by default for maximum reliability
config.ExecutionOptions.ValidationFeedbackEnabled = true  // Default: true
config.ExecutionOptions.MaxValidationRetries = 2          // Default: 2

// For cost-sensitive deployments, disable Layer 3
// (Relies on Layers 1 & 2 only, ~95% success rate)
config.ExecutionOptions.ValidationFeedbackEnabled = false

Observability​

Distributed Tracing (Jaeger/Tempo):

  • type_coercion_applied span event when Layer 2 coerces parameters
  • validation_feedback_started span event when Layer 3 initiates correction
  • validation_feedback_success / validation_feedback_failed span events

Prometheus Metrics:

# Type coercion frequency by capability
sum(rate(orchestration_type_coercion_applied_total[5m])) by (capability)

# Validation feedback success rate
sum(rate(orchestration_validation_feedback_success_total[5m])) /
sum(rate(orchestration_validation_feedback_attempts_total[5m]))

How This Complements Tool-Level Error Handling​

LayerResponsibilityLocation
ToolReport structured errors with ToolErrorTool handlers
AgentRetry with AI-corrected payloadsAgent retry logic
OrchestratorType coercion + validation feedback + semantic retryOrchestration executor

The orchestration layer adds automatic type safety before requests reach tools, reducing the burden on both tools and agents. When combined with proper tool error reporting and agent retry logic, this creates a robust error handling system that achieves ~99% success rates in production.

For detailed implementation information, see the orchestration module documentation.

Orchestration Module: Layer 4 Semantic Retry​

When the multi-layer type safety system exhausts its options—when Layer 3 (Validation Feedback) determines "this error cannot be fixed with different parameters"—there's one more layer of defense: Semantic Retry.

The Problem Semantic Retry Solves​

Consider this hypothetical opt-in workflow. The shipped currency-tool refuses retries for amount validation, so this successful recovery requires a tool that explicitly permits correction for that condition and satisfies executor gates:

User: "Sell 100 Tesla shares and convert the proceeds to EUR"

Step 1 (stock-tool): Returns {symbol: "TSLA", price: 468.285}
Step 2 (currency-tool): Called with {amount: 0} ← MicroResolver couldn't compute this!

The currency tool returns 400: "amount must be greater than 0". At this point:

  • Layer 1 (Prompt Guidance): Already failed
  • Layer 2 (Schema Coercion): Can't help—0 is already a valid number
  • Layer 3 (Validation Feedback): Says "cannot fix—don't know what amount should be"

But wait—the data to compute the correct amount exists! The user said "100 shares" and step 1 returned price: 468.285. A human developer would instantly compute: 100 × 468.285 = 46828.5.

How Semantic Retry Works​

Semantic Retry (Layer 4) uses the full execution trajectory to compute corrected parameters:

┌─────────────────────────────────────────────────────────────┐
│ Layer 4: Contextual Re-Resolution                            │
│                                                              │
│   Input: Full execution context                              │
│   • User's original query (the intent)                       │
│   • Source data from dependent steps (what to compute from)  │
│   • Failed parameters (what went wrong)                      │
│   • Error response (the clue)                                │
│                                                              │
│   Output: Corrected parameters                               │
│   • should_retry: true                                       │
│   • corrected_parameters: {amount: 46828.5}                  │
│   • analysis: "Computed 100 × 468.285 = 46828.5"            │
└─────────────────────────────────────────────────────────────┘

When Semantic Retry Activates​

All positive cases below require an eligible non-transient 4xx, no explicit retry refusal, mutable inputs, a non-orchestrator capability, configured analyzer/resolver and remaining correction budget. Transport and transient failures do not enter parameter correction. Exact-bound values remain protected.

ConditionSemantic Retry?
Tool returns 400/404/409/422 + Layer 3 says "cannot fix"✅ Yes
Tool returns 500 (server error)❌ No (executor's unchanged-input backoff, subject to permission and budget)
Tool returns 401/403 (auth error)❌ No (not retryable)
Layer 3 successfully corrects parameters❌ No (already fixed)
Independent step (no dependencies) with error✅ Yes (uses user query + error context)
Independent step + EnableForIndependentSteps=false❌ No (disabled by config)

Note on Independent Steps: Steps without dependencies (first steps, parallel steps) now trigger Semantic Retry by default. The LLM receives the user's original query and error context to suggest corrections, even without source data from dependent steps.

Transient Error Handling (Transport Failures and HTTP 408/429/5xx)​

Transport failures and HTTP 408/429/5xx are classified before every semantic correction path. The executor retries the same admitted payload using its existing budget and backoff. The error analyzer makes no model call for them.

How the Orchestrator Handles This:

  1. Classify the failure — temporary availability errors retain exact inputs.
  2. Use semantic correction for input failures — a non-transient 4xx can receive authorized changed inputs, subject to existing binding/HITL guards.
  3. Tools express the right status — a fixable input typo uses 400/422, not 503. This deliberately retires the previous 503 semantic-correction exception.
  4. Explicit retry policy is authoritative - retryable:false stops same-step retries before analysis. Omission is distinct from false; non-transient 4xx requests are not resent unchanged when correction cannot proceed. A later planner decision is a new operation, not an override of that retry policy.

Decision Matrix:

This matrix applies only after the executor's retry-permission and terminal-status guards. Explicit retryable:false stops before the analyzer on any HTTP status; non-transient 4xx requests require corrected inputs before resend. A later planner operation has its own authorization and is not a same-step retry.

Failure / LLM ResultAction
Transport / HTTP 408, 429, 5xxBounded retry with unchanged input, no LLM
should_retry=true, suggested_changes non-emptyRetry with new parameters
Non-transient 4xx, empty changes / is_transient_error=trueStop unchanged resend; later planning may recover
should_retry=false, is_transient_error=falseDon't retry

This ensures that transient infrastructure errors don't fail permanently when a simple retry might succeed.

Configuration​

// Semantic retry is enabled by default
config := orchestration.DefaultConfig()
config.SemanticRetry.Enabled = true                   // Default: true
config.SemanticRetry.MaxAttempts = 2                  // Default: 2
config.SemanticRetry.EnableForIndependentSteps = true // Default: true

// Or via environment variables:
// TRUVAG3_SEMANTIC_RETRY_ENABLED=true
// TRUVAG3_SEMANTIC_RETRY_MAX_ATTEMPTS=2
// TRUVAG3_SEMANTIC_RETRY_INDEPENDENT_STEPS=true

Environment Variables:

VariableDefaultDescription
TRUVAG3_SEMANTIC_RETRY_ENABLEDtrueEnable Layer 4 semantic retry
TRUVAG3_SEMANTIC_RETRY_MAX_ATTEMPTS2Maximum semantic retry attempts per step
TRUVAG3_SEMANTIC_RETRY_INDEPENDENT_STEPStrueEnable semantic retry for steps without dependencies

Observability​

Semantic retry generates detailed telemetry:

Distributed Tracing (Jaeger/Tempo):

  • contextual_re_resolution.start - When semantic retry begins
  • contextual_re_resolution.complete - Result with should_retry, analysis, parameter count
  • independent_step attribute - Boolean indicating step had no dependencies

Prometheus Metrics:

# Semantic retry success rate
sum(rate(orchestration_semantic_retry_success_total[5m])) /
sum(rate(orchestration_semantic_retry_attempts_total[5m]))

# LLM latency for semantic retry
histogram_quantile(0.95, orchestration_semantic_retry_llm_latency_ms)

# Independent step semantic retries (new metric)
sum(rate(orchestration_semantic_retry_independent_step_total[5m])) by (capability)

LLM Debug Payload Store:

For deep debugging of error handling flows, enable the LLM Debug Store to capture complete prompts and responses at all stages. Unlike Jaeger spans which truncate large payloads, this stores the full content:

export TRUVAG3_LLM_DEBUG_ENABLED=true

This captures interactions at 6 recording sites including semantic_retry, allowing you to inspect the exact prompts sent to the LLM and the responses received during error recovery.

The Complete Error Handling Stack​

With Semantic Retry, the orchestration module provides a four-layer defense:

LayerStrategyWhen UsedCost
Layer 1Prompt GuidanceAlwaysFree
Layer 2Schema CoercionAlwaysFree
Layer 3Validation FeedbackWhen tool returns correctable error1 LLM call
Layer 4Semantic RetryWhen Layer 3 says "cannot fix"1 LLM call

This creates a robust system that handles the full spectrum of parameter errors—from simple type mismatches to complex computations that require understanding user intent.

Happy error handling!