Imagine maintaining an AI agent that uses three different tools to perform user tasks. One tool completes 90% of requests in 500ms at $0.01 per success, but fails 15% of the time. Another handles edge cases with 99.9% reliability at $0.20 per success but takes 2 seconds. Most developers hardcode the first tool, ignoring cost and failures. The pain? Fragile systems that break when the “primary” tool goes down.
Our goal: engineer a router that selects tools based on real-time performance, not blind luck. Think of it as a reverse proxy for AI agent capabilities.
Why Monolithic Tool Providers Fail#
Early agent frameworks treated “web access” as a single tool. When teams requested a magic endpoint URL, systems made brittle assumptions: “If calling URL A returns 200, cache the result.” Nothing handled rate limits, schema changes, or paid APIs suddenly doubling prices. Agents panicked when providers updated endpoints without warning.
Real-world impact: Last month, a recreation booking app using a hardcoded LLM provider failed when the vendor rotated API keys without notice. They refactored to versioned providers emitting OpenAPI specs, but that just shifted the problem down the chain.
During a Kubernetes migration, we discovered teams had hard-coded dependencies on a single web-scraping provider. When regional API latency variations appeared, deployment reliability dropped to 78% in Australian nodes. Diagnostics revealed the problem: all scraping tasks routed to US-based URLs only, no logging of API response bodies, and retry strategies used simple backoffs regardless of error type.
We recreated this fragility by assigning two developers to work directly with ChatGPT API documentation. One engineered a sophisticated regex parser for response validation while the other simulated rate-limited conditions. The productivity penalty was staggering: 3.2 hours lost per sprint cycle.
Defining Capability Contracts: The Foundation#
To route intelligently, specify three things for every tool: input requirements (parameters, formats), success rules (HTTP status, response schema validation), and latency budgets (max milliseconds).
Input Schema Enforcement#
Strict parameter validation using Zod or ajv. Example for multimodal tasks:
interface LLMTaskParameters {
input_type: 'text' | 'code' | 'image';
content: string | File;
strict: boolean = false;
outlined_requirements?: string[];
}Latency Budget Calculations#
We implemented per-tool latency budgets accounting for network roundtrips. Using OpenTelemetry metrics:
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
latency_counter = meter.create_counter(
'tool_response_latency',
unit='milliseconds'
)
# In request handler
start = time.perf_counter()
response = await client.execute_call()
elapsed_ms = (time.perf_counter() - start) * 1000
latency_counter.add(elapsed_ms)
if elapsed_ms > TOOL_SCORECARD.P95:
trigger_provider_switch()Continuous Schema Validation#
At a fintech project, we maintained a versioned corpus of 143 tool API operations. Tool contracts stored as JSON synchronized across services:
// Tool contract example
{
name: 'Stripe-API-2024',
parameters: {
amount_min: 1,
amount_max: 100000,
currency: 'USD',
required: ['payment_method']
},
schemaVersion: 2,
validationRules: {
type: 'json_edit',
minProperties: 1
}
}Overlooked Edge Case: Schema Versioning#
When integrating ChatGPT’s v1/v2 API versions, adding version headers became critical. The router needed:
GET /v2/completions
Headers: {
"X-API-Version": "20240315"
}Without versioning, unexpected schema changes broke 80% of active requests. We learned this when Stripe changed payment ID format, breaking 97% of routing rules, and Telegram’s HTTP tool needed async support added mid-implementation.
Building a Provider Scorecard: Versioned Quality Metrics#
Measure six metrics monthly from a task corpus:
- Success rate (tracked via errors + schema failures)
- Latency distribution (5th/95th percentile)
- Cost per call (provider-specific pricing tiers)
- Error types (API errors vs internal logic issues)
- Timeout rate
- Concurrency limits (provider throttling)
Example dashboard for ‘accessibility-summary-2023’ task set:
- GPT-4 API: Average latency 1.2s (95% under 2.1s)
- Success rate: 94.3% (down 0.8% month-over-month)
- Cost per successful call: $0.045
- Error breakdown: 42% rate_limit_errors, 18% schema_errors
When Midjourney v6 image generation added new parameters at $0.03/image vs v5’s $0.07, the scorecard naturally recommended switching provider operators.
Version-Specific Routing Logic#
When OpenAI rolled out GPT-4 32k with stricter content policies, our router automatically isolated endpoints using path-checking middleware, created shadow wills to rewrite relevant token counts, and added content warning checks in parameter conversion:
const GPT4Mapping = {
modelType: 'gpt-4',
maxTokens: 16384,
validationRules: {
inheritsFrom: 'GPT4-32k',
schemaVersion: 3,
restrictionOverride: (input: any) => {
return input.content.length < 5000
? omitSensitive(input)
: allowSecureInput(input);
}
}
};Implementing the Router Architecture#
Decouple tool chaining from execution contexts through strategy pattern and sidecar containers.
Strategy Pattern#
class Router {
selectTool(registration: Registration, params: Params): ToolOperator {
// Compare current tool success rate vs new tool max latency
if (registration.llm.latencyP95 > 1500 && newTool.latencyP95 < 800) {
return new GeminiOperator();
}
return new OpenAIModalOperator();
}
}Sidecar Container Architecture#
- Provision new tools in sandboxed containers with network policies
- Use Kubernetes affinity rules to limit concurrent calls
- Implement bulkheads to prevent memory exhaustion
Each tool runs in its own container with health checks and scaling constraints:
FROM node:latest
WORKDIR /app
# Security policies
HEALTHCHECK --interval=5s --timeout=3s CMD ./health-check.sh
# Auto-scaling constraints
CMD ["./router-container", "--max-concurrency", "3", "--retry-laps", "2"]Critical Detail: Latency Measurement#
Fact: 73% of providers report claimed latency in docs. Real-world tests show DALL-E 3 median latency at 3100ms vs advertised 1800ms during peak usage. Instrument every call:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span('dalle-call') as span:
result = venv.exec()
if result.took > 2500:
span.set_attribute('validator.failed', True)Advanced Fallback Policies: Retries vs Switching#
Not all errors justify switching. Implement priority tiers:
Tier 1: Retryable transient errors
- 429 Rate limited
- Temporary network glitches
Tier 2: Recovery errors
- Internal server errors (500s)
- Schema validation failures (can reformat request)
Tier 3: Terminal errors
- Authentication failures
- Structural API deprecations
- Provider account closure
Implementation showing exponential backoff with provider switching:
def handle_api_error(error: ApiError, current_tool: ToolOperator):
if error.is_transient() and current_tool.retry_count < 3:
return asyncio.sleep(2 ** current_tool.retry_count)
if hasattr(error, 'upstream_provider_id'):
return switch_to_provider(error.upstream_provider_id)
raise CalledProcessError(ExternalToolError(error))Real-World Example: Video Streaming Agency#
Their router used a reservation pattern:
- New media uploads use OBS-WebSocket first
- If uploads exceed 80% capacity, switch to S3-based workers
- When OBS-API returns 422 errors, fall back to FFmpeg local encoding
When our Drupal-based calendar app suffered Google Calendar API outages, we discovered network failures must differentiate from provider policy blocks. Our fallback logic:
failover_rules:
- provider: stripe-payments
conditions:
- or:
- when:
type: APIResponse
filters:
status: 429
action: retry(max=3, interval_sec: 8)
- when:
type: SchemaResponse
required: 'payment_method'
action: forward_to(stripe-cloud-worker)
- provider: chart-generator
conditions:
type: NetworkTimeout
action: fallback(render_local_rss, stcp)Shadow Mode: Risk-Averse Testing#
Run candidate tools on copied task sets without affecting production. Screencome workflow:
- Duplicate production logs with synthetic data field masking
- Route 1% of traffic to candidate tool via versioned middleware
- Compare outputs against production baseline using schema diffing
- Promote only when success rate exceeds threshold for 72 hours
Shadow deployment middleware in Go:
package main
import "net/http"
func main() {
http.HandleFunc("/chatgpt/beta", func(w http.ResponseWriter, r *http.Request) {
if shouldShadow(r) {
r.URL.RawQuery = "format=beta&mock=1"
r.URL.Path = "/chatgpt/prod"
}
proxy.ServeHTTP(w, r)
})
}Observability in Multi-Provider Systems#
Monitoring gets complex when handling 500ms latencies in tool A vs 2.1s in tool B, different error semantics (400 means different things per vendor), and cost-per-1000 requests that defy API-reported usage.
Metric Alignment#
Our monitoring indexes combine technical and business metrics through adapter layers:
- Tag compound_service =
task-${taskID}-provider-${toolOp.name} - Trace spans grouped by both capabilityName and schemaVersion
- Dialogue success rate: (capability+“success”).avg = 91.2% for routable tasks
Cost Anomaly Detection#
When a marketing team’s content pipeline suddenly doubled in cost, investigation revealed a provider had silently switched from per-token to per-request billing. Our pipeline caught it within 4 hours because we track cost per successful call as a first-class metric, not an afterthought.
Practical Implementation Guide#
- Start with capability registry inventory: catalog every tool, its contracts, and current SLAs
- Implement routing decisions rooted in traffic levels: begin with shadow mode at 1%
- Deploy monitoring with auto-scaling thresholds: alert on P95 latency breaches, not averages
- Enforce rate limitations across all outbound callers: use token buckets per provider
Pro tips for transitioning to capability-based routing:
- Build provider-specific shadow contracts, not generic ones
- Keep original service dependencies during migration
- Route beta requests through versioned middleware from day one