Routing compatibility

This reference separates Gateway configuration from client-owned request and retry behavior. To inspect captured recommendations and applied decisions, use Understand routing decisions.

Egress resolves an explicit model through the organization's pricing sheet, then forwards through an enabled provider. Auto Router can replace a model on qualified client paths using managed configuration; these are separate decisions.

Provider selection

• The organization manages provider credentials and the model-to-provider mapping.
• A pricing row must name a provider that Egress can route, and that provider must have an enabled configuration.
• Public catalog membership alone does not establish gateway availability.

• Configure provider access through your organization. Request-level provider allow lists, exclude lists, ordering and capability filters do not control Gateway routing.
• A JSON routing object does not constrain provider selection and can be forwarded to the upstream provider as an unknown field.
• Do not use it to enforce a data-handling rule.

Native request example

Use the client-native body with an explicit organization-enabled model:

{
  "model": "<organization-enabled-model>",
  "messages": [{ "role": "user", "content": "Draft the incident update." }]
}

Provider and model fallbacks

• Egress does not accept an ordered provider or fallback-model list for automatic retries.
• Smart Router's fallback to the original request means a replacement was not applied. It is not a retry after a provider failure.
• If your application needs retries, implement and bound them in the caller.
• Distinguish each provider fallback from a model fallback in your own execution record.

Retry only errors that your application and provider allow. Account for SDK retries when setting the total attempt budget. A failed or timed-out request can still incur provider charges.

Performance preferences and caching

• Managed Auto Router maps complexity tiers to configured targets.
• It does not accept request-level quality, cost, latency, throughput, or prompt-caching objectives.
• Use Models to compare base token prices, and your own workload measurements to choose targets.
• The public price catalog contains no quality scores or measured latency. Routing compatibility shows a caller-owned selection recipe.
• Prompt caching follows the native provider's supported request fields and behavior.
• A Smart Router decision cache stores routing decisions. It is separate from provider prompt caching and does not prove a billed cache hit.

Policy constraints

• Enforce access and data-handling requirements with the organization's provider and security configuration.
• Your organization can restrict replacement targets, but a rejected replacement can preserve the original request.
• It does not prevent an otherwise admitted original model from being forwarded.
• If an application imposes a stricter candidate set and no eligible route remains, stop before sending a request.
• Use access rules to refuse disallowed models or providers; an unsupported request field does not provide that protection.

Application-owned recipes

• These recipes run in your application using ordinary Egress requests.
• Each recipe builds ordinary request bodies or calls an already configured OpenAI client. Your application selects the models and combines the results; these recipes do not enable automatic Gateway routing.
• Configure client using OpenAI Python setup.
• Choose organization-enabled models that accept Chat Completions. Each request still passes through the gateway's model and provider checks.

Request builder

Build independent request bodies before executing them. Preserve the prompt, conversation, and compatible request settings when changing the model.

Build request bodies

from copy import deepcopy
 
 
def build_requests(request, models):
    if not models:
        raise ValueError("Choose at least one enabled model")
    return [dict(deepcopy(request), model=model) for model in models]

Call build_requests(request, models) with your native request and approved model IDs. No provider call occurs during construction. Inspect the bodies, then execute each with client.chat.completions.create(**body).

Read execution responses

The caller executes each body and receives an ordinary Chat Completions response. Keep each response's model, choices, and usage alongside its request and any execution error. The gateway does not return a combined requests or cost_latency envelope.

Quality and cost

• Choose one request using your own measured quality and estimated cost.
• Each candidate row below is application data with model, quality, and estimated_cost_usd.
• Use enabled models, finite numeric values, one named evaluation and score scale, and cost estimates for the same workload.
• The public pricing catalog does not supply quality scores.

Select a request

from copy import deepcopy
 
 
def choose_request(request, candidates, minimum_quality):
    eligible = [row for row in candidates
                if row["quality"] >= minimum_quality]
    if not eligible:
        raise ValueError("No candidate meets the quality threshold")
    chosen = min(eligible, key=lambda row: (
        row["estimated_cost_usd"], row["model"]
    ))
    return dict(deepcopy(request), model=chosen["model"])

This application rule chooses the lowest estimated cost among candidates meeting the threshold; model ID breaks equal-cost ties. It fails without sending a request when the candidate set is empty. It does not ask Auto Router to optimize a quality score.

Execute the selected request

Pass the returned body to client.chat.completions.create(**body). Record the evaluation source and workload estimate separately from the provider's actual usage. A cost estimate is not a billed result.

Multi-model deliberation

• The application can call a small panel, ask an analyst to compare the replies, and request a final answer.
• This sequential example permits one to three panel models, disables SDK retries, and sets a per-call timeout.
• It accepts partial panel success, refuses an entirely failed panel, and propagates analyst or final-answer failures.

Run the model roles

from copy import deepcopy
 
 
def deliberate(client, messages, panel_models, analyst_model, final_model):
    if not 1 <= len(panel_models) <= 3:
        raise ValueError("Choose one to three panel models")
    bounded = client.with_options(timeout=30.0, max_retries=0)
    panel = []
    for model in panel_models:
        try:
            response = bounded.chat.completions.create(
                model=model, messages=deepcopy(messages), max_tokens=512
            )
            text = response.choices[0].message.content
            if not text:
                raise ValueError("Panel returned no text")
            panel.append({"model": model, "response": response, "text": text})
        except Exception as error:
            panel.append({"model": model, "error": type(error).__name__})
 
    replies = [entry for entry in panel if "text" in entry]
    if not replies:
        raise RuntimeError("No panel response available")
    comparison = "Compare these candidate answers:\n" + "\n\n".join(
        entry["model"] + ": " + entry["text"] for entry in replies
    )
    analysis = bounded.chat.completions.create(
        model=analyst_model,
        messages=deepcopy(messages) + [{"role": "user", "content": comparison}],
        max_tokens=512,
    )
    analysis_text = analysis.choices[0].message.content
    if not analysis_text:
        raise RuntimeError("Analyst returned no text")
    final = bounded.chat.completions.create(
        model=final_model,
        messages=deepcopy(messages) + [{
            "role": "user",
            "content": "Use this comparison to answer the original task:\n" + analysis_text,
        }],
        max_tokens=512,
    )
    if not final.choices[0].message.content:
        raise RuntimeError("Final model returned no text")
    return {"panel": panel, "analyst": analysis, "final": final}

Call deliberate(client, messages, panel_models, analyst_model, final_model) with your configured client, original conversation, and compatible model IDs. The function sends a separate model request for each role.

Inspect the application result

Read result["final"].choices[0].message.content for the final text. Keep the panel's individual responses or errors and the analyst response for review. The dictionary groups the responses returned by this recipe. Tool calls are not executed by this text-only recipe.

• Up to five model calls compound latency and cost.
• Each successful response carries its own usage.
• Failed calls may incur charges that are not represented by returned usage.
• Add an overall deadline or parallel execution only when the application also defines cancellation and partial-result handling.

Choose the execution model

Auto Router selects one target on a qualified request path. These recipes use explicit models and application-owned execution.

Continue with routing

• Understand routing decisions: inspect captured recommendations, applied decisions, and response evidence.
• Auto Router: use managed single-target routing on qualified paths.
• Connect a client: configure a client and make the first request.