Avoiding LLM provider lock-in: How to switch models without rewriting your app

Common methods of LLM provider lock-in

LLM provider lock-in extends beyond API integration. Replacing a provider may require code changes, prompt recalibration, and rebuilding the evaluation evidence needed to confirm that output quality still meets production requirements. These dependencies fall into three categories.

Code coupling: Applications that import a provider-specific SDK, send requests in its format, and parse its response objects need engineering changes at every affected call site when the provider changes.

Behavioral coupling: Prompts are gradually refined around a model's formatting, verbosity, refusal patterns, and tool-calling behavior. A new model may accept the same prompt while producing materially different results.

Data coupling: Production traces, labeled examples, and evaluation results often accumulate in the first observability or evaluation system a team adopts. Limited export reuse can leave the next provider decision without established baselines or regression cases.

Code coupling has the clearest technical solution because a compatible gateway can provide one interface across multiple providers. Behavioral and data coupling require evaluation infrastructure that remains usable when the underlying model changes.

Code lock-in and the OpenAI-compatible gateway fix

OpenAI-compatible endpoints provide a common request and response format across many model providers. An LLM gateway uses that interface to route requests from one client library to different providers, reducing the provider-specific code that must be maintained throughout an application.

The Braintrust AI Gateway exposes a single endpoint for models from OpenAI, Anthropic, Google, AWS, and other providers. It also supports calling one provider's models through another provider's SDK, allowing teams to standardize the client used across their codebase. Braintrust supports calling Claude with the OpenAI SDK, Gemini with the OpenAI SDK, OpenAI with the Anthropic SDK, and Gemini with the Anthropic SDK.

OpenRouter provides an OpenAI-compatible API for more than 400 models, along with centralized billing, provider routing, and automatic fallbacks. LiteLLM offers an open-source proxy and SDK covering more than 100 models, with load balancing, retries, fallbacks, and spend tracking. Both reduce the engineering work required to route requests across providers.

A shared API confirms that requests reach the new model successfully. Determining whether its outputs meet the application's existing quality requirements still requires evaluation.

Behavioral lock-in that gateways don't touch

Prompts are refined over time around a model's formatting, response length, refusal patterns, tool selection, function arguments, and handling of edge cases. Those expectations can remain hidden until the same prompt runs on a candidate model and produces different behavior.

Manual spot checks may miss regressions that appear only with long documents, unusual inputs, or multi-step tool use. Average output quality can also remain stable while a specific requirement, such as JSON validity or extraction accuracy, declines enough to affect production.

Before changing providers, run an experiment that evaluates the current and candidate models against the same dataset and scorers, and investigate every regression that crosses a release threshold. Behavioral portability is established when the candidate maintains the required quality across production examples and known edge cases.

Data lock-in in your logs and eval history

Production traces, labeled examples, and experiment results provide the historical evidence needed to compare models. When those records cannot be exported or reused independently of the original provider, a migration may require teams to rebuild test cases, quality baselines, and known failure sets before evaluating a replacement.

Braintrust datasets store inputs, expected outputs, metadata, and tags separately from the model configuration. The same versioned dataset can evaluate models from different providers without recreating its test cases. Braintrust also supports exporting logs, datasets, and experiment results through the interface, SDK, and API for use in external evaluation and analysis systems.

Provider-neutral test data supports both the current migration and future model reviews. Production failures collected today remain available as regression cases, while previous experiment results provide a baseline for deciding whether a newly released model is ready for the same workload.

How to switch LLM providers safely

Provider changes may be driven by cost, missing capabilities, service outages, or frequent model releases. Use the sequence below to validate a candidate model before moving production traffic.

Step 1. Route a test request to the candidate model

The Braintrust AI Gateway provides one endpoint for supported models across multiple providers. For compatible requests, changing the model identifier is enough to test a candidate while keeping the existing client and base URL.

const client = new OpenAI({
  baseURL: "https://gateway.braintrust.dev",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

// Call Anthropic's Claude using the OpenAI SDK
const response = await client.responses.create({
  model: "claude-sonnet-4-5",
  input: [{ role: "user", content: "Hello!" }],
});

console.log(response.output_text);

Step 2. Run the same evaluation suite against both models

Evaluate the current and candidate models using the same prompt version, dataset, and scorer definitions. Comparing models on shared test cases makes the model the primary variable behind any score difference.

Step 3. Review score changes on your own data

Public benchmarks can help create a shortlist, but the release decision should rely on production examples, known edge cases, and regression tests from your application. Review overall results alongside each scorer because a candidate may improve answer accuracy while reducing format compliance or tool-call reliability.

Step 4. Move only the traffic that passed

Route the use cases where the candidate meets the required thresholds, then monitor their production traces after the cutover. Keep workloads with unresolved regressions on the current model until the prompt, configuration, or candidate model has been corrected and reevaluated.

Record the final comparison and add any newly discovered edge cases to the regression dataset. The next provider review can then begin with an updated quality baseline.

Connecting model routing with quality evaluation

A gateway can report request success, latency, token usage, provider errors, and fallback events. Those signals measure routing health, the first of two layers teams comparing the best LLM gateways need to evaluate. The second layer is output quality, meaning whether the replacement model preserves extraction accuracy, formatting, tool selection, and other product requirements.

Braintrust's AI Gateway provides multi-provider access and can record requests as traces, while experiments compare models using shared datasets and scorers. When a candidate falls below a required threshold, the experiment results identify the affected criteria before production traffic moves.

Notion uses Braintrust across 70 engineers to run regression and frontier evaluations when new models become available, helping the team deploy suitable new models in under 24 hours.

Start free with Braintrust and run a side-by-side model experiment.

FAQs: Avoiding LLM provider lock-in in 2026

Is a gateway enough to avoid LLM provider lock-in?

A gateway covers inference calls supported by its shared API surface, while provider-specific services can preserve dependencies elsewhere. File APIs, vector databases, fine-tuning jobs, batch processing, prompt caching, multimodal formats, and proprietary tool features may require separate migration work. Audit every provider-specific API and stored asset before treating the application as portable.

What does it cost to switch LLM providers today?

The total cost may include gateway fees, duplicate inference while both models are evaluated, human review, prompt adaptation, unsupported API changes, and data export or retention. A small code change can still require meaningful evaluation and operations work. Estimate the migration using a representative dataset and include model usage, scorer calls, reviewer time, and post-cutover monitoring.

How do I know if a new model is a safe replacement?

Define acceptance thresholds before comparing models and evaluate critical traffic segments separately. A safe replacement must meet the required accuracy, format compliance, latency, cost, safety, and tool-use thresholds for every workload it will serve, then maintain those results during a limited production rollout. Run an experiment to inspect regressions by scorer, use case, language, document length, and other relevant segments.

Does self-hosting a gateway like LiteLLM avoid lock-in better than a hosted one?

Self-hosting provides direct control over credentials, network routing, upgrades, logs, and gateway availability, while placing infrastructure maintenance on the engineering team. Portability still depends on a common API surface, reusable evaluation data, and exportable production history. A hosted gateway can provide comparable provider flexibility when its supported interfaces and data controls meet the application's requirements.

What happens if my LLM provider goes down?

Prepare a provider fallback for each production use case before an outage occurs. Braintrust Gateway can retry the same model through another configured provider after availability failures, rate limits, or server errors. Cross-model fallback requires a mapped replacement model, and the replacement should pass the relevant evaluation thresholds before automatic routing is enabled. If no replacement meets the minimum requirements, return a controlled degraded experience or queue the request until service recovers.