Migrating an LLM application to a new model or API version is more than changing one identifier. This checklist shows how to inventory dependencies, compare prompts and outputs, estimate cost and performance changes, test compatibility, define rollback criteria, and obtain production sign-off before cutover.
Overview
An LLM model migration can change behavior at several layers: request formatting, authentication, token accounting, context limits, tool calling, structured outputs, safety handling, latency, and failure modes. A model that appears compatible in a small manual test may still break an extraction pipeline, alter a customer-facing tone, or produce output that downstream code cannot parse.
Treat the migration as a controlled software change rather than a prompt-only exercise. The safest process has five stages:
- Inventory: identify every model, API version, SDK, prompt, tool, parser, evaluator, and operational dependency.
- Compare: document the old and proposed configurations in one test matrix.
- Evaluate: run representative cases, including failures and adversarial inputs, against both versions.
- Release gradually: use a feature flag, shadow traffic, limited rollout, or another reversible deployment method.
- Review: compare quality, cost, latency, reliability, and safety before removing the rollback path.
This workflow complements a broader prompt QA and regression testing playbook. It is especially useful when a provider deprecates a model, changes an API version, updates an SDK, or introduces a new model family.
How to estimate
Before testing, establish a baseline from the current production configuration. Use a consistent sample of requests rather than relying on a single demonstration. Include common cases, long inputs, empty or malformed inputs, multilingual examples if relevant, tool-use requests, and cases that previously caused failures.
1. Estimate request volume and token usage
Use the following planning formula:
Monthly input tokens = requests per month × average input tokens per request
Monthly output tokens = requests per month × average output tokens per request
If the provider prices input and output separately, estimate them independently. Add retry traffic, background jobs, evaluation runs, and fallback-model traffic rather than treating them as zero. The resulting estimate should be a range, not a false-precision number:
Expected monthly usage = base traffic × (1 + retry rate + fallback rate + evaluation allowance)
Insert current provider pricing only after confirming the applicable model, API mode, cached-input rules, batch options, and billing unit in the provider’s documentation. Pricing is an input that should be refreshed whenever the provider changes its commercial terms.
2. Estimate operational impact
Compare the old and proposed versions using the same measures:
- Quality: task success, factuality, relevance, schema validity, and human review outcomes.
- Reliability: timeout rate, provider errors, invalid responses, tool-call failures, and retry frequency.
- Performance: time to first token, total response time, and throughput under representative load.
- Cost: input tokens, output tokens, retries, tool calls, and any separate embedding or reranking work.
- Risk: privacy exposure, prompt-injection behavior, unsafe completion handling, and changes to logging or retention.
A useful migration decision is not simply “the new model scored higher.” Record whether it meets the application’s minimum requirements at an acceptable cost and operational risk.
3. Create a migration scorecard
| Area | Current version | Proposed version | Decision rule |
|---|---|---|---|
| Prompt behavior | Baseline outputs | Candidate outputs | No unacceptable regression on priority tasks |
| Context limit | Observed maximum | Documented and tested maximum | Supports production input sizes with margin |
| Structured output | Parse success rate | Parse success rate | Meets the application’s schema requirement |
| Tool support | Supported calls and arguments | Supported calls and arguments | All critical tools pass contract tests |
| Latency | Measured distribution | Measured distribution | Within the product’s response-time budget |
| Cost | Usage-based estimate | Usage-based estimate | Fits the approved operating range |
Inputs and assumptions
Build an inventory before changing code. The following template can be copied into a spreadsheet or issue:
| Dependency | What to record | Owner |
|---|---|---|
| Model and API | Provider, model identifier, API version, endpoint, region, and fallback | Platform owner |
| SDK and infrastructure | SDK version, environment variables, request middleware, timeouts, and retry policy | Application owner |
| Prompts | System instructions, user templates, few-shot examples, variables, and version history | Prompt owner |
| Output handling | JSON schema, parser, validators, truncation rules, and repair logic | Application owner |
| Tools and retrieval | Function schemas, permissions, RAG settings, rerankers, and citation handling | Integration owner |
| Operations | Logging, monitoring, alerts, dashboards, budgets, and incident procedures | SRE or operations |
Record assumptions explicitly. For example: average input length is based on the last review period; output limits are enforced by the application; tool results are counted in the context window; and evaluation data excludes confidential production content unless it has been approved for testing. For privacy-sensitive systems, review the AI application data privacy checklist before copying production requests into an evaluation set.
Compatibility questions to answer
- Does the new endpoint accept the same message roles, parameters, headers, and authentication method?
- Are temperature, sampling, token-limit, stop-sequence, or response-format parameters named and interpreted the same way?
- Does the context limit include system messages, tool definitions, retrieved documents, and tool results?
- Are structured outputs enforced, merely encouraged, or unsupported?
- Are tool names, argument schemas, parallel calls, and forced tool selection supported?
- Do error codes, rate-limit headers, streaming events, and usage fields remain compatible?
- Does the SDK silently transform requests or default to a different API version?
Do not assume that identical prompts produce interchangeable outputs. A model upgrade can alter formatting, refusal patterns, verbosity, tool selection, or sensitivity to prompt order. Keep the prompt version fixed during the first comparison so that model behavior is isolated from prompt edits.
Worked examples
Example 1: Structured extraction service
Suppose an application extracts fields from support messages and sends the result to a database. The migration test should use a fixed set of representative messages and compare:
- valid JSON or schema-conforming output;
- correct handling of missing fields;
- consistent data types and date formats;
- behavior when the input is ambiguous or contains extra instructions;
- parser errors, retries, and repair attempts.
Set a release gate such as: no critical field may regress, invalid outputs must remain below the application’s accepted threshold, and every database write must pass validation. The exact thresholds should come from the product’s risk tolerance and existing baseline, not from a generic benchmark.
For extraction-specific prompt patterns, see prompt templates for data extraction. If the input contains untrusted text, also test against instruction-confusion and prompt-injection cases using the prompt injection defense guide.
Example 2: Retrieval-augmented question answering
For a RAG application, model-only testing is insufficient. Keep the retrieved passages and retrieval settings stable, then compare answer quality, citation selection, refusal behavior when evidence is missing, and sensitivity to long context. Test questions with one clear source, multiple conflicting sources, no relevant source, and an irrelevant retrieved passage.
Track token usage for both retrieved content and generated answers. A candidate that appears cheaper per response may create higher total usage if it requires longer prompts, more retries, or additional verification calls. A practical migration outcome could be “approve for low-risk questions, retain the previous model for regulated workflows,” provided that routing and rollback are implemented explicitly.
Reusable side-by-side test case
Case ID: EX-014
Task: Extract customer priority and reason
Input: [sanitized representative message]
System prompt version: system-v12
Tools: none
Expected constraints: valid schema; priority in allowed enum
Old response: [captured output]
New response: [captured output]
Checks: schema / field accuracy / refusal / latency / tokens
Result: pass | fail | review
Notes: [unexpected behavior and follow-up]
Rollback and sign-off template
Rollback if: a critical workflow fails, schema errors exceed the agreed limit, latency breaches the service budget, cost exceeds the approved range, or a safety or privacy issue cannot be mitigated quickly.
Sign-off requires: application owner approval, evaluation results attached, provider and SDK changes documented, monitoring ready, support staff briefed, and a tested rollback command or configuration switch. Keep the old model configuration available until the observation period is complete.
When to recalculate
Revisit the migration estimate and test suite whenever an input changes. At minimum, recalculate after:
- a provider changes pricing, context limits, model availability, or API behavior;
- a model or API version is deprecated, retired, or replaced;
- the application changes its prompt, output schema, tools, retrieval settings, or fallback policy;
- request volume, average input length, output length, or retry behavior changes materially;
- latency, error rates, quality scores, or support incidents show a new pattern;
- new languages, user groups, document types, or sensitive workflows are added.
Schedule a quarterly documentation review even when no migration is planned. Refresh provider lifecycle notes, SDK versions, pricing inputs, capability comparisons, and known compatibility risks. Keep dated snapshots of the scorecard so that a later migration can be compared with the same baseline.
Before production cutover, complete this final sequence: freeze the test set, run old and new configurations side by side, review failures by severity, update the cost estimate, confirm monitoring, perform a controlled rollout, and record the rollback decision. If the result is inconclusive, delay the cutover or limit the new model to a low-risk route. A reversible migration with clear evidence is safer than a fast replacement based on a handful of impressive examples.
For broader quality measurement, use the principles in LLM evaluation metrics, and for usage controls review the LLM cost optimization guide.