LLM Model Migration Runbook: Upgrade Without Surprising Production

Changing an LLM identifier looks like a configuration edit. In production it is a system migration. The replacement may follow instructions differently, call tools with new argument patterns, use more output tokens, refuse different requests, expose new latency tails, or turn a formerly stable parser into a frequent exception.

A good migration runbook treats behavior as an interface. It defines what must remain true, where differences are acceptable, how evidence will be collected, and exactly when traffic returns to the previous model.

Freeze the Current Contract

Before testing the new model, document the current serving path. Capture the model and snapshot, system prompt version, tool schemas, sampling controls, token limits, retry policy, safety filters, response parser, and fallback chain.

Then define service-level expectations for the tasks that matter. Examples include:

• valid structured output rate;
• tool selection and argument accuracy;
• grounded-answer score and citation correctness;
• task completion rate by workflow;
• p50 and p95 latency;
• input and output tokens per successful task;
• escalation, refusal, and unsafe-output rates;
• cost per accepted result.

Avoid a single average quality score. A model can improve general summaries while regressing the one refund workflow that moves money. Segment evaluation by task, language, customer tier, risk class, and input length.

Record the rollback version as an immutable deployment target. “We can switch back” is not a plan if the old prompt, adapter, or routing configuration cannot be restored together.

Replay Representative Production Cases

Build an offline set from privacy-reviewed production traces, known incidents, edge cases, and synthetic boundary tests. Preserve the complete context the model actually sees, including tool descriptions and retrieved evidence. Remove secrets and direct identifiers.

Run old and new models against the same cases. Deterministic checks should validate schemas, required fields, citations, tool permissions, and forbidden actions. Human or model-assisted graders can compare usefulness, but calibrate them against examples and inspect disagreements.

For agent workflows, replay more than the first turn. A new model may choose a valid tool initially but fail after a timeout, return an incompatible follow-up argument, or loop when a tool result is empty. Use recorded tool responses to simulate complete trajectories without touching live systems.

Analyze wins and losses separately. Aggregate improvement does not excuse a severe regression. Create an exception list with an owner, mitigation, and decision for every critical loss.

Test the Whole Serving Stack

Models interact with infrastructure. Verify tokenization assumptions, context truncation, streaming events, finish reasons, JSON or schema modes, tool-call serialization, retry behavior, and rate-limit handling. Confirm that observability captures the new model identifier and usage fields.

Load-test realistic concurrency and prompt sizes. Median latency can improve while p95 worsens. A longer reasoning path may increase queue pressure and timeouts even when individual responses look better.

Recalculate economics using successful tasks, not API calls. If the new model needs fewer retries or shorter prompts, a higher token price may still reduce total cost. If it produces longer outputs that users discard, headline pricing hides the regression.

Security tests should include prompt injection, tool overreach, sensitive-data handling, and refusal consistency. Re-run them with the exact production system prompt and tools; isolated chat tests are insufficient.

Use Shadow, Canary, and Gradual Rollout

Shadow traffic sends a copy of eligible requests to the candidate without using its output. This reveals latency, cost, and behavioral differences on live distributions. Strip side-effecting tools or replace them with simulators. Never let the shadow path send messages, create tickets, or transact.

Next, canary a small, clearly identified share of low-risk traffic. Compare it with a concurrent control group. Increase traffic only after a minimum sample and observation window. Suggested gates might move from internal users to 1%, 5%, 25%, 50%, and then full traffic, but the right ladder depends on volume and risk.

Hold back high-risk workflows until their task-specific evidence is strong. Feature flags should support routing by workflow and customer, not just one global percentage.

Define automatic stops for severe policy violations, schema failure spikes, tool errors, or latency saturation. Define human review gates for subtler quality or support-ticket changes.

Roll Back as a Complete Unit

Rollback triggers must be measurable: a threshold, a time window, and an owner. Restore the model with its compatible prompt, adapters, tool schemas, and parsing rules. Changing only the model can create a hybrid configuration that never passed evaluation.

After rollback, preserve candidate traces and deployment metadata. Classify whether the cause belongs to model behavior, prompting, tool integration, traffic mix, or infrastructure. Repair the evaluation set if the incident revealed a case it did not cover.

A migration is complete only after the new model survives the planned observation period, dashboards and cost forecasts are updated, fallbacks are tested, and the old path has a deliberate retirement date. Model upgrades should feel boring in production. The runbook is what makes that possible.