A/B test models in production

Zain Hasan, Zarni Phyo, Nikitha Suryadevara, Ted Cui

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

A/B experiments allow you to split an endpoint's live traffic into fixed cohorts of one control and up to 20 variants, each with a percentage split of the traffic. This allows you to measure how a candidate model performs with real users at an exposure level you choose. Ramping up a variant can also be done with one call where you can promote the winner using a blue-green rollout. Deleting the experiment returns 100% of traffic to the control without the need to make any client-side or routing logic changes to unwind afterwards. Below we'll run an experiment on a live endpoint where we create at 95%/5%, ramp to 80%/20% and 50%/50%, then delete and check the observed traffic shares at every stage.

Implementing A/B testing for LLMs in production

Sooner or later every team wants to answer the same question: is the new model actually better for our users compared to the current model? Not better on a benchmark but rather better on retention, thumbs-up rate, task completion, whatever your product actually measures.

Shadow traffic can't answer that question. Shadowing tells you the candidate is operationally sound with respect to latency, errors, throughput, but its responses are discarded; no user ever acts on them. Quality questions need real exposure to end users where a cut of your users get model B, and you compare what happens.

Typically teams build this themselves in the application layer using some combination of: