Your team improved the recommendation model's offline accuracy from 78% to 85%. Impressive on paper. You deployed it to production expecting a corresponding lift in click-through rates. Instead, CTR dropped 12%. The new model was more accurate on the test set but worse at the business metric that matters. The model optimized for precision at the expense of diversity — recommending the single most likely item instead of a range of relevant options. Offline metrics did not capture what users actually wanted.
A/B testing AI models is the practice of deploying competing model versions to different segments of production traffic and measuring their impact on business metrics. It bridges the gap between offline evaluation (how the model performs on test data) and real-world impact (how the model affects business outcomes). For AI agencies delivering production systems, A/B testing is the only reliable way to validate that model changes actually improve what clients care about.
Why Offline Metrics Are Insufficient
Proxy Metric Gaps
Offline metrics — accuracy, precision, recall, F1, AUC — are proxies for business value. They measure model quality but not business impact. A model with higher accuracy may not produce higher revenue, lower costs, or better user experiences.
Distribution Differences
Test datasets are static snapshots. Production data evolves continuously. A model that performs well on a historical test set may perform differently on current production data due to distribution shifts, seasonal patterns, and user behavior changes.
System Effects
Models operate within larger systems. A recommendation model's impact depends on the UI that displays recommendations, the timing of recommendations, and the user's context. Offline evaluation cannot capture these system interactions.
A/B Testing Framework
Experiment Design
Hypothesis: Define a clear, testable hypothesis. "We believe the new model will increase click-through rate by at least 5% compared to the current model." Without a clear hypothesis, you cannot design an appropriate experiment or interpret the results.
Primary metric: Choose one primary metric that determines the experiment's outcome. Secondary metrics provide additional context but should not override the primary metric decision.
Minimum detectable effect (MDE): The smallest improvement that would be meaningful enough to justify deploying the new model. If a 1% CTR improvement would not justify the deployment effort, set MDE at 3-5%.
Sample size: Calculate the required sample size to detect your MDE with statistical significance (typically 95% confidence and 80% power). Undersized experiments produce inconclusive results; oversized experiments waste traffic and time.
Duration: Run the experiment long enough to capture natural variation — day-of-week effects, seasonal patterns, and user behavior cycles. Minimum 1-2 weeks for most AI applications.
Traffic Splitting
Random assignment: Assign users or requests randomly to the control (current model) or treatment (new model). Random assignment ensures that differences in outcomes are attributable to the model change, not to differences in the user populations.
Consistent assignment: Once a user is assigned to a group, keep them in that group for the experiment's duration. Switching users between groups mid-experiment confounds results.
Traffic allocation: Start with a small allocation to the treatment — 5-10% — to limit risk. Increase allocation as you gain confidence that the new model is not causing harm. Common progression: 5% for the first 2 days, 20% for the next 5 days, 50% for the remaining duration.
Guardrail Metrics
Define safety boundaries: Guardrail metrics are metrics that must not degrade beyond acceptable thresholds, regardless of the primary metric result. For a recommendation system, guardrails might include latency (must not increase by more than 50ms), error rate (must not increase), and revenue per user (must not decrease by more than 2%).
Automatic stopping: Configure automatic experiment stopping if guardrail metrics are violated. If the new model causes a latency spike or error rate increase, the experiment should stop and traffic should revert to the control model.
Implementation
Infrastructure Requirements
Traffic router: A system that routes incoming requests to either the control or treatment model based on the experiment assignment. This can be a load balancer with routing rules, a feature flag system, or a dedicated experimentation platform.
Model serving for multiple versions: The ability to serve multiple model versions simultaneously. Container-based serving (each model version in a separate container) is the most common approach.
Metric collection: Instrumentation that captures outcome metrics for each experiment group. The metric collection system must associate each outcome with the correct experiment group and model version.
Analysis pipeline: Automated or semi-automated analysis that computes metrics, statistical tests, and confidence intervals for each experiment.
Technology Options
Feature flag platforms (LaunchDarkly, Split): General-purpose feature flag platforms that support A/B testing. Good for simple model switches where the experiment is which model serves the request.
Experimentation platforms (Optimizely, Statsig): Dedicated experimentation platforms with built-in statistical analysis, guardrails, and experiment management. More capable but more expensive.
Custom implementation: For AI-specific needs (model-specific traffic routing, custom metrics, integration with ML serving infrastructure), custom implementation using your existing infrastructure may be necessary.
Running the Experiment
Pre-experiment validation: Before starting the experiment, verify that both models are serving correctly, metrics are being collected, and the traffic split is working as intended. A misconfigured experiment wastes time and may produce misleading results.
Monitoring during experiment: Monitor experiment metrics daily. Watch for early signs of problems — error rate spikes, latency increases, or dramatic metric differences that suggest implementation issues rather than model differences.
Statistical analysis: At the end of the experiment, analyze results with appropriate statistical tests. Calculate the observed difference in the primary metric, the confidence interval, and the p-value. Report practical significance (is the difference large enough to matter?) alongside statistical significance (is the difference real?).
Interpreting Results
Statistically Significant Improvement
If the new model shows a statistically significant improvement in the primary metric without violating guardrails, deploy it as the new production model. Document the experiment results for future reference.
No Significant Difference
If the experiment shows no significant difference, the new model is not providing meaningful value. Keep the current model unless the new model has other advantages (faster inference, lower cost, better maintainability) that justify deployment despite equivalent business metrics.
Significant Degradation
If the new model shows degradation, do not deploy. Analyze why the model performed worse — was it a specific user segment, a particular use case, or a systemic issue? Use these insights to improve the model before testing again.
Conflicting Metrics
If the primary metric improves but secondary metrics degrade, you need judgment. A recommendation model that increases CTR but decreases average order value might be optimizing for clicks at the expense of purchase quality. Discuss the trade-offs with the client and decide based on business priorities.
A/B Testing for Client Delivery
Setting Client Expectations
Educate clients about why A/B testing is necessary and what it involves. Many clients expect that a better model in testing will automatically perform better in production. Help them understand the gap between offline metrics and production impact.
Experiment Roadmaps
For clients with ongoing model improvement programs, create experiment roadmaps — planned experiments for each quarter with hypotheses, metrics, and expected impact. This structures the improvement process and makes progress visible.
Reporting
Report experiment results in business terms, not model metrics. "The new model increased conversion rate by 3.2%, generating an estimated $140,000 in additional annual revenue" is more meaningful to clients than "the new model improved AUC from 0.82 to 0.87."
A/B testing is the bridge between model development and business value. Without it, you are guessing whether model improvements translate to real-world impact. With it, every model change is validated against the metrics that matter — giving clients confidence that their AI investment is producing measurable returns.