⚡️ Saturday ML Spark – ⚖️ A/B Testing for Machine Learning Models
Posted on: October 3, 2026
Description:
Building an accurate Machine Learning model is only part of the journey. A model that performs well during development may not always produce the best results once it's deployed to real users. User behaviour, business goals, and real-world conditions can all influence how successful a model actually is.
This is why organisations use A/B Testing.
Rather than replacing the existing production model immediately, A/B Testing allows two different models to serve different groups of users at the same time. By comparing real-world performance, teams can confidently identify which model delivers better business outcomes before making a full production rollout.
What is A/B Testing?
A/B Testing is a deployment strategy where user traffic is divided between two or more models.
For example:
- Model A may be the current production model.
- Model B may be a newly trained version.
Instead of sending every user to the same model, incoming requests are randomly distributed between them. Each model then serves predictions to its assigned users.
This allows both models to be evaluated under the same production conditions.
How Does It Work?
Imagine an online shopping website with a product recommendation system.
Half of the visitors receive recommendations from the existing production model, while the other half receive recommendations from the new model.
assigned_model = random.choice(["A", "B"])
Each user's interaction—such as clicking a recommended product or making a purchase—is recorded for later analysis.
Because both models receive real production traffic, teams can compare their performance using actual user behaviour rather than relying only on offline evaluation metrics.
Measuring Performance
Unlike Shadow Deployment, where the new model's predictions are hidden from users, A/B Testing measures how users actually respond to each model.
One of the most common business metrics is the conversion rate.
summary = data.groupby("Model")["Converted"].mean() * 100
Other organisations may compare metrics such as:
- Click-through Rate (CTR)
- Conversion Rate
- User Engagement
- Revenue
- Session Duration
- Customer Retention
The most successful model isn't always the one with the highest accuracy—it's the one that delivers the best business results.
Why Use A/B Testing?
The biggest advantage of A/B Testing is that it measures real user impact.
Instead of guessing which model is better, organisations make decisions using production data.
A/B Testing helps teams:
- Compare multiple models fairly.
- Measure business performance objectively.
- Reduce deployment risk.
- Make data-driven deployment decisions.
- Continuously improve production ML systems.
It has become one of the most widely used evaluation strategies in modern MLOps.
A/B Testing vs Shadow Deployment
Although both techniques evaluate new models, they answer different questions.
A simple way to remember the difference is:
- Shadow Deployment → Can the new model safely handle production traffic?
- A/B Testing → Which model performs better for real users?
Many organisations first validate a model using Shadow Deployment and then move to A/B Testing before making it the primary production model.
Final Thoughts
A/B Testing is one of the most practical techniques for evaluating Machine Learning models in production. By exposing different groups of users to different model versions, organisations can measure real business outcomes instead of relying solely on offline evaluation metrics. This leads to more informed deployment decisions and helps ensure that new models improve both user experience and business performance.
In this article, we explored how A/B Testing works, why it's an important production ML strategy, and how it differs from Shadow Deployment. Together, these deployment techniques help organisations release Machine Learning models with greater confidence while continuously optimising their production systems.
Code Snippet:
import random
import pandas as pd
# =========================================================
# Simulate Model A
# =========================================================
def model_a():
return random.random() < 0.62
# =========================================================
# Simulate Model B
# =========================================================
def model_b():
return random.random() < 0.70
# =========================================================
# Create User Traffic
# =========================================================
users = list(range(1, 201))
print("=" * 70)
print("Simulating User Traffic")
print("=" * 70)
print(f"Total Users: {len(users)}")
# =========================================================
# Run A/B Test
# =========================================================
results = []
for user in users:
assigned_model = random.choice(["A", "B"])
if assigned_model == "A":
converted = model_a()
else:
converted = model_b()
results.append({
"User": user,
"Model": assigned_model,
"Converted": converted
})
# =========================================================
# Create Results DataFrame
# =========================================================
data = pd.DataFrame(results)
print("\n" + "=" * 70)
print("A/B Test Results")
print("=" * 70)
print(data.head(10))
# =========================================================
# Check Traffic Distribution
# =========================================================
traffic_distribution = data["Model"].value_counts()
print("\n" + "=" * 70)
print("Traffic Distribution")
print("=" * 70)
print(traffic_distribution)
# =========================================================
# Calculate Conversion Counts
# =========================================================
conversion_counts = data.groupby("Model")["Converted"].sum()
print("\n" + "=" * 70)
print("Conversion Counts")
print("=" * 70)
print(conversion_counts)
# =========================================================
# Calculate Conversion Rates
# =========================================================
conversion_rates = data.groupby("Model")["Converted"].mean() * 100
print("\n" + "=" * 70)
print("Conversion Rates")
print("=" * 70)
print(conversion_rates)
# =========================================================
# Compare Model Performance
# =========================================================
model_a_rate = conversion_rates.get("A", 0)
model_b_rate = conversion_rates.get("B", 0)
difference = model_b_rate - model_a_rate
print("\n" + "=" * 70)
print("Model Performance Comparison")
print("=" * 70)
print(f"Model A Conversion Rate : {model_a_rate:.2f}%")
print(f"Model B Conversion Rate : {model_b_rate:.2f}%")
print(f"Performance Difference : {difference:.2f}%")
# =========================================================
# Select Best Performing Model
# =========================================================
best_model = conversion_rates.idxmax()
best_rate = conversion_rates.max()
print("\n" + "=" * 70)
print("Best Performing Model")
print("=" * 70)
print(f"Winner: Model {best_model}")
print(f"Conversion Rate: {best_rate:.2f}%")
# =========================================================
# Calculate Conversion Improvement
# =========================================================
if model_a_rate > 0:
relative_improvement = ((model_b_rate - model_a_rate) / model_a_rate) * 100
else:
relative_improvement = 0
print("\n" + "=" * 70)
print("Relative Improvement")
print("=" * 70)
print(f"Model B Improvement over Model A: {relative_improvement:.2f}%")
# =========================================================
# Compare Converted vs Non-Converted Users
# =========================================================
conversion_summary = pd.crosstab(
data["Model"],
data["Converted"]
)
print("\n" + "=" * 70)
print("Conversion Summary")
print("=" * 70)
print(conversion_summary)
# =========================================================
# Deployment Recommendation
# =========================================================
print("\n" + "=" * 70)
print("Deployment Recommendation")
print("=" * 70)
if model_b_rate > model_a_rate + 5:
print("Model B shows a strong improvement over Model A.")
print("Consider increasing Model B traffic before full deployment.")
elif model_b_rate > model_a_rate:
print("Model B is performing better than Model A.")
print("Continue the A/B test with more users before making a final decision.")
elif model_a_rate > model_b_rate:
print("Model A is currently performing better.")
print("Keep Model A in production and review Model B before further rollout.")
else:
print("Both models are performing similarly.")
print("Collect more traffic before selecting a winner.")
# =========================================================
# Simulate Additional User Traffic
# =========================================================
print("\n" + "=" * 70)
print("Monitoring Additional Users")
print("=" * 70)
for user in range(201, 211):
assigned_model = random.choice(["A", "B"])
if assigned_model == "A":
converted = model_a()
else:
converted = model_b()
print(f"User={user} | Model={assigned_model} | Converted={converted}")
# =========================================================
# A/B Testing Summary
# =========================================================
print("\n" + "=" * 70)
print("A/B Testing Summary")
print("=" * 70)
print(f"Total Users : {len(data)}")
print(f"Model A Users : {(data['Model'] == 'A').sum()}")
print(f"Model B Users : {(data['Model'] == 'B').sum()}")
print(f"Model A Conversion: {model_a_rate:.2f}%")
print(f"Model B Conversion: {model_b_rate:.2f}%")
print(f"Winning Model : {best_model}")
# =========================================================
# Program Completed
# =========================================================
print("\n" + "=" * 70)
print("A/B Testing Simulation Completed Successfully!")
print("=" * 70)
No comments yet. Be the first to comment!