🧠 AI with Python – 🌗 Shadow Deployment Concept

Deploying a new machine learning model directly into production can be risky. Even if the model performs exceptionally well during development and testing, real-world environments often reveal issues that were never visible offline.

Production data can be messy, user behaviour can change, and system interactions can expose unexpected problems.

This is why many organisations use a technique called Shadow Deployment before rolling out a new model.

In this project, we explore how shadow deployment works, why it is used, and how it helps teams validate machine learning models safely.

---

The Challenge of Deploying New Models

Imagine you have a fraud detection model currently serving millions of transactions every day.

Your data science team develops a new model that:

• achieves higher accuracy
• improves recall
• reduces false positives

during testing.

The natural question becomes:

Can we deploy it immediately?

In most production systems, the answer is no.

Before replacing a proven model, teams need evidence that the new model behaves correctly in real-world conditions.

---

What Is Shadow Deployment?

Shadow deployment is a deployment strategy where:

• the production model continues serving users
• the new model receives the same inputs
• the new model generates predictions
• those predictions are logged but not used

Users never see the output of the shadow model.

The candidate model operates silently in the background.

---

How Shadow Deployment Works

A simplified workflow looks like:

                Incoming Request
                        │
        ┌───────────────┴───────────────┐
        │                               │
        ▼                               ▼

Production Model              Shadow Model

        │                               │
        ▼                               ▼

User Receives Result         Prediction Logged

The production model remains responsible for all user-facing decisions.

The shadow model simply observes.

---

Training the Production Model

The production model represents the currently deployed system.

production_model.fit(
    X_train,
    y_train
)

This model continues serving requests as normal.

---

Training the Shadow Model

The candidate model is trained independently.

shadow_model.fit(
    X_train,
    y_train
)

This model is not exposed to users.

Instead, it silently processes the same requests.

---

Simulating Live Traffic

Incoming requests are duplicated.

live_requests = X_live.iloc[:10]

Both models receive identical inputs.

This allows teams to compare behaviour under realistic conditions.

---

Generating Predictions

Production model:

production_predictions = (
    production_model.predict(
        live_requests
    )
)

Shadow model:

shadow_predictions = (
    shadow_model.predict(
        live_requests
    )
)

The key difference is that only the production predictions are used by the application.

---

Comparing Results

The outputs can then be analysed.

shadow_logs = pd.DataFrame({
    ...
})

Teams often compare:

• prediction agreement
• accuracy
• precision
• recall
• confidence scores

This helps identify unexpected differences before deployment.

---

Why Shadow Deployment Is Useful

Shadow deployment provides several advantages.

Risk-Free Testing

Users continue interacting with the production model.

No business impact occurs if the shadow model behaves unexpectedly.

---

Real Production Data

The shadow model receives actual production traffic rather than synthetic test data.

This often reveals issues missed during development.

---

Performance Validation

Teams can evaluate:

• prediction quality
• latency
• resource consumption
• system stability

under real workloads.

---

Confidence Before Rollout

By the time deployment occurs, the team already understands how the new model behaves.

This significantly reduces rollout risk.

---

Shadow Deployment vs A/B Testing

These concepts are often confused.

Shadow Deployment

Production Model → Users

Shadow Model → Logs Only

Users never see shadow model predictions.

---

A/B Testing

50% Users → Model A

50% Users → Model B

Both models actively influence users.

Shadow deployment typically happens before A/B testing.

---

Real-World Use Cases

Shadow deployments are common in:

• fraud detection systems
• recommendation engines
• search ranking models
• healthcare ML systems
• financial risk platforms

These environments require extremely careful model rollouts.

---

Potential Limitations

Although powerful, shadow deployments are not free.

Challenges include:

• additional infrastructure costs
• duplicate prediction workloads
• monitoring complexity
• storage requirements for prediction logs

Organizations must balance safety against operational cost.

---

Typical Deployment Progression

Many mature ML teams follow a sequence like:

Offline Evaluation
        ↓
Shadow Deployment
        ↓
A/B Testing
        ↓
Canary Release
        ↓
Full Deployment

Each stage progressively increases confidence.

---

Key Takeaways

---

1. Shadow deployment runs a candidate model alongside the production model.
2. The shadow model receives real traffic but does not affect users.
3. Teams can compare predictions safely before deployment.
4. Shadow deployment helps identify issues using real-world data.
5. It is commonly used before A/B testing and full rollout.

---

Conclusion

Shadow deployment is one of the safest ways to evaluate new machine learning models in production environments. By running a candidate model silently alongside the existing system, teams gain valuable insights into real-world behavior without introducing risk to users. This approach helps build confidence, uncover hidden issues, and support smoother production rollouts.

This strengthens the ML Systems track in the AI with Python series — focusing on practical deployment strategies that help machine learning systems evolve safely and reliably.

---

# 📦 Import Required Libraries
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import (
    accuracy_score,
    f1_score,
    classification_report
)


# =========================================================
# 🧩 Load Dataset
# =========================================================

data = load_breast_cancer()

X = pd.DataFrame(
    data.data,
    columns=data.feature_names
)

y = data.target


# =========================================================
# ✂️ Split Dataset
# =========================================================

# X_live simulates live production requests

X_train, X_live, y_train, y_live = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=42,
    stratify=y
)


# =========================================================
# 🤖 Production Model
# =========================================================

production_model = LogisticRegression(
    max_iter=5000
)

production_model.fit(
    X_train,
    y_train
)


# =========================================================
# 🌗 Shadow Model
# =========================================================

shadow_model = RandomForestClassifier(
    n_estimators=200,
    random_state=42
)

shadow_model.fit(
    X_train,
    y_train
)


# =========================================================
# 🚀 Simulate Live Requests
# =========================================================

live_requests = X_live.iloc[:20]
live_labels = y_live[:20]


# =========================================================
# 📊 Production Predictions
# =========================================================

production_predictions = production_model.predict(
    live_requests
)

production_probabilities = production_model.predict_proba(
    live_requests
)[:, 1]


# =========================================================
# 🌗 Shadow Predictions
# =========================================================

shadow_predictions = shadow_model.predict(
    live_requests
)

shadow_probabilities = shadow_model.predict_proba(
    live_requests
)[:, 1]


# =========================================================
# 📝 Create Shadow Deployment Logs
# =========================================================

shadow_logs = pd.DataFrame({
    "actual": live_labels,

    "production_prediction": production_predictions,

    "shadow_prediction": shadow_predictions,

    "production_probability": production_probabilities,

    "shadow_probability": shadow_probabilities
})


# =========================================================
# 🔍 Compare Prediction Behavior
# =========================================================

shadow_logs["prediction_match"] = (
    shadow_logs["production_prediction"]
    ==
    shadow_logs["shadow_prediction"]
)

shadow_logs["probability_difference"] = (
    shadow_logs["shadow_probability"]
    -
    shadow_logs["production_probability"]
).round(4)

print("Shadow Deployment Logs:\n")
print(shadow_logs)


# =========================================================
# 📈 Evaluate Production Model
# =========================================================

prod_accuracy = accuracy_score(
    live_labels,
    production_predictions
)

prod_f1 = f1_score(
    live_labels,
    production_predictions
)


# =========================================================
# 📈 Evaluate Shadow Model
# =========================================================

shadow_accuracy = accuracy_score(
    live_labels,
    shadow_predictions
)

shadow_f1 = f1_score(
    live_labels,
    shadow_predictions
)


# =========================================================
# 📊 Print Evaluation Summary
# =========================================================

print("\n=== Production Model ===")
print("Accuracy:", round(prod_accuracy, 4))
print("F1 Score:", round(prod_f1, 4))

print("\nClassification Report:\n")
print(
    classification_report(
        live_labels,
        production_predictions
    )
)


print("\n=== Shadow Model ===")
print("Accuracy:", round(shadow_accuracy, 4))
print("F1 Score:", round(shadow_f1, 4))

print("\nClassification Report:\n")
print(
    classification_report(
        live_labels,
        shadow_predictions
    )
)


# =========================================================
# 📊 Match Rate Between Models
# =========================================================

match_rate = shadow_logs[
    "prediction_match"
].mean()

print(
    "\nPrediction Match Rate:",
    round(match_rate, 4)
)


# =========================================================
# 🚦 Deployment Decision Simulation
# =========================================================

if (
    shadow_f1 > prod_f1
    and match_rate >= 0.80
):
    print(
        "\n🚀 Shadow model looks promising for further testing."
    )
else:
    print(
        "\n✅ Keep production model active. Continue monitoring shadow model."
    )


# =========================================================
# 💾 Save Shadow Logs
# =========================================================

shadow_logs.to_csv(
    "shadow_deployment_logs.csv",
    index=False
)

print(
    "\nShadow deployment logs saved successfully."
)


# =========================================================
# 📂 Read Saved Logs
# =========================================================

saved_logs = pd.read_csv(
    "shadow_deployment_logs.csv"
)

print("\nSaved Shadow Logs Preview:\n")
print(saved_logs.head())