⚡️ Saturday ML Spark – 📉 Concept Drift Detection Basics

One of the biggest challenges in machine learning is that the world keeps changing. Customer behaviour evolves. Markets shift. Fraud patterns adapt. Business rules change.

A model that performs extremely well today may become less effective tomorrow, even if nothing changes in the code. This phenomenon is known as Concept Drift.

In this project, we explore the basics of concept drift detection and learn how production ML systems identify performance degradation before it becomes a serious problem.

---

Why Models Degrade Over Time

Machine learning models learn patterns from historical data.

For example:

• customer activity → churn probability
• transaction details → fraud risk
• income and credit history → loan approval

The model assumes these relationships remain relatively stable.

However, real-world systems are dynamic.

When those relationships change, the model begins making less accurate predictions.

---

What Is Concept Drift?

Concept drift occurs when: The relationship between input features and the target variable changes over time. The model continues using patterns learned from historical data, but those patterns are no longer valid.

As a result:

• prediction quality decreases
• business decisions become less reliable
• model performance gradually deteriorates

---

A Simple Example

Imagine a recommendation system.

During training:

User watches action movies
        ↓
Recommend action movies

This pattern works well.

Months later, user interests shift.

User now prefers documentaries

The model still recommends action movies. Prediction quality drops because user behaviour changed.

This is concept drift.

---

How Drift Appears in Production

Concept drift usually reveals itself through performance degradation.

For example:

Month	Accuracy
January	96%
February	95%
March	95%
April	94%
May	93%
June	92%
July	90%
August	88%
September	86%
October	84%

At first, the decline seems small.

Over time, the degradation becomes significant.

---

Monitoring Accuracy Trends

One of the simplest drift detection techniques is monitoring model accuracy.

plt.plot(
    performance_data["month"],
    performance_data["accuracy"]
)

Visualising accuracy over time often reveals patterns that are difficult to spot from individual measurements.

---

Measuring Performance Degradation

A common approach is comparing current performance against a baseline.

baseline_accuracy = 0.96

latest_accuracy = 0.84

Calculate the difference:

accuracy_drop = (
    baseline_accuracy -
    latest_accuracy
)

A significant decline may indicate concept drift.

---

Simple Threshold-Based Detection

Many monitoring systems start with threshold-based alerts.

if accuracy_drop > 0.10:
    print("Concept Drift Detected")

Although simple, this approach can provide valuable early warning signals.

---

Using Rolling Averages

Performance metrics naturally fluctuate. Not every dip indicates drift. Rolling averages help smooth short-term noise.

performance_data[
    "rolling_accuracy"
] = (
    performance_data["accuracy"]
    .rolling(window=3)
    .mean()
)

This makes long-term trends easier to identify.

---

Types of Concept Drift

Sudden Drift

Performance changes abruptly.

Examples:

• policy changes
• economic crashes
• regulatory updates

---

Gradual Drift

Performance slowly degrades over time.

Examples:

• evolving customer preferences
• changing market behaviour

---

Recurring Drift

Patterns disappear and later return.

Examples:

• seasonal demand
• holiday purchasing behaviour

---

Why Drift Detection Matters

Without drift detection:

• models silently degrade
• business decisions become less accurate
• customer experience suffers
• operational risk increases

Early detection allows teams to:

• retrain models
• update features
• deploy newer versions
• restore performance

before major issues occur.

---

How Real Systems Detect Drift

Modern ML platforms use techniques such as:

• rolling accuracy monitoring
• statistical drift detection
• Population Stability Index (PSI)
• feature distribution monitoring
• shadow model comparisons

These approaches provide earlier and more reliable drift detection.

---

Where Concept Drift Is Common

Concept drift frequently appears in:

• fraud detection systems
• recommendation engines
• customer churn models
• credit risk models
• marketing prediction systems

Any environment that changes over time is susceptible to drift.

---

Key Takeaways

1. Concept drift occurs when feature-target relationships change over time.
2. Models can degrade even when the code remains unchanged.
3. Performance monitoring is often the first indication of drift.
4. Rolling averages help identify long-term trends.
5. Early drift detection enables timely retraining and model updates.

---

Conclusion

Machine learning models operate in constantly evolving environments. As customer behaviour, business conditions, and external factors change, previously learned relationships can become outdated. Detecting concept drift early helps organisations maintain reliable predictions and avoid silent performance degradation.

---

# 📦 Import Required Libraries
import pandas as pd
import matplotlib.pyplot as plt


# =========================================================
# 📝 Create Historical Performance Data
# =========================================================

performance_data = pd.DataFrame({
    "month": [
        "Jan", "Feb", "Mar", "Apr",
        "May", "Jun", "Jul", "Aug",
        "Sep", "Oct"
    ],

    "accuracy": [
        0.96, 0.95, 0.95, 0.94,
        0.93, 0.92, 0.90, 0.88,
        0.86, 0.84
    ]
})


# =========================================================
# 🔍 View Monitoring Data
# =========================================================

print("Performance Data:\n")
print(performance_data)


# =========================================================
# 📈 Visualize Accuracy Trend
# =========================================================

plt.figure(figsize=(8, 4))

plt.plot(
    performance_data["month"],
    performance_data["accuracy"],
    marker="o"
)

plt.title("Model Accuracy Trend")
plt.xlabel("Month")
plt.ylabel("Accuracy")

plt.grid(True)
plt.tight_layout()
plt.show()


# =========================================================
# 📊 Calculate Performance Degradation
# =========================================================

baseline_accuracy = performance_data[
    "accuracy"
].iloc[0]

latest_accuracy = performance_data[
    "accuracy"
].iloc[-1]

accuracy_drop = (
    baseline_accuracy -
    latest_accuracy
)

print(
    "\nAccuracy Drop:",
    round(accuracy_drop, 4)
)


# =========================================================
# 🚨 Simple Concept Drift Detection
# =========================================================

DRIFT_THRESHOLD = 0.10

print("\n=== Drift Detection ===")

if accuracy_drop > DRIFT_THRESHOLD:
    print("⚠️ Concept Drift Detected")
else:
    print("✅ No Significant Drift")


# =========================================================
# 📈 Rolling Accuracy Monitoring
# =========================================================

performance_data[
    "rolling_accuracy"
] = (
    performance_data["accuracy"]
    .rolling(window=3)
    .mean()
)

print("\nRolling Accuracy:\n")
print(
    performance_data[
        ["month", "rolling_accuracy"]
    ]
)


# =========================================================
# 📊 Plot Actual vs Rolling Accuracy
# =========================================================

plt.figure(figsize=(8, 4))

plt.plot(
    performance_data["month"],
    performance_data["accuracy"],
    marker="o",
    label="Actual Accuracy"
)

plt.plot(
    performance_data["month"],
    performance_data["rolling_accuracy"],
    marker="s",
    label="Rolling Average"
)

plt.title("Concept Drift Monitoring")
plt.xlabel("Month")
plt.ylabel("Accuracy")

plt.legend()

plt.grid(True)
plt.tight_layout()
plt.show()


# =========================================================
# 📉 Calculate Performance Degradation %
# =========================================================

degradation_percent = (
    (
        baseline_accuracy -
        latest_accuracy
    )
    /
    baseline_accuracy
) * 100

print(
    "\nPerformance Degradation (%):",
    round(degradation_percent, 2)
)


# =========================================================
# 📋 Drift Monitoring Summary
# =========================================================

summary = {
    "Baseline Accuracy":
        baseline_accuracy,

    "Latest Accuracy":
        latest_accuracy,

    "Accuracy Drop":
        accuracy_drop,

    "Degradation (%)":
        degradation_percent
}

print("\n=== Monitoring Summary ===")

for metric, value in summary.items():
    print(
        f"{metric}: {round(value, 4)}"
    )


# =========================================================
# 💾 Save Monitoring Results
# =========================================================

performance_data.to_csv(
    "concept_drift_monitoring.csv",
    index=False
)

print(
    "\nMonitoring results saved successfully."
)


# =========================================================
# 📂 Read Saved Results
# =========================================================

saved_data = pd.read_csv(
    "concept_drift_monitoring.csv"
)

print("\nSaved Data Preview:\n")
print(saved_data.head())