⚡️ Saturday ML Spark – 📊 Feature Drift Detection using Population Stability Index (PSI)

Posted on: July 4, 2026

Description:

A machine learning model can continue making predictions even when the data it receives has changed significantly. The model itself hasn’t changed. The code hasn’t changed. Yet, prediction quality gradually begins to decline. One common reason for this is Feature Drift.

In this project, we’ll learn how production ML systems detect changes in input data using one of the most popular monitoring metrics—the Population Stability Index (PSI).

What Is Feature Drift?

Feature drift occurs when the distribution of an input feature changes between the training environment and the production environment.

For example, suppose a credit risk model was trained using customer income data.

During training:

Average Income ≈ ₹50,000

Months later, the production data looks like:

Average Income ≈ ₹56,000

Even though the model hasn’t changed, the incoming data now follows a different distribution.

Over time, this may reduce prediction quality.

Why Feature Drift Matters

Machine learning models assume that production data resembles the data used during training.

When that assumption no longer holds, the model may begin making less reliable predictions.

Feature drift can occur due to:

changing customer behavior
economic conditions
seasonal trends
new business policies
evolving user demographics

Detecting these changes early allows teams to investigate and retrain models when necessary.

What Is Population Stability Index (PSI)?

The Population Stability Index (PSI) is a statistical metric that compares the distribution of a feature across two datasets.

Typically:

Expected Population → Training data
Actual Population → Production data

The larger the PSI value, the greater the difference between the two distributions.

Creating Sample Data

For this demonstration, we simulate a feature collected during training and compare it with a newer production dataset.

training_feature = np.random.normal(
    loc=50000,
    scale=8000,
    size=5000
)

production_feature = np.random.normal(
    loc=56000,
    scale=9000,
    size=5000
)

Here, the production feature has shifted, allowing us to observe feature drift.

Visualizing the Distribution

A histogram provides a quick visual comparison.

plt.hist(
    training_feature,
    alpha=0.6
)

plt.hist(
    production_feature,
    alpha=0.6
)

If the two distributions differ noticeably, feature drift may be occurring.

Visualization offers an intuitive first look before calculating statistical metrics.

Calculating PSI

PSI compares the percentage of observations that fall into identical bins across both datasets.

psi = np.sum(
    (
        train_pct -
        prod_pct
    )
    *
    np.log(
        train_pct /
        prod_pct
    )
)

Although the formula may appear complex, the idea is simple:

Compare how the feature distribution has shifted between training and production.

Interpreting the PSI Score

A common guideline is:

PSI Value	Interpretation
< 0.10	No significant drift
0.10 – 0.25	Moderate drift
> 0.25	Significant drift

A higher PSI indicates that production data is becoming increasingly different from the training data.

Why Production Teams Use PSI

PSI provides a simple yet effective way to monitor feature stability.

Instead of waiting for model accuracy to decline, teams can detect changing data distributions early and investigate potential issues.

It is commonly used as part of automated ML monitoring pipelines.

Real-World Applications

Feature drift monitoring is common in:

fraud detection systems
credit risk scoring
customer churn prediction
recommendation engines
insurance underwriting
demand forecasting

Any production ML system that processes continuously changing data can benefit from PSI monitoring.

Key Takeaways

Feature drift occurs when input feature distributions change over time.
PSI compares training and production feature distributions.
Both datasets must be evaluated using the same bins.
Larger PSI values indicate greater distribution shift.
PSI is one of the most widely used feature monitoring metrics in production ML systems.

Conclusion

Monitoring feature distributions is just as important as monitoring model accuracy. Even when a model remains unchanged, shifts in production data can gradually reduce prediction quality. The Population Stability Index (PSI) provides a practical way to detect these changes early, enabling teams to investigate, retrain models when needed, and maintain reliable machine learning systems over time.

This begins the ML Systems Advanced track in Saturday ML Spark ⚡️, where we’ll explore advanced monitoring, deployment, and production techniques used by modern ML engineering teams.

Code Snippet:

# =========================================================
# 📦 Import Required Libraries
# =========================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


# =========================================================
# 🧩 Simulate Training & Production Data
# =========================================================

np.random.seed(42)

training_feature = np.random.normal(
    loc=50000,
    scale=8000,
    size=5000
)

production_feature = np.random.normal(
    loc=56000,
    scale=9000,
    size=5000
)


# =========================================================
# 📊 Visualize Feature Distribution
# =========================================================

plt.figure(figsize=(8, 5))

plt.hist(
    training_feature,
    bins=20,
    alpha=0.6,
    label="Training Data"
)

plt.hist(
    production_feature,
    bins=20,
    alpha=0.6,
    label="Production Data"
)

plt.title("Training vs Production Feature Distribution")
plt.xlabel("Feature Value")
plt.ylabel("Frequency")

plt.legend()

plt.tight_layout()
plt.show()


# =========================================================
# 🏗️ Create Common Bins
# =========================================================

bins = np.linspace(
    min(
        training_feature.min(),
        production_feature.min()
    ),
    max(
        training_feature.max(),
        production_feature.max()
    ),
    11
)


# =========================================================
# 📈 Calculate Bin Counts
# =========================================================

train_counts, _ = np.histogram(
    training_feature,
    bins=bins
)

prod_counts, _ = np.histogram(
    production_feature,
    bins=bins
)


# =========================================================
# 📊 Convert Counts to Percentages
# =========================================================

train_pct = (
    train_counts /
    len(training_feature)
)

prod_pct = (
    prod_counts /
    len(production_feature)
)


# =========================================================
# ⚠️ Avoid Division by Zero
# =========================================================

epsilon = 1e-6

train_pct = np.where(
    train_pct == 0,
    epsilon,
    train_pct
)

prod_pct = np.where(
    prod_pct == 0,
    epsilon,
    prod_pct
)


# =========================================================
# 🧮 Calculate PSI
# =========================================================

psi_values = (
    (
        train_pct -
        prod_pct
    )
    *
    np.log(
        train_pct /
        prod_pct
    )
)

psi = psi_values.sum()

print("=" * 40)
print("Population Stability Index (PSI)")
print("=" * 40)
print(f"PSI Score : {psi:.4f}")


# =========================================================
# 🚦 Interpret PSI Score
# =========================================================

if psi < 0.10:

    interpretation = "No Significant Drift"

elif psi < 0.25:

    interpretation = "Moderate Drift"

else:

    interpretation = "Significant Drift"

print(
    f"Interpretation : {interpretation}"
)


# =========================================================
# 📋 Create Distribution Summary
# =========================================================

summary = pd.DataFrame({

    "Bin": range(1, len(train_pct) + 1),

    "Training %": train_pct.round(4),

    "Production %": prod_pct.round(4),

    "PSI Contribution": psi_values.round(4)

})

print("\nDistribution Summary:\n")
print(summary)


# =========================================================
# 📊 Visualize Percentage Comparison
# =========================================================

summary.set_index("Bin")[
    [
        "Training %",
        "Production %"
    ]
].plot(
    kind="bar",
    figsize=(10, 5)
)

plt.title("Training vs Production Distribution")
plt.ylabel("Percentage")

plt.grid(
    axis="y",
    linestyle="--",
    alpha=0.5
)

plt.tight_layout()
plt.show()


# =========================================================
# 📈 Visualize PSI Contribution
# =========================================================

plt.figure(figsize=(8, 4))

plt.bar(
    summary["Bin"],
    summary["PSI Contribution"]
)

plt.title("PSI Contribution by Bin")
plt.xlabel("Bin")
plt.ylabel("Contribution")

plt.grid(
    axis="y",
    linestyle="--",
    alpha=0.5
)

plt.tight_layout()
plt.show()


# =========================================================
# 💾 Save Results
# =========================================================

summary.to_csv(
    "psi_feature_drift_results.csv",
    index=False
)

print(
    "\nResults saved to psi_feature_drift_results.csv"
)


# =========================================================
# 📂 Read Saved Results
# =========================================================

saved_results = pd.read_csv(
    "psi_feature_drift_results.csv"
)

print("\nSaved Results Preview:\n")
print(saved_results.head())


# =========================================================
# 📌 Final Summary
# =========================================================

print("\n" + "=" * 45)
print(" Feature Drift Detection Completed")
print("=" * 45)

print(f"Overall PSI Score : {psi:.4f}")
print(f"Drift Status      : {interpretation}")

if interpretation == "Significant Drift":

    print("\n🚨 Recommendation:")
    print("- Investigate production data.")
    print("- Check feature distributions.")
    print("- Consider retraining the model.")

elif interpretation == "Moderate Drift":

    print("\n⚠️ Recommendation:")
    print("- Increase monitoring frequency.")
    print("- Watch future PSI values closely.")

else:

    print("\n✅ Recommendation:")
    print("- Continue regular monitoring.")

← →	move
↑	rotate
↓	soft drop
Space	hard drop
P	pause / resume

⚡️ Saturday ML Spark – 📊 Feature Drift Detection using Population Stability Index (PSI)

Description:

What Is Feature Drift?

Why Feature Drift Matters

What Is Population Stability Index (PSI)?

Creating Sample Data

Visualizing the Distribution

Calculating PSI

Interpreting the PSI Score

Why Production Teams Use PSI

Real-World Applications

Key Takeaways

Conclusion

Code Snippet:

Comments

Add Your Comment

⚡️ Saturday ML Spark – 📊 Feature Drift Detection using Population Stability Index (PSI)

Description:

What Is Feature Drift?

Why Feature Drift Matters

What Is Population Stability Index (PSI)?

Creating Sample Data

Visualizing the Distribution

Calculating PSI

Interpreting the PSI Score

Why Production Teams Use PSI

Real-World Applications

Key Takeaways

Conclusion

Code Snippet:

Comments Show Comments

Add Your Comment

Related Posts

⚡️ Saturday ML Spark – 📉 Concept Drift Detection Basics

⚡️ Saturday ML Spark – 🧪 A/B Testing for ML Models

⚡️ Saturday ML Spark – 📈 Model Monitoring & Performance Tracking

7-Day AI Crash Course

Comments