🧠 AI with Python – 📊 Reliability Diagrams

In many real-world machine learning systems, predicted probabilities are more important than class labels. A model might predict an event with 85% confidence — but does that event actually occur 85% of the time?

Accuracy alone cannot answer this question.

In this project, we explore Reliability Diagrams, an enhanced form of calibration visualisation that helps evaluate how trustworthy predicted probabilities truly are.

Understanding the Problem

Classification models often output probabilities using predict_proba().

However:

• A model can have high accuracy but poor probability estimates.
• Some models are overconfident.
• Others are systematically under-confident.

In domains like healthcare, finance, insurance, and fraud detection, reliable probability estimates are critical for decision-making.

What Is a Reliability Diagram?

A reliability diagram extends a calibration curve by:

• Plotting predicted probability vs observed frequency
• Adding a histogram showing prediction distribution

This gives insight into:

• Calibration quality
• Confidence concentration
• Prediction bias patterns

A perfectly calibrated model lies on the diagonal line.

1. Train a Classification Model

We first train a classifier on structured data.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=200,
    random_state=42
)

model.fit(X_train, y_train)

2. Generate Predicted Probabilities

We extract probabilities for the positive class.

y_probs = model.predict_proba(X_test)[:, 1]

3. Compute Calibration Data

from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(
    y_test,
    y_probs,
    n_bins=10
)

The data is divided into bins, and predicted probability is compared against actual frequency.

4. Plot the Reliability Diagram

plt.plot([0, 1], [0, 1], linestyle="--")
plt.plot(prob_pred, prob_true, marker="o")

Additionally, we visualize the distribution of predictions using a histogram.

How to Interpret the Diagram

• Curve on diagonal → well calibrated
• Below diagonal → overconfident model
• Above diagonal → underconfident model

The histogram reveals whether predictions are concentrated in extreme confidence regions.

Why Reliability Diagrams Matter

Even models with strong ROC-AUC scores can be poorly calibrated.

Reliability diagrams help:

• Validate probability trustworthiness
• Detect systematic bias
• Improve decision thresholds
• Support regulatory compliance

They are essential in probability-driven systems.

Key Takeaways

1. Reliability diagrams evaluate probability accuracy.
2. Calibration is different from classification accuracy.
3. Overconfident models can be risky in production.
4. Histograms reveal prediction confidence distribution.
5. Essential for high-stakes ML systems.

Conclusion

Reliability diagrams provide a complete view of model probability behavior by combining calibration curves with prediction distribution analysis. They are a critical tool for building trustworthy, interpretable machine learning systems — strengthening the Advanced Visualization & Interpretability module in the AI with Python series.