🔐 AI/ML Security Lab – 🎭 Model Evasion 🐍

Machine learning classifiers are widely used in cybersecurity applications to detect suspicious activity, identify fraudulent transactions, and recognise unusual behaviour. However, even a well-trained model can make incorrect predictions when its inputs are deliberately manipulated.

Model Evasion is an adversarial machine learning technique where an attacker modifies input data during inference to influence a model's predictions without changing the model itself.

In this AI/ML Security Lab, we'll use Python and Scikit-learn to simulate a controlled model evasion attack, measure its impact on classification decisions, and explore defensive techniques using feature validation and anomaly detection.

---

🎯 Understanding Model Evasion

A machine learning classifier learns patterns from training data and uses those patterns to classify new inputs.

For example, a security classifier might categorise incoming activity into two classes:

• Class 0: Legitimate Activity
• Class 1: Suspicious Activity

Under normal conditions, the classifier identifies suspicious activity based on learned feature relationships. However, an attacker may deliberately modify certain input features to make suspicious activity appear legitimate. Unlike data poisoning, which manipulates training data, model evasion occurs after the model has been trained, during the prediction process. The attacker's objective is to cross the model's decision boundary and produce an incorrect classification.

---

🟢 Building the Baseline Classifier

We'll begin by generating a synthetic dataset containing 2,000 samples with six numerical features representing simulated activity measurements. Using Scikit-learn, we'll train a Logistic Regression classifier to distinguish legitimate activity from suspicious activity.

X, y = make_classification(
    n_samples=2000,
    n_features=6,
    n_informative=4,
    n_redundant=0,
    n_classes=2,
    class_sep=1.5,
    random_state=42
)

After splitting and standardising the dataset, we'll train the classifier and measure its baseline accuracy.

model = LogisticRegression(max_iter=1000, random_state=42)
model.fit(X_train_scaled, y_train)

baseline_predictions = model.predict(X_test_scaled)
baseline_accuracy = accuracy_score(y_test, baseline_predictions)

print(f"Baseline Accuracy: {baseline_accuracy:.2%}")

This establishes how accurately the classifier performs on unmodified test data. For this experiment, we'll focus on suspicious samples that the model initially classifies correctly.

---

🔴 Simulating a Model Evasion Attack

The objective of our attack is straightforward: manipulate suspicious inputs until the classifier predicts them as legitimate.

We'll simulate a white-box attack, meaning we have access to the classifier's learned parameters. Logistic Regression assigns weights to input features, which influence its classification decisions. By examining these weights, we can determine a direction that reduces the model's suspicious-class score.

weights = model.coef_[0]
direction = -weights / np.linalg.norm(weights)

Next, we'll gradually modify the selected suspicious samples along this direction.

perturbation_levels = np.linspace(0.05, 3.0, 60)

for level in perturbation_levels:
    candidates = target_scaled + level * direction
    candidate_predictions = model.predict(candidates)

    newly_successful = (candidate_predictions == 0) & (~successful_attacks)

    adversarial_scaled[newly_successful] = candidates[newly_successful]
    attack_strengths[newly_successful] = level
    successful_attacks[newly_successful] = True

The attack tests increasing perturbation strengths and records the first successful evasion for each target sample. Importantly, the trained model remains unchanged. Only the inputs supplied to the classifier are modified. This demonstrates how an attacker with knowledge of a model's decision boundary can construct inputs that influence its predictions.

---

🔍 Measuring the Attack's Impact

After generating the manipulated inputs, we'll compare their predictions with those of the original suspicious samples. Two important measurements help us evaluate the attack.

Attack Success Rate: The percentage of targeted suspicious samples that are incorrectly classified as legitimate after manipulation.

Prediction Probability: The classifier's estimated probability that an input belongs to the suspicious class.

adversarial_predictions = model.predict(adversarial_scaled)
adversarial_probabilities = model.predict_proba(adversarial_scaled)[:, 1]

attack_success_rate = successful_attacks.mean()

print(f"Attack Success Rate: {attack_success_rate:.2%}")
print(f"Original Suspicious Probability: {original_probabilities.mean():.2%}")
print(f"Adversarial Suspicious Probability: {adversarial_probabilities.mean():.2%}")

A successful evasion occurs when an originally suspicious sample is classified as legitimate after manipulation. We can also visualise how the average suspicious-class probability changes between the original and manipulated inputs. However, not every attack attempt will succeed. The outcome depends on the model's decision boundary, the selected samples, and the allowed perturbation strength.

---

🛡️ Defending Against Model Evasion

Detecting manipulated inputs is challenging because adversarial samples may resemble ordinary data. For this experiment, we'll introduce two defensive techniques.

1. Feature Range Validation

Feature validation checks whether incoming values fall outside the ranges observed during training.

training_min = X_train.min()
training_max = X_train.max()

def validate_features(samples):
    return ((samples < training_min) | (samples > training_max)).any(axis=1)

This can identify inputs containing unexpected feature values. However, an attacker may construct adversarial inputs that remain within valid ranges, allowing them to bypass simple validation checks.

2. Anomaly Detection with Isolation Forest

Isolation Forest is an unsupervised machine learning algorithm designed to identify unusual observations. We'll train an Isolation Forest model using the original training features and evaluate the manipulated inputs.

anomaly_detector = IsolationForest(
    n_estimators=100,
    contamination=0.05,
    random_state=42
)

anomaly_detector.fit(X_train_scaled)

anomaly_predictions = anomaly_detector.predict(adversarial_scaled)
anomaly_flags = anomaly_predictions == -1

Inputs identified as anomalies can be flagged for additional inspection. Combining anomaly detection with feature validation provides multiple opportunities to identify suspicious inputs. Nevertheless, neither technique guarantees protection against sophisticated evasion attacks.

---

🧪 Verifying the Defence

Finally, we'll measure how effectively the defensive checks identify successful evasion attempts. We'll combine the feature validation and anomaly detection results to determine which manipulated samples are flagged.

combined_flags = range_violations.to_numpy() | anomaly_flags

successful_count = successful_attacks.sum()
detected_successful = np.sum(combined_flags & successful_attacks)

detection_rate = detected_successful / successful_count if successful_count else 0

print(f"Successful Evasions: {successful_count}")
print(f"Detected Successful Evasions: {detected_successful}")
print(f"Detection Rate: {detection_rate:.2%}")

The detection rate indicates what proportion of successful evasion attempts were identified by our defensive checks.

We'll also evaluate how often these checks flag the original, unmodified suspicious samples. This helps us understand whether the defenses are identifying manipulated inputs specifically or frequently flagging suspicious activity in general. A complete evaluation should additionally include legitimate samples to measure false-positive rates.

The key takeaway is that defensive techniques must be evaluated against adversarial inputs rather than relying exclusively on normal classification accuracy.

---

📌 Security Takeaways

• Model evasion manipulates inputs during inference without changing the trained classifier.
• White-box attacks can use model parameters to identify effective manipulation directions.
• Attack success rate measures how frequently manipulated inputs bypass classification.
• Feature validation can identify unexpected values but may miss in-range adversarial inputs.
• Isolation Forest can detect unusual patterns, although it cannot guarantee adversarial detection.
• Evaluating detection rates and false positives is essential when designing ML-based security defences.

---

🚀 Final Note

A machine learning classifier may perform well on ordinary test data while remaining vulnerable to deliberately manipulated inputs. Model evasion highlights an important challenge in AI security: high classification accuracy does not automatically guarantee robustness against adversarial behaviour.

By simulating evasion attempts, measuring their impact, and evaluating defensive controls, developers can better understand the limitations of ML-based security systems and work toward building more resilient applications.