🧠 AI with Python – ⚙️ Feature Engineering for Tabular ML

In machine learning, better performance does not always come from using more advanced models. Very often, the real improvement comes from creating better features. This process is known as feature engineering.

In this project, we explore practical feature engineering techniques for tabular machine learning, including interaction features, ratio features, and log transformations.

---

Understanding the Problem

Raw tabular data is rarely perfect for machine learning models.

Datasets often contain:

• skewed distributions
• hidden relationships
• noisy numerical patterns
• weak standalone signals

If we feed raw data directly into a model, important information may remain hidden.

Feature engineering helps expose those patterns.

---

What Is Feature Engineering?

Feature engineering is the process of:

• transforming existing features
• combining variables
• creating more meaningful representations of data

The goal is to make learning easier for the model.

In many tabular ML tasks, strong features can outperform complex algorithms.

---

1. Baseline Model

We first train a model using the original dataset.

model = LinearRegression()
model.fit(X_train, y_train)

This gives us a baseline performance score for comparison.

---

2. Creating Interaction Features

Interaction features combine multiple variables together.

X["MedInc_HouseAge"] = X["MedInc"] * X["HouseAge"]

This helps the model capture relationships between variables rather than treating them independently.

---

3. Creating Ratio Features

Ratios often reveal more meaningful patterns.

X["Rooms_Per_Person"] = X["AveRooms"] / (X["Population"] + 1)

Such features are common in:

• finance
• analytics
• recommendation systems
• business intelligence

---

4. Applying Log Transformations

Large numerical features are often heavily skewed.

We can stabilise them using logarithmic transformations.

X["Log_Population"] = np.log1p(X["Population"])

This helps models learn more effectively from large-scale values.

---

5. Training with Engineered Features

After feature engineering, we retrain the model.

model.fit(X_train_fe, y_train)

The engineered dataset often performs better than the raw dataset.

---

Why Feature Engineering Matters

Feature engineering helps by:

• exposing hidden relationships
• improving data quality
• reducing noise
• simplifying model learning

For tabular machine learning, feature engineering is often more impactful than switching algorithms.

---

Where It Is Used

Feature engineering is heavily used in:

• credit scoring
• recommendation systems
• demand forecasting
• churn prediction
• Kaggle competitions

---

Key Takeaways

1. Feature engineering improves how models understand data.
2. Interaction features capture combined effects.
3. Ratio features reveal hidden numerical relationships.
4. Log transformations help reduce skewness.
5. Better features can outperform more complex models.

---

Conclusion

Feature engineering is one of the most powerful skills in machine learning, especially for structured tabular data. By transforming and combining features intelligently, we can significantly improve model performance without changing the underlying algorithm.

This strengthens the Advanced ML track in the AI with Python series — helping you move from simply training models to designing better data representations.

---

# 📦 Import Required Libraries
import numpy as np
import pandas as pd

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score


# 🧩 Load Dataset
data = fetch_california_housing()

X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target


# ✂️ Split Data
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)


# =========================================================
# 🚨 Baseline Model
# =========================================================

baseline_model = LinearRegression()
baseline_model.fit(X_train, y_train)

baseline_pred = baseline_model.predict(X_test)

print("Baseline R2 Score:", r2_score(y_test, baseline_pred))


# =========================================================
# ⚙️ Feature Engineering
# =========================================================

X_train_fe = X_train.copy()
X_test_fe = X_test.copy()

# Interaction feature
X_train_fe["MedInc_HouseAge"] = X_train["MedInc"] * X_train["HouseAge"]
X_test_fe["MedInc_HouseAge"] = X_test["MedInc"] * X_test["HouseAge"]

# Ratio feature
X_train_fe["Rooms_Per_Person"] = X_train["AveRooms"] / (X_train["Population"] + 1)
X_test_fe["Rooms_Per_Person"] = X_test["AveRooms"] / (X_test["Population"] + 1)

# Log transformation
X_train_fe["Log_Population"] = np.log1p(X_train["Population"])
X_test_fe["Log_Population"] = np.log1p(X_test["Population"])


# =========================================================
# 🤖 Model with Engineered Features
# =========================================================

fe_model = LinearRegression()
fe_model.fit(X_train_fe, y_train)

fe_pred = fe_model.predict(X_test_fe)

print("Feature Engineered R2 Score:", r2_score(y_test, fe_pred))