🧠 AI with Python – 🔧 ColumnTransformer for Mixed Data

Real-world datasets are rarely clean or uniform. Most of the time, they contain a mix of numeric and categorical features — such as age, income, gender, location, product type, and more.

Handling these different data types correctly is one of the most important steps in building a production-ready machine learning system.

In this project, we use ColumnTransformer to apply appropriate preprocessing steps to different types of features within a single pipeline.

Understanding the Problem

Machine learning models typically expect numeric input.

However, real-world datasets contain:

• Numeric features → require scaling, normalisation
• Categorical features → require encoding
• Missing values → require different handling strategies

If preprocessing is done manually:

• It becomes error-prone
• It may differ between training and inference
• It leads to inconsistent results

We need a structured way to apply different transformations to different columns.

What Is ColumnTransformer?

ColumnTransformer allows us to:

• Select specific columns
• Apply different preprocessing pipelines to each group
• Combine the results into a single transformed dataset

In simple terms:

“Apply the right transformation to the right column — automatically.”

1. Identifying Column Types

We first separate numeric and categorical columns.

numeric_features = ["age", "income"]
categorical_features = ["gender", "city"]

This step is essential for applying the correct transformations.

2. Building Separate Preprocessing Pipelines

We define transformations for each type of data.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

Each pipeline is specialised for its data type.

3. Combining with ColumnTransformer

We merge both pipelines into one unified preprocessing step.

from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
])

Now preprocessing is structured, reusable, and automated.

4. Integrating with a Model Pipeline

We attach the preprocessing step to a machine learning model.

model_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("model", RandomForestClassifier())
])

This creates a full end-to-end pipeline.

Why This Approach Matters

ColumnTransformer solves several real-world challenges:

• Ensures consistent preprocessing
• Prevents manual errors
• Handles mixed data types correctly
• Supports deployment workflows
• Works seamlessly with hyperparameter tuning

It is a fundamental building block for production ML systems.

Key Takeaways

1. Real-world datasets contain mixed feature types.
2. ColumnTransformer applies correct transformations per column group.
3. Separate pipelines keep preprocessing clean and modular.
4. Combining with Pipeline ensures end-to-end consistency.
5. A critical concept for scalable production ML systems.

Conclusion

ColumnTransformer enables us to handle mixed data types in a structured and production-ready way. By applying specialized preprocessing to numeric and categorical features within a unified workflow, we eliminate inconsistencies and build more reliable machine learning systems.

This makes it a key component in the Production ML track of the AI with Python series — bridging the gap between experimentation and real-world deployment.