📊 Python Data Workflows – 🔄 ETL Pipeline 🐍

Posted on: July 3, 2026

Description:

Every data-driven application relies on one fundamental process: ETL.

ETL stands for Extract → Transform → Load, a workflow that collects raw data, prepares it, and stores it in a usable format for reporting or analytics.

What is ETL?

An ETL pipeline consists of three stages:

Extract – Read data from one or more sources.
Transform – Clean, validate, and enrich the data.
Load – Save the processed data for analysis or reporting.

Instead of manually repeating these tasks, ETL automates the entire workflow.

Extract

The first step is reading the raw dataset.

df = pd.read_csv("sales_data.csv")

This data may come from CSV files, databases, APIs, or cloud storage.

Transform

The transformation stage prepares the data.

df = df.drop_duplicates()
df["sales"] = df["sales"].fillna(df["sales"].median())

Additional features can also be created to improve analysis.

Load

Finally, the processed data is stored for downstream systems.

df.to_csv("processed_sales_data.csv")

The output can be consumed by dashboards, reports, or machine learning models.

Key Takeaways

ETL is a core data engineering workflow.
Data transformation improves quality and usability.
Processed data is easier to analyze and share.
ETL pipelines automate repetitive data processing tasks.

Code Snippet:

import pandas as pd


# -------------------------------
# Step 1 — Extract Data
# -------------------------------
df = pd.read_csv("sales_data.csv")

print("✅ Data Extracted\n")
print(df.head(), "\n")


# -------------------------------
# Step 2 — Inspect Dataset
# -------------------------------
print("📌 Dataset Shape:")
print(df.shape)

print("\n📌 Missing Values:")
print(df.isnull().sum())

print("\n📌 Data Types:")
print(df.dtypes)


# -------------------------------
# Step 3 — Transform Data
# -------------------------------
df = df.drop_duplicates()

df["sales"] = pd.to_numeric(df["sales"], errors="coerce")
df["quantity"] = pd.to_numeric(df["quantity"], errors="coerce")

df["sales"] = df["sales"].fillna(df["sales"].median())
df["quantity"] = df["quantity"].fillna(df["quantity"].median())

print("\n✅ Data Cleaned")


# -------------------------------
# Step 4 — Feature Engineering
# -------------------------------
df["revenue_per_item"] = df["sales"] / df["quantity"]

df["sales_category"] = df["sales"].apply(
    lambda x: "High"
    if x >= 2000
    else "Medium"
    if x >= 1000
    else "Low"
)

print("✅ Features Created")


# -------------------------------
# Step 5 — Generate Summary
# -------------------------------
summary = (
    df.groupby("category")
      .agg(
          total_sales=("sales", "sum"),
          average_sales=("sales", "mean"),
          total_orders=("order_id", "count")
      )
      .reset_index()
)

print("\n📊 Sales Summary")
print(summary)


# -------------------------------
# Step 6 — Load Data
# -------------------------------
df.to_csv("processed_sales_data.csv", index=False)

summary.to_csv("sales_summary.csv", index=False)

print("\n💾 Processed data saved")
print("💾 Summary report saved")


# -------------------------------
# Key Takeaways
# -------------------------------
print("\n📌 Key Takeaways:")
print("🔹 ETL automates data processing")
print("🔹 Transformation improves data quality")
print("🔹 Feature engineering adds business value")
print("🔹 Processed datasets are ready for analytics")


# -------------------------------
# Final Note
# -------------------------------
print("\n🚀 ETL is the foundation of every modern data pipeline!")

← →	move
↑	rotate
↓	soft drop
Space	hard drop
P	pause / resume

📊 Python Data Workflows – 🔄 ETL Pipeline 🐍

Description:

What is ETL?

Extract

Transform

Load

Key Takeaways

Code Snippet:

Comments

Add Your Comment

📊 Python Data Workflows – 🔄 ETL Pipeline 🐍

Description:

What is ETL?

Extract

Transform

Load

Key Takeaways

Code Snippet:

Comments Show Comments

Add Your Comment

Related Posts

📊 Python Data Workflows – ⏰ Scheduling Jobs 🐍

📊 Python Data Workflows – ⚙️ Mini Project 🐍

📊 Python Data Workflows – ✅ Data Validation 🐍

7-Day AI Crash Course

Comments