AW Dev Rethought

🕵️ Debugging is like being the detective in a crime movie where you are also the murderer - Filipe Fortes

📊 Python Data Workflows – ⚙️ Mini Project 🐍


Description:

So far in this series, we’ve explored individual parts of a data workflow. We’ve loaded CSV files, cleaned datasets, created features, validated records, queried SQL databases, and fetched API data. Now it’s time to bring everything together into a small end-to-end project.


What Does an End-to-End Workflow Look Like?

A typical workflow follows several stages:

  • Load data
  • Clean data
  • Validate records
  • Create features
  • Generate insights
  • Export results

Instead of treating these as separate tasks, they work together as one continuous process.


Cleaning and Validation

The first responsibility of a workflow is ensuring data quality.

df = df.drop_duplicates()
invalid_sales = df[df["sales"] < 0]

These checks prevent inaccurate reporting later.


Feature Engineering

Raw data often lacks context.

df["revenue_per_item"] = df["sales"] / df["quantity"]

New features help uncover patterns that may not be visible from the original dataset.


Insight Generation

Once the data is ready, analysis becomes much more reliable.

df.groupby("category")["sales"].sum()

This helps answer business questions such as:

  • Which category performs best?
  • Which region generates the most revenue?
  • What are the average sales per order?

Exporting Results

The final step is making the results usable.

category_summary.to_csv("category_summary.csv")

Processed datasets and summary reports can then be shared with stakeholders or used in dashboards.


Key Takeaways

  • Data workflows are made up of connected stages
  • Quality checks should happen before analysis
  • Features add business value to raw data
  • Reports and exports are an important final step

Code Snippet:

import pandas as pd


# Step 1 — Load Dataset
df = pd.read_csv("sales_data.csv")

print("Dataset Loaded\n")


# Step 2 — Clean Data
df = df.drop_duplicates()

df["sales"] = pd.to_numeric(
    df["sales"],
    errors="coerce"
)

df["quantity"] = pd.to_numeric(
    df["quantity"],
    errors="coerce"
)

df["sales"] = df["sales"].fillna(
    df["sales"].median()
)

df["quantity"] = df["quantity"].fillna(
    df["quantity"].median()
)

print("Data Cleaned\n")


# Step 3 — Validate Records
invalid_sales = df[df["sales"] < 0]

print("Invalid Sales Records:")
print(invalid_sales, "\n")


# Step 4 — Feature Engineering
df["revenue_per_item"] = (
    df["sales"] / df["quantity"]
)

df["sales_category"] = df["sales"].apply(
    lambda x: "High"
    if x >= 2000
    else "Medium"
    if x >= 1000
    else "Low"
)

print("Features Created\n")


# Step 5 — Category Insights
category_summary = (
    df.groupby("category")
      .agg(
          total_sales=("sales", "sum"),
          avg_sales=("sales", "mean"),
          total_orders=("order_id", "count")
      )
      .reset_index()
)

print("Category Summary:")
print(category_summary, "\n")


# Step 6 — Regional Insights
region_summary = (
    df.groupby("region")["sales"]
      .sum()
      .sort_values(ascending=False)
)

print("Region Summary:")
print(region_summary, "\n")


# Step 7 — Export Reports
df.to_csv(
    "processed_sales_data.csv",
    index=False
)

category_summary.to_csv(
    "category_summary.csv",
    index=False
)

print("Reports Exported\n")


# Key Takeaways
print("Key Takeaways:")
print("Workflows involve multiple connected stages")
print("Clean data leads to better insights")
print("Features improve analytical value")
print("Reports make insights shareable\n")


# Final Note
print(
    "Raw data becomes valuable when transformed into actionable insights!"
)

Link copied!

Comments

Add Your Comment

Comment Added!