ML in Production: Why the Real Work Is Data Engineering

Introduction:

The public narrative around machine learning in production is dominated by models. New architectures, improved training techniques, larger parameter counts, better benchmark scores — the field's attention is focused almost entirely on the model as the unit of progress. Engineers who want to work in machine learning invest in understanding neural network architectures, optimisation algorithms, and evaluation frameworks.

The reality of operating machine learning systems in production is different. The model is a small fraction of the total engineering work. The majority of the effort — the work that determines whether a machine learning system actually delivers value in production — is data engineering. Collecting data, cleaning it, validating it, transforming it, versioning it, serving it at low latency, and monitoring it for quality and drift consumes far more engineering time than training and deploying models.

Teams that underestimate the data engineering requirements of production machine learning consistently find that their models work and their systems do not.

The Model Is the Easy Part:

Training a machine learning model on a clean, well-structured dataset is a solved problem for most common use cases. The tools are mature, the techniques are well-documented, and the engineering effort required to produce a working model is modest compared to the effort required to produce the data the model trains on.

A model that achieves acceptable performance on a clean dataset can be trained in hours or days. Producing the clean dataset that enables that training may take weeks or months — collecting data from production systems, resolving quality issues, handling missing values, encoding categorical variables, normalising distributions, and constructing the labels that supervised learning requires.

Engineers who spend months on data preparation and days on model training are not doing something wrong. They are experiencing the actual ratio of effort that production machine learning requires. The dissonance between this reality and the model-centric narrative of the field causes teams to understaff data engineering, underinvest in data infrastructure, and then wonder why their machine learning initiatives take longer than expected and deliver less than promised.

Garbage In, Garbage Out Is Not a Cliché:

The relationship between data quality and model quality is direct and unforgiving. A model trained on data that contains labelling errors will learn those errors. A model trained on data that reflects historical biases will reproduce those biases. A model trained on data that does not represent the distribution it will serve in production will perform poorly on that distribution regardless of how sophisticated its architecture is.

These data quality problems are not visible in the model. A model trained on mislabelled data produces predictions with the same confidence as a model trained on correctly labelled data. A model trained on biased data does not signal that its outputs reflect that bias. The data quality problems that exist in the training pipeline emerge as model behaviour problems in production — and diagnosing them requires tracing back through the data pipeline to find the source, not examining the model itself.

Data validation at every stage of the pipeline — schema validation, statistical distribution checks, label quality audits, and consistency verification across data sources — is the engineering practice that catches these problems before they propagate into model behaviour. It is also the practice most consistently skipped when teams are under pressure to deliver a model quickly.

Feature Engineering Determines What the Model Can Learn:

Raw data rarely contains the signals that make machine learning models effective. Feature engineering — transforming raw data into representations that expose the patterns the model needs to learn — is where domain knowledge meets machine learning, and it is one of the highest-leverage activities in the entire machine learning pipeline.

A well-engineered feature set allows a simple model to outperform a complex model trained on raw data. The velocity of transactions on an account in the last hour is a more informative feature for fraud detection than the raw transaction history. The ratio of a product's current price to its historical average is a more informative feature for demand forecasting than the raw price. These features are not produced by the model — they are produced by data engineers who understand both the domain and the learning problem.

Feature engineering also introduces some of the most persistent failure modes in production machine learning. Features that are computed differently at training time and serving time cause training-serving skew. Features that depend on data that is not available at prediction time cause data leakage. Features that encode information about future events into training examples cause models that appear to perform well in evaluation and fail in production.

Data Pipelines Are Production Infrastructure:

In many organisations, data pipelines are treated as supporting infrastructure rather than production systems. They run on best-effort schedules, lack proper monitoring, have no formal on-call ownership, and are documented informally if at all. This treatment is inconsistent with the role they play — data pipelines that fail or produce incorrect output directly affect the quality of every model that depends on them.

A training pipeline that fails silently produces a model trained on stale or incomplete data. A feature pipeline that introduces errors produces incorrect inputs to every prediction made by the model it serves. A data validation pipeline that is disabled to meet a deadline allows quality problems to propagate through the entire system undetected.

Treating data pipelines with the same operational rigour as production services — with monitoring, alerting, on-call ownership, run-books, and incident response procedures — is the engineering practice that keeps the data infrastructure reliable enough to support machine learning systems that users can trust.

Labelling Is an Engineering Problem, Not Just an Annotation Task:

Supervised machine learning requires labelled data — examples paired with the correct output the model should learn to produce. Obtaining those labels at scale is one of the most significant engineering challenges in production machine learning, and it is consistently underestimated as a purely operational task rather than an engineering one.

Labelling pipelines that route examples to human annotators, track annotation quality, resolve disagreements between annotators, and feed confirmed labels back into training pipelines are significant engineering systems. Active learning pipelines that identify which examples would be most valuable to label, and prioritise those for human annotation, require sophisticated engineering to implement correctly. Weak supervision approaches that generate labels programmatically from heuristics and existing knowledge bases require careful quality validation to ensure the generated labels are reliable enough for training.

The engineering complexity of labelling infrastructure is comparable to the engineering complexity of the model training infrastructure it feeds. Teams that treat labelling as an annotation task rather than an engineering problem consistently find that their labelling pipelines are a bottleneck that slows model iteration more than any model-level challenge.

Conclusion:

Machine learning in production is mostly data engineering because models are only as good as the data they train on, the features they receive, and the pipelines that deliver both reliably. The model is the visible output of a machine learning system. The data infrastructure is what makes that output trustworthy, current, and consistent enough to deliver real value.

Teams that invest in data engineering with the same seriousness they invest in model development build machine learning systems that work in production. Teams that treat data engineering as supporting infrastructure for the real work of model development build models that work in evaluation and disappoint in production — not because the models are wrong but because the foundation they depend on was never built to production standards.