ML in Production: Why Feature Stores Solve a Problem You May Not Have

Feature stores arrived with the authority of something obviously correct. Uber built one, then Airbnb, then everyone wrote the blog post, and within about two years “do we need a feature store” had quietly become “which feature store.”

They are good infrastructure. They solve real problems that are genuinely hard to solve by hand. The trouble is that the problems they solve are problems of organisational scale, not problems of doing machine learning — and most teams adopt one while they still have exactly one model, one team, and batch inference.

What a Feature Store Actually Is

Underneath the terminology it’s three things.

An offline store holds historical feature values for building training sets, usually sitting on your warehouse or lake. An online store holds the current value of each feature for serving at inference time, usually something low-latency like Redis, DynamoDB or Cassandra. And a registry holds the definitions — what a feature is, how it’s computed, which entity it belongs to, who owns it.

The pitch is that you define a transformation once and both paths consume it, so your training data and your production inputs cannot drift apart.

The Problem It Was Built For

Two problems, really, and one of them is much harder than the other.

Training/serving skew is the famous one. You compute user_30d_order_count in a Python notebook with pandas for training. Six weeks later someone reimplements it in the serving path, in a different language, under time pressure, and gets the window boundary subtly wrong. The model performs worse in production than it did offline and nobody can explain why for a month.

Point-in-time correctness is the deeper one. When you build a training set, every feature value has to be what it was at the moment of the label event — not what it is now. If you join today’s “average order value” onto a label from six months ago, you’ve fed the model information from the future. It will look superb in evaluation and fail in production, because at inference time the future isn’t available.

Point-in-time correctness is the real problem. Everything else is plumbing.

Getting this right by hand means every training query carries an as_of timestamp and every join respects it. It’s not conceptually difficult. It’s just unforgiving, and one careless join silently inflates your metrics.

Why You Probably Don’t Have That Problem Yet

Here’s where the obviously-correct framing starts to wobble.

If you do batch inference, skew is structurally impossible. You score on a schedule, reading the same warehouse your training reads, running the same SQL. There is no second code path, so there is nothing to drift. A large share of production ML — churn scores, risk ratings, recommendations refreshed nightly, lead scoring — is batch, and gets training/serving consistency for free.

If you have one model and one team, feature reuse isn’t a problem. Discovery and deduplication matter when four teams are independently building user_activity_score. With one team, the registry is a directory of files you already know.

If your features come from the request, point-in-time is trivial. Features like “items in cart” or “device type” are attributes of the event itself. There’s no history to reconstruct. The hard case is aggregations over time, and plenty of models don’t have many.

So the question isn’t whether a feature store is good infrastructure. It’s whether you have the specific conditions that make it necessary: aggregate features over history, and real-time inference, and more than one consumer.

What It Costs

The adoption posts are specific about benefits and vague about costs, so here are the costs.

You’ve added a system to the critical path of inference. When the online store is slow, your model is slow. When it’s down, your model is down. A component that previously didn’t exist can now take your prediction service with it.

You’ve added operational surface. The online store needs running, monitoring and capacity planning. The materialisation jobs that keep it current need scheduling, monitoring and backfilling. Staleness becomes a thing you have to measure, because a feature store serving yesterday’s values is worse than no feature store at all — it fails quietly.

You’ve added conceptual overhead. Entities, feature views, materialisation windows, time-to-live. Every engineer touching the model now learns a vocabulary that exists to serve the infrastructure rather than the problem.

And you’ve made debugging harder. “Why was this prediction wrong?” used to be a question about one pipeline. Now the value could be wrong in the transformation, stale in the online store, correct in the offline store but wrong in the online one, or fine everywhere and wrong only for this entity.

The Version of This That Scales Down

Most of the benefit is available without the platform, if you’re deliberate.

Write feature transformations once, in one place, as a shared library or a set of dbt models, and have both training and serving import it. That’s the skew fix, and it’s most of the value for most teams.

Handle point-in-time correctness explicitly. Make as_of a required argument in your training data builder rather than something each query remembers. Write one correct implementation of the time-travel join and ban ad-hoc joins against feature tables. This is a code review rule, not a product.

Prefer batch scoring until latency genuinely demands otherwise. “Real-time” is frequently a preference rather than a requirement, and the architectural cost of satisfying it is large.

Keep a feature catalogue as a document long before you keep one as a service.

When It Actually Earns Its Place

There’s a threshold, and it’s recognisable.

You cross it when several teams are shipping models that share features, and the duplication is producing genuine inconsistency rather than mild untidiness. When you need sub-100ms inference using features aggregated over months of history, so computing them at request time isn’t viable. When regulators or auditors need you to reproduce the exact inputs to a decision made eighteen months ago. When the number of models in production is large enough that onboarding a new one is dominated by feature plumbing rather than modelling.

A feature store doesn’t make ML easier. It makes ML at organisational scale possible.

Those are good reasons, and teams that have them should adopt one rather than building a worse version internally.

The failure mode is adopting it before any of that is true — spending a quarter on infrastructure that solves coordination problems you don’t have yet, while the actual blocker is that your training data has a leak nobody has found.

That leak is still there afterwards. The feature store doesn’t find it. It just makes it reproducible.