Product Realities: Data Freshness Is a Product Decision, Not a Tech One

Introduction:

Data freshness is almost always treated as a technical problem. Engineers debate streaming versus batch architectures, evaluate change data capture solutions, benchmark message queue throughput, and design complex pipeline orchestration — all in service of making data more current. The assumption underlying all of this work is that fresher data is better data, and that the engineering team's job is to make data as fresh as the infrastructure allows.

That assumption is wrong more often than engineering teams acknowledge. Data freshness is not a technical property to be maximised — it is a product decision to be made deliberately, based on what decisions the data enables, what the cost of staleness actually is for those decisions, and whether the engineering investment required to reduce staleness is justified by the value it delivers.

Treating freshness as a product decision rather than a technical default produces systems that are appropriately fresh for their purpose — not maximally fresh at unnecessary cost.

Freshness Requirements Vary Dramatically by Use Case:

A fraud detection system that evaluates transactions in real time has a freshness requirement measured in milliseconds. A dashboard showing weekly revenue trends has a freshness requirement measured in hours. A report used for quarterly business reviews has a freshness requirement measured in days. These are not different points on the same spectrum — they are fundamentally different requirements that justify fundamentally different engineering investments.

The mistake engineering teams make is applying the freshness standard of their most demanding use case to all use cases. If fraud detection needs millisecond freshness, the instinct is to build a streaming infrastructure that serves that requirement and use it everywhere — because consistency seems cleaner than maintaining multiple pipelines with different freshness characteristics.

This produces systems that are more expensive to build, more complex to operate, and harder to debug than necessary for the majority of use cases they serve. The fraud detection system justifies the streaming infrastructure. The quarterly business review report does not.

The Cost of Staleness Is Rarely Calculated:

Engineering teams that invest in reducing data freshness rarely calculate the actual cost of staleness they are eliminating. The assumption is that staleness is bad and freshness is good, and that the investment in freshness is therefore justified. But the cost of staleness depends entirely on what the data is used for and what decisions are made from it.

If a marketing dashboard shows campaign performance data that is six hours old, what is the actual cost? Marketing managers who review the dashboard in the morning are looking at data from the previous day regardless of whether it is six hours old or six minutes old. The decisions they make — adjusting bids, modifying targeting, reallocating budget — happen on a timescale where six hours of staleness is irrelevant.

Calculating the actual cost of staleness for specific use cases consistently reveals that most data consumers are making decisions on timescales where moderate staleness has no meaningful impact. The engineering investment in reducing that staleness produces infrastructure complexity without producing better decisions.

Real-Time Infrastructure Has Real Ongoing Costs:

Streaming architectures that deliver near-real-time data freshness are significantly more expensive to build and operate than batch pipelines. Apache Kafka clusters require careful capacity planning, replication configuration, and ongoing operational attention. Stream processing jobs require state management, checkpointing, and recovery procedures that batch jobs do not. The operational expertise required to run streaming infrastructure reliably is specialised and expensive.

These costs are ongoing. A batch pipeline that runs nightly has a predictable operational cost. A streaming pipeline runs continuously, consumes resources continuously, and requires continuous monitoring. When something goes wrong — a consumer falls behind, a partition becomes unavailable, a schema change breaks a stream processor — the failure affects all downstream consumers immediately rather than being caught during a scheduled batch run.

Product decisions about freshness requirements need to account for these ongoing costs explicitly. A product manager who requests real-time data without understanding that it requires a fundamentally different and significantly more expensive infrastructure than hourly batch processing is not making a fully informed decision.

Staleness Thresholds Should Be Defined by Business Impact:

The right freshness requirement for any dataset is the staleness threshold beyond which business decisions are meaningfully degraded. For fraud detection, that threshold is seconds. For inventory management in a high-velocity e-commerce system, it might be minutes. For financial reporting, it might be hours. For strategic planning data, it might be days.

Defining these thresholds requires conversation between engineering and the business stakeholders who use the data. Engineers bring knowledge of what freshness levels are achievable and at what cost. Business stakeholders bring knowledge of the decisions being made from the data and how those decisions are affected by staleness. Neither party can define the threshold correctly without the other.

Organisations that establish these thresholds explicitly — and document them as product requirements rather than engineering targets — make better infrastructure decisions and avoid the trap of building for freshness levels that the business never needed.

Near-Real-Time Is Often Good Enough:

For use cases that genuinely require more freshness than hourly batch processing provides, near-real-time — data that is five to fifteen minutes old — is frequently sufficient and significantly less expensive to achieve than true real-time streaming.

Micro-batch architectures using tools like Apache Spark Structured Streaming or scheduled pipeline runs at short intervals can achieve near-real-time freshness without the full operational complexity of event-driven streaming infrastructure. The tradeoff — accepting five minutes of latency instead of five seconds — is irrelevant for most use cases and significant for very few.

Engineering teams that evaluate near-real-time as an intermediate option between batch and streaming consistently find that it satisfies freshness requirements for the majority of use cases at a fraction of the operational cost of true streaming architectures.

Conclusion:

Data freshness decisions made by engineering teams in isolation consistently produce infrastructure that is more complex and more expensive than the business actually needs. Freshness requirements that are defined by product teams in isolation consistently produce systems that are either over-engineered for simple use cases or under-specified for genuinely demanding ones.

The right freshness for any data system emerges from a deliberate conversation about what decisions the data enables, what the actual cost of staleness is for those decisions, and what infrastructure investment is justified by the value that reduced staleness delivers. That conversation is a product conversation, not a technical one — and engineering teams that treat it as technical consistently build systems that are more sophisticated than necessary and less aligned with what the business actually needs.