AW Dev Rethought

⚖️ There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies - C.A.R. Hoare

AWS in Production: The Services That Quietly Cost the Most


Introduction:

Most AWS cost problems don’t come from obvious mistakes. They don’t come from a massive EC2 cluster or a forgotten resource running for months. They come from services that feel harmless early on and only reveal their cost once systems settle into real production usage.

These services usually do their job well. That’s why teams don’t question them. The problem isn’t misuse — it’s accumulation. Costs grow slowly, predictably, and quietly, until they become structural.

This blog looks at where AWS costs tend to grow unnoticed in production and why teams almost always realise it later than they should.


Data Transfer: The Cost No One Designs For:

Data transfer is rarely discussed during design. Early environments are small, traffic stays local, and most communication happens within a single region.

In production, that changes. Services talk across availability zones. Replication spans regions. Traffic flows outward to users, partners, and third-party services. Each hop looks small, but together they form a steady cost stream.

The issue isn’t one large spike. It’s that data transfer becomes part of the system’s shape. Once that shape exists, removing it usually means architectural change.


Managed Databases That Never Shrink:

Managed databases are easy to justify early. They’re reliable, familiar, and operationally simple. What teams underestimate is how permanent they become.

Once a database is in production, it almost never scales down. Storage grows. Backups accumulate. Replicas stay in place “just in case.” Even when usage drops, the database footprint rarely follows.

Over time, databases stop being a design choice and start being a fixed cost. At that point, optimisation feels risky rather than responsible.


Observability That Grows Faster Than the System:

Logging and monitoring feel cheap at first. A few logs, some metrics, basic dashboards — all reasonable decisions.

As systems mature, observability expands:

  • more detailed logs
  • higher-cardinality metrics
  • distributed traces
  • longer retention periods

Each addition improves visibility. None feel optional. Together, they form one of the fastest-growing cost centers in production.

The problem isn’t observability itself. It’s adding it without cost boundaries.


Serverless That Stops Being “Pay for What You Use”:

Serverless services are often introduced with cost efficiency in mind. For irregular workloads, they work exactly as advertised.

In steady production systems, the math changes. Invocation counts stabilise. Execution time increases. Memory allocations stay conservative. Fan-out patterns multiply calls.

Nothing breaks. Nothing spikes. Costs just rise quietly as usage becomes predictable. At that point, serverless isn’t expensive — it’s just no longer cheap by default.


Storage That Outlives Its Purpose:

Storage is rarely revisited. Logs are kept “for now.” Snapshots remain “just in case.” Old datasets linger because no one owns them.

Each decision feels safe. None feel urgent. Over time, storage turns into a long-term archive of assumptions that were never revisited.

Storage costs don’t surprise teams. They simply become part of the background — and that’s why they grow unchecked.


Load Balancers and Gateways You Can’t Remove:

Once load balancers, API gateways, and ingress layers are introduced, they become permanent. Everything flows through them.

Their cost scales with:

  • request volume
  • feature usage
  • traffic growth

Because they sit at the edge, they scale with the entire system. Teams rarely question their cost because they’re foundational — but that also makes them invisible.


The Pattern Behind Quiet AWS Costs:

Across all these services, the pattern is the same.

They are:

  • inexpensive at low scale
  • justified individually
  • hard to remove later

By the time teams notice the cost, it’s tied to core workflows. Reducing it now requires redesign, not cleanup.

This is why AWS costs feel hard to control once production stabilises.


Cost Is Not a Billing Problem:

Teams that manage AWS costs well don’t start with dashboards. They start with architecture.

They ask:

  • what will grow automatically?
  • what becomes permanent once adopted?
  • who owns this service long-term?

Cost-aware systems aren’t underpowered. They’re intentional. They treat cost as a property of design, not a surprise from finance.


Conclusion:

The AWS services that cost the most in production are rarely the ones teams worry about early. They’re the quiet enablers — networking, databases, observability, and supporting infrastructure — that scale naturally with success.

Understanding where these costs hide allows teams to design systems that age well, not just systems that work today. In AWS, cost isn’t an accident. It’s the long-term outcome of many small, reasonable decisions.

Production engineering is about making those decisions consciously — before they harden into constraints.


References:

  • AWS Well-Architected Framework – Cost Optimisation Pillar (🔗 Link)
  • AWS Architecture Blog – Cost Optimisation Best Practices (🔗 Link)
  • AWS Pricing – Data Transfer (🔗 Link)
  • AWS CloudWatch Pricing (🔗 Link)

Rethought Relay:
Link copied!

Comments

Add Your Comment

Comment Added!