Zeke Lares
When I joined Butlr, the API served on the order of two thousand requests a month. Today it serves more than three billion, at 99% uptime. People hear that and assume the story is about raw performance — bigger machines, more caching, a faster language. It mostly wasn't. The throughput problem had known answers. The real work was sequencing a migration so that no customer, internal team, or device ever had to know it was happening.
The shape of the problem
Our original API was a constellation of AWS Lambda functions sitting behind API Gateway. For an early product that pattern is wonderful: near-zero idle cost, no servers to babysit, and you can ship an endpoint in an afternoon. It's also a pattern that quietly punishes you once traffic becomes sustained rather than spiky.
As the device fleet grew, traffic stopped looking like occasional bursts and started looking like a constant firehose. Three things began to hurt at once:
- Cold starts on the long tail of endpoints turned into latency complaints that were maddening to reproduce.
- Connection limits to downstream datastores became a real ceiling — thousands of concurrent Lambdas each wanting their own database connection is a recipe for exhaustion.
- Per-request economics that were a bargain at thousands of requests became indefensible at billions.
The destination was clear: move the hot paths onto long-lived containers on AWS ECS Fargate, where we could pool connections, keep code warm, and pay for sustained throughput instead of per-invocation. The question was never whether. It was how, without a flag day.
Rule one: no flag days
A flag day — the big-bang cutover where everyone switches at midnight and prays — is a bet that you understand every dependency perfectly. At billion-request scale that bet is a fantasy. So the first principle was that the migration had to be incremental, reversible, and observable at every step.
If you can't roll a change back in under a minute, you haven't finished designing the change.
We put a routing layer in front of everything and moved traffic one endpoint at a time, by percentage. A new Fargate-backed implementation would take 1% of an endpoint's traffic, then 10%, then 50%, then 100% — with the Lambda version still warm and one config change away from taking it all back.
Measure the thing customers feel
It is easy to migrate toward a metric that looks good on a dashboard and bad in a customer's hands. We deliberately anchored on the experience, not the average: tail latency (p99, not p50), error budgets per endpoint, and the saturation signals — connection pool usage, queue depth — that predict failure before it shows up as a 500.
p99 latency per endpoint · error rate against a per-route budget · datastore connection saturation · ingestion lag. If any of these moved the wrong way during a traffic shift, the rollout paused itself.
Tying the rollout to those signals meant the system could refuse to proceed on its own. A human deciding "looks fine, push to 100%" at 2am is exactly the kind of judgment you want to remove from the loop.
The boring middle
The most important stretch of any migration like this is the part nobody writes conference talks about: the weeks where two implementations run side by side and you reconcile their behavior down to the edge cases. Idempotency keys. Clock skew. The one client that depended on a header you forgot you emitted. We logged divergences between old and new responses and treated every single one as a bug to explain, even when "explaining" it meant deciding the old behavior was the wrong one.
This is unglamorous and it is where the uptime number actually comes from. 99% uptime at three billion requests isn't a heroic moment. It's the absence of heroic moments — a thousand small decisions that each refused to gamble.
// the migration, compressed to its essence for endpoint in hot_paths { route.shift(endpoint, to: fargate, percent: 1) // watch p99, errors, saturation — auto-pause on regression route.ramp(endpoint, steps: [10, 50, 100], guard: slo_guard) // lambda stays warm until the new path proves itself }
What I'd tell a younger engineer
Scaling stories get told as feats of throughput, but the skill that actually compounds is migration discipline: the ability to change a load-bearing system while it's load-bearing. Throughput you can buy. The confidence to change the engine mid-flight you have to build, one reversible step at a time.
The three-billion number is the headline. The real product was a team that stopped being afraid of large changes — because we'd made large changes small.