There is no exactly-once delivery in asynchronous task queues, at least not in large-scale real-world systems. Even though many online guides and tutorials lead you to believe it exists, it’s logically impossible. This is the single most important lesson I wish someone had shared with us six years ago when we started running Celery on AWS ECS.
In practice, you can get very close to exactly-once delivery depending on your specific use case. To do this, you have to choose which side of the limit you approach: at-most-once or at-least-once delivery. Each side comes with its own trade-offs and subtleties, which is why you have to design your system very deliberately and can’t just take the default settings of any framework like Celery.
Celery is one of the most widely used task queues in Python, but I think its defaults aren’t great for most applications. Therefore, it takes a lot of work and experience to set it up to run reliably. We paid for most of these lessons with many hours of downtime and incident response, before finally arriving at a setup that works well for us and runs hundreds of thousands of tasks per day.
Our lead backend engineer Jan Giacomelli has now consolidated all these hard-learned lessons into a definitive guide for running Celery on AWS ECS. It covers all the gotchas, from why tasks get lost, stuck, or processed twice in the first place to the settings and task design patterns you can use to prevent them.
I wish this guide had existed back then: https://jangiacomelli.com/blog/celery-on-aws-ecs/
Have you ever wondered what “the cloud” actually looks like? It’s a lot more physical than it sounds.
This Google Maps screenshot shows the so-called “Data Center Alley” in Ashburn, Northern Virginia—a cluster of warehouse-sized data centers located between a golf course and a few suburbs that look like they’re straight out of an American movie. AWS’s famous us-east-1 lives here, along with data centers from Microsoft, Google, Meta, IBM, Oracle, and many others. According to a frequently cited estimate from the local economic development office, around 70% of the world’s internet traffic passes through here. I have some doubts about whether this estimate is still accurate, but it is likely the world’s largest concentration of digital infrastructure.
As a software engineer or AI builder, it’s easy to forget that whatever services you call or build on, your code is actually moving photons through optical fibers and electrons across silicon somewhere, drawing real power from an electrical grid. AI is progressing really quickly right now, but the underlying physical infrastructure imposes constraints on how much of that progress is actually deployable.