There is no exactly-once delivery in asynchronous task queues, at least not in large-scale real-world systems. Even though many online guides and tutorials lead you to believe it exists, it’s logically impossible. This is the single most important lesson I wish someone had shared with us six years ago when we started running Celery on AWS ECS.
In practice, you can get very close to exactly-once delivery depending on your specific use case. To do this, you have to choose which side of the limit you approach: at-most-once or at-least-once delivery. Each side comes with its own trade-offs and subtleties, which is why you have to design your system very deliberately and can’t just take the default settings of any framework like Celery.
Celery is one of the most widely used task queues in Python, but I think its defaults aren’t great for most applications. Therefore, it takes a lot of work and experience to set it up to run reliably. We paid for most of these lessons with many hours of downtime and incident response, before finally arriving at a setup that works well for us and runs hundreds of thousands of tasks per day.
Our lead backend engineer Jan Giacomelli has now consolidated all these hard-learned lessons into a definitive guide for running Celery on AWS ECS. It covers all the gotchas, from why tasks get lost, stuck, or processed twice in the first place to the settings and task design patterns you can use to prevent them.
I wish this guide had existed back then: https://jangiacomelli.com/blog/celery-on-aws-ecs/