Sending Celery tasks to AWS Batch for heavy lifting
Celery is a great distributed task processing system and gets used in websites and enterprise apps everywhere. But sometimes those tasks aren’t little things that can be handled by microservices - they might need some heavy CPU or machine-learning element that requires some heavy lifting on some dedicated hardware.
That’s where Angel’s Celery integration for AWS comes in. He blogged about it last week, and came into the Tech Shorts studio to explain how it works, and show it to us in action.
… and this contains links to his repo on GitHub, too.
You can find Angel on twitter here: https://twitter.com/delagoya if you want to find out more.
If you have ideas for technical topics you’d like to see us cover in a future show, let us know by finding us on Twitter (@TechHpc) and DM’ing us with your idea.