Sending Celery tasks to AWS Batch for heavy lifting

Brendan Bouffler
Date : May 11, 2023
Categories : AI/ML ,AWS ParallelCluster ,Amazon NICE DCV ,Elastic Fabric Adapter ,Life Sciences
Tags : cpus ,dcv ,ec2 ,efa ,gpus ,hpc ,high performance computing ,lustre ,mpi ,parallel cluster ,schedulers ,storage ,autoscaling ,bioinformatics ,celery ,cloud computing ,distributed tasks ,elastic ,elastic fabric adapter ,infiniband ,machine learning ,scientific computing ,technical computing ,tightly coupled ,virtualization ,vizualization ,web services ,techshorts

Celery is a great distributed task processing system and gets used in websites and enterprise apps everywhere. But sometimes those tasks aren’t little things that can be handled by microservices - they might need some heavy CPU or machine-learning element that requires some heavy lifting on some dedicated hardware.

That’s where Angel’s Celery integration for AWS comes in. He blogged about it last week, and came into the Tech Shorts studio to explain how it works, and show it to us in action.

The blog is here: https://aws.amazon.com/blogs/hpc/run-celery-workers-for-compute-intensive-tasks-with-aws-batch/

… and this contains links to his repo on GitHub, too.

You can find Angel on twitter here: https://twitter.com/delagoya if you want to find out more.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by finding us on Twitter (@TechHpc) and DM’ing us with your idea.