A deep dive into HTC Grid on AWS

Brendan Bouffler
Date : October 5, 2023
Categories : AI/ML ,AWS Batch ,AWS ParallelCluster ,Amazon NICE DCV ,Elastic Fabric Adapter ,Financial Services ,Life Sciences
Tags : ai ml ,aws ,batch ,cpus ,dcv ,ec2 ,efa ,fsi ,gpus ,hpc ,htc ,high performance computing ,lustre ,mpi ,nccl ,parallel cluster ,schedulers ,storage ,autoscaling ,aws batch ,banking ,bioinformatics ,cloud computing ,elastic ,elastic fabric adapter ,financial services ,grid ,grid scheduler ,hpc instances ,infiniband ,job scheduling ,scientific computing ,supercomputing ,technical computing ,tightly coupled ,virtualization ,vizualization ,techshorts

FSI customers frequently start with a blank page when writing their plans for shifting HPC workloads into the cloud. Our solution architects saw a pattern tho - lots of customers trying to solve the same problems we’d witnessed elsewhere.

So Carlos Manzanedo Rueda, Clement Rey, and Kirill Bogdanov created ‘HTC Grid’. It’s an open source package you can download and use. It’ll get you much closer to your goal than starting from scratch. It navigates you around the problems that slowed other people down. And it’s highly customizable.

Kirill stopped by to explain the architecture and what drove some of their decisions.

You can find out more about the FSI HPC space from a previous talk by Alex Kimber - our chief HPC FSI nerd.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by emailing us at ask-hpc@amazon.com.