Building TByte/s Lustre without pain or remorse

Brendan Bouffler
Date : May 2, 2024
Categories : AI/ML ,AWS Batch ,AWS ParallelCluster ,Amazon NICE DCV ,Elastic Fabric Adapter ,Life Sciences
Tags : ai ml ,aws ,batch ,cpus ,dcv ,ec2 ,efa ,gpus ,hpc ,high performance computing ,lustre ,mpi ,nccl ,parallel cluster ,schedulers ,storage ,amazon fsx ,autoscaling ,aws batch ,bioinformatics ,cloud computing ,elastic ,elastic fabric adapter ,fault tolerance ,hpc instances ,infiniband ,job scheduling ,lustre ,scientific computing ,supercomputing ,technical computing ,tightly coupled ,virtualization ,vizualization ,techshorts

Are you interested in building blazing fast, petabyte-scale storage systems? Today we show you how to easily create a 1 TERABYTE per second Lustre file system on AWS in under an hour.

See how to leverage Amazon FSx for Lustre to instantly deploy this high performance parallel file system without any of the typical complexity or hardware management.

Watch as we launch a 100-node EC2 fleet to drive over 1TB/sec of I/O to the FSx Lustre file system.

Whether you need lightning fast scratch storage for HPC or a performant backend for ML training, this video will show you how FSx makes it almost trivial to deploy Lustre at massive scale.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by emailing us at ask-hpc@amazon.com.