Lustre has gone mainstream - How Amazon File Cache works

Brendan Bouffler
Date : March 2, 2023
Categories : AWS ParallelCluster ,Amazon NICE DCV ,Elastic Fabric Adapter ,Life Sciences
Tags : cpus ,dcv ,ec2 ,efa ,gpus ,hpc ,high performance computing ,lustre ,mpi ,parallel cluster ,schedulers ,storage ,amazon file cache ,autoscaling ,aws ,bioinformatics ,cloud ,cloud computing ,elastic ,elastic fabric adapter ,file cache ,infiniband ,scientific computing ,technical computing ,tightly coupled ,virtualization ,vizualization ,techshorts

Amazon File Cache is a service that’s bringing Lustre mainstream (yes, we know that sounds crazy). Lustre has been serving the HPC community for a few decades now, and it’s weird and wild that it’s now being used as a super high performance caching layer for other more … sedate file services like Amazon S3 and NFS to provide an insane performance boost.

Darryl Osborne from the engineering team that created Amazon File Cache came to Tech Shorts to explain how this all works, and show it to us in action.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by finding us on Twitter (@TechHpc) and DM’ing us with your idea.