Linking Lustre and object storage - Data repository associations in Amazon FSx

Brendan Bouffler
Date : September 19, 2024
Categories : AI/ML ,AWS Batch ,AWS ParallelCluster ,Amazon NICE DCV ,Elastic Fabric Adapter ,Life Sciences
Tags : ai ml ,aws ,batch ,cpus ,dcv ,ec2 ,efa ,gpus ,hpc ,high performance computing ,lustre ,mpi ,nccl ,parallel cluster ,schedulers ,storage ,autoscaling ,aws batch ,bioinformatics ,cloud computing ,elastic ,elastic fabric adapter ,hpc instances ,infiniband ,job scheduling ,scientific computing ,supercomputing ,technical computing ,tightly coupled ,virtualization ,vizualization ,techshorts

HPC and machine learning are increasingly being used to solve complex problems, but working with massive datasets can be prohibitively expensive.

Today’s Tech Short shows how to elevate data in S3 into a high speed POSIX filesystem in just two clicks. Simply specify the S3 path and filesystem location to create an association. The entire bucket structure becomes visible as metadata in the filesystem, while the data remains in low-cost S3 storage.

When files are accessed, they are efficiently streamed from S3 on-demand. Pre-loading and caching commands allow optimizing read performance.

If you need to efficiently process huge datasets for HPC, machine learning or other applications, watch this video to see how FSx for Lustre can save you time and money by seamlessly integrating S3 object storage with a high performance filesystem.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by emailing us at ask-hpc@amazon.com.