Nextflow Tower and how it makes it easy to manage a lot of infrastructure quickly.

Brendan Bouffler
Date : July 22, 2021
Categories : AWS ParallelCluster ,Amazon NICE DCV ,Life Sciences
Tags : cpus ,covid 19 ,dcv ,ec2 ,gpus ,hcls ,hpc ,high performance computing ,lustre ,parallel cluster ,schedulers ,storage ,genomics ,genomics pipelines ,nextflow ,nextflow tower ,nf core ,nf core ,scilife ,seqera ,seqera labs ,spot ,tower ,virtualization ,vizualization ,workflows ,techshorts

Nextflow Tower became necessary when lots of Nextflow users started to leave the confines of their existing environments in search of more compute, more storage, more capabilities. Nextflow made their workflows portable, but there was still a lot of relatively complex work to be done standing up cloud infrastructure, setting limits, choosing instances, pondering VPCs, Batch compute environments … You get the idea.

Nextflow Tower massively simplifies that. It integrates deeply with all these environments (lots of clouds, lots of cluster types, and even in hybrid mode) such that a researcher with a good idea can go from laptop to server to cluster to massive compute farm, step-by-step as their idea and ambition grows.

Evan Floden, the CEO of Seqera Labs and a core developer of Nextflow (along with his friend and business partner Paolo Di Tommaso) steps in to take us for a drive of Tower and explains some of the tech underpinning it.

If you have ideas for technical topics you’d like to see us cover in a future show, let us know by finding us on Twitter (@TechHpc) and DM’ing us with your idea.