Enabling complex scientific computations and efficient use of HPC resources with scientific workflows
Karan will bring the use of workflows to the wider scientific community by developing easy to understand training materials that examine the workflow lifecycle and challenges associated with various steps such as creation, execution, monitoring and debugging. Workflows are needed to capture the complex interdependencies between processing steps in data analysis and simulation pipelines as well as the mechanisms to execute those steps reliably and efficiently. The training will walk users through how to model existing simulation pipelines into workflows, how to package application code in containers, and how to execute the workflow on HPC resources and distributed computing infrastructure such as Open Science Grid. The training materials will also build on existing interactive Jupyter notebooks that guide users on how to develop workflows using Pegasus (http://pegasus.isi.edu). The training materials will be self-guided and also well-suited for use in classroom teaching and virtually.
Karan Vahi is a Senior Computer Scientist in the Science Automation Technologies group at the USC Information Sciences Institute. He has been working in the field of scientific workflows since 2002, and has been closely involved in the development of the Pegasus Workflow Management System. He is currently the architect/lead developer for Pegasus and in charge of the core development of Pegasus. His work on implementing integrity checking in Pegasus for scientific workflows won the best paper and the Phil Andrews Most Transformative Research Award at PEARC19.
Selected Resources
Pegasus tutorial abstract Pegasus tutorial materials Containerize setup for Pegasus and HTCondor Containers for Deploying Workflow Systems and Application Codes
