Understanding unique challenges and practical solutions of reproducibility in High-Performance Computing (HPC).
We are excited to announce our call for presentations/abstracts for the upcoming Workshop on Practical Reproducibility in HPC, to be held on November 18, 2024, in Atlanta, GA. This workshop aims to address the unique challenges of reproducibility in High-Performance Computing (HPC) and advance practical solutions for those challenges. Sign up or apply to present at reproduciblehpc.org.
| Event Information | Details |
|---|---|
| Event Name | Community Workshop on Practical Reproducibility in HPC |
| Event Date | Nov. 18, 2024 |
| Website | https://reproduciblehpc.org |
| Call for Presentations Deadline | Oct. 18, 2024 |
| Early Bird Registration Deadline | Oct. 25, 2024 |
| Registration and Other Information | https://www.eventbrite.com/e/community-workshop-on-practical-reproducibility-in-hpc-tickets-1000658535617 |
Description
This workshop -- co-located with a major Computer Science conference taking place in Atlanta, GA on November 18, 2024 -- aims to address the unique challenges of reproducibility in High-Performance Computing (HPC) and advance practical solutions for those challenges.
There is a broad agreement that in the digital age science should be shared digitally -- through artifacts such as code and data -- and that adopting practices enabling or facilitating reproducibility of computational results can lead to more robust science and increased scientific productivity. Many conferences encourage this practice through artifact evaluation initiatives -- yet, the tools, services, and practices that support both authors and reviewers in sharing the research are still often either insufficient or lacking. This is particularly true in high performance computing (HPC) where packaging experiments for reproducibility presents unique challenges given their need for deep reconfiguration; unique and cutting edge hardware; and significant scales.
The objective of this workshop is to bring together experiment authors and reviewers with experience or insight into packaging and reviewing computer science experiments on open platforms to discuss challenges, share ideas and tips, and formulate a path forward on how to best support reproducibility for computer science research. Since many such initiatives were supported by the Chameleon platform (www.chameleoncloud.org), this workshop will also double as a Chameleon User Meeting with the objective to bring together both existing, potential, and future users of the platform.
As in previous Chameleon User Meetings, the organizers will reimburse travel expenses of up to $1,500 for the presenting authors of the top 10 selected presentation abstracts (one author per abstract). Please, take a look at the Call For Presentations below for details.
Call for Presentation Proposal Guidelines
Presentation proposals should be in PDF format (2 pages maximum), submitted (by Oct. 18, 2024) to presentations@chameleoncloud.org, and should include the following information:
Preamble:
- Presentation title
- Presenter name(s) and affiliation(s)
- Reproducibility platforms/tools used or projected (e.g., Chameleon, other testbed, containers, notebooks)
Proposal Body Sections:
- Project Description (one paragraph or less): A brief but clear description of the Computer Science experiment you are packaging or reproducing: What research problem does it address? (What is the hypothesis? What is the challenge/trade-off? Why is it significant?) How does the experiment capture this research problem?
- Reproducibility Approach (one paragraph or less): What is the description of the experiment you are either packaging or reproducing? What resources does it need and how many? How do those resources need to be configured? What is the experimental environments in which the experiment should run? How are they configured? What does the experiment body consist of? How long and in what manner is the experiment run? What types of data does it produce and how much? How is the data analyzed? What platforms or tools did you use in your experiment?
- Lessons Learned: Good and bad. What were the most important obstacles to either packaging for reproducibility or reproducing the experiment you encountered – and how did you overcome them? Were they on the infrastructure level (hardware availability), configuration level (how the hardware is configured, i.e., “creating an MPI cluster”), executing the experimental workflow, managing or analyzing data? What strategies were effective? How did you decide (or what guidelines did you give to reviewers) as to when the experiment is reproduced? What suggestions or wishes do you have on how support for reproducibility should improve? Specifically, what should improve in (1) the ability to set up an experimental environment, (2) the ability to manage the experiment body, and (3) data analysis? Which of those desired capabilities are the most important ones? Any procedural insights on how to organize reproducibility initiatives?
- Impact and Future Directions (one paragraph or less): How has prioritizing reproducibility influenced your HPC research or applications? How can the HPC community better integrate reproducibility into mainstream research and education?
Selection Criteria
Presentations will be selected based on their relevance to HPC-specific reproducibility challenges, potential for fostering discussion, and the insights they offer to the community. We particularly encourage submissions that:
- Have an insightful, detailed, and well communicated “lessons learned” section
- Have a good discussion/analysis of specific reproducibility challenges in HPC
- Share experiences from HPC-focused reproducibility initiatives
Travel Support
For the top 10 selected abstracts, we will reimburse travel expenses of up to $1,500 for the presenting authors (one per abstract). Submitting the presentation proposal doubles as a travel support application.
Workshop Outcomes
The workshop aims to produce a report capturing the community's collective knowledge and recommendations for advancing practical reproducibility in HPC. Your presentations and participation will directly contribute to this valuable resource for the HPC community.
Please send your submissions to presentations@chameleoncloud.org. Visit our workshop website (reproduciblehpc.org) for more information. For any questions, contact us at contact@chameleoncloud.org or via the Chameleon users list.
We look forward to your contributions and to seeing you in Atlanta this November!
