The ACM REP conference helps build better software by highlighting developments in reproducibility and replicability for computational science.
A few years ago, growing out of work of Carlos Maltzahn, his student Ivo Jimenez, and me, we saw a growing momentum of people interested in reproducibility and replicability challenges and successes for computational science. To help drive the community and grow those efforts, the topic of Ivo Jimenez's PhD thesis, we gathered prominent community members and started an Exploratory Interest Group (EIG) with the Association of Computing Machinery (ACM). Our goal was to explore how big this community is and see whether we could make a catalyst and meeting point for these researchers and practitioners to help drive improving scientific software quality.
Within a few years, we felt like we had enough momentum, and in Summer 2023 we held the first ACM REP conference. Even with short notice and not much advertising, we managed to get a critical mass of research papers and attendance that proved the community was both large enough and interested in participating.
For ACM REP 23, we met in the Hay Barn at the University of California, Santa Cruz and online. With the success of that effort, we took the conference to Rennes, France in the Summer of 2024 to see what a more convenient location for the European community would muster. We had slightly more attendance and a different mix, showing a varied community we could help bring together.
For 2025, we will bring ACM REP to Vancouver, Canada during the last week of July, exploring what attendance may be like for something closer to east Asia, but without the potential visa issues of a meeting hosted in the USA.
At ACM REP 23, we had three keynotes, each looking at a different aspect of sustainable software. Torsten Hoefler described his path into proper statistics for analyzing data and the impacts that had on quality. Juliana Feire presented on the challenges in many domains for data-driven science. The last keynote, from Grigori Fursin, laid out the efforts in which he has been involved in trying to provide tools to support reproducible science. In all of these cases, the depth and breadth of available research topics were made clear, and the challenges in achieving success are shown to be daunting. For us to achieve better scientific software, considerable research will be needed.
At ACM REP 24, our first keynote was from Anne-Laure Boulesteix, who spoke about the challenges in machine learning and how reproducibility is increasingly challenging as ML-infused approaches become an integral part of our computational science. Our second keynote was from Konrad Hinsen, who discussed the problems with platforms and how GUIX can provide a portable, stable, reproducible base upon which to run reproducible computational science. While not a complete solution, GUIX offers stronger infrastructure to make many of the ill-defined elements more about the science rather than the experimental environment.
In both conferences, we featured a wide variety of papers, ranging from success stories to incremental but difficult and important improvements to make reproducibility and replicability achievable. The discussions among attendees have also shown this group to be a vibrant and collaborative, interested in advancing the field rather than trying to promote their own efforts.
Science is defined as something that can be reproduced in a new environment by someone else and achieve statistically the same results. Sometimes that reproduction is exact, while at other times there are minor variations. Overall, the ability to recreate the results is key for something to be considered science. With the importance of computational science to our modern process of scientific inquiry, ensuring it is science is a requirement, or our contributions are little more than, "I got this result and I trust it. Therefore, so should you."
This conference series was started with the hope that we could achieve a critical mass to foster a community for sustainable computational science. The first two years have shown that this critical mass exists and is ready to grow.
Our latest effort is to encourage experience papers that discuss lessons learned from reproducing other computational research. These experience papers can be from any conference or journal and will help justify the educational and research value in exploring existing work by offering a publication credit for writing up the results.
As with both previous years, we plan to continue offering tutorials with a stronger emphasis on hands-on sessions rather than lecture style. This approach will help people learn how to use new tools.
If you wish to participate, please visit https://acm-rep.github.io/2025/ to keep abreast of the conference as it is organized.
Together, we can make better scientific software through better support for reproducibility and replicability.
Further information
Author bio
Jay Lofstead is a Principal Member of Technical Staff at Sandia National Laboratories. His research interests focus on large-scale data management and trusting scientific computing. In particular, he works on storage, IO, metadata, workflows, reproducibility, software engineering, machine learning, and operating system-level support for any of these topics. Broadly across these topics, he is also deeply interested in ethics related to these topics and computing in general and how to drive inclusivity across the computation-related science domains. Dr. Lofstead received his Ph.D. in Computer Science from the Georgia Institute of Technology in 2010.
