At the annual Supercomputing conference and sometimes at the ISC High Performance conference, members of the scientific software developer community come together to share their thoughts, network, and build community. These “Birds of a Feather” (BoF) sessions started in 2015 under the title Software Engineering for Computational Science and Engineering on Supercomputers. While the titles of the sessions have morphed over time, the purpose has always been to focus on the “people” side of high-performance scientific computing. Below, we summarize the 2024 edition of the BoF at SC24 as a sampling of what’s on people’s minds lately.

Lightning talks

As usual, we kicked off the session with a set of lightning talks. The organizers selected these 3-minute talks to highlight recent developments in the community and spur further conversation. This year, we had eight talks covering a wide range of topics.

• Move the Needle by Cristin Merritt (Alces Flight). Cristin described the Move the Needle initiative, which she has led as part of Alces Flight, together with volunteers from Women in HPC. The mission statement for the initiative is, “There is a lack of diversity and inclusion in HPC. Move the Needle’s goal is to discover what people are doing to change this.” Key takeaways from the project include:

  • Appreciate nuance;
    
  • Diversity really does solve problems - different institutions have different wants and needs; and
    
  • There’s still a great deal to learn. Cristin pointed us to the project’s Knowledge Portal, and in early February 2025, the project also released a Facilitation Guide and its Final Report.

• Training and Industry Relations in the DiRAC Facility by Clare Jenner (DiRAC HPC Facility). Clare talked about how the UK Distributed Research using Advanced Computing (DiRAC) program organizes its training program. DiRAC is a collection of systems located at four different universities with the mandate to collaborate with industry and support academic research. Hackathons are the primary approach for DiRAC’s training. Each hackathon has a different focus, often associated with the hosting site. For example, the oneAPI Base Toolkit, A100 and Grace Hopper, or MI300X and MI300A. The feedback from these events suggests that attendees get a lot out of them and appreciate the sharing of ideas and experiences, and that they help attendees develop good practices.

• How RSEs Can Improve Project Quality by Embracing the Role of “Educated Ignoramus” by Spencer Smith (McMaster University). Spencer talked about the valuable role that the “educated ignoramus” can play in helping software teams improve their requirements. The term, which he attributes to a 2002 paper by Daniel M. Berry, is meant to describe someone knowledgeable in computer science, software engineering, math, or other technology areas but who has little knowledge in the project's domain. Such a person comes to the requirements gathering effort without tacit assumptions that every practitioner in the domain “knows” and doesn’t even consider. They’re willing to ask questions to better understand and clarify; non-experts force the experts to gain a better understanding of their domain by asking questions requiring simple answers. Many people can play this role in a project if you just seek them out. Research software engineers are often well-suited for this role.

• Code Review for Scientific Software by Helen Kershaw (NSF National Center for Atmospheric Research). Helen gave an update on her Better Scientific Software Fellowship project on code review. She created https://code-review.org as a training resource for code review, which is described further in a blog article. She has also been holding discussions among her colleagues to better understand their feelings about code review. She finds that many developers are hesitant to share their code and may tend to “gold plate” it before they’re willing to share. On the other side, reviewing is hard – people are unsure how to review and find it hard to give actionable feedback. Helen feels that it is important to humanize code review and that each software team needs to work out for themselves how much structure their code reviews need for them to work well.

• Developing Reusable Tools for Critical Applications, Looking at an Example of a (UK) Hospital Software Tool – How Could We Have Done Better? By Simon Clifford (University of Cambridge). Simon spoke about his experience helping transition a research application into a hospital setting. The application involved using a machine learning algorithm to give a prognostic score for an Alzheimer’s patient based on brain images. They quickly found that the clinical environment was quite different from the research environment in which the application was developed. The tool needed to interface with the hospital’s Picture Archiving and Communication System (PACS); it needed to run on the hardware and network in the hospital; it needed an interface suitable for clinicians to use; and, it needed images with appropriate consent. They ended up creating a “digital close relative” to help address these needs. For others who may be facing challenging situations like this, Simon recommends trying to anticipate the end-use environment from the outset, avoiding siloed thinking, and finding (or starting) a community of people with similar needs to work together.

• Introducing the Consortium for the Advancement of Scientific Software (CASS) by David E. Bernholdt (Member and Chair, CASS Steering Committee). David talked about a newly formed organization focusing on the stewardship and advancement of scientific software. The initial members are eight projects funded by the U.S. Department of Energy to help steward many software products developed or enhanced by the Exascale Computing Project Software Technologies focus area. However, the group is open to public participation and additional members to expand the ecosystem. David encouraged those interested to engage with CASS and its member organizations. Details are available on the https://cass.community website.

• Building a Productive AI-Assisted HPC Software Ecosystem: The Need for a Community-Driven Approach by Harshitha Menon (Lawrence Livermore National Laboratory). Harshitha started by observing the explosion of interest in the use of Large-Language Models (LLMs) in coding and asked the question, “Can LLMs write HPC code?”. She noted many challenges: HPC codes are more complex, they tend to be relatively poorly represented in training corpora, LLMs process source code as text and lack knowledge of parallel programming models, and LLMs are not trained for code performance. She asserts that to be able to effectively harness LLM for HPC code development, we need to improve LLMs effectiveness and trustworthiness in the HPC domain, and we need integration with existing HPC tools. She believes that these challenges can only be addressed by community-based efforts in numerous areas ranging from the development and sharing of HPC-focused datasets to detailed work on code representation, models, evaluation, and more.

• Equity, Diversity and Inclusivity Concerns When Building Communities of Practice by Neil P. Chue Hong (University of Edinburgh). Neil discussed a garden as a metaphor for a community of practice based on resources developed by the Center for Scientific Collaboration and Community Engagement—i.e., many different plants (community members) with different needs and roles. Various tools are needed at different times to help the garden flourish. Interventions need to be intentional. “Gardens thrive when different plants support each other well, rather than being monocultures.” Sometimes there are “weeds” in the garden that need to be dealt with – perhaps using the community’s code of conduct to remove them. Though some disruptive elements may, in fact, be useful to cross-pollinate ideas, so, again, interventions should be thoughtful.

Discussion

Community-building

We began the general discussion following up on some of the community-related themes that arose in a number of the lightning talks. The first question was, “What is the hardest thing in building a community of practice?” Numerous participants chimed in with responses from their own experiences, including getting the attention of the people who might benefit from participating, overcoming the inertia of approaching people to participate (with the further note that many people are happy to write code but less comfortable talking about it). It takes effort to share and discuss with others, and it is not uncommon to find rivalries getting in the way of a free exchange of information. Another participant noted that it is often easier to bring together existing communities than to create a new one.

Someone else asked for experiences and recommendations to start a trans-disciplinary community. Recommendations from other attendees included starting with a vision statement. There are many good examples of vision statements, specifically, as well as general advice on community-building available in the wild – seek them out. The Center for Scientific Collaboration and Community Engagement (CSCCE), which was also mentioned during the lightning talks, was noted as a good source of such advice. Expanding on the theme of community development, attendees mentioned several other books and articles as useful resources (listed below). Additionally, many countries also have organizations that support volunteer efforts and offer useful advice.

A few other observations about communities also came up in our discussions. People can become too invested in one community to the extent that they may become suspicious of another “nearby” community – community organizers need to watch out for this. Someone remarked that the communities that really “stuck” for them tended to be the ones that were somewhat difficult to get into – if it is too easy to join, then it is too easy to leave. Requiring some kind of (non-monetary) investment to get in made the community more important to them. In this context, it is sometimes useful to distinguish between communities of interest and communities of practice, with the former being easy to get into and the latter having a bit of a barrier or being more exclusive. In this case, the community of interest should require minimal effort to sustain, while people can afford to invest more to sustain a community of practice. Communities of interest can be organized around websites or mailing lists – passive for many members. The community starts looking more like a community of practice when people begin desiring to do things together – producing joint outputs.

Uptake of software engineering practices in scientific software

Then, a student, who was just beginning a Ph.D. program, asked those in the audience who identified as research software engineers (RSEs) how they ended up on that path. The answers were varied: some people liked research but didn’t care for paper and grant writing; someone started as an engineer in a commercial organization but liked the programming aspect and ended up finding a fulfilling RSE position. Others observed that RSEs have been around for 30+ years, but it is only recently that the term has gained significant recognition. In the last (roughly) ten years since the term research software engineer was coined, there has also been a significant increase in the adoption of software engineering practices in the scientific community. Participants in the Argonne Training Program for Extreme-Scale Computing (ATPESC) went through a rather sudden phase change in terms of the level of awareness, interest in, and use of software engineering practices. Within a couple of years, the community codes/software engineering track of the summer school went from one of the lowest-scoring to one of the highest-rated tracks. Similarly, the number of participants who were already familiar with version control quickly changed from very few to nearly all (95%+). Another national laboratory transitioned from engineers writing (often bad) code to engineers developing prototypes and letting software engineers write the production code. Attitudes changed from “it's just code” to letting the professionals do it.

Large language models in coding

Having transitioned to talking about software engineering and software development more directly, it may not be surprising that artificial intelligence (AI) and more specifically large language models (LLMs) re-entered the discussion. About a third of the audience raised their hands to affirm they used LLMs in their coding. However, there are numerous challenges. Many institutions (especially government labs and companies) have security and intellectual property concerns about their staff engaging with public LLMs. Some labs have localized LLMs to deal with such concerns. People tend to use LLMs more in the simpler parts of the code because they’re not so good at discovering algorithms. Someone else observed that LLMs might convince people to do test-driven development – since we can’t rely on LLMs to produce correct code! When the question was refined to how many people are looking systematically into using LLMs in coding, the number of hands raised in the room was much smaller. Anshu Dubey noted a paper she published on using LLMs to translate Fortran code to C++.

We then turned to what students should be taught about LLMs in coding. An educator noted that students are already using LLMs all the time and can often get them to generate a lot of the code they need, so perhaps reorienting on design or architecture rather than coding would be one way to address that. Someone else countered that they’re just tools, and the more you use them, the better you understand how to use them well. This comment brought up the idea that the limitations of LLMs should be illustrated better in the classroom. Another concern was that LLMs just give answers rather than teaching (or even demonstrating) skills. Students need to develop many skills to be effective software developers. For example, if an LLM is giving them code that happens to be correct, they’re not learning debugging skills.

Finally, rounding out the discussion, an audience member reminded us that LLMs aren’t the only code-generation option. We know and understand physics fairly well. Can we use physics-based requirements as the basis for generating code (and documentation)? One person noted that requirements gathering is very important and often neglected – “understanding the problem is the problem.” However, the audience generally thought that requirements gathering was not likely to be a strength of LLMs. Another point was that integration of a piece of physics into a larger code system could also be a significant challenge – no matter whether the code is generated by LLMs or other means, and that a lot more attention to verification and validation will be required in any automatic generation scenario. Finally, someone noted that in many cases, the mathematics resulting from an interesting physics problem might be highly nonlinear, which can be very hard to translate automatically into code.

Summary

From this year’s discussions, there is a great deal of interest in building communities. Still, it seems that many people lack the knowledge, experience, or comfort levels to do this work effectively. Several resources were noted that might help, and numerous audience members have had particular experience with the work of the Center for Scientific Collaboration and Community Engagement, which deserves a shout-out. It may be worth exploring developing other venues for people to share knowledge and experiences with building communities on a more regular basis.

The other major topic of discussion was LLMs in coding. There remains a great deal of work to be done to make LLMs useful in the context of high-performance scientific software development. In her lightning talk, Harshitha Menon argued that we need to work as a community to move this forward. We also need to be more direct in showing students the limitations of LLMs.

Resources mentioned

Links to the BoF website, and other resources mentioned in the presentations and discussions.

• BoF series website, including notes from past meetings: http://bit.ly/swe-cse-bof.
  
• Previous blog posts from this BoF series: 2023, 2021
  
• A list of software-focused events at SC24: https://bssw.io/events/sc24-software-related-events
  
• Move the Needle
  
  • Knowledge Portal
    
  • Facilitation Guide
    
  • Final Report
    
• DiRAC training program
  
• Berry’s paper defining the term “educated ignoramus”
  
• Code review
  
  • Tutorial
    
  • Blog article
    
• Consortium for the Advancement of Scientific Software website: https://cass.community
  
• Center for Scientific Collaboration and Community Engagement
  
  • The garden metaphor for community management
    
  • Community participation model
    
  • DEI resources
    
• EDI within research groups and networking communities
  
• Buzzing Communities: How to Build Bigger, Better, and More Active Online Communities by Richard Millington
  
• How do you get programmers to join your project? By Jeremy Garcia
  
• How to attract new contributors to your open source project by Shubheksha Jalan
  
• How to Attract New Contributors by Brian Proffitt
  
• Argonne Training Program for Extreme-Scale Computing
  
• A survey of the state of the practice for research software in the United States
  
• Leveraging Large Language Models for Code Translation and Software Development in Scientific Computing

Author bios

David E. Bernholdt is a Distinguished R&D Staff Member at Oak Ridge National Laboratory. His research interests are broadly in the development of scientific software for high-performance computers, including developer productivity, and software quality and sustainability.

Jeffrey C. Carver is a James R. Cudworth Professor of Computer Science at the University of Alabama. His research focuses on the appropriate use of software engineering practices for developing research software. He is the Editor-in-Chief of IEEE Computing in Science & Engineering.

Ian A. Cosden is the Senior Director of Research Software Engineering at Princeton University. He leads a team of research software engineers who complement multiple traditional academic research groups by offering embedded, long-term software development expertise. Ian is the current and founding chair of the steering committee for the United States Research Software Engineer (US-RSE) Association.

Anshu Dubey is a Senior Computational Scientist known for her work on multiphysics software for high-performance computing. She is the chief architect of FLASH and leads
the development of Flash-X.

Weronika Filinger is an HPC project manager and the Programme Director for an online MSc in HPC and HPC with Data Science at EPCC, the University of Edinburgh. A lot of her work is focused on education, training, and professional skill development in the HPC space. She is also the vice-chair of the ACM SIGHPC Education Chapter.

Sandra Gesing is the Executive Director of the US Research Software Engineer Association and a Senior Research Scientist at the San Diego Supercomputer Center (SDSC). She is passionate about building career paths for Research Software Engineers (RSEs) and advocating for sustainable software practices in scientific research. Her work focuses on science gateways, cyberinfrastructure, and research software engineering, aiming to enhance accessibility to shared computing, data, and software resources. She actively engages in community building, policy discussions, and initiatives that support the RSE community.

Mozhgan Kabiri Chimeh is a GPU Developer Advocate at NVIDIA, specializing in GPU-accelerated computing, high-performance computing (HPC), and AI enablement. She leads hackathons and boot camps across EMEA to accelerate HPC and AI adoption, empowering developers and researchers to harness the full potential of GPU technologies. As a passionate community builder, she actively cultivates a collaborative ecosystem through comprehensive training, technical seminars, and outreach initiatives. Her efforts focus on promoting reproducible, sustainable software practices and expanding access to cutting-edge computing solutions.

Lauren Milechin is the Director of Scientific Consulting at MIT Office of Research Computing and Data, where she leads the training, education, and support of MIT’s research computing resources.

Marion Weinzierl is a Senior Research Software Engineer at the Institute of Computing for Climate Science (ICCS), University of Cambridge.

Harshitha Menon is a Computer Scientist at Lawrence Livermore National Laboratory. Her recent research focuses on developing novel techniques that leverage Large Language Models (LLMs) to enhance HPC software, particularly in the domain of HPC code development and software sustainability.

Addi Malviya-Thakur is the Group Leader for the Software Engineering Group in the Computer Science and Mathematics Division's Advanced Computing Systems Research Section at the Oak Ridge National Laboratory. Her research interests include interconnected science and federated systems, research software engineering, operational workflows, software analytics, software frameworks, and ecosystems for science. Addi leads the Scientific Software Development for data reduction and data analysis software for the neutron science community at ORNL’s High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS). Her work enables scientists to efficiently process and analyze complex neutron scattering data, accelerating scientific discoveries, and advancing innovation in neutron science.