• Information For
    • Computational Molecular Sciences Community
    • Environmental System Science Community
    • Exascale Computing Community
    • Scientific Libraries Community
    • Software Engineering Community
    • Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Contact BSSw
  • Information For
    Computational Molecular Sciences Community Environmental System Science Community Exascale Computing Community Scientific Libraries Community Software Engineering Community Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Resources

    Better

    Planning

    • Software Process Improvement
    • Software Engineering
    • Requirements
    • Design
    • Software Interoperability

    Better

    Development

    • Documentation
    • Configuration and Builds
    • Revision Control
    • Release and Deployment
    • Issue Tracking
    • Programming Languages
    • Development Tools
    • Refactoring

    Better

    Performance

    • High-Performance Computing (HPC)
    • Performance at Leadership Computing Facilities
    • Performance Portability

    Better

    Reliability

    • Testing
    • Continuous Integration Testing
    • Reproducibility
    • Debugging

    Better

    Collaboration

    • Projects and Organizations
    • Strategies for More Effective Teams
    • Inclusivity
    • Funding Sources and Programs
    • Software Publishing and Citation
    • Licensing
    • Discussion and Question Sites
    • Conferences and Workshops

    Better

    Skills

    • Online Learning
    • Personal Productivity and Sustainability

    View All Resources

    • Better Planning
    • Better Development
    • Better Performance
    • Better Reliability
    • Better Collaboration
    • Better Skills
    • View All Resources
  • Blog
  • Events
  • About
    • Site Overview
    • Team
    • Policies and Code of Conduct
    • BSSw Fellowship Program
  • Home
  • Blog
  • Software Team Experiences and Challenges: A Report from Day...

Software Team Experiences and Challenges: A Report from Day 1 of the 2021 Collegeville Workshop on Scientific Software

Share on LinkedIn Share on Facebook Tweet Copied! Permalink

PublishedSep 20, 2021
Authors Cody Balos, Jed Brown, Gerasimos Chourdakis, Ben Cowan, Anshu Dubey, Vadim Dyadechko, Robert Jacob, Sarah Knepper, Jay Lofstead, Lois Curfman McInnes, Reed Milewicz, Jacob Moxley, Todd Munson, Sarah Osborn, Jim Pivarski, Elaine Raybourn, Barry Smith, James Willenbring, Ulrike Meier Yang, Sam Yates, Mike Heroux, and Johanna Cohoon
TOPICS
Better Planning
Software Process Improvement
Software Engineering
Better Collaboration
Projects and Organizations
Strategies for More Effective Teams
Conferences and Workshops

Background: The Collegeville Workshop Series

What do multi-million dollar HPC projects have in common with a PI and their soon-to-be graduating doctoral student? What are the most pressing concerns of teams on the bleeding edge of scientific discovery? The Collegeville Workshop Series on Scientific Software brings together three communities of scientific software contributors: academia, industry, and laboratories. While there are existing exchanges between these communities, this workshop series is dedicated to improving the awareness of common needs, unique contributions, and career paths. Workshop contributions include short white papers and video interviews. The three-day live event includes panels, small-group discussions, and teatime sessions for themed conversations. This blog post is the first of three that summarize the output from the small-group discussions in 2021.

Collegeville 2021 theme: scientific software teams

Research software and scientific discovery are often advanced by collaborative teams rather than individuals. Yet, scientific software teams are often conducting their work with little knowledge of how to best take advantage of their collective experience. A new frontier for scientific software can be met with a better understanding of how software teams function and how teamwork can be improved. As we focus on improving software teams, we see value in augmenting the traditionally-valued expertise of computer scientists, mathematicians, and software engineers with the expertise of social, information, and cognitive scientists

Workshop small group discussions

Each day of the workshop focused on different aspects of software teamwork. The first day of live discussion focused on software team experiences and challenges, the second day on technical strategies for improvement, and the third day on cultural approaches for improvement. Small groups gathered live over video conference to discuss the topic of the day, each group creating a shared notes file. Because the workshop participants come from different work settings and have varying experiences with scientific software (see below), these discussions were a rare opportunity to identify shared concerns and recognize differences among scientific software teams. More than representing their shared knowledge, however, our intention is for these discussions to impact the diverse communities that participated in them.

Day 1: Software team experiences and challenges

To provide a framework for discussing software team improvement, we spent Day 1 of Collegeville 2021 learning about each other, our backgrounds, and what we see as the greatest challenges to improving software teams.

Summary of participant software team experiences

About half of the discussion participants were from research labs; the remaining half were evenly split between universities and industry. Twenty-two participants (listed as authors above) chose to receive attribution for their contributions to these discussions. Discussion participants from labs and universities represented a number of well-known open-source scientific software projects. Others came from industry, representing the oil & gas sector and technical computing software providers. Finally, several participants were from the social, information, or cognitive science communities, where their domain of study includes scientific software teams and developers.

In aggregate, the discussion participants have approximately 300 years of collective software development and software project leadership experience, ranging from individual contributors to leaders of large, multi-team efforts. Together, these participants provide software to thousands of users throughout the world. Participant experiences further represent approximately four decades of focused study of scientific software teams via methodologies from the social, information, and cognitive sciences.

When describing themselves, participants identified the projects they worked on and some provided further details on their team experiences, reflecting on the perspectives that their teammates brought to the collaborations and their means of communication. For instance, one participant described their small, four person team comprised of three mathematicians and one computer scientist, noting that they have excellent communication but few formal methodologies. In another example, a participant described their weekly teleconferences, active Gitter discussions, and (pre-pandemic) evening coding events with students.

Key challenges

In the remainder of this article, we summarize the key challenges identified during the small group discussions. The detailed notes from these discussions are available on the Collegeville 2021 Workshop website.

Staffing diverse software teams: Finding people with the right skills to contribute to a scientific software project is difficult—the talent pool is small, the work is specialized, and there is competition among institutions and industry to attract qualified candidates. In this context, having better strategies for acquiring and retaining the right talent is a growing problem. Some software teams (such as open-source, peer-produced projects) have temporary contributors whose work can be valuable, but discussion participants were generally interested in building lasting teams made of experienced contributors. This is a substantial challenge. Some discussions noted that training efforts might prove fruitless because they are often directed at students who can spend the entirety of an internship becoming accustomed to the software work. That same student might not perceive a rewarding career path and never return to the project. Skilled contributors might be found in-house, but funding structures often mean that their attention is split such that they have only short amounts of time to dedicate to many projects. We are beginning to address the needs of research software engineers (RSEs), and some teams have funding to hire dedicated RSEs, thus avoiding split attentions. However, in general, we need to improve the recognition, value, and career paths for all members of scientific software teams. Without being recognized and funded, non-PIs find it difficult to continue contributing to a software project and establish their career, forcing them to decide between jeopardizing one or the other. We need to distribute credit across the team so that the diverse actors feel valued. We need to address the question one participant posed, “Why can’t I be a phenomenal contributor?”

Aligning incentives and rewards: Highly related to the challenge of staffing a scientific software team is the issue of aligning incentives and rewards for code development with those for research and publishing. Scientific software teams are often comprised of software developers and, distinctly, domain experts; these different career paths thusly depend on the outcomes of a common product. Some necessary work like triaging bug reports and managing infrastructure goes unrecognized in the traditional academic reward system. One discussion group noted, “This is not a side job, but it is often treated as one.” In addition, there is a disconnect between what is highlighted in funding proposals as the primary contribution and what is actually needed to make a software product successful. Testing, integration, deployment, and user support skills are all crucial to sustainable and reliable scientific software, but these skills are seldom listed as explicitly funded activities. This disconnect influences what skills are considered during hiring and makes it more difficult to establish a stable career as an RSE.

Curating and maintaining knowledge: Teams observing best practices may have many conversations and tools that support maintenance of the software product, such as regression testing or continuous integration. However, our discussion groups agreed that less attention is paid to ensuring that knowledge about the product is curated and maintained. As teams grow in size or diversity, knowledge maintenance becomes more complex. Growth is likely to occur across disciplines and institutions, meaning that vocabularies and infrastructure must be aligned. Knowledge curation and maintenance will also differ depending on a team’s purpose. For instance, software development teams will need different approaches from software deployment and support teams. Preparing for staff departure, creating paths for long-term membership, and otherwise committing to the long-term viability of a scientific software project represent opportunities for reducing tacit knowledge and improving knowledge transfer among team members. However, some attempted solutions to these challenges that our discussants have tried, such as teach-ins, are hard to justify when deadlines loom.

Evolving team practices and structure in sync with the software: The initial practices, tools, and infrastructure adopted by a team will need to evolve to take advantage of emerging approaches and tools and to improve effectiveness and efficiency as the team grows. Evolution includes enabling bottom up change, and giving individual contributors autonomy to explore new approaches and receive recognition for their contributions. Additionally, as the teams grow in size and diversity, their communication efforts become more difficult; one discussion group described domain scientists and computer scientists as working with people speaking different languages. While modern tools can facilitate communication across even non-collocated teams, they cannot reconcile differences in vocabulary, priorities, or conceptions of software quality.

Collaborating across software teams: For a healthy software ecosystem, software development teams must communicate and collaborate with one another, but they may approach their work from different viewpoints. Application teams, for instance, develop software that is targeted for a specific computational problem. In contrast, teams that develop libraries and other reusable components focus on a broader set of uses that are based on characteristic examples derived from interaction with their user community. Our discussants noted that application teams have design concerns on one end (dependencies), library teams have concerns on both (dependencies and dependents), but both team types have a joint responsibility to work through application program interfaces (APIs). Collaborations between application and library teams require determining how to design and develop capabilities in the spaces between software providers and clients. Challenges include how to integrate the library team with the application user community, how to foster software ecosystem perspectives (especially around collaborative development of APIs), and designating concrete staffing roles for shepherding API development and adoption.

Characterizing and measuring software quality: During our discussions of software teams, it was clear that participants understood software quality to hold several different meanings. For collaboration purposes and to report meaningful outcomes to funders, we need to establish a shared understanding of quality measures and when a measure is appropriate. This requires recognizing a balance between payoff and effort. We also need to address how much we value innovation vs stability and portability vs cost of special-purpose coding. Teams of different sizes, with different stakeholders, different purposes, and different budgets might answer these questions differently, but they should be answered thoughtfully. We need to develop ways to understand if software is fulfilling its purpose, and we need our sponsors to value and invest in code quality.'

Improving business models for research software sustainment: All scientific software teams have a business model, even if it is implicit. Making the business model explicit and actively managing toward sustainability is much less common. Having a good business model is important for all kinds of software, and critical where there is a distributed and evolving community involved in its development. Each of the above challenges is affected by funding, and our discussion groups recognized this. Seeing the need for more money is easy. However, seeing how funding can affect your teamwork in one, five, or ten years is far more difficult, but crucial to establishing sustainable scientific software.

Final remarks from Day 1 discussions

Day 1 discussions at the Collegeville 2021 Workshop represent the input of a diverse and experienced group of scientific software developers and leaders, and their colleagues from the social, information, and cognitive sciences. We hope that the challenges summarized in this blog post resonate with you and help the scientific software community when prioritizing efforts to improve the quality and impact of software in the pursuit of scientific discovery. We look forward to addressing these challenges with creativity and community-developed insight.

Acknowledgements

The workshop organizers and authors of this article thank all participants for contributions to workshop discussions.


Part 1 | Part 2 | Part 3


Comment

More on Software Process Improvement, Software Engineering, Projects and Organizations, Strategies for More Effective Teams, and Conferences and Workshops

Trusted Cyberinfrastructure Evaluation, Guidance, and Programs for Assurance of Scientific Software

Published May 13, 2022

By Elisa Heymann Pignolo, Barton Miller, and Sean Peisert

Reflecting on Our Community: The SC21 BoF on Software Engineering and Reuse in Modeling, Simulation, and Data Analytics for Science and Engineering

Published Dec 21, 2021

By David Bernholdt, Michael Bader, Michelle Barker, Ben Brown, Anshu Dubey, Nasir Eisty, Sandra Gesing, Patricia Grubel, Rinku Gupta, Mike Heroux, Saswata Hier-Majumder, Axel Huebl, Mozhgan Kabiri Chimeh, Daniel S. Katz, Tomislav Maric, Lois Curfman McInnes, Bill Miller, Manish Parashar, Ulf Schiller, Jean M. Sexton, Peter Vaillancourt, Marion Weinzierl, and Yo Yehudi

Introducing the 2022 BSSw Fellows

Published Jan 14, 2022

By Hai Ah Nam

Experiences Replacing Master/Slave Terminology in ALE3D and Sierra

Published Mar 27, 2022

By Mark C. Miller

The Workshop on Sustainable Software Sustainability

Published Apr 18, 2022

By Carlos Martinez-Ortiz, Shoaib Sufi, and Peter Doorn

logo Better Scientific Software
  • BSSw Fellowship Program
  • Policies
  • Site Contributors
  • Contact BSSw
  • Receive Our Email Digest
  • Follow Our RSS Feed

Copyright © 2022 Better Scientific Software under MIT License

United States Department of Energy Office of Science National Nuclear Security Administration Exascale Computing Project