• Information For
    • Computational Molecular Sciences Community
    • Environmental System Science Community
    • Exascale Computing Community
    • Scientific Libraries Community
    • Software Engineering Community
    • Community of Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Contact BSSw
  • Information For
    Computational Molecular Sciences Community Environmental System Science Community Exascale Computing Community Scientific Libraries Community Software Engineering Community Community of Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Resources

    Better

    Planning

    • Software Process Improvement
    • Software Engineering
    • Requirements
    • Design
    • Software Interoperability

    Better

    Development

    • Documentation
    • Configuration and Builds
    • Revision Control
    • Release and Deployment
    • Issue Tracking
    • Programming Languages
    • Development Tools
    • Refactoring

    Better

    Performance

    • High-Performance Computing (HPC)
    • Performance at Leadership Computing Facilities
    • Performance Portability

    Better

    Reliability

    • Testing
    • Continuous Integration Testing
    • Reproducibility
    • Debugging

    Better

    Collaboration

    • Projects and Organizations
    • Strategies for More Effective Teams
    • Funding Sources and Programs
    • Software Publishing and Citation
    • Licensing
    • Discussion and Question Sites
    • Conferences and Workshops

    Better

    Skills

    • Online Learning
    • Personal Productivity and Sustainability

    View All Resources

    • Better Planning
    • Better Development
    • Better Performance
    • Better Reliability
    • Better Collaboration
    • Better Skills
    • View All Resources
  • Blog
  • Events
  • About
    • Team
    • Policies and Code of Conduct
    • BSSw Fellowship Program
  • Home
  • Blog
  • Useful Practices for Software Engineering on Medium-Sized...

Useful Practices for Software Engineering on Medium-Sized Distributed Scientific Projects

Share on LinkedIn Share on Facebook Tweet Copied! Permalink

PublishedFeb 25, 2021
Authors Keith Beattie and Dan Gunter
TOPICS
Better Planning
Software Engineering
Better Collaboration
Strategies for More Effective Teams
Better Reliability
Testing

Photo © 2010-2019 The Regents of the University of California, Lawrence Berkeley National Laboratory.

Find out how scientists at Lawrence Berkeley National Laboratory have approached producing reliable scientific software in the modern research context with contributors from multiple labs, universities and disciplines, spread across multiple timezones.

Software engineering and distributed teams

Modern science depends heavily on computing and computer software. The correctness, the efficiency and, more generally, the quality of the software are instrumental to scientific advancement. Software engineering — "the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software" [IEEE Standard Glossary of Software Engineering Terminology, IEEE std 610.12-1990, 1990] — is the major tool available to create and maintain software quality. Advances in software engineering, such as "agile" processes and the "devops" revolution, have been important factors in the creation of scientific software.

In our work, we are often in the position of leading, or helping to lead, the software engineering efforts of medium-sized, distributed, multidisciplinary scientific project teams. By medium-sized, we mean from about 10 to 50 people; by distributed, we mean that the team comprises people from multiple institutions that are geographically separated and do not have a common management structure — typically a few universities and one or more national laboratories; and by multidisciplinary, we mean the team is usually composed of people who come from scientific or engineering backgrounds other than computer science or software engineering, with limited experience writing software for use outside their own or their team’s projects. We have often tried to apply software engineering and project management approaches from industry in this milieu but always end up frustrated with some of the assumptions about centralized authority, dedicated software engineering effort and incentive.

Challenges facing a medium-sized project

A good example of such a project team is the Institute for the Design of Advanced Energy Systems (IDAES), which was formed in 2016 to develop new advanced process systems engineering (PSE) capabilities. Funded by the DOE Office of Fossil Energy and led by the National Energy Technology Laboratory (NETL), the project's mission is to improve the efficiency and reliability of the existing fleet of coal-fired power plants while accelerating the development of a broad range of advanced fossil energy systems. The IDAES team spans three national laboratories (NETL, Sandia National Laboratories, and Lawrence Berkeley National Laboratory) and three universities (Carnegie Mellon University, West Virginia University, and Notre Dame University). Most of the team is spread in the two Pennsylvania locations, but half a dozen participants are present in California and New Mexico each, in addition to the professors and several graduate students present in the various universities. From a software engineering perspective, the goal is to build a software package that is capable of simulating important aspects of power plants and power grids. The eventual target users are power plant operators and power grid designers, with a current user base primarly of reseachers, graduate students and the developers themselves.

A three-pronged approach

While the IDAES project runs functionally as a unit, there are sub-hierarchies of control in each national laboratory and university that set local priorities within the framework of the project deliverables. And although most of the effort is being spent on new software to perform chemical process engineering, significant effort is devoted to adding related capabilities to an existing software package, Pyomo, that predates IDAES and continues to be developed at Sandia National Laboratories. This size and variation in scope make project coordination a challenge, but the situation is not unusual in our projects.

Although the specific approaches to increasing overall team productivity through software engineering vary across projects, three elements of what we are doing for IDAES are, we believe, generally applicable to projects with this kind of mix of institutions, disciplines and scale: weekly whole-team developer meetings, incrementally better automation, and “soapboxing” (including software engineering in official goals & deliverables).

Not just another meeting

Many styles of meetings have been discussed in recent years with respect to software development teams, most commonly the daily “standup” meetings combined with some semi-weekly longer “retrospective” meetings. Neither of these fits what we believe is a unique set of constraints of our scientific environment, where we have

  • no common authority structure (due to the multi-institutional nature of our collaborations);
  • contributors split across timezones, sometimes continents; and
  • contributors split across multiple unrelated projects with no inherent interest in accommodating one another.

As a result, people generally have limited slots in their calendar that they can guarantee are open on a regular basis. However, one weekly hour-long meeting is generally possible and serves multiple purposes, by

  • letting people “context-switch” back to what they promised a mere week ago to have done;
  • providing an open forum for cross-cutting issues or questions that don’t get easily addressed in subgroup meetings (because, of course, these also exist);
  • presenting some opportunity to disseminate practices and educate new team members; and
  • building camaraderie through regular (virtual) contact.

For all these purposes, it is essential that this meeting be open to the entire developer team — however daunting that must seem at first. You can think of this meeting as a little bit like going to a weekly religious service or tight-knit social group (church, temple, book group, bicycling club, etc.) where the enthusiasm for the event may vary week to week but the overall experience is as much about the customs and connections as the content.

Specifically what occurs during these meetings will probably depend on your development practices and tools. But in general we have found that two main activities are critical: an open agenda of issues that people can edit before the meeting and a standard task that focuses attention on the active development across the project. For many projects, a modified Kanban “project board” approach provides an excellent focus because it presents an easily summarized list of things completed, in progress, and abandoned. Whoever is leading the meeting — and of course there should be just one person who leads it, since anything else with 20+ people on the line would be chaos — can, in the absence of any other topics, simply walk through all issues and categorize them against release or other timelines. Almost always, this simple practice will bring up interactions between the different pieces of ongoing work that would have otherwise been discovered much later, if at all.

An incremental approach to automated testing

Between meetings, the best friend of overall team productivity is automated practices, and in particular automated tests. Although few people disagree with this idea in principle, the truth is that getting a large suite of automated tests to work and keeping them working in the face of constantly changing software, personnel and dependencies is nontrivial. Of course, most of the tests must be written by people who understand the mathematics and science of the code being tested (or, in the case of infrastructure code, the computer science principles of the code). However, we must face the reality that this is a significant request of someone’s time and is competing with the next publication or result, which the individual may be encouraged to prioritize. Our position is that the right approach to take to this problem is incremental: (1) help people put in a few tests, but don’t require that they pass; (2) add a requirement that the tests pass before the code is merged, but don’t worry about code coverage or style; (3) start informally checking code coverage and style, but don’t enforce anything; (4) start enforcing low levels of code coverage and basic style rules; and (5) finally — well, honestly we’ve never gotten farther than that. But by doing these in baby steps, and talking about them weekly in the developer meetings, you are building up a culture of testing that will lead to eventually having a robust software environment that is broadly supported by the team.

All told, these meetings serve to address the challenges we face developing software with teams composed of people who are not primarily involved in computing or software engineering and are working together without a common authority. The regular meeting "baby steps approach" both educates and builds consensus simultaneously on the best procedures, practices and tools to adopt. Ideally this approach is driven by demonstrated effectiveness from those with the most software development experience.

Soapboxing

The third element that we think is useful is what we call “soapboxing,” not in the negative sense of grandstanding but in the positive sense of putting key elements of software engineering and development practices where they can gain favorable attention from project management and funders. Probably the least important place for this is grant proposals, since these are read at most once every few years, whereas the message needs to be repeated frequently. One of the best places to try to insert some discussion of software practices (and, by extension, developer productivity) is in project meetings. If your PIs and other project leads need any convincing that software engineering is an important topic, an excellent way to do so is to show them the team engaging in a spirited discussion in a public or semi-public forum. Progress reports and general-audience publications about the project are also opportunities to describe software engineering practices.

The goal: developer productivity

We have spoken largely of social challenges, but of course scientific software also presents numerous technical challenges to developer productivity. We believe that conquering these technical challenges goes hand in hand with reducing the friction of the software engineering environment so that developers in medium-sized teams don’t spend all their time working in silos and unintentionally stepping on each other’s toes and, perhaps most important, so that they feel they are working as a single team no matter how different their technical backgrounds.

Author bios

Dan Gunter leads the Integrated Data Frameworks (IDF) group in the Data Science and Technology department of the Computational Research Division at Lawrence Berkeley National Laboratory. Dan's interests include data management and data processing pipelines in heterogeneous environments, software engineering for distributed multidisciplinary scientific teams and building of usable interfaces to enable scientific exploration. He has led software efforts in projects across multiple domains, collaborating with many science divisions at LBNL as well as with scientific and engineering teams across the Unted Statees and internationally. He also brews a mean cup of coffee.

Keith Beattie is a computer systems engineer at LBNL with experience in bringing modern, open-source software engineering practices to academic and research contexts. His interests are in understanding and addressing the unique challenges in leading multi-institutional, geographically dispersed scientific software development teams while still producing effective and usable software, particularly teams composed of members from scientific but not necessarily software engineering backgrounds. He has worked in industry as a software engineer and release manager and has been at LBNL for the past 20 years. He also tortures local music venue attendees playing bass in rock bands.

Comment

More on Software Engineering, Strategies for More Effective Teams, and Testing

When Not to Use Agile in Scientific Software Development

Published Feb 12, 2021

By Anshu Dubey

The RSE-HPC-2020 Workshop: Creating Community, Building Careers, Addressing Challenges

Published Oct 26, 2020

By Charles Ferenbaugh, Sandra Gesing, Simon Hettrick, and Daniel S. Katz

A Community of Practice around Peer Review for Long-Term Research Software Sustainability

Published Aug 28, 2020

By Karthik Ram, Carl Boettiger, Scott Chamberlain, Noam Ross, Maëlle Salmon, and Stefanie Butland

The Lazy Approach to Developing Scientific Research Software

Published Jun 16, 2020

By Carl Simon Adorf and Vyas Ramasubramani

Making Myself Better: What Craftspeople Can Teach Us about Software

Published May 25, 2020

By Wolfgang Bangerth

Working Remotely: The Spack Team

Published May 16, 2020

By Todd Gamblin and Elaine M. Raybourn

Research Software Engineer Stories

Published Apr 24, 2020

By Vanessasaurus

Maintainers Drive Software Sustainability

Published Apr 14, 2020

By Steven Smith

Scientific Software Projects and Their Communities

Published Mar 23, 2020

By Rene Gassmoeller

Spreading Ideas about Better Scientific Software

Published Feb 28, 2020

By David E. Bernholdt

US Research Software Engineer (US-RSE) Association

Published Jan 15, 2020

By Ian Cosden, Chris Hill, Sandra Gesing, and Charles Ferenbaugh

Hello CSE World

Published Dec 05, 2019

By Heather Switzer, Elsa Gonsiorowski, PhD, and Mark C. Miller

Research Software Engineer: A New Career Track?

Published Jun 14, 2018

By Chris Richardson

Can You Teach an Old Code New Tricks?

Published Mar 26, 2018

By Charles Ferenbaugh

Give Thanks!

Published Nov 23, 2020

By Angela Herring

Increasing Productivity by Broadening Participation in Scientific Software Communities

Published Sep 25, 2020

By Mary Ann Leung, Damian Rouson, and Lois Curfman McInnes

Working Remotely: The Exascale Computing Project (ECP) Panel Series

Published Jul 30, 2020

By Elaine M. Raybourn, Reed Milewicz, David M. Rogers, Elsa Gonsiorowski, PhD, Benjamin Sims, and Greg Watson

Research Software Science: A Scientific Approach to Understanding and Improving How We Develop and Use Software for Research

Published Sep 25, 2019

By Mike Heroux

Making Open Source Research Software Visible: A Path to Better Sustainability?

Published Sep 12, 2019

By Neil Chue Hong

Leading a Scientific Software Project: It's All Personal

Published Jun 27, 2019

By Wolfgang Bangerth

Building Connections and Community within an Institution

Published Oct 26, 2018

By Greg Watson and Elsa Gonsiorowski, PhD

Do Social Media and Science Mix? Twitter Use in a Large Research Project

Published Aug 30, 2018

By Tim Scheibe

Scaling Small Teams to a Team of Teams: Shared Consciousness

Published Apr 17, 2018

By Elaine M. Raybourn and David Moulton

Testing Non-Deterministic Research Software

Published Oct 13, 2020

By Nasir Eisty

Porting the Ginkgo Package to AMD's HIP Ecosystem

Published Jun 25, 2020

By Hartwig Anzt

FLASH5 Refactoring and PSIP

Published Aug 27, 2019

By Anshu Dubey and Jared O'Neal

When NOT to Write Automated Tests?

Published Jul 29, 2019

By Roscoe A. Bartlett

Streamlining Software Development through Continuous Integration

Published Apr 26, 2019

By Glenn Hammond

Software As Craft

Published Feb 25, 2019

By Paul Wolfenbarger

The Art of Writing Scientific Software in an Academic Environment

Published Feb 11, 2019

By Hartwig Anzt

Adopting Continuous Integration for Long-Timescale Materials Simulation

Published Sep 25, 2018

By Richard (Rick) Zamora

Software Verification

Published Aug 15, 2018

By Anshu Dubey

SuperLU: How Advances in Software Practices are Increasing Sustainability and Collaboration

Published Apr 30, 2018

By X. Sherry Li

Better Science through Software Testing

Published Feb 02, 2018

By Tom Evans

Improve User Confidence in Your Software Updates

Published Jul 14, 2017

By Mike Heroux

logo Better Scientific Software
  • BSSw Fellowship Program
  • Policies
  • Site Contributors
  • Contact BSSw
  • Receive Our Email Digest
  • Follow Our RSS Feed

Copyright © 2021 Better Scientific Software under MIT License

United States Department of Energy Office of Science National Nuclear Security Administration Exascale Computing Project