• Information For
    • Computational Molecular Sciences Community
    • Environmental System Science Community
    • Exascale Computing Community
    • Scientific Libraries Community
    • Software Engineering Community
    • Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Contact BSSw
  • Information For
    Computational Molecular Sciences Community Environmental System Science Community Exascale Computing Community Scientific Libraries Community Software Engineering Community Supercomputer Facilities and their Users
  • Contribute to BSSw
  • Receive Our Email Digest
  • Resources

    Better

    Planning

    • Software Process Improvement
    • Software Engineering
    • Requirements
    • Design
    • Software Interoperability
    • Software Sustainability

    Better

    Development

    • Documentation
    • Configuration And Builds
    • Revision Control
    • Release And Deployment
    • Issue Tracking
    • Programming Languages
    • Development Tools
    • Refactoring

    Better

    Performance

    • High Performance Computing (Hpc)
    • Performance At Leadership Computing Facilities
    • Performance Portability
    • Cloud Computing
    • Big Data

    Better

    Reliability

    • Peer Code Review
    • Testing
    • Continuous Integration Testing
    • Reproducibility
    • Debugging

    Better

    Collaboration

    • Projects And Organizations
    • Strategies For More Effective Teams
    • Inclusivity
    • Funding Sources And Programs
    • Software Publishing And Citation
    • Licensing
    • Discussion And Question Sites
    • Conferences And Workshops
    • Research Software Engineers

    Better

    Skills

    • Online Learning
    • In Person Learning
    • Personal Productivity And Sustainability

    View All Resources

    • Better Planning
    • Better Development
    • Better Performance
    • Better Reliability
    • Better Collaboration
    • Better Skills
    • View All Resources
  • Blog
  • Events
  • About
    • Site Overview
    • Team
    • Policies and Code of Conduct
    • BSSw Fellowship Program
  • Home
  • Blog
  • Software...

Software Verification

Share on LinkedIn Share on Facebook Tweet Copied! Permalink

PublishedAug 15, 2018
Author Anshu Dubey
TOPICS
Better Reliability
Testing
Track Experience

In the realm of software, verification is often erroneously conflated with testing. In truth, testing is a proper subset of verification for gaining confidence in the correct behavior of one’s software. It is the holistic process by which the developers convince themselves that their software is doing what it was designed to do. In scientific software this could mean verifying the convergence order, numerical stability, and efficacy of the method in the regime of interest over and above testing for expected results. Note that verification is limited to ensuring that the implementation matches the model specification, not that the model itself matches reality. The latter is normally a part of the validation process.

Challenges in verification

For scientific software, verification is as complex and difficult as it is critical. Of course, this claim may apply to software in any environment when it is meant for exploration as opposed to productization. Difficulties begin with the word "exploration." Verification in some sense implies that one knows the expected outcome of running the software and can satisfy oneself that such an outcome is either achieved or not achieved. However, the idea of exploration contradicts that expectation. Software is meant to help understand what the expected outcome ought to be. The developers thus face a conundrum in attempting to incorporate rigorous verification into their scientific process.

Approaches to verification

Many techniques for verification of scientific software come at the problem sideways. Often a new problem being tackled combines components that can be exercised individually against known, simpler problems. Individual components and their combinations can also be verified against known solutions or manufactured solutions. In this way one can build a scaffolding of tests that help build confidence in an increasingly complex collection of permutations and combinations. Additionally, one can resort to indirect verification such as conservation of physics quantities or obeying scientific laws to satisfy oneself that the application is not doing something unphysical.


Figure: Nuclear flame in a controlled model validation simulation


Resources

Furthermore, validation can become a strong component in a good software assurance process. For example, the image above is from a simulation of FLASH performed to validate the model of nuclear flame, which in turn is used in the simulation of supernova explosions. Though direct experimental data is lacking for the behavior of these flames in high-gravity environments, the statistical properties of the flow are known from observations and can be directly compared. The following links provide more resources and examples of the verification process employed in scientific software.

Slides from a tutorial on verification at ISC-2018 Video of a lecture on testing and verification A paper on scientific software verification and testing


Ultimately, when it comes to scientific discovery, any defined process can go only so far. All the tools and techniques can only aid scientific intuition, which in the end has to undergo peer review for acceptance. And even then, the models being solved and methods used to solve them are, at best, approximations based on scientific intuition. All that they guard against is that wrong conclusions are not being drawn through negligence. And that is the foundation stone for a credible scientific process.

Comment

More on Testing

Julia's Value Proposition for Better Scientific Software

Deep Dive

Published Apr 14, 2023

By William F Godoy

Rethinking Software Variants

Deep Dive

Published Sep 29, 2022

By Axel Huebl

Build, Integration, and Testing for Sustainable Scientific Computing Software

Community

Published May 27, 2022

By Keita Teranishi, Roscoe A. Bartlett, Franck Ledoux, Daniel S. Katz, and Anshu Dubey

Overcoming Complexity in Testing Multiphysics Coupling Software

Deep Dive

Published Feb 07, 2022

By Frédéric Simonis, Gerasimos Chourdakis, and Benjamin Uekermann

E4S: Extreme-Scale Scientific Software Stack

Deep Dive

Published Jun 29, 2021

By Sameer Shende, Mike Heroux, and James Willenbring

logo Better Scientific Software
  • BSSw Fellowship Program
  • Policies
  • Site Contributors
  • Contact BSSw
  • Receive Our Email Digest
  • Follow Our RSS Feed

Copyright © 2023 Better Scientific Software under MIT License

United States Department of Energy Office of Science National Nuclear Security Administration Exascale Computing Project