Fundamental Resources for Scientific Computing

This article hosts a number of online documentation resources of fundamental importance in HPC/CSE.

We begin with the most fundamental of topics - formal specifications and standards for programming languages most commonly used in HPC/CSE. These are formal documents primarily for implementers of compilers that define, in excruciating detail, how programs in the language are supposed to behave and attempt to cover all possible corner cases. Language specifications evolve with time and so are versioned, typically by the year, in which the new specification is initiated. But not all languages commonly used in HPC/CSE, notably Python, are defined by a formal specification. Instead, they rely solely on a reference implementation. Python's reference implementation is CPython.

From formal specifications, we move on to language implementations. The implementation of a programming language is typically embodied in a compiler or, for interpretive languages like Python (or Basic), an interpreter. The relevant documentation takes the form of compiler reference manuals. A key bit of information compiler reference manuals provide is how a compiler may deviate from the standard it implements.

A program consisting solely of language statements typically does not make a useful application. Useful applications need to interact with the outside world...memory, a terminal display, disk files, etc. This functionality is typically provided in the form of an accompanying standard library...a set of function calls with well defined interfaces that do things like allocate and free memory, open and close files, read and write data, etc.

A challenge with standard libraries is that they aren't always very standard. Different hardware vendors provide different and sometimes incompatible implementations. POSIX compliance was introduced in the 1990's to address this not only for the C standard library but also for many other aspects of how programs and humans (e.g. command-line shells) interact with an operating system.

As new hardware is introduced, its kinda sorta useless if there doesn't exist any compilers that produce executable (binary) code that runs on that hardware (and takes maximal advantage of any of its unique features). Therefore, hardware vendors are more or less obliged to provide a compiler that supports their hardware. No applications can use their hardware without one. Often, hardware vendors develop and support their own native (and proprietary) compilers. This ensures applications are maximally performant on their hardware. Alternatively (or sometimes additionally), existing compilers are enhanced to target performance features of the vendor's hardware. Notably, GNU, Intel, Portland Group and Clang compilers are not tied to any particular hardware vendor. These compilers are often made available on various vendor's hardware and in some cases can even produce higher performing code than the hardware vendor's native compiler.

It can take many years for compiler vendors to update their implementations to conform to new language standards. It is not uncommon for a language/compiler reference manual or standard library to depart from a formal language specification in various minor ways especially during a period in which it is transitioning to new language standard. For example, the GNU compiler collection (GCC) often supports a number of language extensions some of which eventually make their way into the formal language standard. The use of language features unique to a specific standard and/or compiler can introduce portability issues. To avoid such possibilities, projects often constrain which language standards and compiler versions they agree to use and support. The VisIt project decided to permit C++11 constructs (specific to the 2011 C++ standard) into the code base only in 2018, a full 7 years after the language standard had been released.

Perhaps the next most fundamental aspect of scientific computing after languages, compilers and standard libraries is parallelism; the decomposition of a single, large computing task into many smaller tasks that execute simultaneously on separate copies of hardware resources. Parallelism can manifest in a myriad of ways in both hardware and software creating significant portability challenges. Nonetheless, one critical differentiator is shared memory vs. distributed memory parallelism. Another critical differentiator is whether parallelism manifests as the same computational task running simultaneously everywhere except on different data (e.g. Data parallelism) or something more generalized than this where computational tasks which can be wholly disparate are queued and divvied out to resources as they become available (e.g. Task parallelism). A number of technologies are aimed at addressing portability challenges. Though their key aim is portability of parallelism, these technologies are referred to as performance portability solutions.

We round out this discussion of resources by identifying several other commonly used technologies that are available in the form of either language extensions or application programming interfaces (APIs) to third party libraries. Sometimes whether a technology is considered a language extension or an API isn't always very clear cut. That said, API's are also sometimes decomposed into two pieces - a specification piece and an implementation piece. This is entirely analogous to programming language specification and implementation described above. The canonical example of an API that is managed in this way is the Message Passing Interface (MPI). Another example is OpenGL, a graphics programming API (the L in OpenGL stands for Library but many often treat it as thought it stands for Language). The MPI API has evolved with time and so has multiple specifications, each versioned. The most currently agreed upon MPI interface specification is version 4.0. But, there are a number of earlier interface specifications as well. MPICH serves as a reference implementation of MPI.

Finally, we end with a handful of other miscellaneous programming technologies commonly encountered when contributing to HPC/CSE code projects.

¹Version numbers are the last 2 digits of the year the standard was initiated. Sometimes, standards are formally finalized years after they were initiated.
²Language extensions for special devices (e.g. co-processors, GPUs, FPGAs, accelerators, etc.).
³The most formal resource for Python is the reference implementation, CPython
⁴CPP is sometimes used to process other kinds of text files including those of other languages. CPP #pragma directives are a common way for compiler vendors to extend the language.
⁵The fortran specification does not define a standard library. Nonetheless, there is a community driven effort to develop one.
⁶A critical aspect of these technologies is whether they work on CPUs only, GPUs only or can target both. Some technologies are designed to target a variety of other devices such as FPGAs, etc.
⁷USPSnet is wordplay for sending physical storage media through the US Mail. Another name is FootNet. Sometimes, its the best way to move a lot of data.