Close collaboration and creative problem-solving can overcome significant challenges in integrating packages that employ disparate data models and parallel execution paradigms.

Introduction

Funding from OASIS (now part of RAPIDS), SciDAC (FASTMath), and ECP has facilitated collaborations among developers of tools such as VisIt and ParaView and the broader scientific computing community. Examples include collaborations between developers of VisIt at LLNL and developers of I/O technologies such as ADIOS/ADIOS2 at ORNL, BoxLib/AMReX at LBNL, and MOAB at ANL. This article describes our experiences developing a MOAB database plugin for VisIt. Differences in the two products' scientific data models and parallel execution paradigms have presented several interesting challenges requiring innovative solutions.

What are MOAB and VisIt?

MOAB (Mesh-Oriented datABase) is a scalable in-situ library for managing mesh-based scientific data. It excels at scalable processing of mesh data in bulk parcels or in fine-grained iterations over individual mesh elements. Some aspects of MOAB trace their origins to earlier TSTT and ITAPS SciDAC efforts. VisIt is a scalable scientific visualization tool for analyzing mesh-based scientific data in files or in-situ. Database plugins in VisIt employ various I/O strategies to marshal data from files on disk into Visualization Toolkit (VTK) grid (vtkDataset) and array (vtkDataArray) objects used in VisIt's internal parallel storage and processing framework.

MOAB is more than a database for storing data. VisIt is more than a tool for displaying data. Both also support a variety of scalable data-processing and analysis services such as topological and spatial queries, ghost layer management, and solution transfer. A key MOAB service exploited for the VisIt plugin is scalable, parallel I/O from a single shared file via HDF5.

Early success with the ITAPS/iMesh plugin

As part of the ITAPS project, a VisIt plugin supporting the iMesh interface was initially developed. MOAB, GRUMMP, and FMDB developers participating in ITAPS each implemented the iMesh interface for their respective packages. The same ITAPS plugin source code was then compiled and linked against the three iMesh implementations, producing three shared library plugins: ITAPS-MOAB, ITAPS-GRUMMP, and ITAPS-FMDB. This work demonstrated a key ITAPS goal: that a single interface could support data exchange among a variety of disparate mesh management packages.

The ITAPS-MOAB plugin was used successfully to examine a large MOAB reactor model consisting of hundreds of thousands of subsets for various components of a novel nuclear fuel assembly. However, as funding for the ITAPS SciDAC project ended, so did further development and support of the iMesh interface, which was eventually removed from MOAB. A new VisIt database plugin integrating directly with MOAB's native interface was needed.

Key differences between MOAB and VisIt

MOAB’s data model consists of mesh entities of 0...3 topological dimensions; entity sets; tags (both sparse and dense) attached to mesh entities and/or entity sets; and relations (topological, containment, and/or hierarchical) between these. VisIt’s data model consists of meshes (structured, unstructured, AMR, CSG, etc.) comprised of nodes (0 dimension) and, optionally, zones (1...3 dimensions); decompositions of meshes into different classes of subsets (domains, blocks, materials, enumerations); continuous variables (scalar, vector or tensor); and ordered ensembles of these as time series.

MOAB's data model emphasizes topological and relational aspects of scientific data, whereas VisIt's data model emphasizes computational modeling aspects of scientific data. These differences lead to some challenges in integrating MOAB and VisIt. For example, while a MOAB dense tag is almost certainly a VisIt variable, MOAB tags do not carry information about their tensor rank or continuous interpolation. On the other hand, a MOAB sparse tag almost certainly requires some sort of enumerated subset in VisIt in order to be properly defined. VisIt subsetting is more ad hoc and restrictive than in MOAB, making it difficult to support MOAB's full subsetting generality. Both MOAB and VisIt provide services to manage ghost layers used in parallel processing. It remains to be determined which approach will result in the best user experience.

In addition, like many other scientific data technologies both MOAB and VisIt rely to some extent upon usage conventions. For example, in MOAB, there are tables of conventional tag names. Likewise, in VisIt a time series is little more than a sequence of file names obeying a predefined naming convention. Selecting an appropriate set of conventions for any given MOAB dataset may not always be possible programmatically. VisIt may require explicit user intervention. This can lead to tedious and repetitive GUI activity when using VisIt to simultaneously display multiple MOAB datasets involving different conventions.

Finally, MOAB uses the single shared file (SSF) parallel I/O paradigm, which involves MPI collective calls, whereas VisIt uses the multiple independent file (MIF) parallel I/O paradigm, which involves MPI independent calls. The next section dives more deeply into this particular issue.

Bridging SSF and MIF parallel I/O paradigms

VisIt's database subsystem is designed around the MIF parallel I/O paradigm. In MIF, an upstream data producer once and for all pre-decomposes a massive mesh into K pieces. These pieces are called domains. Domains may be stored among M files (shard-like). When processing the mesh on R MPI ranks, VisIt uses various load balance algorithms to assign domains to ranks. K, M, and R can be chosen mostly independently, allowing for significant flexibility in parallel resource allocation. Typically, K >> M and R ≈ K/n, where n is a small integer divisor of K, often 1.

Consequently, in MIF domains represent an atomic unit of storage at which VisIt makes problem-sized (or bulk) data requests. All requests from VisIt to a plugin are parameterized in terms of a domain (and a timestep) identifier.

For the MOAB plugin, a somewhat unusual design question is what will constitute the all-important mesh domains. In a MOAB (.h5m) file, mesh coordinates and connectivities are stored and expressed in a single, global address space of mesh entities. Except for a handful of simple, structured mesh cases, VisIt itself offers no help in decomposing a globally enumerated mesh into pieces for parallel processing. Fortunately, MOAB combined with the collective parallel I/O capabilities of HDF5 does.

In every MOAB database (.h5m file), there exists a conventional tag named PARALLEL_PARTITION that identifies a collection of K entity sets. These entity sets will serve as the all-important mesh domains for VisIt. Typically, the PARALLEL_PARTITION tag is computed by an upstream partitioner such as Zoltan or METIS. The MOAB plugin uses this tag to define HDF5 dataspace selections so that each MPI rank reads only the subset of the global mesh during collective H5Dread() operations. When R>K, some ranks will not read any entity set of PARALLEL_PARTITION. When R<K, multiple entity sets will be read by the same rank.

Resolving other differences

The remaining issues impacting the MOAB/VisIt user experience involve bridging the MOAB and VisIt data models. MOAB Dense tags on 0D entities (nodes) are used to define node-centered (piecewise-linear interpolation) variables for VisIt. Dense tags on >0D entities are used to define zone-centered (piecewise-constant interpolation) variables. Length 1 tags define scalar variables, whereas length 2 or 3 define vector variables and length >3 tags define array variables. Entity sets for materials and boundary conditions are handled as enumerated subsets in VisIt. With the exception of PARALLEL_PARTITION, sparse tags in MOAB are not yet supported in VisIt, but will likely involve creative uses of VisIt's enumerated subset functionality.

MOAB users often wish to manipulate the mesh in terms of different primitive entity types (e.g. 0D nodes, 1D edges, 2D faces, and 3D volumes). To support this capability in VisIt, a database read option allows users to set which dimensionality entities they wish to have included when the mesh is read. When the same mesh is expressed with different types of entities, this can lead to additional challenges for users in selecting and combining plots in VisIt.

Figure 1: VisIt visualization of MOAB ocean model data.

VisIt and MOAB with MPAS and ROMS ocean models

The visualization in Figure 1, above, demonstrates the use of VisIt and MOAB in analyzing ocean data. A coarse whole global ocean model, using polygonal elements (hexagons extruded in depth), is characterized by the MPAS modeling and simulation application. A handful of much higher resolution regional models (using structured ijk hexahedral meshes), outlined in red, are characterized by the ROMS modeling and simulation application. Some insets represent critical regions of open ocean while others represent inland water bodies such as the Chesapeake Bay. Both MPAS and ROMS use MOAB to store and manage the data and, in particular, also to handle coupling between the global and regional models.

The global ocean visualization involves a number of different MOAB data sources and plots of those sources together with VisIt's Threshold operator set to display different features of interest. These multiple plots were then layered, somewhat tediously, as concentric, spherical shells using VisIt's Transform operator with various cmocean color maps applied. This approach helps to provide context for individual inset regions.

Of the 150+ database plugins in VisIt, MOAB is the only plugin employing SSF collective parallel I/O via HDF5.

Image credits

Images courtesy of Vijay Mahadevan and Iulian Grindeanu.

Author bios

Iulian Grindeanu is a computational scientist with more than two decades of experience in scientific computing, parallel algorithms, and mesh-based simulation technologies. He has made significant contributions to MOAB's parallel infrastructure, distributed mesh algorithms, mesh intersection methods, and climate-modeling applications that rely on scalable mesh data structures and parallel I/O.

Vijay S. Mahadevan is a computational scientist at Argonne National Laboratory. His work focuses on scientific computing, mesh technologies, multiphysics coupling, and high-performance computing. He is a principal contributor to the MOAB mesh infrastructure and related tools for scalable mesh and data management in large-scale simulation workflows.

Mark C. Miller is a retired computer scientist who continues to support the SD program at LLNL. He is a contributor to VisIt, Silo, and HDF5.