Multi-Process HPC Simulator
The thrust area will develop a flexible and extensible fate and transport simulator in support of environmental management applications and performance assessment that is designed to utilize the high-performance computing (HPC) power of modern architectures, from laptops to supercomputers. The graded and iterative approach to assessments naturally generates a suite of conceptual models that span a range of process complexity, potentially coupling hydrological, biogeochemical, geo-mechanical, and thermal processes. The Platform and Integrated toolsets, named Akuna, provides the tools for users to generate this wide range of conceptual models, and Amanzi provides the flexible and extensible computational engine to simulate them.
To support this graded and iterative approach to conceptual model development that is required by environmental applications, Amanzi must be both modular and extensible. This objective is achieved with a hierarchical and modular design that captures all the steps involved in translating a conceptual model to output for analysis. To support this hierarchical view an object oriented programming model is used, with higher-level objects and much of the code being developed in the C++ language. Specifically, the design has high-level objects that abstract process models and their coupling, supporting toolsets that provide the building blocks for these high-level objects, and low-level objects and services that are used by the supporting toolsets. These three component levels are shown schematically in Figure 1.
In this design we use the term Process Model to denote a complete mathematical description of a physical or bio-geochemical process. A Process Kernels (PKs) is then a discrete representation of a Process Model suitable for simulation. The Multi-Process Coordinator handles the coupling and management of a complete coupled system of Process Kernels. This design makes it natural to add new processes by adding new Process Kernels (e.g., thermal energy), which leverage the design and lower-level components that are common to other PKs. In addition, this modular design extends throughout the PK, making it natural to add new parameter relations and constitutive laws. Thus, in the initial phase of development we are establishing the overall framework with a relatively small number of key flow and reactive-transport processes. For example, we are focused on single-phase saturated and unsaturated (Richards) flow models for the initial release of Amanzi, but in the near future we plan to develop a full multi-phase flow capability.
The selection and prioritization of processes for implementation in Amanzi is based on the needs of the risk and performance assessments at the waste sites of interest to our sponsor. Ultimately, simulation capabilities will include multi-phase, multi-component reactive transport models for behavior of residual wastes in closed tanks, cribs, trenches, and landfills, and degradation of waste forms (e.g., glass, cement/grout, waste packages). In addition, models for flow in fractured media, thermal and geo-mechanical processes as well as tightly coupled surface processes (e.g., flow and evapotranspiration), are being considered. Where necessary, new modeling and computational capabilities for the improved treatment of physical-chemical processes will be developed, such as the mesh infrastructure and preconditioning algorithms for efficiently linking flow and transport across the surface-subsurface interface, or new approaches to modeling the degradation of cementitious materials in source areas.
The tasks within this thrust reflect the modular and hierarchical design outlined above:
Process models mathematically represent the physical, chemical, and biological phenomena controlling contaminant release into, and transport through, the subsurface. They may include surface processes, such as surface flow, as well. The Process models task is focused on providing the detailed mathematical description (models) of the relevant subsurface/surface processes. This concise mathematical description and use cases provides critical information for the requirements and design of Amanzi. For example, these process models will include (1) source term behavior needed to predict the performance of residual wastes in closed tanks, cribs, trenches, and landfills and degradation of waste forms and engineered barriers, and (2) subsurface flow and reactive transport processes to predict the behavior of contaminants released from engineered barriers into the vadose zone and groundwater.
HPC Core Framework
This task focuses on the key infrastructure that facilitates the modular design of Amanzi, as well as its portability and its graded Quality Assurance program. This provides a number of low-level services for the HPC Toolsets, which provide the building blocks for the process models. These include data structures, input file specification and utilities, parallel input/output capabilities, and HPC related visualization support. Existing HPC Frameworks, such as Trilinos, and supporting libraries, such as the Hiearchical Data Format version 5 (HDF5) are leveraged here. In addition, portability and performance analysis is addressed by this task, with a portable build system built on CMake, and support for the Verification and Validation task leveraging Ctest.
The HPC Toolsets provide the building blocks that transform the mathematical description of the process models into a discrete form suitable for simulation on a computer, dubbed a process kernel (PK). This task provides the overarching support for the flexible Multi-Process Coordinator (MPC) as well as the data management support for the system state necessary for the abstract interfaces to the process kernels (PKs). In addition, it supports the key toolsets for building PKs, namely meshing, discretization, and solvers. Amanzi supports a structured/unstructured capability, with block structured AMR leveraging BoxLib, and general polyhedral support provide through either the Sierra Toolkit (STK) from Trilinos or the MeSh ToolKit (MSTK). Discretizations are leveraging advanced methods, such as the Mimetic Finite Difference Method, to provide the accuracy and robustness required. Solvers are primarily leveraged from Trilinos, or BoxLib, although techniques for flexible block preconditioning of systems for various Newton or accelerated nonlinear iterations are actively being developed
Verification and Validation
The Verification and Validation (V&V) task will provide a unified hierarchical approach to testing, verification and validation, and benchmarking in order to ensure the reliability and robustness of the HPC Simulator, Amanzi. At the lowest level of the hierarchy, unit tests of individual models will be used to verify the correctness of specific sub-modules. At higher levels various integrated tests will be designed to span multiple coupled processes and detailed conceptual model descriptions with data requirements will be developed. Finally, at the highest level, benchmarks will be developed or gathered from the community that captures realistic site-application scenarios.
Process Models and HPC Simulator - Documents
Mathematical Formulation Requirements and Specifications for the Process Models (pdf)
At the highest level of the HPC Simulator design is a set of process models that mathematically represent the physical, chemical, and biological phenomena controlling contaminant release into, and transport in, the subsurface. The objective of this requirements document is to provide a catalogue of process models, along with their detailed mathematical formulation, for potential implementation in the HPC Simulator.
High-Level Design of Amanzi, the Multi-Process High Performance Computing Simulator (pdf)
The multi-process High Performance Computing (HPC) simulator, named Amanzi, pro- vides a flexible and extensible simulation capability for ASCEM. The goal of this document is to establish a common understanding of the high-level design strategy for all stakeholders in ASCEM as well as to describe the overall design strategy. However, explicit implementation details are beyond the scope of this document, and will be ad- dressed in future Amanzi documentation.