Introduction
What Is GridTools
The GridTools (GT) framework is a set of libraries and utilities to develop performance portable applications in which stencil operations on grids are central. The focus of the project is on regular and block-structured grids as are commonly found in the weather and climate application field. In this context, GT provides a useful level of abstraction to enhance productivity and obtain excellent performance on a wide range of computer architectures. Additionally, it addresses the challenges that arise from integration into production code, such as the expression of boundary conditions, or conditional execution. The framework is structured such that it can be called from different weather models (numerical weather and climate codes) or programming interfaces, and can target various computer architectures. This is achieved by separating the GT core library in a user facing part (frontend) and architecture specific (backend) parts. The core library also abstracts various possible data layouts and applies optimizations on stages with multiple stencils. The core library is complemented by facilities to interoperate with other languages (such as C and Fortran), to aid code development and a communication layer.
GridTools provides optimized backends for GPUs and manycore architectures. Stencils can be run efficiently on different architectures without any code change needed. Stencils can be built up by small composeable units called stages, using GridTools domain-specific language. Such a functor can be as simple as being just a copy stencil, copying data from one field to another:
struct copy_functor {
using in = in_accessor<0>;
using out = inout_accessor<1>;
using param_list = make_param_list<in, out>;
template <typename Evaluation>
GT_FUNCTION static void apply(Evaluation eval) {
eval(out()) = eval(in());
}
};
Several such stages can be composed into a Computation and be applied on each grid-point of a grid. Requiring this abstract descriptions of a stencils, the DSL allows GridTools can apply architecture-specific optimizations to the stencil computations in order to be optimal on the target hardware.
Dependencies
GridTools requires a header-only installation of Boost, a recent version of CMake and a modern compiler. The exact version requirements can be found on GitHub.
Addtionally GridTools requires the following optional dependencies. For the communication module (GCL) MPI is required. For the GPU backends a CUDA or HIP compiler is required. For the CPU backends OpenMP is required.
Using GridTools
GridTools uses CMake as its build system. CMake offers two ways of using a CMake-managed project: from an installation or using FetchContent to pull in a dependency on the fly. GridTools supports both ways.
Installing and Validating GridTools
To install GridTools (in /usr/local
)
and run the tests, use the following commands.
git clone http://github.com/GridTools/gridtools.git
cd gridtools
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
cmake --build --parallel 4
ctest
CMake will detect the optional dependencies and enable the available backends accordingly. During configure the available GridTools targets will be listed.
The following CMake variables are available to customize the installation of GridTools.
Set
GT_INSTALL_EXAMPLES
toON
andselect a directory for installation with
GT_INSTALL_EXAMPLES_PATH
. The examples come with a standalone CMake project and can be built separately.
Additionally, use the following CMake variables to customize building of tests.
Set
BUILD_TESTING
toOFF
to disable building any tests (fast installation without validation).Set
GT_GCL_GPU
toOFF
to disable thegcl_gpu
target (and to disable building of GPU GCL tests). This is useful, if you have CUDA in your environment, but the MPI implementation is not CUDA-aware.Set
GT_CUDA_ARCH
to the compute capability of the GPU device on which you want to run the tests.If your compiler is a CUDA-capable Clang, you can switch how CUDA code will be compiled, by setting
GT_CLANG_CUDA_MODE
to one ofAUTO
(default, prefer Clang-CUDA if available),Clang-CUDA
(compile with Clang),NVCC-CUDA
(compile with NVCC and Clang as the host compiler).
Using a GridTools Installation
Using GridTools follows standard CMake practices. If GridTools was installed to <prefix>,
provide -DCMAKE_PREFIX_PATH=<prefix>
or -DGridTools_ROOT=<prefix>
to indicate where GridTools can be found.
The CMakeLists.txt
file should then contain the following line:
find_package(GridTools VERSION ... REQUIRED)
Note
If GridTools should use CUDA with NVCC, you must call enable_language(CUDA)
before the call to find_package
.
Note
If you are compiling with Clang, set the variable GT_CLANG_CUDA_MODE
before calling find_package
, see
Installing and Validating GridTools. It is recommended to make this variable a CMake cached variable to allow users of your code
to change the mode.
Using GridTools with CMake’s FetchContent
Alternatively to a GridTools installation you can use GridTools with FetchContent. To use FetchContent add the following lines to your CMake project
include(FetchContent)
FetchContent_Declare(GridTools
URL https://github.com/GridTools/gridtools/archive/<release_tag>.tar.gz
)
FetchContent_GetProperties(GridTools)
if(NOT GridTools_POPULATED)
FetchContent_Populate(GridTools)
add_subdirectory(${gridtools_SOURCE_DIR} ${gridtools_BINARY_DIR})
endif()
where <release_tag> is the git tag of the GridTools release, e.g. v2.0.0
.
The following CMake options are available (see also Installing and Validating GridTools).
By default, all GridTools tests are disabled. To enable building of the tests, set
GT_BUILD_TESTING
toON
. If tests are enabled, their behavior can be changed as described in Installing and Validating GridTools.Use
GT_CLANG_CUDA_MODE
to select how CUDA code is compiled, see Installing and Validating GridTools.
Using GridTools
After GridTools was made available by either find_package
or FetchContent
the following targets for the
different GridTools modules are available
stencil_naive
If OpenMP is available
stencil_cpu_ifirst
stencil_cpu_kfirst
storage_cpu_ifirst
storage_cpu_kfirst
layout_transformation_cpu
boundaries_cpu
If OpenMP and MPI is available
gcl_cpu
If a CUDA runtime (or a HIP compiler) is available (no CUDA compiler required)
storage_gpu
If a CUDA compiler or a HIP compiler is available
stencil_gpu
layout_transformation_gpu
boundaries_gpu
If a CUDA compiler and MPI is available
gcl_gpu
(can be disabled by the user if the MPI implementation is not CUDA-aware)
After linking to the GridTools backend, we recommend to call the CMake function
gridtools_setup_target(<target> [CUDA_ARCH <compute_capability>])
on your target.
The function helps abstracting differences in how CUDA code is compiled
(e.g. Clang uses a different flag than NVCC for the CUDA architecture). Additionally, using this function
allows to compile the same .cpp file for both CUDA and host, without having to wrap the implementation in a
.cu file.
add_library(my_library source.cpp)
target_link_libraries(my_library PUBLIC GridTools::stencil_cuda)
gridtools_setup_target(my_library CUDA_ARCH sm_60)
AMD GPU Support
Further, GridTools can also be compiled for AMD GPUs using AMD’s HIP. To compile GridTools you need the Clang-based HIP compiler which is available with ROCm 3.5 and later.
Contributing
Contributions to the GridTools set of libraries are welcome. However, our policy is that we will be the official maintainers and providers of the GridTools code. We believe that this will provide our users with a clear reference point for support and guarantees on timely interactions. For this reason, we require that external contributions to GridTools will be accepted after their authors provide to us a signed copy of a copyright release form to ETH Zurich.