Interfacing deal.II to Trilinos

Trilinos is a software package that provides lots of functionality for linear algebra, among other things. For example, it includes implementations of a variety of linear solvers, as well as various different sparse and dense matrix and vector formats. Trilinos also has many subpackages that deal with problems that go far beyond linear algebra, for example nonlinear solvers, automatic differentiation packages, uncertainty propagation engines, etc. Of particular interest to deal.II is their ability to provide this functionality both on sequential and parallel (using MPI) computers. Compared to PETSc, which is written in C, Trilinos is written in C++ and can be considered to be a more modern version of PETSc though both packages are under continuing development at their respective national laboratories.

deal.II has wrapper classes to the linear algebra parts of Trilinos that provide almost the same interfaces as the built-in deal.II linear algebra classes. We use these interfaces for parallel computations based on MPI since the native deal.II linear algebra classes lack this ability. They are used, among other programs, in step-31 and step-32.

While building deal.II with Trilinos is covered in the ReadMe file, we here give an introduction to building Trilinos in such a way that it contains everything that we need from the deal.II side.

Installing Trilinos

Note: The current version of deal.II requires at least Trilinos 12.4 (12.14.1 if Trilinos includes Kokkos). Deal.II is known to work with Trilinos up to 13.4. Other versions of Trilinos should work too but have not been tested prior to the release.

deal.II uses the following libraries from Trilinos:

Amesos,
AztecOO,
Epetra,
EpetraExt (optional),
Ifpack,
ML,
MueLu (optional, required for TrilinosWrappers::PreconditionAMGMueLu),
NOX (optional, required for TrilinosWrappers::NOXSolver),
ROL (optional),
Sacado (optional),
SEACAS (optional, required for GridIn::read_exodusii()),
Teuchos,
Tpetra (optional),
Zoltan (optional).

Trilinos uses cmake to configure and build. The following slightly longish set of commands will set up a reasonable configuration (trilinos-x.y.z is the version-specific path into which the Trilinos .tar.gz will unpack):


    cd trilinos-x.y.z
    mkdir build
    cd build

    cmake                                            \
    -DTrilinos_ENABLE_Amesos=ON                      \
    -DTrilinos_ENABLE_Epetra=ON                      \
    -DTrilinos_ENABLE_EpetraExt=ON                   \
    -DTrilinos_ENABLE_Ifpack=ON                      \
    -DTrilinos_ENABLE_AztecOO=ON                     \
    -DTrilinos_ENABLE_Sacado=ON                      \
    -DTrilinos_ENABLE_SEACAS=ON                      \
    -DTrilinos_ENABLE_Teuchos=ON                     \
    -DTrilinos_ENABLE_MueLu=ON                       \
    -DTrilinos_ENABLE_ML=ON                          \
    -DTrilinos_ENABLE_NOX=ON                         \
    -DTrilinos_ENABLE_ROL=ON                         \
    -DTrilinos_ENABLE_Tpetra=ON                      \
    -DTrilinos_ENABLE_COMPLEX=ON                     \
    -DTrilinos_ENABLE_FLOAT=ON                       \
    -DTrilinos_ENABLE_Zoltan=ON                      \
    -DTrilinos_VERBOSE_CONFIGURE=OFF                 \
    -DTPL_ENABLE_MPI=ON                              \
    -DBUILD_SHARED_LIBS=ON                           \
    -DCMAKE_VERBOSE_MAKEFILE=OFF                     \
    -DCMAKE_BUILD_TYPE=RELEASE                       \
    -DCMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \
    ../

    make install

You will need to adjust the path into which you want to install Trilinos in the CMAKE_INSTALL_PREFIX line. If your computer has enough memory available, it may also be useful to pass the flag -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON, which will improve compilation times of deal.II programs that use Trilinos.

Parallel builds

If your computer has more than one processor core, use make -jN instead of make in the last line above, where N is the number of processors you have.

BLAS and LAPACK

Note: At the time of writing (November 2016) Trilinos (at least up to v12.8.1) cannot link against LAPACK 3.6.0 or later, due to two symbols that were deprecated (and removed by default) in LAPACK 3.6.0 (see the release notes). To fix this, edit the Trilinos file packages/epetra/src/Epetra_LAPACK_wrappers.h and change the lines


    #define DGGSVD_F77  F77_BLAS_MANGLE(dggsvd,DGGSVD)
    #define SGGSVD_F77  F77_BLAS_MANGLE(sggsvd,SGGSVD)


    #define DGGSVD_F77  F77_BLAS_MANGLE(dggsvd3,DGGSVD3)
    #define SGGSVD_F77  F77_BLAS_MANGLE(sggsvd3,SGGSVD3)

before installing Trilinos. (Credit for this fix goes to this page.)

Trilinos sometimes searches for other libraries but can't find them if they are not in the usual directories or have other names. A common example are BLAS or LAPACK. In a case like this, you may have to specifically pass the directories and/or library names under which they can be found to cmake. For example, this may mean to add the following flags to the call above:


	-DBLAS_LIBRARY_NAMES:STRING=goto \
	-DBLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \
	-DLAPACK_LIBRARY_NAMES:STRING=lapack \
	-DLAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64

Using external direct solvers

Trilinos (via its Amesos package) can interface with a number of direct solvers (see, for example, this page for Trilinos 11.8). Most of them are external packages to Trilinos and you will need to tell Trilinos configuration scripts that you want to use them, for example via the TrilinosWrappers::SolverDirect class. This can be tricky, but adding defines similar to the following to the cmake command line will achieve the goal to enable the UMFPACK and SuperLU/SuperLUDist solvers:


        -DTPL_ENABLE_UMFPACK:BOOL=ON \
        -DTPL_ENABLE_SuperLU:BOOL=ON \
        -DTPL_ENABLE_SuperLUDist:BOOL=ON \
        -DTPL_UMFPACK_INCLUDE_DIRS="/usr/include" \
        -DSuperLUDist_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_DIST_3.2/SRC" \
        -DTPL_SuperLUDist_LIBRARIES:FILEPATH="/path/to/SuperLU_DIST_3.2/lib/libsuperlu_dist.a" \
        -DSuperLU_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_4.3/SRC" \
        -DTPL_SuperLU_LIBRARIES:FILEPATH="/path/to/SuperLU_4.3/lib/libsuperlu_4.3.a"

Similarly, to enable MUMPS, commands should include


        -DTPL_ENABLE_MUMPS:BOOL=ON \
        -DTPL_ENABLE_SCALAPACK:BOOL=ON

and possibly followed by


        -DTPL_MUMPS_INCLUDE_DIRS:PATH=/usr/include/openmpi-x86_64 \
        -DSCALAPACK_LIBRARY_DIRS:PATH=/lib64/openmpi/lib \

where you need to adjust the exact paths, of course.