Interfacing deal.II to Trilinos

Trilinos is a software package that provides lots of functionality for linear algebra, among other things. For example, it includes implementations of a variety of linear solvers, as well as various different sparse and dense matrix and vector formats. Trilinos also has many subpackages that deal with problems that go far beyond linear algebra, for example nonlinear solvers, automatic differentiation packages, uncertainty propagation engines, etc. Of particular interest to deal.II is their ability to provide this functionality both on sequential and parallel (using MPI) computers. Compared to PETSc, which is written in C, Trilinos is written in C++ and can be considered to be a more modern version of PETSc though both packages are under continuing development at their respective national laboratories.

deal.II has wrapper classes to the linear algebra parts of Trilinos that provide almost the same interfaces as the built-in deal.II linear algebra classes. We use these interfaces for parallel computations based on MPI since the native deal.II linear algebra classes lack this ability. They are used, among other programs, in step-31 and step-32.

While building deal.II with Trilinos is covered in the ReadMe file, we here give an introduction to building Trilinos in such a way that it contains everything that we need from the deal.II side.

Installing Trilinos

Note: The current version of deal.II requires at least Trilinos 11.2. Deal.II is known to work with Trilinos up to 11.14 and 12.4. Others versions of Trilinos should work too but we do not do regression tests with them.

Trilinos uses cmake to configure and build. The following slightly longish set of commands will set up a reasonable configuration (we require MueLu starting from 12.0):


	cd trilinos-12.0.1
	mkdir build
	cd build

	cmake \
	-D Trilinos_ENABLE_Sacado=ON \
	-D Trilinos_ENABLE_MueLu:BOOL=ON \
	-D Trilinos_ENABLE_Stratimikos=ON \
	-D CMAKE_BUILD_TYPE=RELEASE \
	-D CMAKE_CXX_FLAGS="-g -O3" \
	-D CMAKE_C_FLAGS="-g -O3" \
	-D CMAKE_FORTRAN_FLAGS="-g -O5" \
	-D Trilinos_EXTRA_LINK_FLAGS="-lgfortran" \
	-D CMAKE_VERBOSE_MAKEFILE=FALSE \
	-D Trilinos_VERBOSE_CONFIGURE=FALSE \
	-D TPL_ENABLE_MPI=ON \
	-D BUILD_SHARED_LIBS=ON \
	-D CMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \
	..

	make install
      
You will need to adjust the path into which you want to install Trilinos in the CMAKE_INSTALL_PREFIX line.

Parallel builds

If your computer has more than one processor core, use make -jN instead of make in the last line above, where N is the number of processors you have.

BLAS and LAPACK

Trilinos sometimes searches for other libraries but can't find them if they are not in the usual directories or have other names. A common example are BLAS or LAPACK. In a case like this, you may have to specifically pass the directories and/or library names under which they can be found to cmake. For example, this may mean to add the following flags to the call above:


	-D BLAS_LIBRARY_NAMES:STRING=goto \
	-D BLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \
	-D LAPACK_LIBRARY_NAMES:STRING=lapack \
	-D LAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64
      

Using external direct solvers

Trilinos (via its Amesos package) can interface with a number of direct solvers (see, for example, this page for Trilinos 11.8). Most of them are external packages to Trilinos and you will need to tell Trilinos configuration scripts that you want to use them, for example via the TrilinosWrappers::SolverDirect class. This can be tricky, but adding defines similar to the following to the cmake command line will achieve the goal to enable the UMFPACK and SuperLU/SuperLUDist solvers:


        -D TPL_ENABLE_UMFPACK:BOOL=ON \
        -D TPL_ENABLE_SuperLU:BOOL=ON \
        -D TPL_ENABLE_SuperLUDist:BOOL=ON \
        -D TPL_UMFPACK_INCLUDE_DIRS="/usr/include" \
        -D SuperLUDist_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_DIST_3.2/SRC" \
	-D TPL_SuperLUDist_LIBRARIES:FILEPATH="/path/to/SuperLU_DIST_3.2/lib/libsuperlu_dist.a" \
        -D SuperLU_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_4.3/SRC" \
        -D TPL_SuperLU_LIBRARIES:FILEPATH="/path/to/SuperLU_4.3/lib/libsuperlu_4.3.a"
      
Similarly, to enable MUMPS, commands should include

        -D TPL_ENABLE_MUMPS:BOOL=ON \
        -D TPL_ENABLE_SCALAPACK:BOOL=ON
      
and possibly followed by

        -D TPL_MUMPS_INCLUDE_DIRS:PATH=/usr/include/openmpi-x86_64 \
        -D SCALAPACK_LIBRARY_DIRS:PATH=/lib64/openmpi/lib \
      
where you need to adjust the exact paths, of course.


Valid HTML 4.01! Valid CSS!