Trilinos is a software package that provides lots of functionality for linear algebra, among other things. For example, it includes implementations of a variety of linear solvers, as well as various different sparse and dense matrix and vector formats. Trilinos also has many subpackages that deal with problems that go far beyond linear algebra, for example nonlinear solvers, automatic differentiation packages, uncertainty propagation engines, etc. Of particular interest to deal.II is their ability to provide this functionality both on sequential and parallel (using MPI) computers. Compared to PETSc, which is written in C, Trilinos is written in C++ and can be considered to be a more modern version of PETSc though both packages are under continuing development at their respective national laboratories.
deal.II has wrapper classes to the linear algebra parts of Trilinos that provide almost the same interfaces as the built-in deal.II linear algebra classes. We use these interfaces for parallel computations based on MPI since the native deal.II linear algebra classes lack this ability. They are used, among other programs, in step-31 and step-32.
While building deal.II with Trilinos is covered in the ReadMe file, we here give an introduction to building Trilinos in such a way that it contains everything that we need from the deal.II side.
Note: The current version of deal.II requires at least Trilinos 12.4. Deal.II is known to work with Trilinos up to 12.12. Other versions of Trilinos should work too but we do not do regression tests with them.
deal.II uses the following libraries from Trilinos:
trilinos-x.y.zis the version-specific path into which the Trilinos
cd trilinos-x.y.z mkdir build cd build cmake \ -DTrilinos_ENABLE_Amesos=ON \ -DTrilinos_ENABLE_Epetra=ON \ -DTrilinos_ENABLE_EpetraExt=ON \ -DTrilinos_ENABLE_Ifpack=ON \ -DTrilinos_ENABLE_AztecOO=ON \ -DTrilinos_ENABLE_Sacado=ON \ -DTrilinos_ENABLE_Teuchos=ON \ -DTrilinos_ENABLE_MueLu=ON \ -DTrilinos_ENABLE_ML=ON \ -DTrilinos_ENABLE_ROL=ON \ -DTrilinos_ENABLE_Tpetra=ON \ -DTpetra_INST_COMPLEX_DOUBLE=ON \ -DTpetra_INST_COMPLEX_FLOAT=ON \ -DTrilinos_ENABLE_Zoltan=ON \ -DTrilinos_VERBOSE_CONFIGURE=OFF \ -DTPL_ENABLE_MPI=ON \ -DBUILD_SHARED_LIBS=ON \ -DCMAKE_VERBOSE_MAKEFILE=OFF \ -DCMAKE_BUILD_TYPE=RELEASE \ -DCMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \ ../ make installYou will need to adjust the path into which you want to install Trilinos in the CMAKE_INSTALL_PREFIX line. If you do not have MPI installed you should use
-DTPL_ENABLE_MPI=OFFinstead. Additionally, if your computer has enough memory available, it may also be useful to pass the flag
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON, which will improve compilation times of deal.II programs that use Trilinos.
If your computer has more than one processor core, use
make -jN instead of
make in the last line
N is the number of processors you have.
Note: At the time of writing (November 2016) Trilinos (at least up to
v12.8.1) cannot link against LAPACK 3.6.0 or later, due to two symbols
that were deprecated (and removed by default) in LAPACK 3.6.0 (see the
release notes). To fix this, edit the Trilinos file
packages/epetra/src/Epetra_LAPACK_wrappers.h and change the
#define DGGSVD_F77 F77_BLAS_MANGLE(dggsvd,DGGSVD) #define SGGSVD_F77 F77_BLAS_MANGLE(sggsvd,SGGSVD)to
#define DGGSVD_F77 F77_BLAS_MANGLE(dggsvd3,DGGSVD3) #define SGGSVD_F77 F77_BLAS_MANGLE(sggsvd3,SGGSVD3)before installing Trilinos. (Credit for this fix goes to this page.)
Trilinos sometimes searches for other libraries but can't find
them if they are not in the usual directories or have other
names. A common example are BLAS or LAPACK. In a case like
this, you may have to specifically pass the directories and/or
library names under which they can be found
cmake. For example, this may mean to add the
following flags to the call above:
-DBLAS_LIBRARY_NAMES:STRING=goto \ -DBLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \ -DLAPACK_LIBRARY_NAMES:STRING=lapack \ -DLAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64
Trilinos (via its Amesos package) can interface with a number of direct
solvers (see, for example,
this page for Trilinos 11.8). Most of them are external
packages to Trilinos and you will need to tell Trilinos configuration
scripts that you want to use them, for example via the
TrilinosWrappers::SolverDirect class. This can be tricky,
but adding defines similar to the following to the cmake command line
will achieve the goal to enable the UMFPACK and SuperLU/SuperLUDist
-DTPL_ENABLE_UMFPACK:BOOL=ON \ -DTPL_ENABLE_SuperLU:BOOL=ON \ -DTPL_ENABLE_SuperLUDist:BOOL=ON \ -DTPL_UMFPACK_INCLUDE_DIRS="/usr/include" \ -DSuperLUDist_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_DIST_3.2/SRC" \ -DTPL_SuperLUDist_LIBRARIES:FILEPATH="/path/to/SuperLU_DIST_3.2/lib/libsuperlu_dist.a" \ -DSuperLU_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_4.3/SRC" \ -DTPL_SuperLU_LIBRARIES:FILEPATH="/path/to/SuperLU_4.3/lib/libsuperlu_4.3.a"Similarly, to enable MUMPS, commands should include
-DTPL_ENABLE_MUMPS:BOOL=ON \ -DTPL_ENABLE_SCALAPACK:BOOL=ONand possibly followed by
-DTPL_MUMPS_INCLUDE_DIRS:PATH=/usr/include/openmpi-x86_64 \ -DSCALAPACK_LIBRARY_DIRS:PATH=/lib64/openmpi/lib \where you need to adjust the exact paths, of course.