Utilities::MPI::MPI_InitFinalize Class Reference

#include <deal.II/base/mpi.h>

Inheritance diagram for Utilities::MPI::MPI_InitFinalize:

Public Member Functions
	MPI_InitFinalize (int &argc, char **&argv, const unsigned int max_num_threads=numbers::invalid_unsigned_int)
	~MPI_InitFinalize ()=default
void	finalize ()

Static Public Member Functions
static void	register_request (MPI_Request &request)
static void	unregister_request (MPI_Request &request)

Static Public Attributes
static Signals	signals = InitFinalize::Signals()

Private Attributes
InitializeLibrary	libraries
bool	is_finalized = false
bool	finalize_petscslepc

Static Private Attributes
static std::set< MPI_Request * >	requests

Detailed Description

A class that is used to initialize the MPI system at the beginning of a program and to shut it down again at the end. It also allows you to control the number of threads used within each MPI process.

If deal.II is configured with PETSc, PETSc will be initialized via PetscInitialize in the beginning (constructor of this class) and de-initialized via PetscFinalize at the end (i.e., in the destructor of this class). The same is true for SLEPc.

If deal.II is configured with p4est, that library will also be initialized in the beginning, and de-initialized at the end (by calling sc_init(), p4est_init(), and sc_finalize()).

If a program uses MPI one would typically just create an object of this type at the beginning of main(). The constructor of this class then runs MPI_Init() with the given arguments and also initializes the other libraries mentioned above. At the end of the program, the compiler will invoke the destructor of this object which in turns calls MPI_Finalize to shut down the MPI system.

This class is used in step-17, step-18, step-40, step-32, and several others.

Note

This class performs initialization of the MPI subsystem as well as the dependent libraries listed above through the MPI_COMM_WORLD communicator. This means that you will have to create an MPI_InitFinalize object on all MPI processes, whether or not you intend to use deal.II on a given processor. In most use cases, one will of course want to work on all MPI processes using essentially the same program, and so this is not an issue. But if you plan to run deal.II-based work on only a subset of MPI processes, using an MPI communicator that is a subset of MPI_COMM_WORLD (for example, in client-server settings where only a subset of processes is responsible for the finite element communications and the remaining processes do other things), then you still need to create this object here on all MPI processes at the beginning of the program because it uses MPI_COMM_WORLD during initialization.

Definition at line 1041 of file mpi.h.

Constructor & Destructor Documentation

MPI_InitFinalize()

Utilities::MPI::MPI_InitFinalize::MPI_InitFinalize	(	int &	argc,
		char **&	argv,
		const unsigned int	max_num_threads = numbers::invalid_unsigned_int
	)

Initialize MPI (and, if deal.II was configured to use it, PETSc) and set the number of threads used by deal.II (via the underlying Threading Building Blocks library) to the given parameter.

Parameters

[in,out]	argc	A reference to the 'argc' argument passed to main. This argument is used to initialize MPI (and, possibly, PETSc) as they read arguments from the command line.
[in,out]	argv	A reference to the 'argv' argument passed to main.
[in]	max_num_threads	The maximal number of threads this MPI process should utilize. If this argument is set to numbers::invalid_unsigned_int (the default value), then the number of threads is determined automatically in the following way: the number of threads to run on this MPI process is set in such a way that all of the cores in your node are spoken for. In other words, if you have started one MPI process per node, setting this argument is equivalent to setting it to the number of cores present in the node this MPI process runs on. If you have started as many MPI processes per node as there are cores on each node, then this is equivalent to passing 1 as the argument. On the other hand, if, for example, you start 4 MPI processes on each 16-core node, then this option will start 4 worker threads for each node. If you start 3 processes on an 8 core node, then they will start 3, 3 and 2 threads, respectively.

Note

This function calls MultithreadInfo::set_thread_limit() with either max_num_threads or, following the discussion above, a number of threads equal to the number of cores allocated to this MPI process. However, MultithreadInfo::set_thread_limit() in turn also evaluates the environment variable DEAL_II_NUM_THREADS. Finally, the worker threads can only be created on cores to which the current MPI process has access to; some MPI implementations limit the number of cores each process may access to one or a subset of cores in order to ensure better cache behavior. Consequently, the number of threads that will really be created will be the minimum of the argument passed here, the environment variable (if set), and the number of cores accessible to the thread.

MultithreadInfo::set_thread_limit() can only work if it is called before any threads are created. The safest place for a call to it is therefore at the beginning of main(). Consequently, this extends to the current class: the best place to create an object of this type is also at or close to the top of main().

Definition at line 667 of file mpi.cc.

~MPI_InitFinalize()

Utilities::MPI::MPI_InitFinalize::~MPI_InitFinalize ( )

default

Destructor. Calls MPI_Finalize() in case this class owns the MPI process.

Member Function Documentation

finalize()

void InitFinalize::finalize ( )

inherited

Finalize deal.II and the external libraries initialize by deal.II.

Definition at line 328 of file init_finalize.cc.

register_request()

void InitFinalize::register_request ( MPI_Request &  request )

staticinherited

Register a reference to an MPI_Request on which we need to call MPI_Wait before calling MPI_Finalize.

The object request needs to exist when MPI_Finalize is called, which means the request is typically statically allocated. Otherwise, you need to call unregister_request() before the request goes out of scope. Note that it is acceptable for a request to be already waited on (and consequently reset to MPI_REQUEST_NULL).

It is acceptable to call this function more than once with the same instance (as it is done in the example below).

Typically, this function is used by CollectiveMutex and not directly, but it can also be used directly like this:

void my_fancy_communication()
{
  static MPI_Request request = MPI_REQUEST_NULL;
  MPI_InitFinalize::register_request(request);
  MPI_Wait(&request, MPI_STATUS_IGNORE);
  // [some algorithm that is not safe to be executed twice in a row.]
  MPI_IBarrier(comm, &request);
}

Definition at line 303 of file init_finalize.cc.

unregister_request()

void InitFinalize::unregister_request ( MPI_Request &  request )

staticinherited

Unregister a request previously added using register_request().

Definition at line 312 of file init_finalize.cc.

Member Data Documentation

signals

InitFinalize::Signals InitFinalize::signals = InitFinalize::Signals()

staticinherited

Definition at line 201 of file init_finalize.h.

libraries

InitializeLibrary InitFinalize::libraries

privateinherited

Definition at line 204 of file init_finalize.h.

requests

std::set< MPI_Request * > InitFinalize::requests

staticprivateinherited

Requests to MPI_Wait before finalizing

Definition at line 209 of file init_finalize.h.

is_finalized

bool InitFinalize::is_finalized = false

privateinherited

Definition at line 211 of file init_finalize.h.

finalize_petscslepc

bool InitFinalize::finalize_petscslepc

privateinherited

Definition at line 214 of file init_finalize.h.

---

The documentation for this class was generated from the following files:

• include/deal.II/base/mpi.h
• source/base/mpi.cc