Classes
class	DataSet

class	File

class	Group

class	HDF5Object

Detailed Description

Namespace containing the HDF5 interface.

The Hierarchical Data Format (HDF) is a cross platform and a high I/O performance format designed to store large amounts of data. It supports serial and MPI I/O access. This set of classes provides an interface to the HDF5 library.

Groups, Datasets and attributes

An HDF5 file is organized in groups and datasets. Groups can contain datasets and other groups. Datasets are objects composed by a collection of data elements. Datasets are equivalent to tensors and matrices. In addition, attributes can be attached to the root file, a group or a dataset. An HDF5 attribute is a small meta data. The methods HDF5Object::get_attribute() and HDF5Object::set_attribute() can be used to get and set attributes.

An example is shown below

HDF5::File data_file(filename, HDF5::File::Mode::create);
double double_attribute = 2.2;
data_file.set_attribute("double_attribute", double_attribute);
auto group = data_file.create_group("group");
group.set_attribute("simulation_type", std::string("elastic_equation"));
auto dataset = group.create_dataset<double>("dataset_name", dimensions);
dataset.set_attribute("complex_double_attribute",
                      std::complex<double>(2,2.3));

MPI I/O

An HDF5 file can be opened/created with serial (one single process) or MPI support (several processes access the same HDF5 file). File::File(const std::string &, const FileAccessMode) opens/creates an HDF5 file for serial operations. File::File(const std::string &, const FileAccessMode, const MPI_Comm) creates or opens an HDF5 file in parallel using MPI. The HDF5 calls that modify the structure of the file are always collective, whereas writing and reading raw data in a dataset can be done independently or collectively. Collective access is usually faster since it allows MPI to do optimizations. In the deal.II's HDF5 interface all the calls are set to collective in order to maximize the performance. This means that all the MPI processes have to contribute to every single call, even if they don't have data to write. MPI HDF5 requires that deal.II and HDF5 have been compiled with MPI support.

Write a hyperslab in parallel

Hyperslabs are portions of datasets. A hyperslab can be a contiguous collection of points in a dataset, or it can be a regular pattern of points or blocks in a datataset. Hyperslabs are equivalent to python numpy and h5py slices. See the Dataspaces and Data Transfer section in the HDF5 User's Guide. See as well the H5Sselect_hyperslab definition.

The example below shows how to write a simple rectangular hyperslab. The offset defines the origin of the hyperslab in the original dataset. The dimensions of the hyperslab are hyperslab_dimensions = {2, 5}.

std::vector<hsize_t> dataset_dimensions = {50, 30};
auto dataset = group.create_dataset<double>("name", dataset_dimensions);
if (Utilities::MPI::this_mpi_process(mpi_communicator) == 0)
  {
    // data can be std::vector, FullMatrix or Vector
    FullMatrix<double> data = {...};
    std::vector<hsize_t> hyperslab_dimensions = {2, 3};
    std::vector<hsize_t> hyperslab_offset     = {1, 2};
    dataset.write_hyperslab(hyperslab_data,
                            hyperslab_offset,
                            hyperslab_dimensions);
  }
else
  {
    dataset.write_none<double>();
  }

The function DataSet::write_hyperslab(const Container &,const std::vector<hsize_t> &, const std::vector<hsize_t> &) is used to write and simple hyperslabs and the function DataSet::write_hyperslab(const Container &,const std::vector<hsize_t> &, const std::vector<hsize_t> &, const std::vector<hsize_t> &, const std::vector<hsize_t> &, const std::vector<hsize_t> &) is used to write complex hyperslabs.

Write unordered data in parallel

The example below shows how to write a selection of data.

std::vector<hsize_t> dataset_dimensions = {50, 30};
auto dataset = group.create_dataset<double>("name", dataset_dimensions);
if (Utilities::MPI::this_mpi_process(mpi_communicator) == 0)
  {
    std::vector<hsize_t> coordinates = {0,
                                        0, // first point
                                        0,
                                        2, // second point
                                        3,
                                        4, // third point
                                        25,
                                        12}; // fourth point
    std::vector<double>  data        = {2, 3, 5, 6};
    dataset.write_selection(data, coordinates);
  }
else if (Utilities::MPI::this_mpi_process(mpi_communicator) == 1)
  {
    std::vector<hsize_t> coordinates = {5,
                                        0, // first point
                                        0,
                                        4, // second point
                                        5,
                                        4, // third point
                                        26,
                                        12}; // fourth point
    std::vector<double>  data        = {9, 4, 7, 6};
    dataset.write_selection(data, coordinates);
  }
else
  {
    dataset.write_none<double>();
  }

Query the I/O mode that HDF5 used in the last parallel I/O call

The default access mode in the deal.II's HDF5 C++ interface is collective which is typically faster since it allows MPI to do more optimizations. In some cases, such as when there is type conversion, the HDF5 library can decide to do independent I/O instead of collective I/O, even if the user asks for collective I/O. See the following article. In cases where maximum performance is a requirement, it is important to make sure that all MPI read/write operations are collective. The HDF5 library provides API routines that can be used after the read/write I/O operations to query the I/O mode. If DataSet::query_io_mode is True, then after every read/write operation the deal.II's HDF5 interface calls the routines H5Pget_mpio_actual_io_mode() and H5Pget_mpio_no_collective_cause(). The results are stored in DataSet::io_mode, DataSet::local_no_collective_cause and DataSet::get_global_no_collective_cause. We suggest to query the I/O mode only in Debug mode because it requires calling additional HDF5 routines.

The following code can be used to query the I/O method.

auto dataset = group.create_dataset<double>("name", dimensions);
#ifdef DEBUG
dataset.set_query_io_mode(true);
#endif
if (Utilities::MPI::this_mpi_process(mpi_communicator) == 0)
  {
    dataset.write(data);
  }
else
  {
    dataset.write_none<double>();
  }
if(dataset.get_query_io_mode()){
  pcout << "IO mode: " << dataset.io_mode() << std::endl;
  pcout << "Local no collective cause: "
        << dataset.local_no_collective_cause() << std::endl;
  pcout << "Global no collective cause: "
        << dataset.get_global_no_collective_cause() <<
std::endl;
}

If the write operation was collective then the output should be

IO mode: H5D_MPIO_CONTIGUOUS_COLLECTIVE
Local no collective cause: H5D_MPIO_COLLECTIVE
Global no collective cause: H5D_MPIO_COLLECTIVE

See DataSet::get_io_mode(), DataSet::get_local_no_collective_cause() and DataSet::get_global_no_collective_cause() for all the possible return codes.

Rank of HDF5 datasets and hyperslabs

The deal.II's HDF5 interface can be used to write/read data to datasets and hyperslabs of any particular rank. FullMatrix can only be used to write/read data to datasets and hyperslabs of rank 2. In the other hand, std::vector and Vector can be used to write/read data to datasets and hyperslabs of rank 1, 2, 3 and higher, the data is organized in row-major order which is commonly used in C and C++ matrices. We can re-write the code from the previous section using std::vector

// Dataset of rank 2. dim_0 = 50, dim_1 = 30
std::vector<hsize_t> dataset_dimensions = {50, 30};
auto dataset = group.create_dataset<double>("name", dataset_dimensions);
if (Utilities::MPI::this_mpi_process(mpi_communicator) == 0)
  {
    // data can be std::vector, FullMatrix or Vector
    std::vector<double> data = {0,1,2,3,4,5};
    // hyperslab of rank 2. dim_0 = 2 and dim_1 = 3
    std::vector<hsize_t> hyperslab_dimensions = {2, 3};
    std::vector<hsize_t> hyperslab_offset     = {1, 2};
    dataset.write_hyperslab(hyperslab_data,
                            hyperslab_offset,
                            hyperslab_dimensions);
  }
else
  {
    dataset.write_none<double>();
  }

The previous code writes the following hyperslab matrix

1
3
5

Data exchange with python scripts

The HDF5 format can be used to exchange data with python scripts. The strings are stored as HDF5 variable-length UTF-8 strings and the complex numbers are stored as HDF5 compound datatypes compatible with h5py and numpy.

The following python script writes the parameters for a deal.II simulation:

h5_file = h5py.File('simulation.hdf5','w')
data = h5_file.create_group('data')
data.attrs['nb_frequency_points'] = 50 # int
data.attrs['rho'] = 2300.5 # double
data.attrs['save_vtk_files'] = True # bool
data.attrs['simulation_type'] = 'elastic_equation' # utf8 string

C++ deal.II simulation with MPI HDF5:

hdf5::File data_file("simulation.hdf5",
                     MPI_COMM_WORLD,
                     HDF5::FileAccessMode::Mode::open);
hdf5::Group data = data_file.open_group("data");
auto nb_frequency_points = data.get_attribute<int>("nb_frequency_points");
auto rho = data.get_attribute<double>("rho");
auto save_vtk_files = data.get_attribute<bool>("save_vtk_files");
auto simulation_type = data.get_attribute<std::string>("simulation_type");
std::vector<std::complex<double>> displacement = {...};
data.write_dataset("displacement", displacement);
// Write the simulation metadata
data.set_attribute("active_cells", triangulation.n_active_cells());

Read the simulation results with python:

h5_file = h5py.File('simulation.hdf5','r+')
data = h5_file['data']
displacement = data['displacement'] # complex128 dtype
active_cells = data.attrs['degrees_of_freedom'])

HDF5 and thread safety

By default HDF5 is not thread-safe. The HDF5 library can be configured to be thread-safe, see the HDF5 documentation. The thread-safe HDF5 version serializes the API but does not provide any level of concurrency. To achieve high parallel performance with HDF5, we advice to use HDF5 with MPI.

Author: Daniel Garcia-Sanchez, 2018, 2019.

Classes