The growing availability and power of computing resources, has led in recent years to the production of a huge amount of data. However, writing results of numerical simulations from memory to disk is a well known bottleneck in high-performance computing. Moreover, sometimes data are stored in files without a specific organization or format, making them no portable neither accessible on other resources, severely limiting their use.
Thus, crucial issues are both the choice of the better way to write and to read data taking into account of the hardware resources available, like the bandwidth and the filesystem's properties; and the choice of the better techniques to write and to read data in order to obtain portable and accessible files.
Aim of this course is to introduce the basis for the usage of the MPI-2 I/O library for binary data, and the standard data formats as HDF5 and NetCDF, that allow for data writing and reading in an organized, efficiency and portable way
Some note about the hardware features affecting the I/O performance are also provided, in order to choose the better way to write and to read data.
Topics
Students and researchers of any scientific field, interested in improving their knowledge in production and managment of scientific data.
Pre-requisites
Knowledge and experience of Fortran, C, or C++. Experience with UNIX operating systems
Thus, crucial issues are both the choice of the better way to write and to read data taking into account of the hardware resources available, like the bandwidth and the filesystem's properties; and the choice of the better techniques to write and to read data in order to obtain portable and accessible files.
Aim of this course is to introduce the basis for the usage of the MPI-2 I/O library for binary data, and the standard data formats as HDF5 and NetCDF, that allow for data writing and reading in an organized, efficiency and portable way
Some note about the hardware features affecting the I/O performance are also provided, in order to choose the better way to write and to read data.
Topics
- I/O: basic concepts
- Managment of large scientific data
- Parallel file I/O with MPI2
- HDF5 library
Students and researchers of any scientific field, interested in improving their knowledge in production and managment of scientific data.
Pre-requisites
Knowledge and experience of Fortran, C, or C++. Experience with UNIX operating systems
