Implementation of parallel I/O

[frank_stulle]

To get a dedicated and parallel I/O capable fileserver for our "cluster" I was asked by our IT department to provide some information on the implementation of parallel I/O and the requirements. Could you please tell me how you have implemented this in Pelegant?

Best regards
Frank

[ywang25]

Pelegant integrated parallel SDDS I/O for parallel file operations. The parallel SDDS I/O is implemented on top of MPI-IO library, which usually comes with an MPI distribution. More technical details about how the parallel I/O was implemented in Pelegant can be found in the attached paper: "Parallel SDDS: A Scientific High-Performance I/O Interface". Pelegant has been used successfully on several parallel file systems, such as GPFS, Lustre, PVFS. If you use openMPI, it is not recommended to use Pelegant with Infiniband network, as some system calls in Pelegant are not supported for this openMPI-Infiniband combination at the current stage. MPICH2, or openMPI with other high bandwidth networks should have no such issue.

Yusong

[frank_stulle]

Many thanks for the information. Could you tell me even more details, e.g. which MPI functions are used for I/O, and some more technical aspects of the implementation? Which functionality has a file system to provide to allow proper parallel I/O?

Frank

[ywang25]

Here are the list of the MPI functions used in Pelegant
MPI_File_open
MPI_File_write
MPI_File_set_view
MPI_File_close
MPI_File_seek
MPI_File_read
MPI_File_read_all
MPI_File_sync
MPI_File_set_size
MPI_File_write_at
MPI_File_write_all

For information about the supported file systems, the following link can provide some useful information, although it is a little bit outdated.
http://www.mcs.anl.gov/research/projects/romio/
ROMIO is the library distributed with MPICH for MPI-IO. I think openMPI uses it too.

Yusong
Yusong

[frank_stulle]

After some discussions with IT guys it seems the basic question is: Why does parallel I/O not work on AFS? What functionality is missing which would be required to make it work? That seems to be a pretty general question. But I failed to find anything on this in the internet. Is this question too simple, too complicated or too absurd? Do you have an answer?

Many thanks
Frank

[ywang25]

"The reason is the last process to close file overwrites the updates of all the other processes on AFS."
AFS might not be designed for parallel file operations, unless they have a new version to support the parallel file system. An easy way to try it is using the IO examples distributed with your MPI library. Pelegant should have no problem for parallel I/O if the IO example tests pass on a particular file system. The best place to answer the question would be AFS support team. You can also post the question to the mpich-discuss list (mpich-discuss@mcs.anl.gov):
http://www.mcs.anl.gov/research/project ... ?s=support
Several HPC experts on the list can answer the parallel file system related questions.

Yusong

[frank_stulle]

Many thanks for answering the question. And sorry for not remembering your post in the previous discussion where you already answered everything.

Frank