Hi, I have taken some time to have a relative "small" code to reproduce the problem. The good thing is that now I can extract the exact call sequence of almost any MPI I/O call we make in our code to have a pure "mpi" calls in s simple C program, independent of our in-house code. To reproduce the bug, just compile the attached file with any mpich/master since commit b4ab2f118d (nov 8) , and launch the resulting executable with 3 processes along with the second attachment (file_for_bug.data) saved in the pwd on an *NFS* path. You should see comething like this: ERROR Returned by MPI: 604040736 ERROR_string Returned by MPI: Other I/O error , error stack: ADIOI_NFS_READSTRIDED(523): Other I/O error Bad file descriptor ERROR Returned by MPI: 268496416 ERROR_string Returned by MPI: Other I/O error , error stack: ADIOI_NFS_READSTRIDED(523): Other I/O error Operation now in progress application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 If you launch it on a local drive, it works. Can someone confirm it as reproduce the problem please? Moreover, if I launch it with valgrind, even on a local disk, it complains like this, on process 0: ==99023== Warning: invalid file descriptor -1 in syscall read() ==99023== at 0x53E2CB0: __read_nocancel (in /lib64/libc-2.19.so) ==99023== by 0x5041606: file_to_info_all (system_hints.c:101) ==99023== by 0x5041606: ADIOI_process_system_hints (system_hints.c:150) ==99023== by 0x50311B8: ADIO_Open (ad_open.c:123) ==99023== by 0x50161FD: PMPI_File_open (open.c:154) ==99023== by 0x400F91: main (mpich_mpiio_nfs_bug_read.c:42) Thanks, Eric On 21/11/17 02:49 PM, Eric Chamberland wrote:
Hi M. Latham,
I have more information now.
When I try to run my example on NFS, I have the following error code:
error #812707360 Other I/O error , error stack: ADIOI_NFS_READSTRIDED(523): Other I/O error Success
that is returned by MPI_File_read_all_begin
When I try on a local disk, everything is fine.
Here are all files about my actual build:
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_conf...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_c.tx...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_m.tx...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mi.t...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpl_...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_pm_h...
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.21.05h40m02s_mpie...
Hope this help to dig further into this issue.
Thanks,
Eric
On 15/11/17 03:55 PM, Eric Chamberland wrote:
Hi,
We are compiling with mpich/master each night since august 2016...
since nov 8, the mpich/master branch is buggy with our nighlty build tests.
Here is the nov 8 config.log:
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.08.05h36m02s_conf...
For nov 7 the configure log:
http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2017.11.07.05h36m01s_conf...
Since nov 8, on a specific ROMIO test, it is hanging indefinitely in optimized mode, and into DEBUG mode, I have a strange (yet to be debugged) assertion in our code.
I reran the test manually, and when I wrote the results on a local disk, everything is fine.
However, when I write over *NFS*, the test is faulty.
I have not yet debugged enough through this, but, I suspect something related with one of:
MPI_File_write_all_begin MPI_File_write_all_end MPI_File_read_all_begin MPI_File_read_all_end MPI_File_set_view MPI_Type_free
Am I alone to see these problems?
Thanks, Eric
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss