May 2015 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 08 May '15

08 May '15

On 05/08/2015 04:04 AM, Lisandro Dalcin wrote: > On 7 May 2015 at 22:00, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >> >> Thanks for reporting these two issues. These two patches (attached) address >> your test case, but it's a part of the code I don't play in often, so I'm >> going to need a review on them before committing: >> > > Rob, I think you patch 2 is not right. It will endup calling the error > handler set in COMM_WORLD. So if you set the FATAL errhandler in a > file, but the RETURNS in comm world, then things will not work as > expected. > > I think a better approach would be to somehow call > > handleFatalError(NULL, "MPI_File_call_errhandler", errorcode); > > in errutil.c, but that would require some code refactoring to access > that function. I've done that refactoring -- possibly not in the most MPICH-y of ways -- in the attached patch 0002 ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 08 May '15

08 May '15

On 05/08/2015 04:04 AM, Lisandro Dalcin wrote: > On 7 May 2015 at 22:00, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >> >> Thanks for reporting these two issues. These two patches (attached) address >> your test case, but it's a part of the code I don't play in often, so I'm >> going to need a review on them before committing: >> > > Rob, I think you patch 2 is not right. It will endup calling the error > handler set in COMM_WORLD. So if you set the FATAL errhandler in a > file, but the RETURNS in comm world, then things will not work as > expected. > > I think a better approach would be to somehow call > > handleFatalError(NULL, "MPI_File_call_errhandler", errorcode); > > in errutil.c, but that would require some code refactoring to access > that function. Thought I had found a shortcut to that function, but I guess not. Well, I'll have to bite the bullet and write a little accessor function for files like there is for comm and win. ==rob > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Lisandro Dalcin 08 May '15

08 May '15

On 7 May 2015 at 22:00, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > Thanks for reporting these two issues. These two patches (attached) address > your test case, but it's a part of the code I don't play in often, so I'm > going to need a review on them before committing: > Rob, I think you patch 2 is not right. It will endup calling the error handler set in COMM_WORLD. So if you set the FATAL errhandler in a file, but the RETURNS in comm world, then things will not work as expected. I think a better approach would be to somehow call handleFatalError(NULL, "MPI_File_call_errhandler", errorcode); in errutil.c, but that would require some code refactoring to access that function. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 06:58 AM, Lisandro Dalcin wrote: > This is using the preview release 3.2b2, I discovered this issue > months ago and it seems I forgot to report it. Thanks for reporting these two issues. These two patches (attached) address your test case, but it's a part of the code I don't play in often, so I'm going to need a review on them before committing: ==rob > > 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS > instead of value under normal operation (as the MPI spec says), and > particularly if the error handler is set to MPI_ERRORS_RETURN. This > bug is trivial to fix. > > 2) Setting the predefined error handler MPI_ERRORS_ARE_FATAL and > calling MPI_File_call_errhandler(file, errcode) should "gracefully" > abort the processes the MPI way, instead of segfaulting. The problem > here is that ROMIO has to somehow handle MPI_ERRORS_ARE_FATAL in some > special way I'm not sure how to implement, right now it is trying to > invoke a callback that is set to NULL. > > I'm attaching a test case showcasing the two issues, and pasting output below: > > $ cat file_errhdl.c > #include <mpi.h> > #include <stdio.h> > > int main(int argc, char *argv[]) > { > int ierr; > MPI_File fh; > MPI_Init(&argc, &argv); > > MPI_File_open(MPI_COMM_WORLD,"/tmp/datafile", > MPI_MODE_CREATE|MPI_MODE_RDWR|MPI_MODE_DELETE_ON_CLOSE, > MPI_INFO_NULL,&fh); > > MPI_File_set_errhandler(fh,MPI_ERRORS_RETURN); > ierr = MPI_File_call_errhandler(fh,MPI_ERR_FILE); > printf("ierr: %d, expected: %d\n", ierr, (int)MPI_SUCCESS); > > MPI_File_set_errhandler(fh,MPI_ERRORS_ARE_FATAL); > MPI_File_call_errhandler(fh,MPI_ERR_FILE); /* should abort */ > MPI_File_close(&fh); > > MPI_Finalize(); > return 0; > } > > $ mpicc file_errhdl.c > > $ ./a.out > ierr: 27, expected: 0 > Segmentation fault (core dumped) > > $ valgrind -q ./a.out > ierr: 27, expected: 0 > ==30036== Jump to the invalid address stated on the next line > ==30036== at 0x0: ??? > ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler > (file_call_errhandler.c:105) > ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) > ==30036== Address 0x0 is not stack'd, malloc'd or (recently) free'd > ==30036== > ==30036== > ==30036== Process terminating with default action of signal 11 (SIGSEGV) > ==30036== Bad permissions for mapped region at address 0x0 > ==30036== at 0x0: ??? > ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler > (file_call_errhandler.c:105) > ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) > Segmentation fault (core dumped) > > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 11:34 AM, Lisandro Dalcin wrote: > On 7 May 2015 at 18:44, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >>> 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS >>> instead of value under normal operation (as the MPI spec says), and >>> particularly if the error handler is set to MPI_ERRORS_RETURN. This >>> bug is trivial to fix. >> >> >> Trivial to fix...if we only had to worry about C and Fortran! > > I'm probably missing something here, so I have to ask. > > I said this was "trivial to fix" for the particular case of > MPI_ERRORS_RETURN. How should C++ bindings behave in case > MPI_ERRORS_RETURN is set? In other cases, I agree with you, fixing > this beast is not easy. However, you have a point about C++ bindings > being deprecated. It is I who was missing something. I read the code too fast, saw the special C++ case, and thought it mattered for MPI_ERRORS_RETURN. This one is easy. The core dumping one is a little more work but I see an angle of attack now. ==rob > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Lisandro Dalcin 07 May '15

07 May '15

On 7 May 2015 at 18:44, Rob Latham <robl(a)mcs.anl.gov> wrote: > >> 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS >> instead of value under normal operation (as the MPI spec says), and >> particularly if the error handler is set to MPI_ERRORS_RETURN. This >> bug is trivial to fix. > > > Trivial to fix...if we only had to worry about C and Fortran! I'm probably missing something here, so I have to ask. I said this was "trivial to fix" for the particular case of MPI_ERRORS_RETURN. How should C++ bindings behave in case MPI_ERRORS_RETURN is set? In other cases, I agree with you, fixing this beast is not easy. However, you have a point about C++ bindings being deprecated. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 06:58 AM, Lisandro Dalcin wrote: > This is using the preview release 3.2b2, I discovered this issue > months ago and it seems I forgot to report it. We really should get mpi4py into the nightly tests. You seem to find a lot of interesting cases. > 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS > instead of value under normal operation (as the MPI spec says), and > particularly if the error handler is set to MPI_ERRORS_RETURN. This > bug is trivial to fix. Trivial to fix...if we only had to worry about C and Fortran! (and maybe we can ignore C++ regressions now that the bindings are deprecated? ) It looks like we have hijacked the error code (years and years ago, in to send information to the C++ binidngs: http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/errhan/file_call_errhandl… The change dates back to 2008: http://git.mpich.org/mpich.git/commit/abbe596a and it looks like the Comm and Window error handlers stomp on the error code, too. This is ticket #2259 if you want to follow along https://trac.mpich.org/projects/mpich/ticket/2259 I'll respond to #2 in another email ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Issue with too many calls to MPIR_Add_finalize
by Lisandro Dalcin 07 May '15

07 May '15

I'm running the testsuite of my Python wrappers with MPICH 3.2b2. The runs are aborting this way: $ python test/runtests.py -q overflow in finalize stack! internal ABORT - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=13 : system msg for write_line failure : Bad file descriptor It seems that the routine MPIR_Add_finalize() is being called more than 32 times, exhausting the static allocation space of 32 entries. I'm attaching a text file with a gdb sessions showing all the calls to this routine. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Two issues with MPI_File error handling
by Lisandro Dalcin 07 May '15

07 May '15

This is using the preview release 3.2b2, I discovered this issue months ago and it seems I forgot to report it. 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS instead of value under normal operation (as the MPI spec says), and particularly if the error handler is set to MPI_ERRORS_RETURN. This bug is trivial to fix. 2) Setting the predefined error handler MPI_ERRORS_ARE_FATAL and calling MPI_File_call_errhandler(file, errcode) should "gracefully" abort the processes the MPI way, instead of segfaulting. The problem here is that ROMIO has to somehow handle MPI_ERRORS_ARE_FATAL in some special way I'm not sure how to implement, right now it is trying to invoke a callback that is set to NULL. I'm attaching a test case showcasing the two issues, and pasting output below: $ cat file_errhdl.c #include <mpi.h> #include <stdio.h> int main(int argc, char *argv[]) { int ierr; MPI_File fh; MPI_Init(&argc, &argv); MPI_File_open(MPI_COMM_WORLD,"/tmp/datafile", MPI_MODE_CREATE|MPI_MODE_RDWR|MPI_MODE_DELETE_ON_CLOSE, MPI_INFO_NULL,&fh); MPI_File_set_errhandler(fh,MPI_ERRORS_RETURN); ierr = MPI_File_call_errhandler(fh,MPI_ERR_FILE); printf("ierr: %d, expected: %d\n", ierr, (int)MPI_SUCCESS); MPI_File_set_errhandler(fh,MPI_ERRORS_ARE_FATAL); MPI_File_call_errhandler(fh,MPI_ERR_FILE); /* should abort */ MPI_File_close(&fh); MPI_Finalize(); return 0; } $ mpicc file_errhdl.c $ ./a.out ierr: 27, expected: 0 Segmentation fault (core dumped) $ valgrind -q ./a.out ierr: 27, expected: 0 ==30036== Jump to the invalid address stated on the next line ==30036== at 0x0: ??? ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler (file_call_errhandler.c:105) ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) ==30036== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==30036== ==30036== ==30036== Process terminating with default action of signal 11 (SIGSEGV) ==30036== Bad permissions for mapped region at address 0x0 ==30036== at 0x0: ??? ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler (file_call_errhandler.c:105) ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) Segmentation fault (core dumped) -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0