December 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] FW: Potential MPICH problem
by Rob Latham 31 Dec '16

31 Dec '16

On 12/30/2016 07:48 AM, Richard Warren wrote: > Hi Rob, > Many thanks for your note and heads-up message about your vacation. The vacation situation applies to most of our group, so it’s a good time for me to investigate the problem a bit more. Along with your suggestions, I’m thinking that I might work on writing an MPI only reproducer, e.g. I can probably produce a trace of the MPI function calls that happen during the file close operation and simply package that into a test that I can share with you… > I'd be surprised if it's a problem with MPI_FILE_SET_VIEW itself. like, if you put an MPI_BARRIER in there before the call, I bet you'd find a problem in MPI_BARRIER. ==rob > Thanks again, > Richard > > On 12/29/16, 10:24 PM, "Rob Latham" <robl(a)mcs.anl.gov> wrote: > > > > On 12/29/2016 08:28 AM, Richard Warren wrote: > > > > > > Hi All, > > > > I’m writing to get some advice and possibly report a bug. The > > circumstances are that we are currently working on updating HDF5 > > functionality and have run into an issue running a parallel test of a > > CFD code (benchmark.hdf) from the CGNS code base > > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that > > our failure occurs during a call to MPI_File_set_view, with the failure > > signature as follows: > > Interesting. I can take a look at it but I'll warn you I'm on vacation > until 9 January. > > Stuff I would do: > - hook up valgrind or address sanitizer to see if anything unexpected > does crop up (your "message queue corrupted" theory). > - get a gdb backtrace of the failing MPI_File_set_view including > caller's stack. > - maybe try out with older/newer versions, though lots of tests invoke > MPI_FILE_SET_VIEW without problems, so that's a bit of a long shot. > > ==rob > > > > > > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > > > MPIR_Barrier_impl(337)..............: Failure during collective > > > > MPIR_Barrier_impl(330)..............: > > > > MPIR_Barrier(294)...................: > > > > MPIR_Barrier_intra(151).............: > > > > barrier_smp_intra(111)..............: > > > > MPIR_Bcast_impl(1462)...............: > > > > MPIR_Bcast(1486)....................: > > > > MPIR_Bcast_intra(1295)..............: > > > > MPIR_Bcast_binomial(241)............: > > > > MPIC_Recv(352)......................: > > > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > > received but buffer size is 1 > > > > [cli_1]: aborting job: > > > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > > > MPIR_Barrier_impl(337)..............: Failure during collective > > > > MPIR_Barrier_impl(330)..............: > > > > MPIR_Barrier(294)...................: > > > > MPIR_Barrier_intra(151).............: > > > > barrier_smp_intra(111)..............: > > > > MPIR_Bcast_impl(1462)...............: > > > > MPIR_Bcast(1486)....................: > > > > MPIR_Bcast_intra(1295)..............: > > > > MPIR_Bcast_binomial(241)............: > > > > MPIC_Recv(352)......................: > > > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > > received but buffer size is 1 > > > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > > H5F_close: Assertion `f->file_id > 0' failed. > > > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > > > PMPI_Allgather(1002)......................: > > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > > > MPIR_Allgather_impl(842)..................: > > > > MPIR_Allgather(801).......................: > > > > MPIR_Allgather_intra(216).................: > > > > MPIC_Sendrecv(475)........................: > > > > MPIC_Wait(243)............................: > > > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > > handling an event returned by MPIDU_Sock_Wait() > > > > MPIDI_CH3I_Progress_handle_sock_event(451): > > > > MPIDU_Socki_handle_read(649)..............: connection failure > > (set=0,sock=1,errno=104:Connection reset by peer) > > > > [cli_0]: aborting job: > > > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > > > PMPI_Allgather(1002)......................: > > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > > > MPIR_Allgather_impl(842)..................: > > > > MPIR_Allgather(801).......................: > > > > MPIR_Allgather_intra(216).................: > > > > MPIC_Sendrecv(475)........................: > > > > MPIC_Wait(243)............................: > > > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > > handling an event returned by MPIDU_Sock_Wait() > > > > MPIDI_CH3I_Progress_handle_sock_event(451): > > > > MPIDU_Socki_handle_read(649)..............: connection failure > > (set=0,sock=1,errno=104:Connection reset by peer) > > > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > > H5F_close: Assertion `f->file_id > 0’ failed. > > > > > > > > Please note that the above trace was the original stacktrace which > > appears to utilize sockets, though I’ve reproduced the same problem by > > running on an SMP with shared memory. > > > > > > > > While it’s not definitive that the issue has anything to do with the > > above stack trace, the very same benchmark runs perfectly well utilizing > > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > > available from your download site and with OpenMPI 2.0.1 (also their > > latest download). Both MPI releases were built from source on my Fedora > > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > > > > > Given that the synchronous calls into MPI_File_set_view appear to be > > coded correctly AND that there isn’t much in the way of input parameters > > that would cause problems (other than incorrect coding), we tend to > > believe that the internal message queues between processes may somehow > > be corrupted. This impression is strengthened by the fact that our > > recent codebase changes (which are unrelated to the actual calls to > > MPI_File_set_view) may have introduced this issue. Note too, that the > > code paths to MPI_File_set_view have been taken many times previously > > and those function calls have all succeeded. > > > > > > > > Are there any suggestions out there as to how to further debug this > > potential corruption issue? > > > > Many thanks, > > > > Richard A. Warren > > > > > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] FW: Potential MPICH problem
by Richard Warren 30 Dec '16

30 Dec '16

Hi Rob, Many thanks for your note and heads-up message about your vacation. The vacation situation applies to most of our group, so it’s a good time for me to investigate the problem a bit more. Along with your suggestions, I’m thinking that I might work on writing an MPI only reproducer, e.g. I can probably produce a trace of the MPI function calls that happen during the file close operation and simply package that into a test that I can share with you… Thanks again, Richard On 12/29/16, 10:24 PM, "Rob Latham" <robl(a)mcs.anl.gov> wrote: On 12/29/2016 08:28 AM, Richard Warren wrote: > > > Hi All, > > I’m writing to get some advice and possibly report a bug. The > circumstances are that we are currently working on updating HDF5 > functionality and have run into an issue running a parallel test of a > CFD code (benchmark.hdf) from the CGNS code base > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that > our failure occurs during a call to MPI_File_set_view, with the failure > signature as follows: Interesting. I can take a look at it but I'll warn you I'm on vacation until 9 January. Stuff I would do: - hook up valgrind or address sanitizer to see if anything unexpected does crop up (your "message queue corrupted" theory). - get a gdb backtrace of the failing MPI_File_set_view including caller's stack. - maybe try out with older/newer versions, though lots of tests invoke MPI_FILE_SET_VIEW without problems, so that's a bit of a long shot. ==rob > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > [cli_1]: aborting job: > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0' failed. > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > [cli_0]: aborting job: > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0’ failed. > > > > Please note that the above trace was the original stacktrace which > appears to utilize sockets, though I’ve reproduced the same problem by > running on an SMP with shared memory. > > > > While it’s not definitive that the issue has anything to do with the > above stack trace, the very same benchmark runs perfectly well utilizing > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > available from your download site and with OpenMPI 2.0.1 (also their > latest download). Both MPI releases were built from source on my Fedora > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > Given that the synchronous calls into MPI_File_set_view appear to be > coded correctly AND that there isn’t much in the way of input parameters > that would cause problems (other than incorrect coding), we tend to > believe that the internal message queues between processes may somehow > be corrupted. This impression is strengthened by the fact that our > recent codebase changes (which are unrelated to the actual calls to > MPI_File_set_view) may have introduced this issue. Note too, that the > code paths to MPI_File_set_view have been taken many times previously > and those function calls have all succeeded. > > > > Are there any suggestions out there as to how to further debug this > potential corruption issue? > > Many thanks, > > Richard A. Warren > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] FW: Potential MPICH problem
by Rob Latham 29 Dec '16

29 Dec '16

On 12/29/2016 08:28 AM, Richard Warren wrote: > > > Hi All, > > I’m writing to get some advice and possibly report a bug. The > circumstances are that we are currently working on updating HDF5 > functionality and have run into an issue running a parallel test of a > CFD code (benchmark.hdf) from the CGNS code base > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that > our failure occurs during a call to MPI_File_set_view, with the failure > signature as follows: Interesting. I can take a look at it but I'll warn you I'm on vacation until 9 January. Stuff I would do: - hook up valgrind or address sanitizer to see if anything unexpected does crop up (your "message queue corrupted" theory). - get a gdb backtrace of the failing MPI_File_set_view including caller's stack. - maybe try out with older/newer versions, though lots of tests invoke MPI_FILE_SET_VIEW without problems, so that's a bit of a long shot. ==rob > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > [cli_1]: aborting job: > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0' failed. > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > [cli_0]: aborting job: > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0’ failed. > > > > Please note that the above trace was the original stacktrace which > appears to utilize sockets, though I’ve reproduced the same problem by > running on an SMP with shared memory. > > > > While it’s not definitive that the issue has anything to do with the > above stack trace, the very same benchmark runs perfectly well utilizing > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > available from your download site and with OpenMPI 2.0.1 (also their > latest download). Both MPI releases were built from source on my Fedora > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > Given that the synchronous calls into MPI_File_set_view appear to be > coded correctly AND that there isn’t much in the way of input parameters > that would cause problems (other than incorrect coding), we tend to > believe that the internal message queues between processes may somehow > be corrupted. This impression is strengthened by the fact that our > recent codebase changes (which are unrelated to the actual calls to > MPI_File_set_view) may have introduced this issue. Note too, that the > code paths to MPI_File_set_view have been taken many times previously > and those function calls have all succeeded. > > > > Are there any suggestions out there as to how to further debug this > potential corruption issue? > > Many thanks, > > Richard A. Warren > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

FW: Potential MPICH problem
by Richard Warren 29 Dec '16

29 Dec '16

Hi All, I’m writing to get some advice and possibly report a bug. The circumstances are that we are currently working on updating HDF5 functionality and have run into an issue running a parallel test of a CFD code (benchmark.hdf) from the CGNS code base<https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that our failure occurs during a call to MPI_File_set_view, with the failure signature as follows: [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 Fatal error in PMPI_Barrier: Message truncated, error stack: PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed MPIR_Barrier_impl(337)..............: Failure during collective MPIR_Barrier_impl(330)..............: MPIR_Barrier(294)...................: MPIR_Barrier_intra(151).............: barrier_smp_intra(111)..............: MPIR_Bcast_impl(1462)...............: MPIR_Bcast(1486)....................: MPIR_Bcast_intra(1295)..............: MPIR_Bcast_binomial(241)............: MPIC_Recv(352)......................: MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes received but buffer size is 1 [cli_1]: aborting job: Fatal error in PMPI_Barrier: Message truncated, error stack: PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed MPIR_Barrier_impl(337)..............: Failure during collective MPIR_Barrier_impl(330)..............: MPIR_Barrier(294)...................: MPIR_Barrier_intra(151).............: barrier_smp_intra(111)..............: MPIR_Bcast_impl(1462)...............: MPIR_Bcast(1486)....................: MPIR_Bcast_intra(1295)..............: MPIR_Bcast_binomial(241)............: MPIC_Recv(352)......................: MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes received but buffer size is 1 benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: H5F_close: Assertion `f->file_id > 0' failed. Fatal error in PMPI_Allgather: Unknown error class, error stack: PMPI_Allgather(1002)......................: MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed MPIR_Allgather_impl(842)..................: MPIR_Allgather(801).......................: MPIR_Allgather_intra(216).................: MPIC_Sendrecv(475)........................: MPIC_Wait(243)............................: MPIDI_CH3i_Progress_wait(239).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(451): MPIDU_Socki_handle_read(649)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer) [cli_0]: aborting job: Fatal error in PMPI_Allgather: Unknown error class, error stack: PMPI_Allgather(1002)......................: MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed MPIR_Allgather_impl(842)..................: MPIR_Allgather(801).......................: MPIR_Allgather_intra(216).................: MPIC_Sendrecv(475)........................: MPIC_Wait(243)............................: MPIDI_CH3i_Progress_wait(239).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(451): MPIDU_Socki_handle_read(649)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer) benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: H5F_close: Assertion `f->file_id > 0’ failed. Please note that the above trace was the original stacktrace which appears to utilize sockets, though I’ve reproduced the same problem by running on an SMP with shared memory. While it’s not definitive that the issue has anything to do with the above stack trace, the very same benchmark runs perfectly well utilizing PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 available from your download site and with OpenMPI 2.0.1 (also their latest download). Both MPI releases were built from source on my Fedora 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). Given that the synchronous calls into MPI_File_set_view appear to be coded correctly AND that there isn’t much in the way of input parameters that would cause problems (other than incorrect coding), we tend to believe that the internal message queues between processes may somehow be corrupted. This impression is strengthened by the fact that our recent codebase changes (which are unrelated to the actual calls to MPI_File_set_view) may have introduced this issue. Note too, that the code paths to MPI_File_set_view have been taken many times previously and those function calls have all succeeded. Are there any suggestions out there as to how to further debug this potential corruption issue? Many thanks, Richard A. Warren _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Problem using MPICH 3.2 compiled with --with-device=ch3:sock
by Anatoly G 21 Dec '16

21 Dec '16

Dear MPICH, I downloaded a MPICH 3.2 package. When I configure it using: *./configure --prefix=/space/mpich3.2 --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-totalview --enable-static --disable-f77 --disable-fc --enable-shared=no --disable-fortran* My test program (simple send/receive) works fine on single & multiple computers. But each process activates "busy wait on each receive". I'd like to get program without busy wait. Than, I configured MPICH 3.2 with following: *./configure --prefix=/space/mpich3.2_no_busy_wait --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-totalview --enable-static --disable-f77 --disable-fc --enable-shared=no --disable-fortran --with-device=ch3:sock* After build my test program with this mpi, my test works fine on single computer two processes, but doesn't work on two computers (process per computer) with same configuration. Can you please advice? Thank you, Anatoly. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Abort not working with multinode jobs launched by hydra-3.2
by Doug Johnson 13 Dec '16

13 Dec '16

Hi Min, Thanks for the clarification about avoiding version mismatch. Looking more closely at the mpiexec hydra included with Intel MPI, they include a library named libjmi_pbs.so.1.0 that includes the necessary TM calls to fully integrate with Torque for job launch. So, no need for an out of tree mpiexec hydra. Doug Min Si <msi(a)anl.gov> writes: > Hi Doug, > > We have changed the internal implementation of MPI_Abort in both MPICH > code and hydra in 3.2. However, the Intel MPI 5.1.3.210 was based on an > old MPICH version v3.1.2. Thus if you compile the program with Intel MPI > which is based on MPICH v3.1.2 and execute the binary with hydra v3.2, > processes in remote nodes might not be able to exit. > > We do not support the usage with mismatched versions of MPICH and hydra. > You should try a new version of Intel MPI if you want to use hydra 3.2. > AFAIK, Intel MPI 2017.0.064 is based on MPICH v3.2. > > Min > > On 12/12/16 7:16 AM, Doug Johnson wrote: >> Hi, >> >> We've encountered a problem with hydra-3.2 and Intel MPI 5.1.3.210 with >> multi-node MPI programs. A call to MPI_Abort results in the all MPI >> ranks running on the same node as the rank that called MPI_Abort to >> exit, but leaves the other ranks running. The program hangs on the >> other nodes interminably (at least until the time limit of the batch job >> is reached.) The behavior is the same with hydra-3.3a2. The problem >> does not exist when using hydra-3.1.4, all processes exit on all nodes. >> >> Reverting commit 9882227414439a4a79edd49ec10261742bb60108 fixes this >> problem with 3.2. >> >> The hydra shipped with Intel MPI does not exhibit this problem, but we >> are using an out-of-tree hydra as want the pbs launcher enabled. Is >> there a better mechanism for process cleanup other than reverting the >> patch above? Let me know if there's other information needed. >> >> Thanks, >> Doug >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Abort not working with multinode jobs launched by hydra-3.2
by Min Si 12 Dec '16

12 Dec '16

Hi Doug, We have changed the internal implementation of MPI_Abort in both MPICH code and hydra in 3.2. However, the Intel MPI 5.1.3.210 was based on an old MPICH version v3.1.2. Thus if you compile the program with Intel MPI which is based on MPICH v3.1.2 and execute the binary with hydra v3.2, processes in remote nodes might not be able to exit. We do not support the usage with mismatched versions of MPICH and hydra. You should try a new version of Intel MPI if you want to use hydra 3.2. AFAIK, Intel MPI 2017.0.064 is based on MPICH v3.2. Min On 12/12/16 7:16 AM, Doug Johnson wrote: > Hi, > > We've encountered a problem with hydra-3.2 and Intel MPI 5.1.3.210 with > multi-node MPI programs. A call to MPI_Abort results in the all MPI > ranks running on the same node as the rank that called MPI_Abort to > exit, but leaves the other ranks running. The program hangs on the > other nodes interminably (at least until the time limit of the batch job > is reached.) The behavior is the same with hydra-3.3a2. The problem > does not exist when using hydra-3.1.4, all processes exit on all nodes. > > Reverting commit 9882227414439a4a79edd49ec10261742bb60108 fixes this > problem with 3.2. > > The hydra shipped with Intel MPI does not exhibit this problem, but we > are using an out-of-tree hydra as want the pbs launcher enabled. Is > there a better mechanism for process cleanup other than reverting the > patch above? Let me know if there's other information needed. > > Thanks, > Doug > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Abort not working with multinode jobs launched by hydra-3.2
by Doug Johnson 12 Dec '16

12 Dec '16

Hi, We've encountered a problem with hydra-3.2 and Intel MPI 5.1.3.210 with multi-node MPI programs. A call to MPI_Abort results in the all MPI ranks running on the same node as the rank that called MPI_Abort to exit, but leaves the other ranks running. The program hangs on the other nodes interminably (at least until the time limit of the batch job is reached.) The behavior is the same with hydra-3.3a2. The problem does not exist when using hydra-3.1.4, all processes exit on all nodes. Reverting commit 9882227414439a4a79edd49ec10261742bb60108 fixes this problem with 3.2. The hydra shipped with Intel MPI does not exhibit this problem, but we are using an out-of-tree hydra as want the pbs launcher enabled. Is there a better mechanism for process cleanup other than reverting the patch above? Let me know if there's other information needed. Thanks, Doug _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Unsubscribe
by Peter Ljungberg 10 Dec '16

10 Dec '16

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_gather slow on a single node
by Keith.Bannister＠csiro.au 08 Dec '16

08 Dec '16

On 9 Dec 2016, at 9:53 am, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Why are you building MPICH from source on a Cray machine? Cray MPI is based on MPICH and highly `` for that system. > Because I get the same results with cray_mpi :-( -- KEITH BANNISTER CSIRO Astronomy and Space Science T +61 2 9372 4295 E keith.bannister(a)csiro.au _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0