February 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI I/O spilt collective implementation: MPI_File_write_all_begin blocking or non-blocking?
by pramod kumbhar 11 Feb '16

11 Feb '16

thanks! I see expected behaviour by setting MPICH_ASYNC_PROGRESS (mpich compiled with --enable-threads=multiple) On Tue, Feb 9, 2016 at 10:22 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Rob Latham <robl(a)mcs.anl.gov> writes: > > MPICH takes a weak interpretation of the progress rule, but one can > > emulate strict interpretation by spawning a thread whose one job in life > > is to call MPI_Test repeatedly. > > ... or define the MPICH_ASYNC_PROGRESS environment variable. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] any clue on this mpiio error
by Jaln 10 Feb '16

10 Feb '16

Hi David, That sounds great, I appreciate for your input. I'm going to try it with two I/O tickets on hand, and hopefully to update you guys shortly. Best, Jialin On Wed, Feb 10, 2016 at 1:53 PM, David Knaak <knaak(a)cray.com> wrote: > > > On 02/10/2016 11:39 AM, Jaln wrote: > > > My jobs on Edison die on IO errors like this: > > > > > > ADIOI_CRAY_WRITECONTIG(284): filename='OUT/rei20_0.g029' > > > error='Input/output error' errno=5 PE=00044 W_rec=33518 > > > off=2232920756 len=0000524288 > > > See MPICH_MPIIO_ABORT_ON_RW_ERROR. > > > > > > Any Ideas about this error infor? I couldn't find anything on Google. > > > Thanks > > > On Wed, Feb 10, 2016 at 9:46 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > > Cray doesn't share their modifications to ROMIO with us. You'll have > > more luck with your cray support contact. > > > > errno 5 might be indicative of a general I/O error of some kind. Your > > offset is just large enough that maybe you are hitting some kind of cray > > 32 bit limitation? > > > > but I'm only guessing as cray doesn't share source with us. > > > > ==rob > > Hi Jaln, > > This message means that Cray's MPIIO had just made a system write() call > and write() returned a status of -1. The MPIIO routine that made the > call, "ADIOI_CRAY_WriteContig", is giving you as much information as it > can. "errno" has a value of 5, which translates to the general I/O > error message "Input/output error". That doesn't really tell you much. > The message recommends that you look at MPICH_MPIIO_ABORT_ON_RW_ERROR in > the Cray MPI "intro_mpi" man page: > > MPICH_MPIIO_ABORT_ON_RW_ERROR > If set to enable, causes MPI-IO to abort immediately after > issuing an error message if an I/O error occurs during a > system read() or write() call. This applies only to I/O > errors for system read() and write() calls made as a result > of MPI I/O calls. It does not apply to I/O errors for other > MPI I/O calls such as MPI_File_open(), nor does it apply to > read() and write() calls made by means other than MPI I/O > calls. > > Abort on error is not standard behavior. The MPI Standard > specifies that the default error handling for MPI I/O calls > is to return an error code to the application rather than > aborting the application, but since errors on write or read > are almost always unexpected and usually not recoverable, it > may be preferable to abort as soon as the error is detected. > Doing so does not allow any recovery, but does provide the > most information about the error and terminates the job > quickly. > > If the Cray Abnormal Termination Processing (ATP) feature is > enabled, the abort will result in a full stack backtrace > written to stderr and a graphical merged stack backtrace > tree (a "dot" file) that shows exactly where each process > was at the time of the abort. > > This environment variable is global for all files opened by > MPI_File_open(). To enable this behavior only for specific > files, use the MPICH_MPIIO_HINTS abort_on_rw_error option. > > Default: disable > > If this problem is repeatable and if you enable ATP (see man atp), > set the environment variable and see what the backtrace tells you. > > You can contact me directly (knaak(a)cray.com) > > David Knaak > > > > -- Genius only means hard-working all one's life _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] any clue on this mpiio error
by David Knaak 10 Feb '16

10 Feb '16

> > On 02/10/2016 11:39 AM, Jaln wrote: > > My jobs on Edison die on IO errors like this: > > > > ADIOI_CRAY_WRITECONTIG(284): filename='OUT/rei20_0.g029' > > error='Input/output error' errno=5 PE=00044 W_rec=33518 > > off=2232920756 len=0000524288 > > See MPICH_MPIIO_ABORT_ON_RW_ERROR. > > > > Any Ideas about this error infor? I couldn't find anything on Google. > > Thanks > On Wed, Feb 10, 2016 at 9:46 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > Cray doesn't share their modifications to ROMIO with us. You'll have > more luck with your cray support contact. > > errno 5 might be indicative of a general I/O error of some kind. Your > offset is just large enough that maybe you are hitting some kind of cray > 32 bit limitation? > > but I'm only guessing as cray doesn't share source with us. > > ==rob Hi Jaln, This message means that Cray's MPIIO had just made a system write() call and write() returned a status of -1. The MPIIO routine that made the call, "ADIOI_CRAY_WriteContig", is giving you as much information as it can. "errno" has a value of 5, which translates to the general I/O error message "Input/output error". That doesn't really tell you much. The message recommends that you look at MPICH_MPIIO_ABORT_ON_RW_ERROR in the Cray MPI "intro_mpi" man page: MPICH_MPIIO_ABORT_ON_RW_ERROR If set to enable, causes MPI-IO to abort immediately after issuing an error message if an I/O error occurs during a system read() or write() call. This applies only to I/O errors for system read() and write() calls made as a result of MPI I/O calls. It does not apply to I/O errors for other MPI I/O calls such as MPI_File_open(), nor does it apply to read() and write() calls made by means other than MPI I/O calls. Abort on error is not standard behavior. The MPI Standard specifies that the default error handling for MPI I/O calls is to return an error code to the application rather than aborting the application, but since errors on write or read are almost always unexpected and usually not recoverable, it may be preferable to abort as soon as the error is detected. Doing so does not allow any recovery, but does provide the most information about the error and terminates the job quickly. If the Cray Abnormal Termination Processing (ATP) feature is enabled, the abort will result in a full stack backtrace written to stderr and a graphical merged stack backtrace tree (a "dot" file) that shows exactly where each process was at the time of the abort. This environment variable is global for all files opened by MPI_File_open(). To enable this behavior only for specific files, use the MPICH_MPIIO_HINTS abort_on_rw_error option. Default: disable If this problem is repeatable and if you enable ATP (see man atp), set the environment variable and see what the backtrace tells you. You can contact me directly (knaak(a)cray.com) David Knaak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] any clue on this mpiio error
by Jaln 10 Feb '16

10 Feb '16

I see, Thanks Rob, Best, Jialin On Wed, Feb 10, 2016 at 9:46 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 02/10/2016 11:39 AM, Jaln wrote: > >> Hi, >> >> My jobs on Edison die on IO errors like this: >> >> ADIOI_CRAY_WRITECONTIG(284): filename='OUT/rei20_0.g029' >> error='Input/output er >> ror' errno=5 PE=00044 W_rec=33518 off=2232920756 len=0000524288 See MPICH_ >> MPIIO_ABORT_ON_RW_ERROR. >> >> Any Ideas about this error infor? I couldn't find anything on Google. >> Thanks >> > > Cray doesn't share their modifications to ROMIO with us. You'll have more > luck with your cray support contact. > > errno 5 might be indicative of a general I/O error of some kind. Your > offset is just large enough that maybe you are hitting some kind of cray 32 > bit limitation? > > but I'm only guessing as cray doesn't share source with us. > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Genius only means hard-working all one's life _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] any clue on this mpiio error
by Rob Latham 10 Feb '16

10 Feb '16

On 02/10/2016 11:39 AM, Jaln wrote: > Hi, > > My jobs on Edison die on IO errors like this: > > ADIOI_CRAY_WRITECONTIG(284): filename='OUT/rei20_0.g029' > error='Input/output er > ror' errno=5 PE=00044 W_rec=33518 off=2232920756 len=0000524288 See MPICH_ > MPIIO_ABORT_ON_RW_ERROR. > > Any Ideas about this error infor? I couldn't find anything on Google. > Thanks Cray doesn't share their modifications to ROMIO with us. You'll have more luck with your cray support contact. errno 5 might be indicative of a general I/O error of some kind. Your offset is just large enough that maybe you are hitting some kind of cray 32 bit limitation? but I'm only guessing as cray doesn't share source with us. ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

any clue on this mpiio error
by Jaln 10 Feb '16

10 Feb '16

Hi, My jobs on Edison die on IO errors like this: ADIOI_CRAY_WRITECONTIG(284): filename='OUT/rei20_0.g029' error='Input/output er ror' errno=5 PE=00044 W_rec=33518 off=2232920756 len=0000524288 See MPICH_ MPIIO_ABORT_ON_RW_ERROR. Any Ideas about this error infor? I couldn't find anything on Google. Thanks. Best, Jialin Liu LBNL _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O spilt collective implementation: MPI_File_write_all_begin blocking or non-blocking?
by Jeff Hammond 09 Feb '16

09 Feb '16

On Tuesday, February 9, 2016, Jed Brown <jed(a)jedbrown.org> wrote: > Rob Latham <robl(a)mcs.anl.gov <javascript:;>> writes: > > MPICH takes a weak interpretation of the progress rule, but one can > > emulate strict interpretation by spawning a thread whose one job in life > > is to call MPI_Test repeatedly. > > ... or define the MPICH_ASYNC_PROGRESS environment variable. > Or extend Casper to support MPI IO... :-) Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O spilt collective implementation: MPI_File_write_all_begin blocking or non-blocking?
by Jed Brown 09 Feb '16

09 Feb '16

Rob Latham <robl(a)mcs.anl.gov> writes: > MPICH takes a weak interpretation of the progress rule, but one can > emulate strict interpretation by spawning a thread whose one job in life > is to call MPI_Test repeatedly. ... or define the MPICH_ASYNC_PROGRESS environment variable. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O spilt collective implementation: MPI_File_write_all_begin blocking or non-blocking?
by Rob Latham 09 Feb '16

09 Feb '16

On 02/09/2016 01:42 PM, pramod kumbhar wrote: > > In this case I still see the similar time spent in MPI_Wait. I added > MPI_Test for the non-blocking request as: > > some_compute_fun() { > MPI_Test(&req, &flag, &status); > .....some_flops_here...... > } > > err = MPI_File_iwrite_all( fh, buf, bufsize, MPI_INT, &req); > some_compute_fun(); > MPI_Wait(&req, &status); > > Now I see that i/o operation progresses in background and don't see any > time spent in MPI_Wait. It's a subtle but important point: the standard suggests might progress happen in the background. It does not mandate it. We have the notions of "strict interpretation of the progress rule" where background progress will happen and "weak interpretation", where progress only happens when one is in an MPI call. When you call MPI_Test, the mpi library can, as we say "kick the progress engine" and allow outstanding requests to make some function calls. MPICH takes a weak interpretation of the progress rule, but one can emulate strict interpretation by spawning a thread whose one job in life is to call MPI_Test repeatedly. ==rob > > Few months ago I was testing non-blocking collective operations in > mvapich and saw similar behaviour where I have to use few MPI_Test calls > to make progress with non-blocking collectives (this will be addressed > in upcoming releases). > > I am testing above i/o example on my laptop and haven't tested other > non-blocking collectives. Before analysing/profiling this further on > production cluster / bg-q, I would like to know: > - what is the expected behaviour? > - are there any specific options that I have to use to make sure > internal progress? > > -Pramod > > p.s. I just built mpich-3.2 with configure,make,make install on OS X, > without looking into all configure options... > > On Tue, Feb 9, 2016 at 3:51 AM, Rob Latham <robl(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > > > On 02/08/2016 06:04 PM, pramod kumbhar wrote: > > thanks! MPI_File_iwrite_all working as expected. > > > I'm always glad to see folks looking at the MPI-IO routines. Let us > know how your research progresses. > > ==rob > > > -Pramod > > On Tue, Feb 9, 2016 at 12:52 AM, Balaji, Pavan <balaji(a)anl.gov > <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > I believe it is valid to do that in split collectives. If > you want > truly nonblocking nature, the standard way to do it would > be to use > nonblocking collective I/O, i.e., MPI_File_iwrite_all. This is > guaranteed to not block during the MPI_File_iwrite_all call. > > -- Pavan > > > On Feb 8, 2016, at 5:48 PM, pramod kumbhar > <pramod.s.kumbhar(a)gmail.com > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > Hello All, > > > > I am testing MPI_FIle_write_all_begin/_end and it seems like > > MPI_File_write_all_begin waits for i/o completion. > > > > Could someone confirm the implementation of these routines? > > > > Thanks, > > Pramod > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI I/O spilt collective implementation: MPI_File_write_all_begin blocking or non-blocking?
by pramod kumbhar 09 Feb '16

09 Feb '16

I have additional question about the internal progress of these non-blocking routines. I have simple test case: err = MPI_File_iwrite_all( fh, buf, bufsize, MPI_INT, &req); MPI_Wait(&req, &status); Here MPI_File_iwrite_all returns immediately and profiling shows time spent in MPI_Wait. Next, I modified example to overlap i/o with some computations: some_compute_fun() { .....some_flops_here...... } err = MPI_File_iwrite_all( fh, buf, bufsize, MPI_INT, &req); some_compute_fun(); MPI_Wait(&req, &status); In this case I still see the similar time spent in MPI_Wait. I added MPI_Test for the non-blocking request as: some_compute_fun() { MPI_Test(&req, &flag, &status); .....some_flops_here...... } err = MPI_File_iwrite_all( fh, buf, bufsize, MPI_INT, &req); some_compute_fun(); MPI_Wait(&req, &status); Now I see that i/o operation progresses in background and don't see any time spent in MPI_Wait. Few months ago I was testing non-blocking collective operations in mvapich and saw similar behaviour where I have to use few MPI_Test calls to make progress with non-blocking collectives (this will be addressed in upcoming releases). I am testing above i/o example on my laptop and haven't tested other non-blocking collectives. Before analysing/profiling this further on production cluster / bg-q, I would like to know: - what is the expected behaviour? - are there any specific options that I have to use to make sure internal progress? -Pramod p.s. I just built mpich-3.2 with configure,make,make install on OS X, without looking into all configure options... On Tue, Feb 9, 2016 at 3:51 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 02/08/2016 06:04 PM, pramod kumbhar wrote: > >> thanks! MPI_File_iwrite_all working as expected. >> > > I'm always glad to see folks looking at the MPI-IO routines. Let us know > how your research progresses. > > ==rob > > >> -Pramod >> >> On Tue, Feb 9, 2016 at 12:52 AM, Balaji, Pavan <balaji(a)anl.gov >> <mailto:[email protected]>> wrote: >> >> >> I believe it is valid to do that in split collectives. If you want >> truly nonblocking nature, the standard way to do it would be to use >> nonblocking collective I/O, i.e., MPI_File_iwrite_all. This is >> guaranteed to not block during the MPI_File_iwrite_all call. >> >> -- Pavan >> >> > On Feb 8, 2016, at 5:48 PM, pramod kumbhar >> <pramod.s.kumbhar(a)gmail.com <mailto:[email protected]>> >> wrote: >> > >> > Hello All, >> > >> > I am testing MPI_FIle_write_all_begin/_end and it seems like >> > MPI_File_write_all_begin waits for i/o completion. >> > >> > Could someone confirm the implementation of these routines? >> > >> > Thanks, >> > Pramod >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0