June 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Controlling MPICH busy-wait behavior
by Jeff Hammond 08 Jun '20

08 Jun '20

I wrote an example of an interposition library that inserts usleep or nanosleep calls at microsecond granularity to blocking MPI operations. I have done minimal testing and not in the relevant circumstances (load-imbalance) so take it with a huge grain of salt. The intent is to be educational. https://github.com/jeffhammond/NiceWait Jeff On Thu, Jun 4, 2020 at 8:38 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > You might be able to implement this across the board using > https://github.com/LLNL/wrap and interposing all calls to MPI_Wait > transparently rather than changing the application code. > > Jeff > > On Thu, Jun 4, 2020 at 8:30 AM Mccall, Kurt E. (MSFC-EV41) via discuss < > discuss(a)mpich.org> wrote: > >> Thanks Hui. I will use MPI_Iprobe + MPI_Test. >> >> >> >> Kurt >> >> >> >> *From:* Zhou, Hui <zhouh(a)anl.gov> >> *Sent:* Thursday, June 4, 2020 10:27 AM >> *To:* discuss(a)mpich.org >> *Cc:* Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> >> *Subject:* [EXTERNAL] Re: [mpich-discuss] Controlling MPICH busy-wait >> behavior >> >> >> >> There is no such option for MPICH. If you configure with ch3:sock, which >> waits for the progress with blocking `poll`, I think it will not hog the >> CPU by default, for other configurations, it always busy waits. >> >> Is using MPI_Iprobe + MPI_Test an option for you? If your process can >> afford latency, then you can control your own sleep cycles between >> MPI_Test. If MPICH will implement such non-busy-wait option, internally it >> will just do the same test-sleep cycles. >> >> >> >> -- >> Hui Zhou >> >> >> >> >> >> *From: *"Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> >> *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Date: *Wednesday, June 3, 2020 at 3:57 PM >> *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Cc: *"Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> >> *Subject: *[mpich-discuss] Controlling MPICH busy-wait behavior >> >> >> >> Is there a means to tell MPICH to yield to other processes when >> performing a busy-wait for incoming messages? The page below >> distinguishes between “aggressive busy-wait” and “degraded busy-wait”, >> where the latter is what I’d like to try. >> >> >> >> >> https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu… >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_ques…> >> >> >> >> >> >> Thanks, >> >> Kurt >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Controlling MPICH busy-wait behavior
by Jeff Hammond 04 Jun '20

04 Jun '20

You might be able to implement this across the board using https://github.com/LLNL/wrap and interposing all calls to MPI_Wait transparently rather than changing the application code. Jeff On Thu, Jun 4, 2020 at 8:30 AM Mccall, Kurt E. (MSFC-EV41) via discuss < discuss(a)mpich.org> wrote: > Thanks Hui. I will use MPI_Iprobe + MPI_Test. > > > > Kurt > > > > *From:* Zhou, Hui <zhouh(a)anl.gov> > *Sent:* Thursday, June 4, 2020 10:27 AM > *To:* discuss(a)mpich.org > *Cc:* Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > *Subject:* [EXTERNAL] Re: [mpich-discuss] Controlling MPICH busy-wait > behavior > > > > There is no such option for MPICH. If you configure with ch3:sock, which > waits for the progress with blocking `poll`, I think it will not hog the > CPU by default, for other configurations, it always busy waits. > > Is using MPI_Iprobe + MPI_Test an option for you? If your process can > afford latency, then you can control your own sleep cycles between > MPI_Test. If MPICH will implement such non-busy-wait option, internally it > will just do the same test-sleep cycles. > > > > -- > Hui Zhou > > > > > > *From: *"Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Wednesday, June 3, 2020 at 3:57 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *"Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> > *Subject: *[mpich-discuss] Controlling MPICH busy-wait behavior > > > > Is there a means to tell MPICH to yield to other processes when performing > a busy-wait for incoming messages? The page below distinguishes between > “aggressive busy-wait” and “degraded busy-wait”, where the latter is what > I’d like to try. > > > > > https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_ques…> > > > > > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Controlling MPICH busy-wait behavior
by Mccall, Kurt E. (MSFC-EV41) 04 Jun '20

04 Jun '20

Thanks Hui. I will use MPI_Iprobe + MPI_Test. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Thursday, June 4, 2020 10:27 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Controlling MPICH busy-wait behavior There is no such option for MPICH. If you configure with ch3:sock, which waits for the progress with blocking `poll`, I think it will not hog the CPU by default, for other configurations, it always busy waits. Is using MPI_Iprobe + MPI_Test an option for you? If your process can afford latency, then you can control your own sleep cycles between MPI_Test. If MPICH will implement such non-busy-wait option, internally it will just do the same test-sleep cycles. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, June 3, 2020 at 3:57 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Controlling MPICH busy-wait behavior Is there a means to tell MPICH to yield to other processes when performing a busy-wait for incoming messages? The page below distinguishes between “aggressive busy-wait” and “degraded busy-wait”, where the latter is what I’d like to try. https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu…<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_ques…> Thanks, Kurt

1 0

Re: [mpich-discuss] Controlling MPICH busy-wait behavior
by Zhou, Hui 04 Jun '20

04 Jun '20

There is no such option for MPICH. If you configure with ch3:sock, which waits for the progress with blocking `poll`, I think it will not hog the CPU by default, for other configurations, it always busy waits. Is using MPI_Iprobe + MPI_Test an option for you? If your process can afford latency, then you can control your own sleep cycles between MPI_Test. If MPICH will implement such non-busy-wait option, internally it will just do the same test-sleep cycles. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, June 3, 2020 at 3:57 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Controlling MPICH busy-wait behavior Is there a means to tell MPICH to yield to other processes when performing a busy-wait for incoming messages? The page below distinguishes between “aggressive busy-wait” and “degraded busy-wait”, where the latter is what I’d like to try. https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu… Thanks, Kurt

1 0

Re: [mpich-discuss] Controlling MPICH busy-wait behavior
by Jeff Hammond 04 Jun '20

04 Jun '20

There is an FAQ entry on this topic, but it isn't named in a way that would have made it easy for you to find. https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_do… While ch3:sock is deprecated, I believe the PETSc team uses it on a regular basis, so it should work for common MPI operations. It seems that libfabric supports efficient waiting ( https://ofiwg.github.io/libfabric/v1.10.0/man/fabtests.7.html) but I have no idea if or how this would work inside of MPICH. MPICH-derived and ABI-compatible Intel MPI supports wait backoff options, although I have never used them. https://software.intel.com/en-us/forums/intel-clusters-and-hpc-technology/t… You can install Intel MPI with Apt for zero cost. https://software.intel.com/content/www/us/en/develop/articles/installing-in… Best, Jeff (who works for Intel) On Wed, Jun 3, 2020 at 1:57 PM Mccall, Kurt E. (MSFC-EV41) via discuss < discuss(a)mpich.org> wrote: > Is there a means to tell MPICH to yield to other processes when performing > a busy-wait for incoming messages? The page below distinguishes between > “aggressive busy-wait” and “degraded busy-wait”, where the latter is what > I’d like to try. > > > > > https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu… > > > > > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Controlling MPICH busy-wait behavior
by Mccall, Kurt E. (MSFC-EV41) 03 Jun '20

03 Jun '20

Is there a means to tell MPICH to yield to other processes when performing a busy-wait for incoming messages? The page below distinguishes between "aggressive busy-wait" and "degraded busy-wait", where the latter is what I'd like to try. https://stackoverflow.com/questions/14560714/probe-seems-to-consume-the-cpu… Thanks, Kurt

1 0

Re: [mpich-discuss] Excessive disk activity in MPI processes
by Mccall, Kurt E. (MSFC-EV41) 02 Jun '20

02 Jun '20

Sorry, I should have mentioned that our file system is NFS hosted on the cluster head node, with infiniband from the compute nodes to the head. ****** I am attempting to run an MPI Monte Carlo: multiple instances of a simulation "Mav" under the control of MPI. Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds. The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to "iotop", the MPI-supervised Mav instances regularly become blocked waiting for I/O. iotop often shows the Mavs's spending 99% of their time waiting for I/O. The independent Torque jobs don't do that. I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb -k n) to a file. My question is: are there any other optional types of disk activity performed by MPI that I can disable? What else could be happening here? I'm using MPICH 3.3.2 compiled with pgc++, PBS Torque 5.1.1, CentOS 3.10.0. Here are my MPICH configure options: '--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl' Thanks, Kurt

1 0

Excessive disk activity in MPI processes
by Mccall, Kurt E. (MSFC-EV41) 02 Jun '20

02 Jun '20

I am attempting to run an MPI Monte Carlo: multiple instances of a simulation "Mav" under the control of MPI. Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds. The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to "iotop", the MPI-supervised Mav instances regularly become blocked waiting for I/O. iotop often shows the Mavs's spending 99% of their time waiting for I/O. The independent Torque jobs don't do that. I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb -k n) to a file. My question is: are there any other optional types of disk activity performed by MPI that I can disable? What else could be happening here? I'm using MPICH 3.3.2 compiled with pgc++, PBS Torque 5.1.1, CentOS 3.10.0. Here are my MPICH configure options: '--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl' Thanks, Kurt

1 0