September 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Hang in MPI_Win_Flush during scenario in one-sided communication
by Rajeev Thakur 23 Sep '14

23 Sep '14

It results in MPICH creating a thread that checks for pending operations. Having such a thread always present costs some in terms of performance so it is not set up by default as most applications don't need it. In its absence, progress may not happen until an MPI function is called. Rajeev On Sep 23, 2014, at 1:57 PM, Neil Spruit <nrspruit(a)gmail.com> wrote: > Yep, that did it, setting the MPICH_ASYNC_PROGRESS env var to 1 fixed the issue, thanks! > > So, why does that fix the issue? What is that env var causing internally in mpich to fix this scenario if you don't mind me asking. > > Thanks, > Neil > > On Mon, Sep 22, 2014 at 5:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > For this to work you need to enable asynchronous progress in MPICH at least on the target side. You can do that by setting the environment variable MPICH_ASYNC_PROGRESS to 1 before running mpiexec. > > Rajeev > > On Sep 22, 2014, at 4:53 PM, Neil Spruit <nrspruit(a)gmail.com> > wrote: > > > Hello, > > > > > > I have been experimenting with mpich’s implementation of one-sided communication with no remote synchronization that was introduced in the mpi3.0 spec and I have run into a hang issue during connect/accept MPI scenarios. > > > > > > > > Here is the situation, I have two applications that connect to each other after they have been launched separately with mpiexec on two different hosts. > > > > > > After connecting, both hosts allocate memory and run MPI_Win_Create as the following: > > > > Host1_program: > > > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > > > MPI_Comm_rank(comm, &myrank); > > > > MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, comm, &win); > > > > Host2_program: > > > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > > > MPI_Comm_rank(comm, &myrank); > > > > MPI_Win_create(buffer, buf_length, 1, MPI_INFO_NULL, comm, &win); > > > > > > > > Once the MPI_Window has been created I execute the following on host1: > > > > Host1_program(continued): > > > > err = MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, buf_length, MPI_BYTE, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Win_flush(target_rank, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Win_unlock(target_rank, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > While this is running, host2’s program is simply sleeping in a getchar(). > > > > > > This should run and complete, however, if the other host has not called another mpi function such as MPI_Recv and instead is in a sleep, getchar(), etc. the MPI_Win_flush above will simply hang. > > > > > > Why is the one-sided operation hanging until another MPI operation is called on the target host? My understanding is that utilizing MPI_Win_Flush does not require any action by the target host to complete the operation. > > > > > > This is a stack trace of where the flush gets stuck: > > > > #0 0x0000003b456dea28 in poll () from /lib64/libc.so.6 > > > > #1 0x00007fe1c08004ee in MPID_nem_tcp_connpoll () > > > > from /usr/local/lib/libmpich.so.12 > > > > #2 0x00007fe1c07efab7 in MPIDI_CH3I_Progress () > > > > from /usr/local/lib/libmpich.so.12 > > > > #3 0x00007fe1c07d35ff in MPIDI_CH3I_Wait_for_lock_granted () > > > > from /usr/local/lib/libmpich.so.12 > > > > #4 0x00007fe1c07d88bf in MPIDI_Win_flush () from /usr/local/lib/libmpich.so.12 > > > > #5 0x00007fe1c08c1b46 in PMPI_Win_flush () from /usr/local/lib/libmpich.so.12 > > > > #6 0x000000000040115c in main () > > > > > > According to this it seems that somehow the remote host is holding something after MPI_Win_Create that in my scenario gets unlocked if I call MPI_Recv on the connected_comm from above after the MPI_Win_Create. > > > > > > Thank you for your time and I look forward to your reply. > > > > > > Respectfully, > > > > Neil Spruit > > > > > > > > P.S. MPICH version info > > > > Version: 3.1 > > > > Release Date: Thu Feb 20 11:41:13 CST 2014 > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hang in MPI_Win_Flush during scenario in one-sided communication
by Neil Spruit 23 Sep '14

23 Sep '14

Yep, that did it, setting the MPICH_ASYNC_PROGRESS env var to 1 fixed the issue, thanks! So, why does that fix the issue? What is that env var causing internally in mpich to fix this scenario if you don't mind me asking. Thanks, Neil On Mon, Sep 22, 2014 at 5:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > For this to work you need to enable asynchronous progress in MPICH at > least on the target side. You can do that by setting the environment > variable MPICH_ASYNC_PROGRESS to 1 before running mpiexec. > > Rajeev > > On Sep 22, 2014, at 4:53 PM, Neil Spruit <nrspruit(a)gmail.com> > wrote: > > > Hello, > > > > > > I have been experimenting with mpich’s implementation of one-sided > communication with no remote synchronization that was introduced in the > mpi3.0 spec and I have run into a hang issue during connect/accept MPI > scenarios. > > > > > > > > Here is the situation, I have two applications that connect to each > other after they have been launched separately with mpiexec on two > different hosts. > > > > > > After connecting, both hosts allocate memory and run MPI_Win_Create as > the following: > > > > Host1_program: > > > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, > MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > > > MPI_Comm_rank(comm, &myrank); > > > > MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, comm, &win); > > > > Host2_program: > > > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, > MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > > > MPI_Comm_rank(comm, &myrank); > > > > MPI_Win_create(buffer, buf_length, 1, MPI_INFO_NULL, comm, &win); > > > > > > > > Once the MPI_Window has been created I execute the following on host1: > > > > Host1_program(continued): > > > > err = MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, > buf_length, MPI_BYTE, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Win_flush(target_rank, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > err = MPI_Win_unlock(target_rank, win); > > > > if(err != MPI_SUCCESS) > > > > { > > > > return 1; > > > > } > > > > While this is running, host2’s program is simply sleeping in a getchar(). > > > > > > This should run and complete, however, if the other host has not called > another mpi function such as MPI_Recv and instead is in a sleep, getchar(), > etc. the MPI_Win_flush above will simply hang. > > > > > > Why is the one-sided operation hanging until another MPI operation is > called on the target host? My understanding is that utilizing MPI_Win_Flush > does not require any action by the target host to complete the operation. > > > > > > This is a stack trace of where the flush gets stuck: > > > > #0 0x0000003b456dea28 in poll () from /lib64/libc.so.6 > > > > #1 0x00007fe1c08004ee in MPID_nem_tcp_connpoll () > > > > from /usr/local/lib/libmpich.so.12 > > > > #2 0x00007fe1c07efab7 in MPIDI_CH3I_Progress () > > > > from /usr/local/lib/libmpich.so.12 > > > > #3 0x00007fe1c07d35ff in MPIDI_CH3I_Wait_for_lock_granted () > > > > from /usr/local/lib/libmpich.so.12 > > > > #4 0x00007fe1c07d88bf in MPIDI_Win_flush () from > /usr/local/lib/libmpich.so.12 > > > > #5 0x00007fe1c08c1b46 in PMPI_Win_flush () from > /usr/local/lib/libmpich.so.12 > > > > #6 0x000000000040115c in main () > > > > > > According to this it seems that somehow the remote host is holding > something after MPI_Win_Create that in my scenario gets unlocked if I call > MPI_Recv on the connected_comm from above after the MPI_Win_Create. > > > > > > Thank you for your time and I look forward to your reply. > > > > > > Respectfully, > > > > Neil Spruit > > > > > > > > P.S. MPICH version info > > > > Version: 3.1 > > > > Release Date: Thu Feb 20 11:41:13 CST 2014 > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich-3.2a for Blue Gene /Q omits spi libraries from the wrappers
by Rob Latham 23 Sep '14

23 Sep '14

On 09/16/2014 04:06 PM, William Scullin wrote: > When building mpich-3.2a for Blue Gene /Q and following the same > procedure used for building mpich-3.1.2 for BG/Q, the required linking > information for spi and pami is omitted from the wrappers. I'm not > sure if this is intentional or not. Looks like we finally fixed that one: https://trac.mpich.org/projects/mpich/changeset/e8b11d0a ==rob > > Thanks, > William > > > Patching the wrappers to reflect the required linking gives the following diff : > > [wscullin@vestasn1 ~]$ diff > /projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/bin/mpicc > /projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/bin/mpicc-postpatch > 265c265 > < $Show $CC $PROFILE_INCPATHS > -L/bgsys/drivers/V1R2M2/ppc64/spi/lib > -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir > -L$libdir $PROFILE_PRELIB $PROFILE_FOO $rpath_flags -lmpi > $PROFILE_POSTLIB > --- >> $Show $CC $PROFILE_INCPATHS -L/bgsys/drivers/V1R2M2/ppc64/spi/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir -L$libdir $PROFILE_PRELIB $PROFILE_FOO $rpath_flags -lmpi $PROFILE_POSTLIB -lpami-gcc -lpthread -lSPI -lSPI_cnk -lrt -lstdc++ > > Full diff of mpich-3.1.2 versus 3.2a wrappers: > > [wscullin@vestasn1 ~]$ diff > /projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7/bin/mpicc > /projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/bin/mpicc > 30,34c30,34 > < prefix=/projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7 > < exec_prefix=/projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7 > < sysconfdir=/projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7/etc > < includedir=/projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7/include > < libdir=/projects/Operations/wscullin/mpich-3.1.2/gcc-4.4.7/lib > --- >> prefix=/projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7 >> exec_prefix=/projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7 >> sysconfdir=/projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/etc >> includedir=/projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/include >> libdir=/projects/Operations/wscullin/mpich-3.2a1/gcc-4.4.7/lib > 40c40 > < MPICH_VERSION="3.1.2" > --- >> MPICH_VERSION="3.2a1" > 262c262 > < $Show $CC $PROFILE_INCPATHS > -L/bgsys/drivers/V1R2M2/ppc64/spi/lib > -L/bgsys/drivers/V1R2M2/ppc64/comm/lib > -L/bgsys/drivers/V1R2M2/ppc64/spi/lib > -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir > --- >> $Show $CC $PROFILE_INCPATHS -L/bgsys/drivers/V1R2M2/ppc64/spi/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir > 265c265 > < $Show $CC $PROFILE_INCPATHS > -L/bgsys/drivers/V1R2M2/ppc64/spi/lib > -L/bgsys/drivers/V1R2M2/ppc64/comm/lib > -L/bgsys/drivers/V1R2M2/ppc64/spi/lib > -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir > -L$libdir $PROFILE_PRELIB $PROFILE_FOO $rpath_flags -lmpi > $PROFILE_POSTLIB -lpami-gcc -lpthread -lSPI -lSPI_cnk -lrt -lstdc++ > --- >> $Show $CC $PROFILE_INCPATHS -L/bgsys/drivers/V1R2M2/ppc64/spi/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/lib "${allargs[@]}" -I$includedir -L$libdir $PROFILE_PRELIB $PROFILE_FOO $rpath_flags -lmpi $PROFILE_POSTLIB > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Does MPICH implements polling mechanism on MPI_Recv?
by Rob Latham 23 Sep '14

23 Sep '14

On 09/23/2014 09:38 AM, Wesley Bland wrote: > Out of curiosity, what is the issue with using 100% of the CPU? If > you’re not using it for your application (which it appears that you > aren’t since you’re calling MPI_Wait), what difference does it make if > MPI uses all of it? power consumption, I'd imagine. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Does MPICH implements polling mechanism on MPI_Recv?
by Wesley Bland 23 Sep '14

23 Sep '14

> On Sep 22, 2014, at 11:42 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Thank you very much. > > I'm using MPICH 3.1. > > The application purpose is continue execution until at least Master process is live. The assumption is that processes can fail, but no more than single failure at a time. In this case survived processes must continue "like a Terminator". (-: > In run-time I can't execute "manually" MPI_Test on all requests, it's too heavy. I have a network traffic ~ 400-500 Mb/s. > > Implementation: > I set -disable-auto-cleanup flag to activate Fault tolerance. > All my processes activate MPI_Irecv on other processes and then execute MPI_Waitany on all active requests. When data arrives correctly, I process it and execute new MPI_Irecv on source process. If MPI_Waitany returns error (some process failed) I recognize failed rank and stop communication with it on application level (no more Sends & Recvs to it). In this mode system continues execution with survived processes. I don't use any collective operations, I simulate them using MPI_Irecv & MPI_Isend + MPI_Waitany or MPI_Waitall (returns error if some process failed). > > I think it's ugly solution, but I can't think on any more elegant solution. > Any other solution at all would be welcome. You could use MPI_Testall/any if you’d like, but of course that may not fit your application usage. Overall, I don’t see an issue with your model. > > The problem in "ready" state, when there is almost only waiting 3-4 messages may be received by process. In this state all CPUs are busy by executing polling mechanism. That’s to be expected. While waiting for messages using MPI_Wait, it’s expected that nothing else is happening (unless you’re using threads as well, which it appears that you aren’t). That puts MPI into a busy wait to receive the message. As I mentioned previously, if you want to poll less frequently, you’ll have to use some flavor of MPI_Test. Out of curiosity, what is the issue with using 100% of the CPU? If you’re not using it for your application (which it appears that you aren’t since you’re calling MPI_Wait), what difference does it make if MPI uses all of it? Thanks, Wesley > > > > Regards, > Anatoly. > > > On Mon, Sep 22, 2014 at 4:46 PM, Wesley Bland <wbland(a)anl.gov <mailto:[email protected]>> wrote: > Which version of MPICH are you using? > > Which fault tolerance features are you using? Fault tolerance is currently undergoing some changes and has different features than it used to have. > > AFAIK, neither version of FT has been tested with ch3:sock. It’s possible that it will work, but FT is still a very experimental feature and hasn’t been widely tested. > > If you want to avoid polling more, you can use non-blocking receive calls to post a receive and poll the system yourself periodically (using MPI_TEST). This will give your application an opportunity to do something else while waiting for the receives to complete. > > Thanks, > Wesley > >> On Sep 22, 2014, at 8:30 AM, Anatoly G <anatolyrishon(a)gmail.com <mailto:[email protected]>> wrote: >> >> Dear MPICH. >> I have a problem with poling MPICH mechanism. >> I'm working on cluster. There are 2-4 processes on each computer (I can't execute single process per computer because of application requirements). >> My system has 2 states: >> Ready - slaves listen to master (but no data flow) >> Run - masters start communication, then there is data flow. >> When system in ready state (all processes except master executed MPI_Recv requests on master) but Master process still net sending data I see CPU usage > 100% (more than 1 core used) per process. When 4 processes are in ready state (waiting for data) computer begins to slow down other processes, I think because of polling. >> I tried to build MPICH with --with-device=ch3:sock, then I get 0% CPU usage in ready state, but I have a problem with Fault tolerance feature. >> My questions are: >> 1) Is such behavior expected that build with --with-device=ch3:sock causes Fault tolerance not work? Does Fault tolerance based on polling mechanism? >> 2) Can I change polling rate to reduce CPU payload? I understand that penalty is transfer rate slow down. >> 3) Can I use any other MPI APIs to check if message from master is arrived w/o activating polling mechanism? >> >> Regards, >> Anatoly. >> >> >> On Thu, May 8, 2014 at 3:57 PM, Balaji, Pavan <balaji(a)anl.gov <mailto:[email protected]>> wrote: >> >> This is expected. Currently, the only way to not have MPICH poll is to configure with --with-device=ch3:sock. Please note that this can cause performance loss (the polling is helpful for performance in the common case). >> >> We are planning to allow this in the default build as well in the future. >> >> — Pavan >> >> On May 8, 2014, at 7:54 AM, Anatoly G <anatolyrishon(a)gmail.com <mailto:[email protected]>> wrote: >> >> > Dear MPICH forum. >> > I created an endless MPI program. >> > In this program each process calls MPI_Recv from other process, w/o any MPI_Send. >> > When I execute this program I see each process takes ~ 100% CPU core. >> > Is this behavior (I suppose polling) is normal? >> > May I reduce MPI_Recv CPU penalty? >> > >> > Regards, >> > Anatoly. >> > <mpi_polling.cpp>_______________________________________________ >> > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Does MPICH implements polling mechanism on MPI_Recv?
by Anatoly G 22 Sep '14

22 Sep '14

Thank you very much. I'm using MPICH 3.1. The application purpose is continue execution until at least Master process is live. The assumption is that processes can fail, but no more than single failure at a time. In this case survived processes must continue "like a Terminator". (-: In run-time I can't execute "manually" MPI_Test on all requests, it's too heavy. I have a network traffic ~ 400-500 Mb/s. Implementation: I set -disable-auto-cleanup flag to activate Fault tolerance. All my processes activate MPI_Irecv on other processes and then execute MPI_Waitany on all active requests. When data arrives correctly, I process it and execute new MPI_Irecv on source process. If MPI_Waitany returns error (some process failed) I recognize failed rank and stop communication with it on application level (no more Sends & Recvs to it). In this mode system continues execution with survived processes. I don't use any collective operations, I simulate them using MPI_Irecv & MPI_Isend + MPI_Waitany or MPI_Waitall (returns error if some process failed). I think it's ugly solution, but I can't think on any more elegant solution. Any other solution at all would be welcome. The problem in "ready" state, when there is almost only waiting 3-4 messages may be received by process. In this state all CPUs are busy by executing polling mechanism. Regards, Anatoly. On Mon, Sep 22, 2014 at 4:46 PM, Wesley Bland <wbland(a)anl.gov> wrote: > Which version of MPICH are you using? > > Which fault tolerance features are you using? Fault tolerance is currently > undergoing some changes and has different features than it used to have. > > AFAIK, neither version of FT has been tested with ch3:sock. It’s possible > that it will work, but FT is still a very experimental feature and hasn’t > been widely tested. > > If you want to avoid polling more, you can use non-blocking receive calls > to post a receive and poll the system yourself periodically (using > MPI_TEST). This will give your application an opportunity to do something > else while waiting for the receives to complete. > > Thanks, > Wesley > > On Sep 22, 2014, at 8:30 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Dear MPICH. > I have a problem with poling MPICH mechanism. > I'm working on cluster. There are 2-4 processes on each computer (I can't > execute single process per computer because of application requirements). > My system has 2 states: > Ready - slaves listen to master (but no data flow) > Run - masters start communication, then there is data flow. > When system in ready state (all processes except master executed MPI_Recv > requests on master) but Master process still net sending data I see CPU > usage > 100% (more than 1 core used) per process. When 4 processes are in > ready state (waiting for data) computer begins to slow down other > processes, I think because of polling. > I tried to build MPICH with --with-device=ch3:sock, then I get 0% CPU > usage in ready state, but I have a problem with Fault tolerance feature. > My questions are: > 1) Is such behavior expected that build with --with-device=ch3:sock > causes Fault tolerance not work? Does Fault tolerance based on polling > mechanism? > 2) Can I change polling rate to reduce CPU payload? I understand that > penalty is transfer rate slow down. > 3) Can I use any other MPI APIs to check if message from master is arrived > w/o activating polling mechanism? > > Regards, > Anatoly. > > > On Thu, May 8, 2014 at 3:57 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> >> This is expected. Currently, the only way to not have MPICH poll is to >> configure with --with-device=ch3:sock. Please note that this can cause >> performance loss (the polling is helpful for performance in the common >> case). >> >> We are planning to allow this in the default build as well in the future. >> >> — Pavan >> >> On May 8, 2014, at 7:54 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Dear MPICH forum. >> > I created an endless MPI program. >> > In this program each process calls MPI_Recv from other process, w/o any >> MPI_Send. >> > When I execute this program I see each process takes ~ 100% CPU core. >> > Is this behavior (I suppose polling) is normal? >> > May I reduce MPI_Recv CPU penalty? >> > >> > Regards, >> > Anatoly. >> > <mpi_polling.cpp>_______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hang in MPI_Win_Flush during scenario in one-sided communication
by Rajeev Thakur 22 Sep '14

22 Sep '14

For this to work you need to enable asynchronous progress in MPICH at least on the target side. You can do that by setting the environment variable MPICH_ASYNC_PROGRESS to 1 before running mpiexec. Rajeev On Sep 22, 2014, at 4:53 PM, Neil Spruit <nrspruit(a)gmail.com> wrote: > Hello, > > > I have been experimenting with mpich’s implementation of one-sided communication with no remote synchronization that was introduced in the mpi3.0 spec and I have run into a hang issue during connect/accept MPI scenarios. > > > > Here is the situation, I have two applications that connect to each other after they have been launched separately with mpiexec on two different hosts. > > > After connecting, both hosts allocate memory and run MPI_Win_Create as the following: > > Host1_program: > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > MPI_Comm_rank(comm, &myrank); > > MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, comm, &win); > > Host2_program: > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > MPI_Comm_rank(comm, &myrank); > > MPI_Win_create(buffer, buf_length, 1, MPI_INFO_NULL, comm, &win); > > > > Once the MPI_Window has been created I execute the following on host1: > > Host1_program(continued): > > err = MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, buf_length, MPI_BYTE, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Win_flush(target_rank, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Win_unlock(target_rank, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > While this is running, host2’s program is simply sleeping in a getchar(). > > > This should run and complete, however, if the other host has not called another mpi function such as MPI_Recv and instead is in a sleep, getchar(), etc. the MPI_Win_flush above will simply hang. > > > Why is the one-sided operation hanging until another MPI operation is called on the target host? My understanding is that utilizing MPI_Win_Flush does not require any action by the target host to complete the operation. > > > This is a stack trace of where the flush gets stuck: > > #0 0x0000003b456dea28 in poll () from /lib64/libc.so.6 > > #1 0x00007fe1c08004ee in MPID_nem_tcp_connpoll () > > from /usr/local/lib/libmpich.so.12 > > #2 0x00007fe1c07efab7 in MPIDI_CH3I_Progress () > > from /usr/local/lib/libmpich.so.12 > > #3 0x00007fe1c07d35ff in MPIDI_CH3I_Wait_for_lock_granted () > > from /usr/local/lib/libmpich.so.12 > > #4 0x00007fe1c07d88bf in MPIDI_Win_flush () from /usr/local/lib/libmpich.so.12 > > #5 0x00007fe1c08c1b46 in PMPI_Win_flush () from /usr/local/lib/libmpich.so.12 > > #6 0x000000000040115c in main () > > > According to this it seems that somehow the remote host is holding something after MPI_Win_Create that in my scenario gets unlocked if I call MPI_Recv on the connected_comm from above after the MPI_Win_Create. > > > Thank you for your time and I look forward to your reply. > > > Respectfully, > > Neil Spruit > > > > P.S. MPICH version info > > Version: 3.1 > > Release Date: Thu Feb 20 11:41:13 CST 2014 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hang in MPI_Win_Flush during scenario in one-sided communication
by Junchao Zhang 22 Sep '14

22 Sep '14

Hello, Neil, Could you send us a copy of the code? Thanks. --Junchao Zhang On Mon, Sep 22, 2014 at 4:53 PM, Neil Spruit <nrspruit(a)gmail.com> wrote: > Hello, > > > > I have been experimenting with mpich’s implementation of one-sided > communication with no remote synchronization that was introduced in the > mpi3.0 spec and I have run into a hang issue during connect/accept MPI > scenarios. > > > > Here is the situation, I have two applications that connect to each other > after they have been launched separately with mpiexec on two different > hosts. > > > > After connecting, both hosts allocate memory and run MPI_Win_Create as the > following: > > Host1_program: > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, > MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > MPI_Comm_rank(comm, &myrank); > > MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, comm, &win); > > Host2_program: > > void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, > MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); > > MPI_Intercomm_merge(connected_comm, 1, &comm); > > MPI_Comm_rank(comm, &myrank); > > MPI_Win_create(buffer, buf_length, 1, MPI_INFO_NULL, comm, &win); > > > > > > Once the MPI_Window has been created I execute the following on host1: > > Host1_program(continued): > > err = MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, > buf_length, MPI_BYTE, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Win_flush(target_rank, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > err = MPI_Win_unlock(target_rank, win); > > if(err != MPI_SUCCESS) > > { > > return 1; > > } > > While this is running, host2’s program is simply sleeping in a getchar(). > > > > This should run and complete, however, if the other host has not called > another mpi function such as MPI_Recv and instead is in a sleep, getchar(), > etc. the MPI_Win_flush above will simply hang. > > > > Why is the one-sided operation hanging until another MPI operation is > called on the target host? My understanding is that utilizing MPI_Win_Flush > does not require any action by the target host to complete the operation. > > > > This is a stack trace of where the flush gets stuck: > > #0 0x0000003b456dea28 in poll () from /lib64/libc.so.6 > > #1 0x00007fe1c08004ee in MPID_nem_tcp_connpoll () > > from /usr/local/lib/libmpich.so.12 > > #2 0x00007fe1c07efab7 in MPIDI_CH3I_Progress () > > from /usr/local/lib/libmpich.so.12 > > #3 0x00007fe1c07d35ff in MPIDI_CH3I_Wait_for_lock_granted () > > from /usr/local/lib/libmpich.so.12 > > #4 0x00007fe1c07d88bf in MPIDI_Win_flush () from > /usr/local/lib/libmpich.so.12 > > #5 0x00007fe1c08c1b46 in PMPI_Win_flush () from > /usr/local/lib/libmpich.so.12 > > #6 0x000000000040115c in main () > > > > According to this it seems that somehow the remote host is holding > something after MPI_Win_Create that in my scenario gets unlocked if I call > MPI_Recv on the connected_comm from above after the MPI_Win_Create. > > > > Thank you for your time and I look forward to your reply. > > > > Respectfully, > > Neil Spruit > > > P.S. MPICH version info > > Version: 3.1 > > Release Date: Thu Feb 20 11:41:13 CST 2014 > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Hang in MPI_Win_Flush during scenario in one-sided communication
by Neil Spruit 22 Sep '14

22 Sep '14

Hello, I have been experimenting with mpich’s implementation of one-sided communication with no remote synchronization that was introduced in the mpi3.0 spec and I have run into a hang issue during connect/accept MPI scenarios. Here is the situation, I have two applications that connect to each other after they have been launched separately with mpiexec on two different hosts. After connecting, both hosts allocate memory and run MPI_Win_Create as the following: Host1_program: void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); MPI_Intercomm_merge(connected_comm, 1, &comm); MPI_Comm_rank(comm, &myrank); MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, comm, &win); Host2_program: void* buffer = mmap(0, buf_length, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_POPULATE|MAP_ANON, -1, 0); MPI_Intercomm_merge(connected_comm, 1, &comm); MPI_Comm_rank(comm, &myrank); MPI_Win_create(buffer, buf_length, 1, MPI_INFO_NULL, comm, &win); Once the MPI_Window has been created I execute the following on host1: Host1_program(continued): err = MPI_Win_lock(MPI_LOCK_SHARED, target_rank, 0, win); if(err != MPI_SUCCESS) { return 1; } err = MPI_Put(buffer, buf_length, MPI_BYTE, target_rank, 0, buf_length, MPI_BYTE, win); if(err != MPI_SUCCESS) { return 1; } err = MPI_Win_flush(target_rank, win); if(err != MPI_SUCCESS) { return 1; } err = MPI_Win_unlock(target_rank, win); if(err != MPI_SUCCESS) { return 1; } While this is running, host2’s program is simply sleeping in a getchar(). This should run and complete, however, if the other host has not called another mpi function such as MPI_Recv and instead is in a sleep, getchar(), etc. the MPI_Win_flush above will simply hang. Why is the one-sided operation hanging until another MPI operation is called on the target host? My understanding is that utilizing MPI_Win_Flush does not require any action by the target host to complete the operation. This is a stack trace of where the flush gets stuck: #0 0x0000003b456dea28 in poll () from /lib64/libc.so.6 #1 0x00007fe1c08004ee in MPID_nem_tcp_connpoll () from /usr/local/lib/libmpich.so.12 #2 0x00007fe1c07efab7 in MPIDI_CH3I_Progress () from /usr/local/lib/libmpich.so.12 #3 0x00007fe1c07d35ff in MPIDI_CH3I_Wait_for_lock_granted () from /usr/local/lib/libmpich.so.12 #4 0x00007fe1c07d88bf in MPIDI_Win_flush () from /usr/local/lib/libmpich.so.12 #5 0x00007fe1c08c1b46 in PMPI_Win_flush () from /usr/local/lib/libmpich.so.12 #6 0x000000000040115c in main () According to this it seems that somehow the remote host is holding something after MPI_Win_Create that in my scenario gets unlocked if I call MPI_Recv on the connected_comm from above after the MPI_Win_Create. Thank you for your time and I look forward to your reply. Respectfully, Neil Spruit P.S. MPICH version info Version: 3.1 Release Date: Thu Feb 20 11:41:13 CST 2014 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Does MPICH implements polling mechanism on MPI_Recv?
by Wesley Bland 22 Sep '14

22 Sep '14

Which version of MPICH are you using? Which fault tolerance features are you using? Fault tolerance is currently undergoing some changes and has different features than it used to have. AFAIK, neither version of FT has been tested with ch3:sock. It’s possible that it will work, but FT is still a very experimental feature and hasn’t been widely tested. If you want to avoid polling more, you can use non-blocking receive calls to post a receive and poll the system yourself periodically (using MPI_TEST). This will give your application an opportunity to do something else while waiting for the receives to complete. Thanks, Wesley > On Sep 22, 2014, at 8:30 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Dear MPICH. > I have a problem with poling MPICH mechanism. > I'm working on cluster. There are 2-4 processes on each computer (I can't execute single process per computer because of application requirements). > My system has 2 states: > Ready - slaves listen to master (but no data flow) > Run - masters start communication, then there is data flow. > When system in ready state (all processes except master executed MPI_Recv requests on master) but Master process still net sending data I see CPU usage > 100% (more than 1 core used) per process. When 4 processes are in ready state (waiting for data) computer begins to slow down other processes, I think because of polling. > I tried to build MPICH with --with-device=ch3:sock, then I get 0% CPU usage in ready state, but I have a problem with Fault tolerance feature. > My questions are: > 1) Is such behavior expected that build with --with-device=ch3:sock causes Fault tolerance not work? Does Fault tolerance based on polling mechanism? > 2) Can I change polling rate to reduce CPU payload? I understand that penalty is transfer rate slow down. > 3) Can I use any other MPI APIs to check if message from master is arrived w/o activating polling mechanism? > > Regards, > Anatoly. > > > On Thu, May 8, 2014 at 3:57 PM, Balaji, Pavan <balaji(a)anl.gov <mailto:[email protected]>> wrote: > > This is expected. Currently, the only way to not have MPICH poll is to configure with --with-device=ch3:sock. Please note that this can cause performance loss (the polling is helpful for performance in the common case). > > We are planning to allow this in the default build as well in the future. > > — Pavan > > On May 8, 2014, at 7:54 AM, Anatoly G <anatolyrishon(a)gmail.com <mailto:[email protected]>> wrote: > > > Dear MPICH forum. > > I created an endless MPI program. > > In this program each process calls MPI_Recv from other process, w/o any MPI_Send. > > When I execute this program I see each process takes ~ 100% CPU core. > > Is this behavior (I suppose polling) is normal? > > May I reduce MPI_Recv CPU penalty? > > > > Regards, > > Anatoly. > > <mpi_polling.cpp>_______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0