June 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] error with MPI_Reduce running cpi
by Jinang_Shah 17 Jun '19

17 Jun '19

1) $ ./configure --prefix=/users/misc/sjinang/mpich-install 2) No. It is showing an error : mpiexec -n 2 -f hostfile ./mpi/mpich-3.3.1/examples/send_recv Fatal error in PMPI_Send: Unknown error class, error stack: PMPI_Send(159).............: MPI_Send(buf=0x7ffe5d59e2a4, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1845): Communication error with rank 1: Connection refused 3) I have done ssh from both the sides and its working fine. So now can you see the problem... Thanks. On 17-06-2019 21:28, Zhou, Hui wrote: > Hi Jinang_Shah, > > Could you list your configure line ( try `head config.log`)? > > If you try a simple example where process 0 sends a short message to process 1, would you result in similar error? > > Can csews1 and csews2 connect to each other freely, i.e. is there firewall, router, etc. between these two hosts? > > -- > Hui Zhou > > On Jun 17, 2019, at 10:11 AM, Jinang_Shah via discuss <discuss(a)mpich.org> wrote: > > $ mpiexec -n 2 -f hostfile ./mpi/mpich-3.3.1/examples/cpi > Process 1 of 2 is on csews2 > Process 0 of 2 is on csews1 > Fatal error in PMPI_Reduce: Unknown error class, error stack: > PMPI_Reduce(523)................: MPI_Reduce(sbuf=0x7ffeab580c10, rbuf=0x7ffeab580c18, count=1, datatype=MPI_DOUBLE, op=MPI_SUM, root=0, comm=MPI_COMM_WORLD) failed > PMPI_Reduce(509)................: > MPIR_Reduce_impl(316)...........: > MPIR_Reduce_intra_auto(231).....: > MPIR_Reduce_intra_binomial(125).: > MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 1 > > Can anyone explain this error and how to overcome it.I have installed just MPICH and not HYDRA package. > > By the way this same command works fine with hello program the one where each node says their identity. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] error with MPI_Reduce running cpi
by Zhou, Hui 17 Jun '19

17 Jun '19

Hi Jinang_Shah, Could you list your configure line ( try `head config.log`)? If you try a simple example where process 0 sends a short message to process 1, would you result in similar error? Can csews1 and csews2 connect to each other freely, i.e. is there firewall, router, etc. between these two hosts? — Hui Zhou On Jun 17, 2019, at 10:11 AM, Jinang_Shah via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: $ mpiexec -n 2 -f hostfile ./mpi/mpich-3.3.1/examples/cpi Process 1 of 2 is on csews2 Process 0 of 2 is on csews1 Fatal error in PMPI_Reduce: Unknown error class, error stack: PMPI_Reduce(523)................: MPI_Reduce(sbuf=0x7ffeab580c10, rbuf=0x7ffeab580c18, count=1, datatype=MPI_DOUBLE, op=MPI_SUM, root=0, comm=MPI_COMM_WORLD) failed PMPI_Reduce(509)................: MPIR_Reduce_impl(316)...........: MPIR_Reduce_intra_auto(231).....: MPIR_Reduce_intra_binomial(125).: MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 1 Can anyone explain this error and how to overcome it.I have installed just MPICH and not HYDRA package. By the way this same command works fine with hello program the one where each node says their identity. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

error with MPI_Reduce running cpi
by Jinang_Shah 17 Jun '19

17 Jun '19

>> $ mpiexec -n 2 -f hostfile ./mpi/mpich-3.3.1/examples/cpi >> Process 1 of 2 is on csews2 >> Process 0 of 2 is on csews1 >> Fatal error in PMPI_Reduce: Unknown error class, error stack: >> PMPI_Reduce(523)................: MPI_Reduce(sbuf=0x7ffeab580c10, rbuf=0x7ffeab580c18, count=1, datatype=MPI_DOUBLE, op=MPI_SUM, root=0, comm=MPI_COMM_WORLD) failed >> PMPI_Reduce(509)................: >> MPIR_Reduce_impl(316)...........: >> MPIR_Reduce_intra_auto(231).....: >> MPIR_Reduce_intra_binomial(125).: >> MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 1 >> >> Can anyone explain this error and how to overcome it.I have installed just MPICH and not HYDRA package. >> >> By the way this same command works fine with hello program the one where each node says their identity.

1 0

Re: [mpich-discuss] mpich/master: Something got wrong since may 24
by Eric Chamberland 13 Jun '19

13 Jun '19

Hi Rob, I am very sorry for wrongly suspecting MPI I/O... :/ Everything is effectively fine with MPI I/O itself! :) Regards, Eric On 2019-06-11 1:59 p.m., Latham, Robert J. wrote: > On Wed, 2019-06-05 at 09:18 -0400, Eric Chamberland via discuss wrote: >> Hi, >> >> I am in the early debugging stages of something that broke on may 24 >> on >> our test database with 17 tests against mpich/master branch. >> >> It seems related to MPI I/O, since we have observed the bugs just >> after >> MPI I/O reads. > Thanks for the report and the reproducer. > > Since you mentioned MPI-IO, I suspect most of the MPICH team will say > "Oh, that must be RobL's problem", but I'm (at least partially) off the > hook! Your test case doesn't have any call to MPI_File routines. > > I created https://github.com/pmodels/mpich/issues/3846 to track this > defect. > > ==rob > >> Everything is fine with other MPI libs, even older mpich. This is >> only >> a mpich/master problem. >> >> My early query: am I alone to have observed this? >> >> Thanks, >> >> Eric >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] RMA and multithreading (MPI_THREAD_MULTIPLE)
by Alexey Paznikov 12 Jun '19

12 Jun '19

Thank you for the answer! Second solution just suits me. I wanted to implement simple release-acquire semantics through atomic flag. For example, two processes include two threads, first of which waits for the flag by MPI_Get_accumulate and the other one remotely sets flag by MPI_Accumulate on the other process. In order to keep atomicity of operations, I have to use one window. Now all works fine: MPI_Win win; int flag = 0; int nproc = 0; int myrank = 0; void *thread(void *arg) { int flag_get = 0; do { MPI_Get_accumulate(NULL, 0, MPI_INT, &flag_get, 1, MPI_INT, myrank, 0, 1, MPI_INT, MPI_NO_OP, win); MPI_Win_flush(myrank, win); } while (flag_get == 0); printf("%d \t I got a flag!\n", myrank); return NULL; } int main(int argc, char *argv[]) { MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); MPI_Comm_size(MPI_COMM_WORLD, &nproc); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); assert(nproc == 2); int *buf = NULL; const int bufsize = 1; MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win); MPI_Win_lock_all(0, win); pthread_t tid; pthread_create(&tid, NULL, thread, NULL); int peer = 1 - myrank; int flag_put = 1; MPI_Accumulate(&flag_put, 1, MPI_INT, peer, 0, 1, MPI_INT, MPI_REPLACE, win); MPI_Win_flush(peer, win); pthread_join(tid, NULL); MPI_Win_unlock_all(win); MPI_Win_free(&win); MPI_Finalize(); return 0; } On Thu, Jun 6, 2019 at 23:18 Jeff Hammond wrote: > Your threads are trying to lock a window that is already locked or unlock > a window that is not locked. That's because you are using one window with > multiple threads. > > If you want to use multiple threads like this, do one of two things: > 1) use a window per thread > 2) synchronize with flush(_all). only lock and unlock the window once > (immediately after construction and immediately before destruction, > respectively) > > Jeff > > On Thu, Jun 6, 2019 at 12:30 PM Alexey Paznikov via discuss < > discuss(a)mpich.org> wrote: > >> Hi, >> >> I have multithreaded MPI program (MPI_THREAD_MULTIPLE mode) with RMA >> calls.This is a simplified example: >> >> #include <stdio.h> >> #include <mpi.h> >> #include <pthread.h> >> >> MPI_Win win; >> pthread_mutex_t lock; >> >> void *thread(void *arg) >> { >> MPI_Win_lock_all(0, win); >> MPI_Win_unlock_all(win); >> return NULL; >> } >> >> int main(int argc, char *argv[]) >> { >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); >> pthread_mutex_init(&lock, NULL); >> >> int *buf = NULL; >> const int bufsize = 1; >> MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); >> MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win); >> >> pthread_t tid; >> pthread_create(&tid, NULL, thread, NULL); >> >> MPI_Win_lock_all(0, win); >> MPI_Win_unlock_all(win); >> >> pthread_join(tid, NULL); >> >> MPI_Win_free(&win); >> >> MPI_Finalize(); >> >> return 0; >> } >> >> If I run such program, it crashes with an error message: >> >> Fatal error in PMPI_Win_lock_all: Wrong synchronization of RMA calls , >> error stack: >> PMPI_Win_lock_all(149).: MPI_Win_lock_all(assert=0, win=0xa0000000) failed >> MPID_Win_lock_all(1522): Wrong synchronization of RMA calls >> >> If I replace MPI_Win_lock_all with MPI_Win_lock the problem remains: >> >> Fatal error in PMPI_Win_lock: Wrong synchronization of RMA calls , error >> stack: >> PMPI_Win_lock(157).: MPI_Win_lock(lock_type=234, rank=0, assert=0, >> win=0xa0000000) failed >> MPID_Win_lock(1163): Wrong synchronization of RMA calls >> >> (this message is repeated many times) >> >> If I protect RMA operations with a mutex, the problem disappears. >> >> MPICH version 3.3. Similar problem is also in MVAPICH2 2.3.1. >> >> Are the RMA operations not thread safe at the moment? Could you tell me >> how to deal with this problem? >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Alexey Paznikov

1 0

Re: [mpich-discuss] mpich/master: Something got wrong since may 24
by Latham, Robert J. 11 Jun '19

11 Jun '19

On Wed, 2019-06-05 at 09:18 -0400, Eric Chamberland via discuss wrote: > Hi, > > I am in the early debugging stages of something that broke on may 24 > on > our test database with 17 tests against mpich/master branch. > > It seems related to MPI I/O, since we have observed the bugs just > after > MPI I/O reads. Thanks for the report and the reproducer. Since you mentioned MPI-IO, I suspect most of the MPICH team will say "Oh, that must be RobL's problem", but I'm (at least partially) off the hook! Your test case doesn't have any call to MPI_File routines. I created https://github.com/pmodels/mpich/issues/3846 to track this defect. ==rob > Everything is fine with other MPI libs, even older mpich. This is > only > a mpich/master problem. > > My early query: am I alone to have observed this? > > Thanks, > > Eric > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich/master: Something got wrong since may 24
by Eric Chamberland 06 Jun '19

06 Jun '19

Hi, I have worked to reproduce the bug in a somewhat small example. On on any good MPI version, the output of the attached example is: Rank 1: Element # 33 : {40,106,105,204,208,200},8 Rank 1: Element # 34 : {61,105,106,201,208,205},8 Rank 1: Element # 35 : {40,105,107,200,212,209},8 Rank 1: Element # 36 : {62,107,105,210,212,202},8 Rank 1: Element # 37 : {40,108,106,213,216,204},8 Rank 1: Element # 38 : {42,110,85,223,227,108},8 On the mpich/master branch, since may 24, commit 60035bb7b9766db15041033959e4209f0ad936b, the output is: Rank 1: Element # 33 : {40,106,105,204,208,200},8 Rank 1: Element # 34 : {52,76,104,74,197,195},8 Rank 1: Element # 35 : {62,105,65,202,203,40},8 Rank 1: Element # 36 : {63,66,106,41,207,206},8 Rank 1: Element # 37 : {40,106,105,204,208,200},8 Rank 1: Element # 38 : {42,110,85,223,227,108},8 I hope you wil be able to work with my example to reproduce and solve this problem! Compile & run with: mpic++ -o bugmaster mpich_bug_indexed_type.cc mpirun -n 2 bugmaster Thanks a lot, Eric Le 2019-06-05 à 9:18 a.m., Eric Chamberland via discuss a écrit : > Hi, > > I am in the early debugging stages of something that broke on may 24 on > our test database with 17 tests against mpich/master branch. > > It seems related to MPI I/O, since we have observed the bugs just after > MPI I/O reads. > > Everything is fine with other MPI libs, even older mpich. This is only > a mpich/master problem. > > My early query: am I alone to have observed this? > > Thanks, > > Eric > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] A possible bug in HYD_pmcd_pmi_allocate_kvs
by Xiaopeng Duan 06 Jun '19

06 Jun '19

Got it. Thank you, Giuseppe and Ken. Regards, Xiaopeng On Thu, Jun 6, 2019, 2:56 PM Congiu, Giuseppe <gcongiu(a)anl.gov> wrote: > Actually, the fix uses a combination of hostname and random number, which > seed is a time stamp. > I don’t remember why exactly we didn’t go for the hostname only but I > suspect it is because this might not be > unique. Adding the random number with a timestamp seed should be robust > enough against collisions. > > Giuseppe > > > On Jun 6, 2019, at 8:25 AM, Raffenetti, Kenneth J. via discuss < > discuss(a)mpich.org> wrote: > > > > On 6/6/19 12:52 AM, Xiaopeng Duan wrote: > >> Thank you, Ken. > >> > >> We were having another problem with 3.3, and will try it once we fixed > >> our issue. > >> > >> Just my couriosity, why a random number was chosen for the fix instead > >> of hostname or address? Looks to me the random number still has some > >> possibility to repeat (although very rare), but hostnames and addresses > >> should be unique in a system. > > > > I had the same thought when looking back at this patch. Maybe Giuseppe > > can share why that was added. I'm fairly sure it can be safely removed. > > > > Ken > > > >> > >> Regards, > >> Xiaopeng > >> > >> On Wed, Jun 5, 2019, 8:29 AM Raffenetti, Kenneth J. > >> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > >> > >> We added a similar fix in https://github.com/pmodels/mpich/pull/2788 > . > >> This was included in the MPICH 3.3 release. > >> > >> Ken > >> > >> On 6/4/19 11:27 PM, Xiaopeng Duan via discuss wrote: > >>> Hi, MPICH experts, > >>> > >>> We are working on a dynamic master-worker flow using > >>> mpi_comm_connect/mpi_com_accept. In some cases when the total > >> number of > >>> worker process is large, they may get the same kvs_name and > >> confuse the > >>> internal group identifiers. This was traced to the naming > >> convention in > >>> HYD_pmcd_pmi_allocate_kvs() that considers only process id, while > >> two > >>> processes on different machines may have the same pid. I tried to > >> add > >>> host name (from unistd.h>gethostname) to the name, i.e. > >>> 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our > >> testing. > >>> > >>> So I'm wondering if this change is safe (we may need it for our > >> release) > >>> and if it would go into the official MPICH release some time. > >>> > >>> Thank you very much. > >>> > >>> Xiaopeng > >>> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >>> > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] RMA and multithreading (MPI_THREAD_MULTIPLE)
by Jeff Hammond 06 Jun '19

06 Jun '19

Your threads are trying to lock a window that is already locked or unlock a window that is not locked. That's because you are using one window with multiple threads. If you want to use multiple threads like this, do one of two things: 1) use a window per thread 2) synchronize with flush(_all). only lock and unlock the window once (immediately after construction and immediately before destruction, respectively) Jeff On Thu, Jun 6, 2019 at 12:30 PM Alexey Paznikov via discuss < discuss(a)mpich.org> wrote: > Hi, > > I have multithreaded MPI program (MPI_THREAD_MULTIPLE mode) with RMA > calls.This is a simplified example: > > #include <stdio.h> > #include <mpi.h> > #include <pthread.h> > > MPI_Win win; > pthread_mutex_t lock; > > void *thread(void *arg) > { > MPI_Win_lock_all(0, win); > MPI_Win_unlock_all(win); > return NULL; > } > > int main(int argc, char *argv[]) > { > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); > pthread_mutex_init(&lock, NULL); > > int *buf = NULL; > const int bufsize = 1; > MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); > MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win); > > pthread_t tid; > pthread_create(&tid, NULL, thread, NULL); > > MPI_Win_lock_all(0, win); > MPI_Win_unlock_all(win); > > pthread_join(tid, NULL); > > MPI_Win_free(&win); > > MPI_Finalize(); > > return 0; > } > > If I run such program, it crashes with an error message: > > Fatal error in PMPI_Win_lock_all: Wrong synchronization of RMA calls , > error stack: > PMPI_Win_lock_all(149).: MPI_Win_lock_all(assert=0, win=0xa0000000) failed > MPID_Win_lock_all(1522): Wrong synchronization of RMA calls > > If I replace MPI_Win_lock_all with MPI_Win_lock the problem remains: > > Fatal error in PMPI_Win_lock: Wrong synchronization of RMA calls , error > stack: > PMPI_Win_lock(157).: MPI_Win_lock(lock_type=234, rank=0, assert=0, > win=0xa0000000) failed > MPID_Win_lock(1163): Wrong synchronization of RMA calls > > (this message is repeated many times) > > If I protect RMA operations with a mutex, the problem disappears. > > MPICH version 3.3. Similar problem is also in MVAPICH2 2.3.1. > > Are the RMA operations not thread safe at the moment? Could you tell me > how to deal with this problem? > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] A possible bug in HYD_pmcd_pmi_allocate_kvs
by Congiu, Giuseppe 06 Jun '19

06 Jun '19

Actually, the fix uses a combination of hostname and random number, which seed is a time stamp. I don’t remember why exactly we didn’t go for the hostname only but I suspect it is because this might not be unique. Adding the random number with a timestamp seed should be robust enough against collisions. Giuseppe > On Jun 6, 2019, at 8:25 AM, Raffenetti, Kenneth J. via discuss <discuss(a)mpich.org> wrote: > > On 6/6/19 12:52 AM, Xiaopeng Duan wrote: >> Thank you, Ken. >> >> We were having another problem with 3.3, and will try it once we fixed >> our issue. >> >> Just my couriosity, why a random number was chosen for the fix instead >> of hostname or address? Looks to me the random number still has some >> possibility to repeat (although very rare), but hostnames and addresses >> should be unique in a system. > > I had the same thought when looking back at this patch. Maybe Giuseppe > can share why that was added. I'm fairly sure it can be safely removed. > > Ken > >> >> Regards, >> Xiaopeng >> >> On Wed, Jun 5, 2019, 8:29 AM Raffenetti, Kenneth J. >> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: >> >> We added a similar fix in https://github.com/pmodels/mpich/pull/2788. >> This was included in the MPICH 3.3 release. >> >> Ken >> >> On 6/4/19 11:27 PM, Xiaopeng Duan via discuss wrote: >>> Hi, MPICH experts, >>> >>> We are working on a dynamic master-worker flow using >>> mpi_comm_connect/mpi_com_accept. In some cases when the total >> number of >>> worker process is large, they may get the same kvs_name and >> confuse the >>> internal group identifiers. This was traced to the naming >> convention in >>> HYD_pmcd_pmi_allocate_kvs() that considers only process id, while >> two >>> processes on different machines may have the same pid. I tried to >> add >>> host name (from unistd.h>gethostname) to the name, i.e. >>> 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our >> testing. >>> >>> So I'm wondering if this change is safe (we may need it for our >> release) >>> and if it would go into the official MPICH release some time. >>> >>> Thank you very much. >>> >>> Xiaopeng >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0