November 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Amin Hassani 25 Nov '14

25 Nov '14

Here I attached config.log exits in the root folder where it is compiled. I'm not too familiar with MPICH but, there are other config.logs in other directories also but not sure if you needed them too. I don't have any specific environment variable that can relate to MPICH. Also tried with export HYDRA_HOST_FILE=<address to host file>, but have the same problem. I don't do anything FT related in MPICH, I don't think this version of MPICH has anything related to FT in it. Thanks. Amin Hassani, CIS department at UAB, Birmingham, AL, USA. On Tue, Nov 25, 2014 at 9:02 PM, Bland, Wesley B. <wbland(a)anl.gov> wrote: > Can you also provide your config.log and any CVARs or other relevant > environment variables that you might be setting (for instance, in relation > to fault tolerance)? > > Thanks, > Wesley > > > On Nov 25, 2014, at 3:58 PM, Amin Hassani <ahassani(a)cis.uab.edu> wrote: > > This is the simplest code I have that doesn't run. > > > #include <mpi.h> > #include <stdio.h> > #include <malloc.h> > #include <unistd.h> > #include <stdlib.h> > > int main(int argc, char** argv) > { > int rank, size; > int i, j, k; > double t1, t2; > int rc; > > MPI_Init(&argc, &argv); > MPI_Comm world = MPI_COMM_WORLD, newworld, newworld2; > MPI_Comm_rank(world, &rank); > MPI_Comm_size(world, &size); > > t2 = 1; > MPI_Allreduce(&t2, &t_avg, 1, MPI_DOUBLE, MPI_SUM, world); > t_avg = t_avg / size; > > MPI_Finalize(); > > return 0; > } > > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > On Tue, Nov 25, 2014 at 2:46 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> > wrote: > >> >> Hi Amin, >> >> Can you share with us a minimal piece of code with which you can >> reproduce this issue? >> >> Thanks, >> Antonio >> >> >> >> On 11/25/2014 12:52 PM, Amin Hassani wrote: >> >> Hi, >> >> I am having problem running MPICH, on multiple nodes. When I run an >> multiple MPI processes on one node, it totally works, but when I try to run >> on multiple nodes, it fails with the error below. >> My machines have Debian OS, Both infiniband and TCP interconnects. I'm >> guessing it has something do to with the TCP network, but I can run openmpi >> on these machines with no problem. But for some reason I cannot run MPICH >> on multiple nodes. Please let me know if more info is needed from my side. >> I'm guessing there are some configuration that I am missing. I used MPICH >> 3.1.3 for this test. I googled this problem but couldn't find any solution. >> >> In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD. >> >> my host file (hosts-hydra) is something like this: >> oakmnt-0-a:1 >> oakmnt-0-b:1 >> >> I get this error: >> >> $ mpirun -hostfile hosts-hydra -np 2 test_dup >> Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: >> status->MPI_TAG == recvtag >> Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: >> status->MPI_TAG == recvtag >> internal ABORT - process 1 >> internal ABORT - process 0 >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 30744 RUNNING AT oakmnt-0-b >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> =================================================================================== >> [mpiexec@vulcan13] HYDU_sock_read >> (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file >> descriptor) >> [mpiexec@vulcan13] control_cb >> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read >> command from proxy >> [mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event >> (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned >> error status >> [mpiexec@vulcan13] HYD_pmci_wait_for_completion >> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for >> event >> [mpiexec@vulcan13] main >> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error >> waiting for completion >> >> Thanks. >> Amin Hassani, >> CIS department at UAB, >> Birmingham, AL, USA. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847apenya(a)mcs.anl.govwww.mcs.anl.gov/~apenya >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Bland, Wesley B. 25 Nov '14

25 Nov '14

Can you also provide your config.log and any CVARs or other relevant environment variables that you might be setting (for instance, in relation to fault tolerance)? Thanks, Wesley On Nov 25, 2014, at 3:58 PM, Amin Hassani <ahassani(a)cis.uab.edu<mailto:[email protected]>> wrote: This is the simplest code I have that doesn't run. #include <mpi.h> #include <stdio.h> #include <malloc.h> #include <unistd.h> #include <stdlib.h> int main(int argc, char** argv) { int rank, size; int i, j, k; double t1, t2; int rc; MPI_Init(&argc, &argv); MPI_Comm world = MPI_COMM_WORLD, newworld, newworld2; MPI_Comm_rank(world, &rank); MPI_Comm_size(world, &size); t2 = 1; MPI_Allreduce(&t2, &t_avg, 1, MPI_DOUBLE, MPI_SUM, world); t_avg = t_avg / size; MPI_Finalize(); return 0; } Amin Hassani, CIS department at UAB, Birmingham, AL, USA. On Tue, Nov 25, 2014 at 2:46 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hi Amin, Can you share with us a minimal piece of code with which you can reproduce this issue? Thanks, Antonio On 11/25/2014 12:52 PM, Amin Hassani wrote: Hi, I am having problem running MPICH, on multiple nodes. When I run an multiple MPI processes on one node, it totally works, but when I try to run on multiple nodes, it fails with the error below. My machines have Debian OS, Both infiniband and TCP interconnects. I'm guessing it has something do to with the TCP network, but I can run openmpi on these machines with no problem. But for some reason I cannot run MPICH on multiple nodes. Please let me know if more info is needed from my side. I'm guessing there are some configuration that I am missing. I used MPICH 3.1.3 for this test. I googled this problem but couldn't find any solution. In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD. my host file (hosts-hydra) is something like this: oakmnt-0-a:1 oakmnt-0-b:1 I get this error: $ mpirun -hostfile hosts-hydra -np 2 test_dup Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag internal ABORT - process 1 internal ABORT - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 30744 RUNNING AT oakmnt-0-b = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [mpiexec@vulcan13] HYDU_sock_read (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file descriptor) [mpiexec@vulcan13] control_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read command from proxy [mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@vulcan13] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@vulcan13] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thanks. Amin Hassani, CIS department at UAB, Birmingham, AL, USA. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov<mailto:[email protected]> www.mcs.anl.gov/~apenya<http://www.mcs.anl.gov/~apenya> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Amin Hassani 25 Nov '14

25 Nov '14

This is the simplest code I have that doesn't run. #include <mpi.h> #include <stdio.h> #include <malloc.h> #include <unistd.h> #include <stdlib.h> int main(int argc, char** argv) { int rank, size; int i, j, k; double t1, t2; int rc; MPI_Init(&argc, &argv); MPI_Comm world = MPI_COMM_WORLD, newworld, newworld2; MPI_Comm_rank(world, &rank); MPI_Comm_size(world, &size); t2 = 1; MPI_Allreduce(&t2, &t_avg, 1, MPI_DOUBLE, MPI_SUM, world); t_avg = t_avg / size; MPI_Finalize(); return 0; } Amin Hassani, CIS department at UAB, Birmingham, AL, USA. On Tue, Nov 25, 2014 at 2:46 PM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> wrote: > > Hi Amin, > > Can you share with us a minimal piece of code with which you can reproduce > this issue? > > Thanks, > Antonio > > > > On 11/25/2014 12:52 PM, Amin Hassani wrote: > > Hi, > > I am having problem running MPICH, on multiple nodes. When I run an > multiple MPI processes on one node, it totally works, but when I try to run > on multiple nodes, it fails with the error below. > My machines have Debian OS, Both infiniband and TCP interconnects. I'm > guessing it has something do to with the TCP network, but I can run openmpi > on these machines with no problem. But for some reason I cannot run MPICH > on multiple nodes. Please let me know if more info is needed from my side. > I'm guessing there are some configuration that I am missing. I used MPICH > 3.1.3 for this test. I googled this problem but couldn't find any solution. > > In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD. > > my host file (hosts-hydra) is something like this: > oakmnt-0-a:1 > oakmnt-0-b:1 > > I get this error: > > $ mpirun -hostfile hosts-hydra -np 2 test_dup > Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: > status->MPI_TAG == recvtag > Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: > status->MPI_TAG == recvtag > internal ABORT - process 1 > internal ABORT - process 0 > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 30744 RUNNING AT oakmnt-0-b > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > [mpiexec@vulcan13] HYDU_sock_read > (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file > descriptor) > [mpiexec@vulcan13] control_cb > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read > command from proxy > [mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned > error status > [mpiexec@vulcan13] HYD_pmci_wait_for_completion > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for > event > [mpiexec@vulcan13] main > (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error > waiting for completion > > Thanks. > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847apenya(a)mcs.anl.govwww.mcs.anl.gov/~apenya > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by "Antonio J. Peña" 25 Nov '14

25 Nov '14

Hi Amin, Can you share with us a minimal piece of code with which you can reproduce this issue? Thanks, Antonio On 11/25/2014 12:52 PM, Amin Hassani wrote: > Hi, > > I am having problem running MPICH, on multiple nodes. When I run an > multiple MPI processes on one node, it totally works, but when I try > to run on multiple nodes, it fails with the error below. > My machines have Debian OS, Both infiniband and TCP interconnects. I'm > guessing it has something do to with the TCP network, but I can run > openmpi on these machines with no problem. But for some reason I > cannot run MPICH on multiple nodes. Please let me know if more info is > needed from my side. I'm guessing there are some configuration that I > am missing. I used MPICH 3.1.3 for this test. I googled this problem > but couldn't find any solution. > > In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD. > > my host file (hosts-hydra) is something like this: > oakmnt-0-a:1 > oakmnt-0-b:1 > > I get this error: > > $ mpirun -hostfile hosts-hydra -np 2 test_dup > Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: > status->MPI_TAG == recvtag > Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: > status->MPI_TAG == recvtag > internal ABORT - process 1 > internal ABORT - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 30744 RUNNING AT oakmnt-0-b > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > [mpiexec@vulcan13] HYDU_sock_read > (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file > descriptor) > [mpiexec@vulcan13] control_cb > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read > command from proxy > [mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback > returned error status > [mpiexec@vulcan13] HYD_pmci_wait_for_completion > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error > waiting for event > [mpiexec@vulcan13] main > (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager > error waiting for completion > > Thanks. > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

having problem running MPICH on multiple nodes
by Amin Hassani 25 Nov '14

25 Nov '14

Hi, I am having problem running MPICH, on multiple nodes. When I run an multiple MPI processes on one node, it totally works, but when I try to run on multiple nodes, it fails with the error below. My machines have Debian OS, Both infiniband and TCP interconnects. I'm guessing it has something do to with the TCP network, but I can run openmpi on these machines with no problem. But for some reason I cannot run MPICH on multiple nodes. Please let me know if more info is needed from my side. I'm guessing there are some configuration that I am missing. I used MPICH 3.1.3 for this test. I googled this problem but couldn't find any solution. In my MPI program, I am doing a simple allreduce over MPI_COMM_WORLD. my host file (hosts-hydra) is something like this: oakmnt-0-a:1 oakmnt-0-b:1 I get this error: $ mpirun -hostfile hosts-hydra -np 2 test_dup Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag Assertion failed in file ../src/mpi/coll/helper_fns.c at line 490: status->MPI_TAG == recvtag internal ABORT - process 1 internal ABORT - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 30744 RUNNING AT oakmnt-0-b = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [mpiexec@vulcan13] HYDU_sock_read (../../../../src/pm/hydra/utils/sock/sock.c:239): read error (Bad file descriptor) [mpiexec@vulcan13] control_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:199): unable to read command from proxy [mpiexec@vulcan13] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@vulcan13] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@vulcan13] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thanks. Amin Hassani, CIS department at UAB, Birmingham, AL, USA. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion failed in file src/mpid/ch3/src/ch3u_handle_send_req.c at line 61 (RMA && Derived datatypes)
by Victor Vysotskiy 25 Nov '14

25 Nov '14

Dear Xin, I just checked out the latest nightly snapshot 'v3.2a2-16-g8a0887b9'. So, everything works out on my laptop. Thank you very much for your help and support! With best regards, Victor. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion in MX netmod
by Kuleshov Aleksey 24 Nov '14

24 Nov '14

> Our policy is to keep unsupported code for a while in our releases in a > best effort practice, at least while it seems to be working and does not > bother us in our developments. The reality is that we do not further > have any hardware nor specific funding to keep supporting this netmod. Ok, got it. >> 2) netmod newmad has the same routing as mx (which calls assertion) but newmad is still in 3.2a2 => MPICH still has broken code unless something was fixed in subroutings? > > Are you saying that you can reproduce the issue with the newmad netmod? No. > Otherwise, similar code paths do not necessarily mean that we will be > hitting the same bug. I really hope so! The last assertion was called in sending, when the path had it way from user program's MPI_Send directly to the netmod's function 'send' and newmad and mx codes are almost identical on this way. I said, "unless something was fixed". May be it was fixed after 3.1.2 release, may be it is in MX netmod -- I don't know. Now I see that 3.2a2 version has new mxm netmod which also uses that routing which causes assertion. So I'll try to use the latest MPICH code and port MX to it and see if I hit the same assertion. > We do extensive automated testing in multiple > netmods, architectures, compilers, and compiling configurations. Without > being able to reproduce the problem in other than MX, we cannot conclude > other than that the bug was specifically located in that netmod. In case > you confirm you are reproducing the same bug in the newmad netmod, we > will contact the external person who contributed and used to maintain it. Ok. I tried newmad several months ago with MPICH (during very short time -- never tried to use it on these tests), but, unfortunately, I don't have time to resurrect it and try it on these tests. Anyway, thank you for conversation. It helped me to move on -- to the newest MPICH version! _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion in MX netmod
by "Antonio J. Peña" 24 Nov '14

24 Nov '14

On 11/24/2014 12:23 PM, Kuleshov Aleksey wrote: > Thank you Antonio for this information. This is very sad, because: > 1) in the releases 3.1.2 and 3.1.3 (which are stable releases!) MPICH has broken netmod? Our policy is to keep unsupported code for a while in our releases in a best effort practice, at least while it seems to be working and does not bother us in our developments. The reality is that we do not further have any hardware nor specific funding to keep supporting this netmod. > 2) netmod newmad has the same routing as mx (which calls assertion) but newmad is still in 3.2a2 => MPICH still has broken code unless something was fixed in subroutings? Are you saying that you can reproduce the issue with the newmad netmod? Otherwise, similar code paths do not necessarily mean that we will be hitting the same bug. We do extensive automated testing in multiple netmods, architectures, compilers, and compiling configurations. Without being able to reproduce the problem in other than MX, we cannot conclude other than that the bug was specifically located in that netmod. In case you confirm you are reproducing the same bug in the newmad netmod, we will contact the external person who contributed and used to maintain it. Antonio > > 24.11.2014, 18:09, "Antonio J. Peña" <apenya(a)mcs.anl.gov>: >> Dear Kuleshov, >> >> In order to accomodate resources for more recent networking APIs we >> dropped support for the mx netmod, which in fact has been completely >> removed in our most recent 3.2 releases. So, unfortunately, we are not >> able to assist you with this issue. >> >> Best regards, >> Antonio >> >> On 11/22/2014 01:52 PM, Kuleshov Aleksey wrote: >>> And the same problem with different approach: >>> I downloaded from http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html mpi2test.tar.gz, built it and try >>> to run pingping test: >>>> MPITEST_VERBOSE=1 MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 2 /tests/pingping >>> [stdout] >>> Get new datatypes: send = MPI_INT, recv = MPI_INT >>> Get new datatypes: send = MPI_INT, recv = MPI_INT >>> Sending count = 1 of sendtype MPI_INT of total size 4 bytes >>> Sending count = 1 of sendtype MPI_INT of total size 4 bytes >>> Get new datatypes: send = MPI_DOUBLE, recv = MPI_DOUBLE >>> Get new datatypes: send = MPI_DOUBLE, recv = MPI_DOUBLE >>> Sending count = 1 of sendtype MPI_DOUBLE of total size 8 bytes >>> Sending count = 1 of sendtype MPI_DOUBLE of total size 8 bytes >>> Get new datatypes: send = MPI_FLOAT_INT, recv = MPI_FLOAT_INT >>> Sending count = 1 of sendtype MPI_FLOAT_INT of total size 8 bytes >>> Get new datatypes: send = MPI_FLOAT_INT, recv = MPI_FLOAT_INT >>> Sending count = 1 of sendtype MPI_FLOAT_INT of total size 8 bytes >>> Get new datatypes: send = dup of MPI_INT, recv = dup of MPI_INT >>> Get new datatypes: send = dup of MPI_INT, recv = dup of MPI_INT >>> Sending count = 1 of sendtype dup of MPI_INT of total size 4 bytes >>> Sending count = 1 of sendtype dup of MPI_INT of total size 4 bytes >>> Get new datatypes: send = int-vector, recv = MPI_INT >>> Sending count = 1 of sendtype int-vector of total size 4 bytes >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_send.c at line 435: n_iov > 0 >>> internal ABORT - process 0 >>> [/stdout] >>> >>> 22.11.2014, 18:39, "Kuleshov Aleksey" <rndfax(a)yandex.ru>: >>>> Hello! Can you please help me with problem? >>>> >>>> I'm working on custom myriexpress library and I'm using MX netmod in MPICH v.3.1.2. >>>> For testing purposes I built OSU Micro Benchmarks v3.8. >>>> >>>> To run it on 7 nodes I execute test osu_alltoall as follows: >>>>> MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 7 /osu_alltoall >>>> It passed successfully (I also tried it on 2, 3, 4, 5 and 6 nodes - everything is alright). >>>> >>>> But now I want to run it on 8 nodes: >>>>> MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 8 /osu_alltoall >>>> [stdout] >>>> # OSU MPI All-to-All Personalized Exchange Latency Test v3.8 >>>> # Size Avg Latency(us) >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>>> internal ABORT - process 4 >>>> internal ABORT - process 7 >>>> internal ABORT - process 2 >>>> internal ABORT - process 6 >>>> internal ABORT - process 3 >>>> internal ABORT - process 0 >>>> internal ABORT - process 5 >>>> internal ABORT - process 1 >>>> [/stdout] >>>> >>>> So, what does these assertions mean? >>>> Is it something wrong with MX netmod? >>>> Or in myriexpress library? >>>> Or in test osu_alltoall itself? >>>> >>>> BTW, osu_alltoall on 8 nodes passed successfully for TCP netmod. >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion failed in file src/mpid/ch3/src/ch3u_handle_send_req.c at line 61 (RMA && Derived datatypes)
by Zhao, Xin 24 Nov '14

24 Nov '14

Hi Victor, The bug is recently fixed in mpich/master (see https://trac.mpich.org/projects/mpich/ticket/2204) Could you try tonight's nightly snapshot? Thanks, Xin ________________________________________ From: Victor Vysotskiy [victor.vysotskiy(a)teokem.lu.se] Sent: Wednesday, November 19, 2014 3:00 AM To: Zhao, Xin Subject: RE: [mpich-discuss] Assertion failed in file src/mpid/ch3/src/ch3u_handle_send_req.c at line 61 (RMA && Derived datatypes) Hi Xin, >I think it is due to a bug in our MPICH RMA thanks for your email! It is a great news, that you have found a problematic place inside MPICH. >I created a ticket for this: https://trac.mpich.org/projects/mpich/ticket/2204, you can track the progress of this bug on it. I will keep an eye on it. With best regards, Victor. ________________________________________ From: Zhao, Xin [xinzhao3(a)illinois.edu] Sent: Wednesday, November 19, 2014 4:40 AM To: discuss(a)mpich.org Cc: Victor Vysotskiy Subject: RE: [mpich-discuss] Assertion failed in file src/mpid/ch3/src/ch3u_handle_send_req.c at line 61 (RMA && Derived datatypes) Hi Victor, I looked your test code and I think it is due to a bug in our MPICH RMA that the request handler is not re-entrant safe, which makes win_ptr->at_completion_counter being decremented twice for one GET operation. We will fix it as soon as possible. I created a ticket for this: https://trac.mpich.org/projects/mpich/ticket/2204, you can track the progress of this bug on it. Thanks, Xin ________________________________________ From: Victor Vysotskiy [victor.vysotskiy(a)teokem.lu.se] Sent: Tuesday, November 18, 2014 6:00 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Assertion failed in file src/mpid/ch3/src/ch3u_handle_send_req.c at line 61 (RMA && Derived datatypes) Hi Pavan, >FYI, this is what we heard back from the Mellanox folks: >I think, issue could be because of older MXM (part of MOFED) being used. We can ask him to try latest MXM from HPCX (http://bgate.mellanox.com/products/hpcx) Indeed, I just checked the latest software stack, including: - hpcx-v1.2.0-258-icc-OFED-3.12-redhat6.5; - MPICH v3.2a2 ('--with-device=ch3:nemesis:mxm'); And, there is no problem with 'assertion failed in ch3u_handle_send_req.c' anymore! Many thanks for your help and support! With best regards, Victor. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion in MX netmod
by Kuleshov Aleksey 24 Nov '14

24 Nov '14

Thank you Antonio for this information. This is very sad, because: 1) in the releases 3.1.2 and 3.1.3 (which are stable releases!) MPICH has broken netmod? 2) netmod newmad has the same routing as mx (which calls assertion) but newmad is still in 3.2a2 => MPICH still has broken code unless something was fixed in subroutings? 24.11.2014, 18:09, "Antonio J. Peña" <apenya(a)mcs.anl.gov>: > Dear Kuleshov, > > In order to accomodate resources for more recent networking APIs we > dropped support for the mx netmod, which in fact has been completely > removed in our most recent 3.2 releases. So, unfortunately, we are not > able to assist you with this issue. > > Best regards, > Antonio > > On 11/22/2014 01:52 PM, Kuleshov Aleksey wrote: >> And the same problem with different approach: >> I downloaded from http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html mpi2test.tar.gz, built it and try >> to run pingping test: >>> MPITEST_VERBOSE=1 MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 2 /tests/pingping >> [stdout] >> Get new datatypes: send = MPI_INT, recv = MPI_INT >> Get new datatypes: send = MPI_INT, recv = MPI_INT >> Sending count = 1 of sendtype MPI_INT of total size 4 bytes >> Sending count = 1 of sendtype MPI_INT of total size 4 bytes >> Get new datatypes: send = MPI_DOUBLE, recv = MPI_DOUBLE >> Get new datatypes: send = MPI_DOUBLE, recv = MPI_DOUBLE >> Sending count = 1 of sendtype MPI_DOUBLE of total size 8 bytes >> Sending count = 1 of sendtype MPI_DOUBLE of total size 8 bytes >> Get new datatypes: send = MPI_FLOAT_INT, recv = MPI_FLOAT_INT >> Sending count = 1 of sendtype MPI_FLOAT_INT of total size 8 bytes >> Get new datatypes: send = MPI_FLOAT_INT, recv = MPI_FLOAT_INT >> Sending count = 1 of sendtype MPI_FLOAT_INT of total size 8 bytes >> Get new datatypes: send = dup of MPI_INT, recv = dup of MPI_INT >> Get new datatypes: send = dup of MPI_INT, recv = dup of MPI_INT >> Sending count = 1 of sendtype dup of MPI_INT of total size 4 bytes >> Sending count = 1 of sendtype dup of MPI_INT of total size 4 bytes >> Get new datatypes: send = int-vector, recv = MPI_INT >> Sending count = 1 of sendtype int-vector of total size 4 bytes >> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_send.c at line 435: n_iov > 0 >> internal ABORT - process 0 >> [/stdout] >> >> 22.11.2014, 18:39, "Kuleshov Aleksey" <rndfax(a)yandex.ru>: >>> Hello! Can you please help me with problem? >>> >>> I'm working on custom myriexpress library and I'm using MX netmod in MPICH v.3.1.2. >>> For testing purposes I built OSU Micro Benchmarks v3.8. >>> >>> To run it on 7 nodes I execute test osu_alltoall as follows: >>>> MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 7 /osu_alltoall >>> It passed successfully (I also tried it on 2, 3, 4, 5 and 6 nodes - everything is alright). >>> >>> But now I want to run it on 8 nodes: >>>> MPICH_NEMESIS_NETMOD=mx mpiexec -f /tmp/m -n 8 /osu_alltoall >>> [stdout] >>> # OSU MPI All-to-All Personalized Exchange Latency Test v3.8 >>> # Size Avg Latency(us) >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> Assertion failed in file ../src/mpid/ch3/channels/nemesis/netmod/mx/mx_poll.c at line 784: n_iov > 0 >>> internal ABORT - process 4 >>> internal ABORT - process 7 >>> internal ABORT - process 2 >>> internal ABORT - process 6 >>> internal ABORT - process 3 >>> internal ABORT - process 0 >>> internal ABORT - process 5 >>> internal ABORT - process 1 >>> [/stdout] >>> >>> So, what does these assertions mean? >>> Is it something wrong with MX netmod? >>> Or in myriexpress library? >>> Or in test osu_alltoall itself? >>> >>> BTW, osu_alltoall on 8 nodes passed successfully for TCP netmod. >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0