December 2013 - discuss - mailman3.cels.anl.gov

Alternative for -nolocal
by Madhawa Bandara 23 Dec '13

23 Dec '13

Is there an alternative for the -nolocal option when I execute mpiexec? What's the difference between mpirun and mpiexec? -- Regards, *H.K. Madhawa Bandara* *Microsoft Student Ambassador* *IEEE UCSC Student Branch* *Computer Society of University of Colombo* *University of Colombo School of Computing*

1 0

Re: [mpich-discuss] Erroneous MPI type tag warning
by Jeff Hammond 23 Dec '13

23 Dec '13

On Mon, Dec 23, 2013 at 10:55 AM, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: > Hi, > >> I guess the warning has nothing to do with padding. > > Agreed. The warning is due to the type_tag attributes defined in > MPICH's 'mpi.h' header file. > >> For MPI_Reduce(in, out, 1, MPI_2INT, ...), the compiler tries to >> match int, i.e., typeof(*in), with MPI_2INT. Whatever the layout of >> MPI_2INT is defined (e.g., struct {int i1; int i2} or int[2]), the >> compiler fails at matching. If you want to use MPI_2INT, then in, out >> should be defined as of type struct {int v1; int v2}. This is not what the MPI standard says. MPI-3 Section 5.9.4 defines MPI_2INT _as if_ created by the following call: MPI_Type_contiguous(2, MPI_INT, MPI_2INT); Therefore, the resulting type corresponds to "int[2]" not "struct {int i1; int i2}". > Well, according to the C99 standard "Each non-bit-field member of a > structure or union object is aligned in an *implementation-defined* > manner appropriate to its type." (§6.7.2.1) It is therefore not > guaranteed that the two ints in the struct are contiguous -- which > means that, strictly speaking, int[2] is the only portable choice. > (Although I'm not aware of any compiler that would insert padding > between the two ints, i.e., in practice the struct should be fine, > too.) This is interesting, because it seems to suggest that either the MPI standard is wrong in its definition of mixed pair types (e.g. MPI_FLOAT_INT) or at least assumes that the compiler does no padding. To be explicit, the MPI standard (again, MPI-3 Section 5.9.4) says the following: type[0] = MPI_FLOAT type[1] = MPI_INT disp[0] = 0 disp[1] = sizeof(float) block[0] = 1 block[1] = 1 MPI_TYPE_CREATE_STRUCT(2, block, disp, type, MPI_FLOAT_INT) It would seem that disp[1] is wrong in the event that the compiler adds any padding between the float and the int. If it is intended to be absolutely correct, the the definition of MPI_FLOAT_INT demands the user take whatever steps are required to ensure that the compiler does no padding. >> I agree that MPI_Reduce(in, out, 1, MPI_2INT, ...) is absolutely >> right. I think the only reason for the warning is that the compiler is >> not smart enough. > > Compilers could always be smarter ;-) It would be great if you could > come up with a datatype that is -- from the compiler's perspective -- > compatible with both struct {int i1; int i2} and int[2]. Otherwise, > I suggest to remove the type check for MPI_2INT as the warning is > misleading. If compilers were capable of being smart, everyone would just write naive UPC code and let the compiler do all the work instead of bother with MPI :-D >From what you've stated above, this requires a lot of introspection of the compiler. How does one know for sure that the compiler pads the same way every time? What if the user employs a flag that induces more or less padding? There seems to be no reasonable way to address this. Fortunately, the MPI standard is quite clear and the valid type check is for equivalence to "int[2]" and not any manner of struct, and thus the following line of mpi.h.in should be changed accordingly: struct mpich_struct_mpi_2int { int i1; int i2; }; Best, Jeff >> On Sun, Dec 22, 2013 at 3:56 AM, Markus Geimer <m.geimer(a)fz-juelich.de >> <mailto:[email protected]>> wrote: >> >> Dear MPICH developers, >> >> When compiling the attached example with MPICH 3.1rc2 using Clang 3.3, >> I get the following compiler warnings: >> >> -------------------- snip -------------------- >> mpi2int.c:17:20: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~~ ~~~~~~~~ >> mpi2int.c:17:16: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~ ~~~~~~~~ >> 2 warnings generated. >> -------------------- snip -------------------- >> >> According to the MPI standard, however, MPI_2INT is a datatype as if >> defined by >> >> MPI_Type_contiguous(2, MPI_INT, MPI_2INT) >> >> i.e., 'int[2]' should be a perfect match. This is not necessarily true >> for the type used for comparison, >> >> struct mpich_struct_mpi_2int { int i1; int i2; }; >> >> which will only be contiguous if the compiler does not add any padding. >> >> Is there any chance this gets fixed for the final 3.1 release? Or did >> I miss something? >> >> Thanks, >> Markus >> > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Erroneous MPI type tag warning
by Rajeev Thakur 23 Dec '13

23 Dec '13

We will fix it. A ticket has been created. http://trac.mpich.org/projects/mpich/ticket/1993 Rajeev On Dec 23, 2013, at 10:55 AM, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: > Hi, > >> I guess the warning has nothing to do with padding. > > Agreed. The warning is due to the type_tag attributes defined in > MPICH's 'mpi.h' header file. > >> For MPI_Reduce(in, out, 1, MPI_2INT, ...), the compiler tries to >> match int, i.e., typeof(*in), with MPI_2INT. Whatever the layout of >> MPI_2INT is defined (e.g., struct {int i1; int i2} or int[2]), the >> compiler fails at matching. If you want to use MPI_2INT, then in, out >> should be defined as of type struct {int v1; int v2}. > > Well, according to the C99 standard "Each non-bit-field member of a > structure or union object is aligned in an *implementation-defined* > manner appropriate to its type." (§6.7.2.1) It is therefore not > guaranteed that the two ints in the struct are contiguous -- which > means that, strictly speaking, int[2] is the only portable choice. > (Although I'm not aware of any compiler that would insert padding > between the two ints, i.e., in practice the struct should be fine, > too.) > >> I agree that MPI_Reduce(in, out, 1, MPI_2INT, ...) is absolutely >> right. I think the only reason for the warning is that the compiler is >> not smart enough. > > Compilers could always be smarter ;-) It would be great if you could > come up with a datatype that is -- from the compiler's perspective -- > compatible with both struct {int i1; int i2} and int[2]. Otherwise, > I suggest to remove the type check for MPI_2INT as the warning is > misleading. > > Thanks, > Markus > > >> On Sun, Dec 22, 2013 at 3:56 AM, Markus Geimer <m.geimer(a)fz-juelich.de >> <mailto:[email protected]>> wrote: >> >> Dear MPICH developers, >> >> When compiling the attached example with MPICH 3.1rc2 using Clang 3.3, >> I get the following compiler warnings: >> >> -------------------- snip -------------------- >> mpi2int.c:17:20: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~~ ~~~~~~~~ >> mpi2int.c:17:16: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~ ~~~~~~~~ >> 2 warnings generated. >> -------------------- snip -------------------- >> >> According to the MPI standard, however, MPI_2INT is a datatype as if >> defined by >> >> MPI_Type_contiguous(2, MPI_INT, MPI_2INT) >> >> i.e., 'int[2]' should be a perfect match. This is not necessarily true >> for the type used for comparison, >> >> struct mpich_struct_mpi_2int { int i1; int i2; }; >> >> which will only be contiguous if the compiler does not add any padding. >> >> Is there any chance this gets fixed for the final 3.1 release? Or did >> I miss something? >> >> Thanks, >> Markus >> > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Questions about using MPIR_proctable interface in a parallel debugger
by Ashley Pittman 23 Dec '13

23 Dec '13

On 23 Dec 2013, at 14:17, David Wootton <drwootton(a)hvc.rr.com> wrote: > I have a couple questions > 1) Is this the correct way to do this or is there another method I should be using to get this information? It’s a way and it’s not incorrect ;) It’s the most universal method however isn’t as convenient as some other ways. There are often easier ways to get this information directly from any resource manager that you’re using, for example slurm has a “scontrol listpids” command. > 2) I read somewhere that the MPIR_proctable was limited to 64 proceses initially. Is this only an initial size limitation or does MPIR_proctable grow to hold the info all processes regardless of how many processes exist in the application? I’m not familiar with this restriction and I’ve certainly used MPIR_proctable with systems much larger than that. > I also ran into a problem experimenting with this at the MPICH 3.0.4 level. It seems that the pid in all entries of MPIR_proctable is the same for each process and is the pid of the process that is rank zero. All processes are running on the same host that mpirun is also running on. This sounds like a bug in mpich2. Ashley.

1 0

Re: [mpich-discuss] Erroneous MPI type tag warning
by Markus Geimer 23 Dec '13

23 Dec '13

Hi, > I guess the warning has nothing to do with padding. Agreed. The warning is due to the type_tag attributes defined in MPICH's 'mpi.h' header file. > For MPI_Reduce(in, out, 1, MPI_2INT, ...), the compiler tries to > match int, i.e., typeof(*in), with MPI_2INT. Whatever the layout of > MPI_2INT is defined (e.g., struct {int i1; int i2} or int[2]), the > compiler fails at matching. If you want to use MPI_2INT, then in, out > should be defined as of type struct {int v1; int v2}. Well, according to the C99 standard "Each non-bit-field member of a structure or union object is aligned in an *implementation-defined* manner appropriate to its type." (§6.7.2.1) It is therefore not guaranteed that the two ints in the struct are contiguous -- which means that, strictly speaking, int[2] is the only portable choice. (Although I'm not aware of any compiler that would insert padding between the two ints, i.e., in practice the struct should be fine, too.) > I agree that MPI_Reduce(in, out, 1, MPI_2INT, ...) is absolutely > right. I think the only reason for the warning is that the compiler is > not smart enough. Compilers could always be smarter ;-) It would be great if you could come up with a datatype that is -- from the compiler's perspective -- compatible with both struct {int i1; int i2} and int[2]. Otherwise, I suggest to remove the type check for MPI_2INT as the warning is misleading. Thanks, Markus > On Sun, Dec 22, 2013 at 3:56 AM, Markus Geimer <m.geimer(a)fz-juelich.de > <mailto:[email protected]>> wrote: > > Dear MPICH developers, > > When compiling the attached example with MPICH 3.1rc2 using Clang 3.3, > I get the following compiler warnings: > > -------------------- snip -------------------- > mpi2int.c:17:20: warning: argument type 'int *' doesn't match specified > 'MPI' type tag [-Wtype-safety] > MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); > ^~~ ~~~~~~~~ > mpi2int.c:17:16: warning: argument type 'int *' doesn't match specified > 'MPI' type tag [-Wtype-safety] > MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); > ^~ ~~~~~~~~ > 2 warnings generated. > -------------------- snip -------------------- > > According to the MPI standard, however, MPI_2INT is a datatype as if > defined by > > MPI_Type_contiguous(2, MPI_INT, MPI_2INT) > > i.e., 'int[2]' should be a perfect match. This is not necessarily true > for the type used for comparison, > > struct mpich_struct_mpi_2int { int i1; int i2; }; > > which will only be contiguous if the compiler does not add any padding. > > Is there any chance this gets fixed for the final 3.1 release? Or did > I miss something? > > Thanks, > Markus > -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] Questions about using MPIR_proctable interface in a parallel debugger
by Dave Goodell (dgoodell) 23 Dec '13

23 Dec '13

I don't have specific answers to your questions, but you might wish to refer to this document instead of the link you mention: http://www.mpi-forum.org/docs/mpir-specification-10-11-2010.pdf Also, depending on what you are trying to accomplish, several parallel debuggers already exist for MPI (including MPICH). For certain kinds of debugging, I've found the freely available padb debugger to be extremely helpful: http://padb.pittman.org.uk/ ANL folks: please consider updating that page at least mention the MPI Forum's document. -Dave On Dec 23, 2013, at 8:17 AM, David Wootton <drwootton(a)hvc.rr.com> wrote: > I'm trying to get a parallel debugger to support debugging MPICH programs. This involves getting the list of application process pids and the hostname or IP address of the host the process is running on so the debugger can attach to those processes. > > I found a web page at http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/ that discusses using MPIR_proctable and MPIR_proctable_size to get this information from the mpirun or mpiexec process. > > I have a couple questions > 1) Is this the correct way to do this or is there another method I should be using to get this information? > 2) I read somewhere that the MPIR_proctable was limited to 64 proceses initially. Is this only an initial size limitation or does MPIR_proctable grow to hold the info all processes regardless of how many processes exist in the application? > > I also ran into a problem experimenting with this at the MPICH 3.0.4 level. It seems that the pid in all entries of MPIR_proctable is the same for each process and is the pid of the process that is rank zero. All processes are running on the same host that mpirun is also running on. > > I'm running the program as 'mpirun -n 5 ./shallow' > > Thanks > > Dave > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Questions about using MPIR_proctable interface in a parallel debugger
by David Wootton 23 Dec '13

23 Dec '13

I'm trying to get a parallel debugger to support debugging MPICH programs. This involves getting the list of application process pids and the hostname or IP address of the host the process is running on so the debugger can attach to those processes. I found a web page at http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/ that discusses using MPIR_proctable and MPIR_proctable_size to get this information from the mpirun or mpiexec process. I have a couple questions 1) Is this the correct way to do this or is there another method I should be using to get this information? 2) I read somewhere that the MPIR_proctable was limited to 64 proceses initially. Is this only an initial size limitation or does MPIR_proctable grow to hold the info all processes regardless of how many processes exist in the application? I also ran into a problem experimenting with this at the MPICH 3.0.4 level. It seems that the pid in all entries of MPIR_proctable is the same for each process and is the pid of the process that is rank zero. All processes are running on the same host that mpirun is also running on. I'm running the program as 'mpirun -n 5 ./shallow' Thanks Dave

1 0

MPI_Send/MPI_Recv - getting delayed indeffinitely
by Madhawa Bandara 23 Dec '13

23 Dec '13

Hi, I use mpich2 on a small cluster of 3 nodes and each node has Ubuntu 12.04 installed. I use this cluster to do the following. 1. *Master *node send some matrices to the 2 *workers* 2. Workers perform some calculations and send the resulted matrices back to the master. 3. Master perform some final calculations. code snippet: //master(taskid=0) MPI_Send(ha11, n / 2 * n / 2, MPI_DOUBLE, 1, 1, MPI_COMM_WORLD); //to worker 1 MPI_Send(ha11, n / 2 * n / 2, MPI_DOUBLE, 2, 1, MPI_COMM_WORLD); //to worker 2 MPI_Recv(hM1, n / 2 * n / 2, MPI_DOUBLE, 1, 2, MPI_COMM_WORLD,&status); //from worker 1 MPI_Recv(hM2, n / 2 * n / 2, MPI_DOUBLE, 2, 2, MPI_COMM_WORLD,&status);//from worker 2 //final calculations using hM1,hM2 //worker 1 (taskid=1) MPI_Recv(ha11, n / 2 * n / 2, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD,&status); //does some calculations MPI_Send(hM1, n / 2 * n / 2, MPI_DOUBLE, 0, 2, MPI_COMM_WORLD); //sends back //worker 2(taskid=2) MPI_Recv(ha11, n / 2 * n / 2, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD,&status); //does some calculations MPI_Send(hM2, n / 2 * n / 2, MPI_DOUBLE, 0, 2, MPI_COMM_WORLD); //sends back This worked fine at first, for n=128 to n=2048. But after I pushed 'n' beyond 2048 I got a segmentation fault from the worker 1. Since then, code works fine for the small n values. But whenever I set the value n=128 or greater, worker 1 is getting delayed indefinitely while the rest of the nodes works fine. What could be the reason for this? And how can I resolve this? If I have done any mistakes please point out. Thanks in advance. -- Regards, *H.K. Madhawa Bandara*

1 0

Re: [mpich-discuss] MPI memory allocation.
by Anatoly G 23 Dec '13

23 Dec '13

Hi. Do you need more info to reproduce problem? Anatoly. On Sun, Dec 22, 2013 at 7:54 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Good morning Pavan. > Do you see process memory raise & network throughput low behavior? > > Anatoly. > > > On Thu, Dec 19, 2013 at 7:09 PM, Anatoly G <anatolyrishon(a)gmail.com>wrote: > >> No. You can see memory increase using monitor like "qps", or any other. >> The stack trace that I send I get when I stop my real program (not >> simulation) with TotalView debugger. >> if you monitor network traffic, you will see network speed which becomes >> low until it reaches almost 0. >> >> Anatoly. >> >> >> On Thu, Dec 19, 2013 at 5:16 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >>> >>> Are you printing it out? >>> >>> — Pavan >>> >>> On Dec 19, 2013, at 10:51 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >>> >>> > Can you please remove comment from section >>> > /* >>> > // swap tag & enter blocked recv >>> > MPI_Status stat; >>> > tags[slaveIdx] = (tags[slaveIdx] == TAG1) ? TAG2 : TAG1; >>> > MPI_Recv(RcvBufs[slaveIdx], BUF_SZ, MPI::CHAR, slaveRank, >>> tags[slaveIdx], MPI_COMM_WORLD, &stat); >>> > >>> > ++SlavesRcvIters[slaveIdx]; >>> > */ >>> > >>> > And then run it. >>> > Do you see memory allocation increase? >>> > >>> > Regards, >>> > Anatoly. >>> > >>> > >>> > On Thu, Dec 19, 2013 at 4:29 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >>> wrote: >>> > >>> > I’m not sure what I should look for. I ran the program and it >>> completed fine. >>> > >>> > — Pavan >>> > >>> > On Dec 19, 2013, at 7:16 PM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > >>> > > Good afternoon. >>> > > My program enters a lot to attached stack functions. >>> > > Can you please explain if it's ok? >>> > > Did you success to execute simulation from previous mail. >>> > > Did you see the memory raise when MPI_Recv not in comments? >>> > > >>> > > Regards, >>> > > Anatoly. >>> > > >>> > > >>> > > On Thu, Dec 12, 2013 at 9:36 PM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > Hi. >>> > > Finally, I got an additional info. >>> > > I build short simulation of my real application. >>> > > >>> > > The short description of real scenario. >>> > > I have Master + N slaves. Each slave sends to Master 2 types of >>> messages: >>> > > • constant length message with predefined fields (one of it's >>> fields is length of second message). >>> > > • second message - length of this message each time is >>> different and passed in first message. >>> > > Master should use MPI_Irecv commands, in order to be tolerant to >>> slaves failure (blocked MPI_Recv is blocking Master in the failure case). >>> > > Master executes MPI_Irecv to each slave with buffer size equal to >>> the constant size of first message type. After receiving first type of >>> message, Master allocates expected buffer for second message and performs >>> receive too. This happens in endless loop for each slaves. I use >>> MPI_Waitany to monitor all receives. >>> > > In order to separate messages Master & slaves use different tags (as >>> ids) for first & second messages. >>> > > >>> > > Simulation description: >>> > > All passed buffers (first & second) have the same size. >>> > > Slave (SndSyncSlave) sends messages and swaps 2 tags between them >>> (like 2 types of messages, but second one has constant size too). >>> > > Master routine (Rcv_WaitAny function) executes MPI_Irecv for first >>> message, and after receive executes MPI_Irecv for the second one. >>> > > >>> > > In this scenario: 5 processes works fine, but if I execute 20 >>> processes and remove comment from line "usleep(200000)" I see 800 Mbit/s >>> on network at the test beginning, but after 1-2 second network speed become >>> 200-300Kbit/s and not increased back any more. >>> > > >>> > > If I add MPI_Recv block in Master (remove comment from "MPI_Recv" >>> and line around) I see that Master starts increase memory as my real >>> application, but again on 5 processes this not happens. This is scenario >>> used in my real application. >>> > > >>> > > Command line: mpiexec.hydra -genvall -f MpiConfigMachines.txt >>> -launcher=ssh -n 20 mpi_rcv_any_multithread 100000 1000000 out >>> > > >>> > > where >>> > > 100000 - number of sends from each slave >>> > > 1000000 - scale to separate input from each scale (used for debug >>> only) >>> > > out - prefix of output file. Each process produce out_"rank".txt >>> file. >>> > > >>> > > MpiConfigMachines.txt - configuration file for my computers, 2 >>> computers back to back 1 Gbit/s network. >>> > > >>> > > >>> > > Can you please test this case, and give me yours suggestions. >>> > > >>> > > Thank you, >>> > > Anatoly. >>> > > >>> > > >>> > > >>> > > On Mon, Dec 9, 2013 at 9:55 PM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > Yes, I understand that. I"ll try to make my stand alone test closer >>> to real application. Thank you. >>> > > >>> > > >>> > > On Mon, Dec 9, 2013 at 9:31 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >>> wrote: >>> > > >>> > > It sounds like MPICH is working correctly. Without a test case, >>> it’s unfortunately quite hard for us to even know what to look for. It’s >>> also possible that there’s a bug in your code which might be causing some >>> bad behavior. >>> > > >>> > > — Pavan >>> > > >>> > > On Dec 9, 2013, at 1:27 PM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > >>> > > > Yes, I"m actually need Fault tolerance, and it was the main reason >>> for choosing MPICH2. I use fault tolerance for unpredictable bugs in the >>> future. My system should survive partially. But in the regular case I just >>> need full performance. I"m suspect that I don't use MPI correctly, but on >>> slow rate everything works fine. The fail caused by increasing rate of >>> MPI_Isend or increasing data buffer size. I didn't find yet any strong >>> dependence, only main stream. >>> > > > >>> > > > Unfortunately I have a complex system which has a number of >>> threads in each process. Part of the threads use different communicators. >>> > > > >>> > > > I try to simulate the same MPI behavior in simple stand alone >>> test, but stand alone test works fine. It shows a full network performance, >>> when I slow down master (on stand alone test), all slaves are stopped too >>> and are waiting for master to continue. Can I open any MPICH log to send >>> you the results? >>> > > > >>> > > > >>> > > > On Mon, Dec 9, 2013 at 8:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >>> wrote: >>> > > > >>> > > > Do you actually need Fault Tolerance (one of your previous emails >>> seemed to indicate that)? >>> > > > >>> > > > It sounds like there a bug in either your application or in the >>> MPICH stack and you are trying to trace that down, and don’t really care >>> about fault tolerance. Is that a correct assessment? >>> > > > >>> > > > Do you have a simplified program that reproduces this error, that >>> we can try? >>> > > > >>> > > > — Pavan >>> > > > >>> > > > On Dec 9, 2013, at 11:44 AM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > > >>> > > > > No. Hardware is Ok. Master process allocates memory (check with >>> MemoryScape doesn't show any sufficient memory allocation in my code). Then >>> network traffic become low, and then Master process crashes w/o saving core >>> file. I have unlimited size of core files. The same fail (w/o core) I see >>> when I call MPI_Abort, but I don't call it. >>> > > > > >>> > > > > >>> > > > > On Mon, Dec 9, 2013 at 7:28 PM, Wesley Bland <wbland(a)mcs.anl.gov> >>> wrote: >>> > > > > Are you actually seeing hardware failure or is your code just >>> crashing? It's odd that one specific process would fail so often in the >>> same way if it were a hardware problem. >>> > > > > >>> > > > > Thanks, >>> > > > > Wesley >>> > > > > >>> > > > > On Dec 9, 2013, at 11:15 AM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > > > >>> > > > >> One more interesting fact. Each time I have a failure, the >>> fails only master process, but slaves are still exists together with >>> mpiexec.hydra. I thought that slaves should fail too, but slaves are live. >>> > > > >> >>> > > > >> >>> > > > >> On Mon, Dec 9, 2013 at 10:30 AM, Anatoly G < >>> anatolyrishon(a)gmail.com> wrote: >>> > > > >> I configure MPICH-3.1rc2 build w/o "so" files. But instead of >>> MPICH2 & MPICH-3.0.4 I get so files. What should I change in configure line >>> to link MPI with my application statically. >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> On Mon, Dec 9, 2013 at 9:47 AM, Pavan Balaji < >>> balaji(a)mcs.anl.gov> wrote: >>> > > > >> >>> > > > >> Can you try mpich-3.1rc2? There were several fixes for this in >>> this version and it’ll be good to try that out before we go digging too far >>> into this. >>> > > > >> >>> > > > >> — Pavan >>> > > > >> >>> > > > >> On Dec 9, 2013, at 1:46 AM, Anatoly G <anatolyrishon(a)gmail.com> >>> wrote: >>> > > > >> >>> > > > >> > With MPICH - 3.0.4 the situation repeated. It looks like MPI >>> allocates memory for messages. >>> > > > >> > Can you please advice about scenario when MPI or may be TCP >>> under MPI allocates memory due to high transfer rate? >>> > > > >> > >>> > > > >> > >>> > > > >> > On Mon, Dec 9, 2013 at 9:32 AM, Anatoly G < >>> anatolyrishon(a)gmail.com> wrote: >>> > > > >> > Thank you very much. >>> > > > >> > Issend - is not so good, It can't support me Fault tolerance. >>> If slave process fails, the master stall. >>> > > > >> > I tried mpich-3.0.4 with hydra-3.0.4 but my program which >>> uses MPI Fault tolerance doesn't recognize failure of slave process, but >>> recognizes failure with MPICH2. May be you can suggest solution? >>> > > > >> > I tried to use hydra from MPICH2 but link my program with >>> MPICH3. This combination recognizes failures, but I"m not sure that such >>> combination is stable enough. >>> > > > >> > Can you please advice? >>> > > > >> > Anatoly. >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > On Sat, Dec 7, 2013 at 5:20 PM, Pavan Balaji < >>> balaji(a)mcs.anl.gov> wrote: >>> > > > >> > >>> > > > >> > As much as I hate saying this — some people find it easier to >>> think of it as “MPICH3”. >>> > > > >> > >>> > > > >> > — Pavan >>> > > > >> > >>> > > > >> > On Dec 7, 2013, at 7:37 AM, Wesley Bland <wbland(a)mcs.anl.gov> >>> wrote: >>> > > > >> > >>> > > > >> > > MPICH is just the new version of MPICH2. We renamed it when >>> we went past version 3.0. >>> > > > >> > > >>> > > > >> > > On Dec 7, 2013, at 3:55 AM, Anatoly G < >>> anatolyrishon(a)gmail.com> wrote: >>> > > > >> > > >>> > > > >> > >> Ok. I"ll try both Issend, and next step to upgrade MPICH >>> to 3.0.4. >>> > > > >> > >> I thought before that MPICH & MPICH2 are two different >>> branches, when MPICH2 partially supports Fault tolerance, but MPICH not. >>> Now I understand, that I was wrong and MPICH2 is just main version of MPICH. >>> > > > >> > >> >>> > > > >> > >> Thank you very much, >>> > > > >> > >> Anatoly. >>> > > > >> > >> >>> > > > >> > >> >>> > > > >> > >> >>> > > > >> > >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur < >>> thakur(a)mcs.anl.gov> wrote: >>> > > > >> > >> The master is receiving too many incoming messages than it >>> can match quickly enough with Irecvs. Try using MPI_Issend instead of >>> MPI_Isend. >>> > > > >> > >> >>> > > > >> > >> Rajeev >>> > > > >> > >> >>> > > > >> > >> On Dec 5, 2013, at 2:58 AM, Anatoly G < >>> anatolyrishon(a)gmail.com> wrote: >>> > > > >> > >> >>> > > > >> > >> > Hello. >>> > > > >> > >> > I"m using MPICH2 1.5. >>> > > > >> > >> > My system contains master and 16 slaves. >>> > > > >> > >> > System uses number of communicators. >>> > > > >> > >> > The single communicator used for below scenario: >>> > > > >> > >> > Each slave sends non-stop 2Kbyte data buffer using >>> MPI_Isend and waits using MPI_Wait. >>> > > > >> > >> > Master starts with MPI_Irecv to each slave >>> > > > >> > >> > Then in endless loop: >>> > > > >> > >> > MPI_Waitany and MPI_Irecv on rank returned by >>> MPI_Waitany. >>> > > > >> > >> > >>> > > > >> > >> > Another communicator used for broadcast communication >>> (commands between master + slaves), >>> > > > >> > >> > but it's not used in parallel with previous communicator, >>> > > > >> > >> > only before or after data transfer. >>> > > > >> > >> > >>> > > > >> > >> > The system executed on two computers linked by 1Gbit/s >>> Ethernet. >>> > > > >> > >> > Master executed on first computer, all slaves on other >>> one. >>> > > > >> > >> > Network traffic is ~800Mbit/s. >>> > > > >> > >> > >>> > > > >> > >> > After 1-2 minutes of execution, master process starts to >>> increase it's memory allocation and network traffic becomes low. >>> > > > >> > >> > This memory allocation & network traffic slow down >>> continues until fail of MPI, >>> > > > >> > >> > without core file save. >>> > > > >> > >> > My program doesn't allocate memory. Can you please >>> explain this behaviour. >>> > > > >> > >> > How can I cause MPI to stop sending slaves if Master >>> can't serve such traffic, instead of memory allocation and fail? >>> > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > Thank you, >>> > > > >> > >> > Anatoly. >>> > > > >> > >> > >>> > > > >> > >> > P.S. >>> > > > >> > >> > On my stand alone test, I simulate similar behaviour, >>> but with single thread on each process (master & hosts). >>> > > > >> > >> > When I start stand alone test, master stops slaves until >>> it completes accumulated data processing and MPI doesn't increase memory >>> allocation. >>> > > > >> > >> > When Master is free slaves continue to send data. >>> > > > >> > >> > _______________________________________________ >>> > > > >> > >> > discuss mailing list discuss(a)mpich.org >>> > > > >> > >> > To manage subscription options or unsubscribe: >>> > > > >> > >> > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> > >> >>> > > > >> > >> _______________________________________________ >>> > > > >> > >> discuss mailing list discuss(a)mpich.org >>> > > > >> > >> To manage subscription options or unsubscribe: >>> > > > >> > >> https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> > >> >>> > > > >> > >> _______________________________________________ >>> > > > >> > >> discuss mailing list discuss(a)mpich.org >>> > > > >> > >> To manage subscription options or unsubscribe: >>> > > > >> > >> https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> > > _______________________________________________ >>> > > > >> > > discuss mailing list discuss(a)mpich.org >>> > > > >> > > To manage subscription options or unsubscribe: >>> > > > >> > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> > >>> > > > >> > -- >>> > > > >> > Pavan Balaji >>> > > > >> > http://www.mcs.anl.gov/~balaji >>> > > > >> > >>> > > > >> > _______________________________________________ >>> > > > >> > discuss mailing list discuss(a)mpich.org >>> > > > >> > To manage subscription options or unsubscribe: >>> > > > >> > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> > >>> > > > >> > >>> > > > >> > _______________________________________________ >>> > > > >> > discuss mailing list discuss(a)mpich.org >>> > > > >> > To manage subscription options or unsubscribe: >>> > > > >> > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> >>> > > > >> -- >>> > > > >> Pavan Balaji >>> > > > >> http://www.mcs.anl.gov/~balaji >>> > > > >> >>> > > > >> _______________________________________________ >>> > > > >> discuss mailing list discuss(a)mpich.org >>> > > > >> To manage subscription options or unsubscribe: >>> > > > >> https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >> >>> > > > >> >>> > > > >> _______________________________________________ >>> > > > >> discuss mailing list discuss(a)mpich.org >>> > > > >> To manage subscription options or unsubscribe: >>> > > > >> https://lists.mpich.org/mailman/listinfo/discuss >>> > > > > >>> > > > > _______________________________________________ >>> > > > > discuss mailing list discuss(a)mpich.org >>> > > > > To manage subscription options or unsubscribe: >>> > > > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > > >>> > > > > _______________________________________________ >>> > > > > discuss mailing list discuss(a)mpich.org >>> > > > > To manage subscription options or unsubscribe: >>> > > > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >>> > > > -- >>> > > > Pavan Balaji >>> > > > http://www.mcs.anl.gov/~balaji >>> > > > >>> > > > _______________________________________________ >>> > > > discuss mailing list discuss(a)mpich.org >>> > > > To manage subscription options or unsubscribe: >>> > > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > > >>> > > > _______________________________________________ >>> > > > discuss mailing list discuss(a)mpich.org >>> > > > To manage subscription options or unsubscribe: >>> > > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > >>> > > -- >>> > > Pavan Balaji >>> > > http://www.mcs.anl.gov/~balaji >>> > > >>> > > _______________________________________________ >>> > > discuss mailing list discuss(a)mpich.org >>> > > To manage subscription options or unsubscribe: >>> > > https://lists.mpich.org/mailman/listinfo/discuss >>> > > >>> > > >>> > > >>> > > <backtrace2.txt>_______________________________________________ >>> > > discuss mailing list discuss(a)mpich.org >>> > > To manage subscription options or unsubscribe: >>> > > https://lists.mpich.org/mailman/listinfo/discuss >>> > >>> > -- >>> > Pavan Balaji >>> > http://www.mcs.anl.gov/~balaji >>> > >>> > _______________________________________________ >>> > discuss mailing list discuss(a)mpich.org >>> > To manage subscription options or unsubscribe: >>> > https://lists.mpich.org/mailman/listinfo/discuss >>> > >>> > _______________________________________________ >>> > discuss mailing list discuss(a)mpich.org >>> > To manage subscription options or unsubscribe: >>> > https://lists.mpich.org/mailman/listinfo/discuss >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >

1 0

Re: [mpich-discuss] Erroneous MPI type tag warning
by Junchao Zhang 22 Dec '13

22 Dec '13

Oops, MPI_MAXLOC requires MPI_DOUBLE_INT and the like. The MPI_Reduce(in, out, 2, INT, MPI_MAXLOC, ...) approach is incorrect. --Junchao Zhang On Sun, Dec 22, 2013 at 10:58 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > Hi, Markus, > I think you should change MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, > 0, MPI_COMM_WORLD) to MPI_Reduce(in, out, 2, INT, MPI_MAXLOC, 0, > MPI_COMM_WORLD). > I guess the warning has nothing to do with padding. For MPI_Reduce(in, > out, 1, MPI_2INT, ...), the compiler tries to match int, i.e., typeof(*in), > with MPI_2INT. Whatever the layout of MPI_2INT is defined (e.g., struct > {int i1; int i2} or int[2]), the compiler fails at matching. If you want to > use MPI_2INT, then in, out should be defined as of type struct {int v1; > int v2}. > > I agree that MPI_Reduce(in, out, 1, MPI_2INT, ...) is absolutely right. > I think the only reason for the warning is that the compiler is not smart > enough. For example, when matching a pointer P with a type T, the compiler > should try to match typeof(*P) ~ T, typeof(*P)[2] ~T, typeof(*P)[3] ~ T, > etc. > > > --Junchao Zhang > > > On Sun, Dec 22, 2013 at 3:56 AM, Markus Geimer <m.geimer(a)fz-juelich.de>wrote: > >> Dear MPICH developers, >> >> When compiling the attached example with MPICH 3.1rc2 using Clang 3.3, >> I get the following compiler warnings: >> >> -------------------- snip -------------------- >> mpi2int.c:17:20: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~~ ~~~~~~~~ >> mpi2int.c:17:16: warning: argument type 'int *' doesn't match specified >> 'MPI' type tag [-Wtype-safety] >> MPI_Reduce(in, out, 1, MPI_2INT, MPI_MAXLOC, 0, MPI_COMM_WORLD); >> ^~ ~~~~~~~~ >> 2 warnings generated. >> -------------------- snip -------------------- >> >> According to the MPI standard, however, MPI_2INT is a datatype as if >> defined by >> >> MPI_Type_contiguous(2, MPI_INT, MPI_2INT) >> >> i.e., 'int[2]' should be a perfect match. This is not necessarily true >> for the type used for comparison, >> >> struct mpich_struct_mpi_2int { int i1; int i2; }; >> >> which will only be contiguous if the compiler does not add any padding. >> >> Is there any chance this gets fixed for the final 3.1 release? Or did >> I miss something? >> >> Thanks, >> Markus >> >> -- >> Dr. Markus Geimer >> Juelich Supercomputing Centre >> Institute for Advanced Simulation >> Forschungszentrum Juelich GmbH >> 52425 Juelich, Germany >> >> Phone: +49-2461-61-1773 >> Fax: +49-2461-61-6656 >> E-mail: m.geimer(a)fz-juelich.de >> WWW: http://www.fz-juelich.de/jsc/ >> >> >> >> ------------------------------------------------------------------------------------------------ >> >> ------------------------------------------------------------------------------------------------ >> Forschungszentrum Juelich GmbH >> 52425 Juelich >> Sitz der Gesellschaft: Juelich >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >> Prof. Dr. Sebastian M. Schmidt >> >> ------------------------------------------------------------------------------------------------ >> >> ------------------------------------------------------------------------------------------------ >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > >

1 0