September 2015 - discuss - mailman3.cels.anl.gov

Possible integer overflows at scale in gather.c
by Ignacio Laguna 30 Sep '15

30 Sep '15

Hi, I found some 'potential' integer overflows when running mpich at large scale and/or with large inputs in gather.c. I believe that they are related to ticket #1767 (http://trac.mpich.org/projects/mpich/ticket/1767) but I didn't see any other bug reports about them so I thought I should confirm with mpich developers. In addition to this case in line 171: 98 int tmp_buf_size, missing; ... 169 if (nbytes < MPIR_CVAR_GATHER_VSMALL_MSG_SIZE) tmp_buf_size++; 170 171 tmp_buf_size *= nbytes; which I believe it's fixed in mpich-3.2b4 (where tmp_buf_size is declared as a 64-bit MPI_Aint), I found the following cases in the same file src/mpi/coll/gather.c (in both mpich-3.1.4 and mpich-3.2b4): Case 1: 222 mpi_errno = MPIC_Recv(((char *)recvbuf + 223 (((rank + mask) % comm_size)*recvcount*extent)), 224 recvblks * recvcount, recvtype, src, 225 MPIR_GATHER_TAG, comm, 226 &status, errflag); In line 223 I believe we get an integer overflow as follows. Suppose I run 2^20 = 1,048,576 ranks and do a gather with 4,096 elements. In this case (if I understand the algorithm well), ((rank + mask) % comm_size) would be 2^20 / 2 = 524,288, and recvcount = 4,096. Then the ((rank + mask) % comm_size)*recvcount expression would overflow: 524,288 * 4,096 = 2,147,483,648, and become negative. When multiplied with 'extent', which is size_t or MPI_Aint, it will become negative I believe or a huge positive which in any case will point to the wrong location in the recvbuf buffer, unless of course this wraparound behavior is intended. Case 2: There might be a similar problem in line 224 in the above code. With 2^20 ranks, recvblks becomes 524,288 (again if I understand well the algorithm), so the recvblks * recvcount operation will also overflow. I might be wrong on this -- I'm catching these issues with LLVM symbolic analysis -- so they can be totally false positives, but I just wanted to check with the mpich developers if they are valid issues or not. If they are, I believe fixes can be easy to implement (just make all these computations size_t). Thanks! -- Ignacio Laguna Center for Applied Scientific Computing (CASC) Lawrence Livermore National Laboratory Phone: 925-422-7308, Fax: 925-422-6287 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Displaying source after attaching to an MPI job
by Peter Thompson 28 Sep '15

28 Sep '15

Understood. People have been using it like this for a while, and it's not earth shattering, so no need to rush it in. It just seems like a nice thing to have for debugger users, and it shouldn't have any harmful side effects that I can think of. Cheers, PeterT On 09/28/2015 09:52 AM, Rob Latham wrote: > > > On 09/22/2015 03:38 PM, Peter Thompson wrote: >> This question came to me in my support role for TotalVIew. When you >> are debugging an MPI job, and attach after the job goes parallel, why >> don't you see the source for the main program? Instead you are usually >> located in the assembler code someplace, in some library above >> MPI_Init, and you have to go down the stack frame to see the source for >> your program. > > We're trying our hardest to get a release out the door, so since this doesn't > quite rise to the level of freeze-breaking worthy, I've stashed this in a > ticket to help us not forget about it: > > http://trac.mpich.org/projects/mpich/ticket/2305 > > ==rob > >> >> There is a solution though. Way back when, when the MPIR_proctable >> interface was designed to help the debugger acquire processes >> automatically, there was a global variable defined MPIR_force_to_main. >> If this is defined in debugger.c as a global variable, then indeed, when >> you start up TotalView. Here's a diff of the change: >> >> *** debugger.c~ 2015-07-22 17:59:51.000000000 -0400 >> --- debugger.c 2015-09-22 16:18:21.000000000 -0400 >> *************** >> *** 11,17 **** >> int MPIR_proctable_size = 0; >> int MPIR_i_am_starter = 0; >> int MPIR_partial_attach_ok = 0; >> ! >> volatile int MPIR_debug_state = 0; >> char *MPIR_debug_abort_string = 0; >> >> --- 11,17 ---- >> int MPIR_proctable_size = 0; >> int MPIR_i_am_starter = 0; >> int MPIR_partial_attach_ok = 0; >> ! int MPIR_force_to_main = 0; >> volatile int MPIR_debug_state = 0; >> char *MPIR_debug_abort_string = 0; >> >> >> The MPI being discussed was Intel MPI, but I've always understood that >> to be based on MPICH, and I thought this would be a good place to >> start. I had the same discussion in the Open MPI group a few years >> ago, and they had dropped the variable at one point since it did not >> appear to be used in any way. The code certainly doesn't make use of >> it, but when the debugger starts looking for it, that's when the magic >> happens. ;-) I tried just defining the variable in debugger.h, but I >> think it got optimized away since I just set up >> >> extern int MPIR_force_to_main; >> >> and didn't assign a value at that point. >> >> Cheers, >> PeterT >> > -- Peter Thompson | Principal Technical Support Engineer | Rogue Wave Software, Inc. Accelerating Great Code | P 508-652-7734 | F 508-652-7701 | www.roguewave.com / peter.thompson(a)roguewave.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Displaying source after attaching to an MPI job
by Rob Latham 28 Sep '15

28 Sep '15

On 09/22/2015 03:38 PM, Peter Thompson wrote: > This question came to me in my support role for TotalVIew. When you > are debugging an MPI job, and attach after the job goes parallel, why > don't you see the source for the main program? Instead you are usually > located in the assembler code someplace, in some library above > MPI_Init, and you have to go down the stack frame to see the source for > your program. We're trying our hardest to get a release out the door, so since this doesn't quite rise to the level of freeze-breaking worthy, I've stashed this in a ticket to help us not forget about it: http://trac.mpich.org/projects/mpich/ticket/2305 ==rob > > There is a solution though. Way back when, when the MPIR_proctable > interface was designed to help the debugger acquire processes > automatically, there was a global variable defined MPIR_force_to_main. > If this is defined in debugger.c as a global variable, then indeed, when > you start up TotalView. Here's a diff of the change: > > *** debugger.c~ 2015-07-22 17:59:51.000000000 -0400 > --- debugger.c 2015-09-22 16:18:21.000000000 -0400 > *************** > *** 11,17 **** > int MPIR_proctable_size = 0; > int MPIR_i_am_starter = 0; > int MPIR_partial_attach_ok = 0; > ! > volatile int MPIR_debug_state = 0; > char *MPIR_debug_abort_string = 0; > > --- 11,17 ---- > int MPIR_proctable_size = 0; > int MPIR_i_am_starter = 0; > int MPIR_partial_attach_ok = 0; > ! int MPIR_force_to_main = 0; > volatile int MPIR_debug_state = 0; > char *MPIR_debug_abort_string = 0; > > > The MPI being discussed was Intel MPI, but I've always understood that > to be based on MPICH, and I thought this would be a good place to > start. I had the same discussion in the Open MPI group a few years > ago, and they had dropped the variable at one point since it did not > appear to be used in any way. The code certainly doesn't make use of > it, but when the debugger starts looking for it, that's when the magic > happens. ;-) I tried just defining the variable in debugger.h, but I > think it got optimized away since I just set up > > extern int MPIR_force_to_main; > > and didn't assign a value at that point. > > Cheers, > PeterT > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Argonne Spam Quarantine Notification
by Argonne Anti Spam Quarantine 28 Sep '15

28 Sep '15

++ Note: This message has been sent by a notification only system. Please do not reply ++ IronPort Spam Quarantine Notification If a spam message was missed, or if a message was incorrectly flagged as spam, you can add the sender to a personal safelist or blocklist via the quarantine. Senders on the safelist will not be flagged as spam, and senders on the blocklist will be automatically quarantined. For more information, visit: https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Documentat… The message(s) below have been quarantined by the Argonne National Laboratory spam filtering system. There are 2 new messages in your Email Quarantine since you received your last Spam Quarantine Notification. If the messages below are spam, you do not need to take any action. Messages will be automatically deleted from the quarantine after 28 day(s). If any of the messages below are not spam, click the 'Release' link to have them sent to your Inbox. Messages that have been correctly flagged as spam do not need to be reported. ----------- New Quarantine Messages --------------- Message 1 From: "Michael Howell"<azlina(a)prubsnadvisor.com.my> Subject: *****HIGH SPAM*****Attention winner Dear Google User, Date: 28 Sep 2015 Release: http://mailgateway.anl.gov:8080/Message?action=Release&mid=95079361&h=29b39… Message 2 From: "Michael Howell"<miguel(a)ubannet.com.br> Subject: *****HIGH SPAM*****Attention winner Dear Google User, Date: 27 Sep 2015 Release: http://mailgateway.anl.gov:8080/Message?action=Release&mid=95064795&h=189ab… --------------------------------------------------- To manage your quarantine please visit the URL below: http://mailgateway.anl.gov:8080/Search?h=21f1498397973e308911f990a5530a1b&e… If you have any questions regarding the Laboratory spam filtering product, please contact the CIS helpdesk at help(a)anl.gov or x-9999 option 2.

1 0

Argonne Spam Quarantine Notification
by Argonne Anti Spam Quarantine 27 Sep '15

27 Sep '15

++ Note: This message has been sent by a notification only system. Please do not reply ++ IronPort Spam Quarantine Notification If a spam message was missed, or if a message was incorrectly flagged as spam, you can add the sender to a personal safelist or blocklist via the quarantine. Senders on the safelist will not be flagged as spam, and senders on the blocklist will be automatically quarantined. For more information, visit: https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Documentat… The message(s) below have been quarantined by the Argonne National Laboratory spam filtering system. There are 1 new messages in your Email Quarantine since you received your last Spam Quarantine Notification. If the messages below are spam, you do not need to take any action. Messages will be automatically deleted from the quarantine after 28 day(s). If any of the messages below are not spam, click the 'Release' link to have them sent to your Inbox. Messages that have been correctly flagged as spam do not need to be reported. ----------- New Quarantine Messages --------------- Message 1 From: "Michael Howell"<miguel(a)ubannet.com.br> Subject: *****HIGH SPAM*****Google Notification. Date: 26 Sep 2015 Release: http://mailgateway.anl.gov:8080/Message?action=Release&mid=95029045&h=ac829… --------------------------------------------------- To manage your quarantine please visit the URL below: http://mailgateway.anl.gov:8080/Search?h=21f1498397973e308911f990a5530a1b&e… If you have any questions regarding the Laboratory spam filtering product, please contact the CIS helpdesk at help(a)anl.gov or x-9999 option 2.

1 0

Argonne Spam Quarantine Notification
by Argonne Anti Spam Quarantine 26 Sep '15

26 Sep '15

++ Note: This message has been sent by a notification only system. Please do not reply ++ IronPort Spam Quarantine Notification If a spam message was missed, or if a message was incorrectly flagged as spam, you can add the sender to a personal safelist or blocklist via the quarantine. Senders on the safelist will not be flagged as spam, and senders on the blocklist will be automatically quarantined. For more information, visit: https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Documentat… The message(s) below have been quarantined by the Argonne National Laboratory spam filtering system. There are 1 new messages in your Email Quarantine since you received your last Spam Quarantine Notification. If the messages below are spam, you do not need to take any action. Messages will be automatically deleted from the quarantine after 28 day(s). If any of the messages below are not spam, click the 'Release' link to have them sent to your Inbox. Messages that have been correctly flagged as spam do not need to be reported. ----------- New Quarantine Messages --------------- Message 1 From: "Michael Howell"<customerservice(a)kratos.com.mx> Subject: *****HIGH SPAM*****Dear Google User, Date: 26 Sep 2015 Release: http://mailgateway.anl.gov:8080/Message?action=Release&mid=95001253&h=6230b… --------------------------------------------------- To manage your quarantine please visit the URL below: http://mailgateway.anl.gov:8080/Search?h=21f1498397973e308911f990a5530a1b&e… If you have any questions regarding the Laboratory spam filtering product, please contact the CIS helpdesk at help(a)anl.gov or x-9999 option 2.

1 0

Displaying source after attaching to an MPI job
by Peter Thompson 22 Sep '15

22 Sep '15

This question came to me in my support role for TotalVIew. When you are debugging an MPI job, and attach after the job goes parallel, why don't you see the source for the main program? Instead you are usually located in the assembler code someplace, in some library above MPI_Init, and you have to go down the stack frame to see the source for your program. There is a solution though. Way back when, when the MPIR_proctable interface was designed to help the debugger acquire processes automatically, there was a global variable defined MPIR_force_to_main. If this is defined in debugger.c as a global variable, then indeed, when you start up TotalView. Here's a diff of the change: *** debugger.c~ 2015-07-22 17:59:51.000000000 -0400 --- debugger.c 2015-09-22 16:18:21.000000000 -0400 *************** *** 11,17 **** int MPIR_proctable_size = 0; int MPIR_i_am_starter = 0; int MPIR_partial_attach_ok = 0; ! volatile int MPIR_debug_state = 0; char *MPIR_debug_abort_string = 0; --- 11,17 ---- int MPIR_proctable_size = 0; int MPIR_i_am_starter = 0; int MPIR_partial_attach_ok = 0; ! int MPIR_force_to_main = 0; volatile int MPIR_debug_state = 0; char *MPIR_debug_abort_string = 0; The MPI being discussed was Intel MPI, but I've always understood that to be based on MPICH, and I thought this would be a good place to start. I had the same discussion in the Open MPI group a few years ago, and they had dropped the variable at one point since it did not appear to be used in any way. The code certainly doesn't make use of it, but when the debugger starts looking for it, that's when the magic happens. ;-) I tried just defining the variable in debugger.h, but I think it got optimized away since I just set up extern int MPIR_force_to_main; and didn't assign a value at that point. Cheers, PeterT -- Peter Thompson | Principal Technical Support Engineer | Rogue Wave Software, Inc. Accelerating Great Code | P 508-652-7734 | F 508-652-7701 | www.roguewave.com / peter.thompson(a)roguewave.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] ROMIO configure looks for lstat in the wrong header
by Rob Latham 21 Sep '15

21 Sep '15

On 09/14/2015 06:09 PM, William Throwe wrote: > Hello, > > I hope this is the correct place to report a minor bug. If not, please > point me in the right direction. thanks for sending this patch. Since ROMIO happily provides its own lstat declaration and continues on its way, I never noticed this error. I committed this change to master: http://git.mpich.org/mpich.git/commit/7b8bd055dfdeb ==rob > > The ROMIO configure script checks for a declaration of lstat in > unistd.h, but, at least on the Linux machines I checked, lstat is in > sys/stat.h. (The detection failure led to a linker error when building > ROMIO as part of OpenMPI on one of my admittedly strangely configured > machines, somehow.) It appears from the man page that either location > is possible, so I suggest checking both. Suggested patch below. > > Thanks, > William Throwe > > diff -Naur a/src/mpi/romio/configure.ac b/src/mpi/romio/configure.ac > --- a/src/mpi/romio/configure.ac 2015-09-14 18:33:06.000000000 -0400 > +++ b/src/mpi/romio/configure.ac 2015-09-14 18:36:06.000000000 -0400 > @@ -1562,7 +1562,8 @@ > AC_CHECK_FUNCS(lstat) > if test "$ac_cv_func_lstat" = "yes" ; then > # Do we need to declare lstat? > - PAC_FUNC_NEEDS_DECL([#include <unistd.h>],lstat) > + PAC_FUNC_NEEDS_DECL([#include <unistd.h> > + #include <sys/stat.h>],lstat) > fi > AC_CHECK_FUNCS(readlink) > if test "$ac_cv_func_readlink" = "yes" ; then > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi client/server : problem with mpich3
by Kenneth Raffenetti 21 Sep '15

21 Sep '15

If you build MPICH 3.1.4 with the option "--with-namepublisher=file", it will use the same method as previous versions. Ken On 09/21/2015 09:36 AM, PEDRONO Annaig wrote: > Hello, > > I am using MPI Client/Server features to couple 2 codes. > You can find a short example in attached file. > In the Makefile, mpif90 is the wrapper for mpich compiled with gfortran. > > With mpich2, I manage to run a client/server case without any problem > with the following script : > > mpirun -np 1 ./Serveur > Serveur.log & > sleep 5 > mpirun -np 1 ./Client > gradym_client.log > > > and I am able to run many coupled simulations on the same computer. > > > With mpich3, the script above doesn't work any more. I have found > another way to run my coupled case in the mpich3 manual : > > #/bin/csh > set hydraPID=`ps -u $USER | grep hydra_nameserve | cut -d ' ' -f 1` > kill -9 $hydraPID > > /PRODCOM/Ubuntu12.04/hydra/3.1.4/intel-15.0/bin/hydra_nameserver & > > > mpiexec -hosts `hostname` -n 1 -nameserver `hostname` ./Serveur & > > sleep 1 > > mpiexec -hosts `hostname` -n 1 -nameserver `hostname` ./Client > > set hydraPID=`ps -u $USER | grep hydra_nameserve | cut -d ' ' -f 1` > echo $hydraPID > kill -9 $hydraPID > > > It is more complicated and I can only run one simulation by computer at > the same time because I can launch only one hydra server. > Do you know another way to run client/server simulation with mpich3? > What can I do to run many client server simulation on the same host? > > Thanks a lot for your help. > > A. Pedrono > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpi client/server : problem with mpich3
by PEDRONO Annaig 21 Sep '15

21 Sep '15

Hello, I am using MPI Client/Server features to couple 2 codes. You can find a short example in attached file. In the Makefile, mpif90 is the wrapper for mpich compiled with gfortran. With mpich2, I manage to run a client/server case without any problem with the following script : mpirun -np 1 ./Serveur > Serveur.log & sleep 5 mpirun -np 1 ./Client > gradym_client.log and I am able to run many coupled simulations on the same computer. With mpich3, the script above doesn't work any more. I have found another way to run my coupled case in the mpich3 manual : #/bin/csh set hydraPID=`ps -u $USER | grep hydra_nameserve | cut -d ' ' -f 1` kill -9 $hydraPID /PRODCOM/Ubuntu12.04/hydra/3.1.4/intel-15.0/bin/hydra_nameserver & mpiexec -hosts `hostname` -n 1 -nameserver `hostname` ./Serveur & sleep 1 mpiexec -hosts `hostname` -n 1 -nameserver `hostname` ./Client set hydraPID=`ps -u $USER | grep hydra_nameserve | cut -d ' ' -f 1` echo $hydraPID kill -9 $hydraPID It is more complicated and I can only run one simulation by computer at the same time because I can launch only one hydra server. Do you know another way to run client/server simulation with mpich3? What can I do to run many client server simulation on the same host? Thanks a lot for your help. A. Pedrono -- -------------------------------------------------------------------------- Annaig PEDRONO Institut de Mecanique des Fluides Tel : (33) (0)5 34 32 29 06 Allee du Professeur Camille SOULA Fax : (33) (0)5 34 32 28 99 31400 TOULOUSE, FRANCE e-mail :[email protected] --------------------------------------------------------------------------- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0