June 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Help troubleshooting a session
by "Antonio J. Peña" 30 Jun '14

30 Jun '14

Hi Mark, That's not enough information for us to determine if your code is doing something inappropriate or if you're hitting a bug in MPICH. Usually we request reporters to submit the smallest piece of code with which that they can reproduce the problem. From your description, though, I'd say you could have something wrong on your code, so I'd suggest first asking for assistance in some place like stackoverflow to discard this before you file a possible bug to us. Thanks, Antonio On 06/30/2014 06:56 AM, mark wrote: > Στις 30/06/2014 01:09 μμ, ο/η mark έγραψε: >> Hello. >> I wrote a program in MPI and for some combinations of data and number >> of nodes it runs ok, but other times it fails to run and returns >> errors (i.e. rank 0 in job 29 Calliope_50394 caused collective abort >> of all ranks exit status of rank 0: killed by signal 9). How i see it, >> for some reason, the division of data into the nodes keeps failing the >> execution. Because if i use 16 nodes and use for example a 320-line >> array as data collection, it executes fine. But if i use a 50-line >> array it fails the execution. Is there a way to troubleshoot the code >> and find where it fails? Also some examples would be great. >> >> >> PS. Because i have wrote other programs in MPI that worked, the only >> difference they have with this one, that keeps failing, is that in >> this one i use loops like this: >> >> for(i=id*(n/p); i< (id+1)*n/p; i++){..... >> >> to parse the data accordingly, but again for some combinations of data >> and number of nodes it worked... > Sorry for double posting. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

Re: [mpich-discuss] Help troubleshooting a session
by mark 30 Jun '14

30 Jun '14

Στις 30/06/2014 01:09 μμ, ο/η mark έγραψε: > Hello. > I wrote a program in MPI and for some combinations of data and number > of nodes it runs ok, but other times it fails to run and returns > errors (i.e. rank 0 in job 29 Calliope_50394 caused collective abort > of all ranks exit status of rank 0: killed by signal 9). How i see it, > for some reason, the division of data into the nodes keeps failing the > execution. Because if i use 16 nodes and use for example a 320-line > array as data collection, it executes fine. But if i use a 50-line > array it fails the execution. Is there a way to troubleshoot the code > and find where it fails? Also some examples would be great. > > > PS. Because i have wrote other programs in MPI that worked, the only > difference they have with this one, that keeps failing, is that in > this one i use loops like this: > > for(i=id*(n/p); i< (id+1)*n/p; i++){..... > > to parse the data accordingly, but again for some combinations of data > and number of nodes it worked... Sorry for double posting.

1 0

Need help troubleshooting
by mark 30 Jun '14

30 Jun '14

Hello. I need some help troubleshooting a program i wrote. For a combination of data and nodes the program runs fine, but for others not. For example i use 2d arrays for data collections and divide them into the nodes. If the number of the lines in the array are 320 and the nodes are 16 ( 8 physical nodes with multi-threading) the program runs fine. But if the lines in the array are 50 and the nodes 16 the program fails, but again if the nodes are 2 or 4 the program runs ok. Is there a way to define the exact spot where the code is failing? Also, some examples would do wonders. Thanks Ps-1: The errors that the program returns are in the form of: rank 25 in job 29 Calliope_50394 caused collective abort of all ranks - exit status of rank 25: killed by signal 11 Ps-2: I wrote other programs in MPI that worked, and the only difference is, that in this program i use loops like these: for(i=id*n/p; i< (id+1)*n/p; i++){..... (where id are the id's of the nodes, n is the data collection and p are the number of nodes) to parse the data accordingly.

1 0

Help troubleshooting a session
by mark 30 Jun '14

30 Jun '14

Hello. I wrote a program in MPI and for some combinations of data and number of nodes it runs ok, but other times it fails to run and returns errors (i.e. rank 0 in job 29 Calliope_50394 caused collective abort of all ranks exit status of rank 0: killed by signal 9). How i see it, for some reason, the division of data into the nodes keeps failing the execution. Because if i use 16 nodes and use for example a 320-line array as data collection, it executes fine. But if i use a 50-line array it fails the execution. Is there a way to troubleshoot the code and find where it fails? Also some examples would be great. PS. Because i have wrote other programs in MPI that worked, the only difference they have with this one, that keeps failing, is that in this one i use loops like this: for(i=id*(n/p); i< (id+1)*n/p; i++){..... to parse the data accordingly, but again for some combinations of data and number of nodes it worked...

1 0

Re: [mpich-discuss] MPICH 3.1.1 and MPI_*_c2f
by Balaji, Pavan 29 Jun '14

29 Jun '14

On Jun 28, 2014, at 3:31 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > p19 of MPI-3.0 says those _c2f / _f2c routines can be implemented in macros. But we don’t, in MPICH. So that’s irrelevant here. > Also, MPI Standard only defines them in C. Their Fortran versions are a non-standard extension. I’m only referring to the C symbols. We are just discussing which library those symbols need to go into. libmpi seems to be the right choice. — Pavan

1 0

Re: [mpich-discuss] MPICH 3.1.1 and MPI_*_c2f
by Junchao Zhang 28 Jun '14

28 Jun '14

p19 of MPI-3.0 says those _c2f / _f2c routines can be implemented in macros. Also, MPI Standard only defines them in C. Their Fortran versions are a non-standard extension. --Junchao Zhang On Sat, Jun 28, 2014 at 11:43 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Chris, > > Thanks for the report. What you suggested below seems reasonable. > > I was the one who made the change when I was cleaning up the separation of > symbols into the appropriate libraries. The intention was that the f2c > symbols go into libmpifort, while the c2f symbols go into libmpi. But > maybe there is a case for f2c symbols to show up in the C library as well > (if I got an object from a Fortran module and want to convert it to C). > > In any case, it sounds like the execution didn’t match the intention. I > remember I had some issue with the c2f/f2c symbols, but I don’t remember > what that was exactly. Let me go back and see what the reason for that was. > > I think the cleanest solution is to put both c2f and f2c in libmpi, since > libmpifort has a dependency on libmpi anyway. > > I’ve created a ticket so you can keep track of the progress on this: > > https://trac.mpich.org/projects/mpich/ticket/2122 > > Thanks, > > — Pavan > > On Jun 26, 2014, at 5:00 AM, Chris January <chris.january(a)allinea.com> > wrote: > > > Hello all, > > > > I noticed that MPICH 3.1.1 changed which libraries symbols go in. The > > MPI_*_c2f functions are no longer available to source files compiled > > with mpicc and mpicxx (unless you explicitly add -lmpifort). > > > > In our opinion the MPI_*_c2f and MPI_* symbols should be available from > > all compiler drivers (mpicc, mpicxx, mpif77, mpif90, etc.) without the > > need to explicitly add -lmpifort. Since they are used to convert values > > between C and Fortran and vice versa it is natural that one would want > > to use them from both languages. > > > > Existing code which assumes, I believe quite reasonably, that MPI_*_c2f, > > etc. can be used from mpicc with no extra effort fails to compile with > > MPICH 3.1.1. I think this is a regression which should be fixed. > > > > Yours, > > Chris January - VP Engineering - Allinea Software Ltd. > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH 3.1.1 and MPI_*_c2f
by Balaji, Pavan 28 Jun '14

28 Jun '14

Chris, Thanks for the report. What you suggested below seems reasonable. I was the one who made the change when I was cleaning up the separation of symbols into the appropriate libraries. The intention was that the f2c symbols go into libmpifort, while the c2f symbols go into libmpi. But maybe there is a case for f2c symbols to show up in the C library as well (if I got an object from a Fortran module and want to convert it to C). In any case, it sounds like the execution didn’t match the intention. I remember I had some issue with the c2f/f2c symbols, but I don’t remember what that was exactly. Let me go back and see what the reason for that was. I think the cleanest solution is to put both c2f and f2c in libmpi, since libmpifort has a dependency on libmpi anyway. I’ve created a ticket so you can keep track of the progress on this: https://trac.mpich.org/projects/mpich/ticket/2122 Thanks, — Pavan On Jun 26, 2014, at 5:00 AM, Chris January <chris.january(a)allinea.com> wrote: > Hello all, > > I noticed that MPICH 3.1.1 changed which libraries symbols go in. The > MPI_*_c2f functions are no longer available to source files compiled > with mpicc and mpicxx (unless you explicitly add -lmpifort). > > In our opinion the MPI_*_c2f and MPI_* symbols should be available from > all compiler drivers (mpicc, mpicxx, mpif77, mpif90, etc.) without the > need to explicitly add -lmpifort. Since they are used to convert values > between C and Fortran and vice versa it is natural that one would want > to use them from both languages. > > Existing code which assumes, I believe quite reasonably, that MPI_*_c2f, > etc. can be used from mpicc with no extra effort fails to compile with > MPICH 3.1.1. I think this is a regression which should be fixed. > > Yours, > Chris January - VP Engineering - Allinea Software Ltd. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q
by Michael Blocksome 26 Jun '14

26 Jun '14

Halim, This problem sounds similar to another issue we are debugging related to cancel and multiple endpoints in per-object locking mode. I'll try a few things and post status. Thanks, Michael Blocksome Parallel Environment MPI Middleware Team Lead, TCEM POWER, x86, and Blue Gene HPC Messaging blocksom(a)us.ibm.com From: Halim <halim.amer(a)gmail.com> To: discuss(a)mpich.org, Date: 06/26/2014 02:31 PM Subject: [mpich-discuss] Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q Sent by: discuss-bounces(a)mpich.org Hi, I have a specific issue that arises with MPICH (I use 3.1.1 built with gcc) + MPI_THREAD_MULTIPLE + per-object critical sections on BG/Q. A deadlock happens in the attached hybrid MPI+OpenMP example code with 2 processes and more than one thread per process. Debugging shows that one process is stuck in MPI_Allreduce while the other is blocked in MPI_Finalize. A similar communication pattern happens in my application, but in this case both processes are stuck in MPI_Allreduce. Note that the problem disappears when removing MPI_Allreduce, or avoiding request cancellation (cancel+wait+test_cancelled). Both Allreduce and cancellation operations can be avoided in this test while ensuring a correct result. But in my application, both operations are necessary. In addition, using a global critical section (default) results in a correct execution. My configure line is as follows: ./configure --prefix=/home/aamer/usr --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs:BGQ --with-file-system=bg+bglockless --with-atomic-primitives --enable-handle-allocation=tls --enable-refcount=lock-free -disable-predefined-refcount --disable-error-checking --without-timing --without-mpit-pvars --enable-fast=O3,ndebug --enable-thread-cs=per-object I appreciate any advice to solve this issue? Regards, --Halim [attachment "allred_cancel.c" deleted by Michael Blocksome/Rochester/IBM] _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q
by Halim 26 Jun '14

26 Jun '14

Hi, I have a specific issue that arises with MPICH (I use 3.1.1 built with gcc) + MPI_THREAD_MULTIPLE + per-object critical sections on BG/Q. A deadlock happens in the attached hybrid MPI+OpenMP example code with 2 processes and more than one thread per process. Debugging shows that one process is stuck in MPI_Allreduce while the other is blocked in MPI_Finalize. A similar communication pattern happens in my application, but in this case both processes are stuck in MPI_Allreduce. Note that the problem disappears when removing MPI_Allreduce, or avoiding request cancellation (cancel+wait+test_cancelled). Both Allreduce and cancellation operations can be avoided in this test while ensuring a correct result. But in my application, both operations are necessary. In addition, using a global critical section (default) results in a correct execution. My configure line is as follows: ./configure --prefix=/home/aamer/usr --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs:BGQ --with-file-system=bg+bglockless --with-atomic-primitives --enable-handle-allocation=tls --enable-refcount=lock-free -disable-predefined-refcount --disable-error-checking --without-timing --without-mpit-pvars --enable-fast=O3,ndebug --enable-thread-cs=per-object I appreciate any advice to solve this issue? Regards, --Halim

1 0

Re: [mpich-discuss] couldn't build mpich-3.1.1 on Solaris10withgcc-4.9.0
by Siegmar Gross 26 Jun '14

26 Jun '14

Hi, > >> Can you try with a simple configure and just CC, CXX, and FC set, > >> and nothing else? > > > > Yes, now I used the following commands to configure the system. > > > > ../mpich-3.1.1/configure --prefix=/usr/local/mpich-3.1.1_64_gcc \ > > --libdir=/usr/local/mpich-3.1.1_64_gcc/lib64 \ > > --includedir=/usr/local/mpich-3.1.1_64_gcc/include64 \ > > CC="gcc" CXX="g++" F77="gfortran" FC="gfortran" \ > > CFLAGS="-m64" CXXFLAGS="-m64" FFLAGS="-m64" FCFLAGS="-m64" \ > > LDFLAGS="-m64 -L/usr/lib/sparcv9 -Wl,-rpath /usr/lib/sparcv9" \ > > |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_gcc > > Can you not set the additional flags and try? Which additional flags do you mean? Originally I used the following command, from which I removed all options (as you requested). ../mpich-3.1.1/configure --prefix=/usr/local/mpich-3.1.1_64_gcc \ --libdir=/usr/local/mpich-3.1.1_64_gcc/lib64 \ --includedir=/usr/local/mpich-3.1.1_64_gcc/include64 \ CC="gcc" CXX="g++" F77="gfortran" FC="gfortran" \ CFLAGS="-m64" CXXFLAGS="-m64" FFLAGS="-m64" FCFLAGS="-m64" \ LDFLAGS="-m64 -L/usr/lib/sparcv9 -Wl,-rpath /usr/lib/sparcv9" \ --enable-f77 --enable-fc --enable-cxx --enable-romio \ --enable-debuginfo --enable-smpcoll \ --enable-threads=runtime --with-thread-package=posix \ --enable-shared \ |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_gcc I can try to build "mpich" once more with gcc-4.9.0 on Solaris, if you tell me which configure-command I shall use. Thank you very much for your reply in advance. Kind regards Siegmar

1 0