February 2013 - discuss - mailman3.cels.anl.gov

Debugging mpi program using mpich
by Mathieu Dutour 26 Feb '13

26 Feb '13

Dear all, I used mpich-3.0.1 with debugging options and the program ran correctly but at the end returned some errors indicated later. I thank mpich for finding those errors that other mpi implementation did not find but I wonder if there is a way to transform this into more useful debugging informations. Mathieu PS: The errors retuned after leaving: leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff In direct memory block for handle type GROUP, 3 handles are still allocated leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff In direct memory block for handle type GROUP, 3 handles are still allocated leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff In direct memory block for handle type GROUP, 3 handles are still allocated In direct memory block for handle type REQUEST, 8 handles are still allocated In indirect memory block 0 for handle type REQUEST, 191 handles are still allocated In direct memory block for handle type REQUEST, 8 handles are still allocated In indirect memory block 0 for handle type REQUEST, 190 handles are still allocated In direct memory block for handle type COMM, 4 handles are still allocated In direct memory block for handle type REQUEST, 8 handles are still allocated In indirect memory block 0 for handle type REQUEST, 187 handles are still allocated In direct memory block for handle type COMM, 4 handles are still allocated In direct memory block for handle type DATATYPE, 3 handles are still allocated In indirect memory block 0 for handle type DATATYPE, 1 handles are still allocated In direct memory block for handle type DATATYPE, 3 handles are still allocated In indirect memory block 0 for handle type DATATYPE, 1 handles are still allocated In direct memory block for handle type COMM, 4 handles are still allocated In indirect memory block 0 for handle type COMM, 2 handles are still allocated In direct memory block for handle type DATATYPE, 3 handles are still allocated In indirect memory block 0 for handle type DATATYPE, 1 handles are still allocated

1 0

Re: [mpich-discuss] MPICH2 -Problem
by Andrey V. Malyshev 25 Feb '13

25 Feb '13

Hello, hamideh! You wrote to <discuss(a)mpich.org> on Tue, 26 Feb 2013 00:29:05 +0330: hj> Dear Andrey hj> I checked this directory. there is correct. I use the following your hj> ls -la /home/inet/src/applications/sct.. hj> ls -la inet/src/applications/sctpapp/SCTPlikeAPI/include You have checked relative path, not absolute. Check full path, beginning with '/', and pass it to configure. ??>> 2) /usr/bin/ld: cannot find -lsctplike ??>> ??>> Have you correctly installed your SCTPlikeAPI? Add its 'lib' directory ??>> to ldconfig paths (/etc/ld.so.conf.d). This directory have to contain ??>> the file (or softlink) libsctplike.so. Then run from root ldconfig -v, ??>> and try to configure again. ??>> hj> I do all of your suggestion.. Oh, no 8-) Remove from /etc/ld.so.conf.d all libraries you have copied there, and do this: sudo echo /home/my-ubuntu/inet/src/applications/sctpapp/SCTPlikeAPI/lib > /etc/ld.so.conf.d/sctp.conf NB! It must be absolute (with beginning slash) correct path. Then sudo ldconfig -v and retry configure... hj> sudo su hj> cp /home/my-ubuntu/inet/src/applications/sctpapp/SCTPlikeAPI/lib hj> /etc/ld.so.conf.d hj> ldconfig -v hj> but after configure again, the config.log was the same previous with hj> this problem! I attached it to this email hj> Please help me again... :( hj> Excuse me... hj>>> On Sun, Feb 24, 2013 at 12:22 PM, Andrey V. Malyshev hj>>> <amal(a)icm.krasn.ru>wrote: ??>> ??>>>> Hello, hamideh! ??>>>> You wrote to <discuss(a)mpich.org> on Sun, 24 Feb 2013 10:12:01 +0330: ??>>>> hj>>>>> I think configure doesn't done correctly! ??>>>> ??>>>> That's right. ??>>>> hj>>>>> configure command is : hj>>>>> ../configure --with-device=ch3:sctp hj>>>>> --with-sctp=/home/inet/src/applications/sctpapp/SCTPlikeAPI hj>>>>> --prefix=/home/mpich2-hginit ï¿½disable-mpe hj>>>>> What I am doing? :( ??>>>> ??>>>> See the file 'config.log' from the same directory where the ??>>>> 'configure' was started, or where the 'configure' file is ??>>>> located. Pthread library was not found, you should understand, why, ??>>>> and maybe then it becomes clear how to fix. ??>>>> hj>>>>> On Sun, Feb 24, 2013 at 10:07 AM, Andrey V. Malyshev hj>>>>> <amal(a)icm.krasn.ru>wrote: ??>>>> ??>>>>>> Hello, hamideh! ??>>>>>> You wrote to <discuss(a)mpich.org> on Sun, 24 Feb 2013 08:23:37 ??>> +0330: ??>>>>>> hj>>>>>>> checking for pthread.h... yes ??>>>>>> checking for pthread_yield... no ??>>>>>> checking for library containing pthread_key_create... no ??>>>>>> configure: error: unable to find pthreads library ??>>>>>> ??>>>>>> See config.log, what error occurs during this check? ??>>>>>> ??>>>>>> -- ??>>>>>> With best regards, Andrey V. Malyshev. ??>>>>>> E-mail: amal(a)krasn.ru ??>>>>>> _______________________________________________ ??>>>>>> discuss mailing list discuss(a)mpich.org ??>>>>>> To manage subscription options or unsubscribe: ??>>>>>> https://lists.mpich.org/mailman/listinfo/discuss ??>>>>>> hj>>>>> -- hj>>>>> *Best Regards* hj>>>>> * ??>>>> hj>>>>> Hamideh Jabalameli* ??>>>> hj>>>>> M.Sc Candidate, hj>>>>> School of Electrical and Computer Engineering, hj>>>>> College of Engineering, hj>>>>> Electronici Shahid Beheshti Tehran University hj>>>>> h.jabal(a)Mail.sbu.ac.ir hj>>>>> hamideh.jabalameli(a)gmail.com ??>>>> ??>>>> -- ??>>>> With best regards, Andrey V. Malyshev. ??>>>> E-mail: amal(a)krasn.ru ??>>>> ??>>>> _______________________________________________ ??>>>> discuss mailing list discuss(a)mpich.org ??>>>> To manage subscription options or unsubscribe: ??>>>> https://lists.mpich.org/mailman/listinfo/discuss ??>>>> hj>>> -- hj>>> *Best Regards* hj>>> * ??>> hj>>> Hamideh Jabalameli* ??>> hj>>> M.Sc Candidate, hj>>> School of Electrical and Computer Engineering, hj>>> College of Engineering, hj>>> Electronici Shahid Beheshti Tehran University hj>>> h.jabal(a)Mail.sbu.ac.ir hj>>> hamideh.jabalameli(a)gmail.com ??>> ??>> -- ??>> With best regards, Andrey V. Malyshev. ??>> E-mail: amal(a)krasn.ru ??>> ??>> _______________________________________________ ??>> discuss mailing list discuss(a)mpich.org ??>> To manage subscription options or unsubscribe: ??>> https://lists.mpich.org/mailman/listinfo/discuss ??>> hj> -- hj> *Best Regards* hj> * hj> Hamideh Jabalameli* hj> M.Sc Candidate, hj> School of Electrical and Computer Engineering, hj> College of Engineering, hj> Electronici Shahid Beheshti Tehran University hj> h.jabal(a)Mail.sbu.ac.ir hj> hamideh.jabalameli(a)gmail.com -- With best regards, Andrey V. Malyshev. E-mail: amal(a)krasn.ru

1 0

Re: [mpich-discuss] MPICH2 -Problem
by hamideh jabalameli 25 Feb '13

25 Feb '13

On Sun, Feb 24, 2013 at 10:37 PM, Andrey V. Malyshev <amal(a)icm.krasn.ru>wrote: > Hello, hamideh! > You wrote to <discuss(a)mpich.org> on Sun, 24 Feb 2013 18:30:43 +0330: > > hj> I see file 'config.log' from the same directory where the 'configure' > hj> was started. I attached its to this email. In the content of this > file, > hj> pthread libraries such as "pthread.h,pthread_yield" have make. but > hj> there are related error... > hj> Do you know that what do we do? > > 1) /home/inet/src/applications/sctpapp/SCTPlikeAPI/include: No such file or > directory > Is this directory exists? If yes, maybe the path at all is incorrect? Try > exactly > 'ls -la /home/inet/src/applications/sctpapp/SCTPlikeAPI/include' > Dear Andrey I checked this directory. there is correct. I use the following your command :ls -la /home/inet/src/applications/sctpapp/SCTPlikeAPI/include, and I saw : ls -la inet/src/applications/sctpapp/SCTPlikeAPI/include total 8 drwxrwxr-x 2 my-ubuntu my-ubuntu 4096 Feb 22 22:07 . drwx------ 4 my-ubuntu my-ubuntu 4096 Feb 22 22:07 .. lrwxrwxrwx 1 my-ubuntu my-ubuntu 19 Feb 22 22:07 sctplikeapi.h -> ../../sctplikeapi.h Therefore, this directory is correct... > 2) /usr/bin/ld: cannot find -lsctplike > > Have you correctly installed your SCTPlikeAPI? Add its 'lib' directory to > ldconfig paths (/etc/ld.so.conf.d). This directory have to contain the file > (or softlink) libsctplike.so. Then run from root ldconfig -v, and try to > configure again. > I do all of your suggestion.. sudo su cp /home/my-ubuntu/inet/src/applications/sctpapp/SCTPlikeAPI/lib /etc/ld.so.conf.d ldconfig -v but after configure again, the config.log was the same previous with this problem! I attached it to this email Please help me again... :( Excuse me... > > hj> On Sun, Feb 24, 2013 at 12:22 PM, Andrey V. Malyshev > hj> <amal(a)icm.krasn.ru>wrote: > > ??>> Hello, hamideh! > ??>> You wrote to <discuss(a)mpich.org> on Sun, 24 Feb 2013 10:12:01 +0330: > ??>> > hj>>> I think configure doesn't done correctly! > ??>> > ??>> That's right. > ??>> > hj>>> configure command is : > hj>>> ../configure --with-device=ch3:sctp > hj>>> --with-sctp=/home/inet/src/applications/sctpapp/SCTPlikeAPI > hj>>> --prefix=/home/mpich2-hginit –disable-mpe > hj>>> What I am doing? :( > ??>> > ??>> See the file 'config.log' from the same directory where the > ??>> 'configure' was started, or where the 'configure' file is > ??>> located. Pthread library was not found, you should understand, why, > ??>> and maybe then it becomes clear how to fix. > ??>> > hj>>> On Sun, Feb 24, 2013 at 10:07 AM, Andrey V. Malyshev > hj>>> <amal(a)icm.krasn.ru>wrote: > ??>> > ??>>>> Hello, hamideh! > ??>>>> You wrote to <discuss(a)mpich.org> on Sun, 24 Feb 2013 08:23:37 > +0330: > ??>>>> > hj>>>>> checking for pthread.h... yes > ??>>>> checking for pthread_yield... no > ??>>>> checking for library containing pthread_key_create... no > ??>>>> configure: error: unable to find pthreads library > ??>>>> > ??>>>> See config.log, what error occurs during this check? > ??>>>> > ??>>>> -- > ??>>>> With best regards, Andrey V. Malyshev. > ??>>>> E-mail: amal(a)krasn.ru > ??>>>> _______________________________________________ > ??>>>> discuss mailing list discuss(a)mpich.org > ??>>>> To manage subscription options or unsubscribe: > ??>>>> https://lists.mpich.org/mailman/listinfo/discuss > ??>>>> > hj>>> -- > hj>>> *Best Regards* > hj>>> * > ??>> > hj>>> Hamideh Jabalameli* > ??>> > hj>>> M.Sc Candidate, > hj>>> School of Electrical and Computer Engineering, > hj>>> College of Engineering, > hj>>> Electronici Shahid Beheshti Tehran University > hj>>> h.jabal(a)Mail.sbu.ac.ir > hj>>> hamideh.jabalameli(a)gmail.com > ??>> > ??>> -- > ??>> With best regards, Andrey V. Malyshev. > ??>> E-mail: amal(a)krasn.ru > ??>> > ??>> _______________________________________________ > ??>> discuss mailing list discuss(a)mpich.org > ??>> To manage subscription options or unsubscribe: > ??>> https://lists.mpich.org/mailman/listinfo/discuss > ??>> > hj> -- > hj> *Best Regards* > hj> * > > hj> Hamideh Jabalameli* > > hj> M.Sc Candidate, > hj> School of Electrical and Computer Engineering, > hj> College of Engineering, > hj> Electronici Shahid Beheshti Tehran University > hj> h.jabal(a)Mail.sbu.ac.ir > hj> hamideh.jabalameli(a)gmail.com > > -- > With best regards, Andrey V. Malyshev. > E-mail: amal(a)krasn.ru > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- *Best Regards* * Hamideh Jabalameli* M.Sc Candidate, School of Electrical and Computer Engineering, College of Engineering, Electronici Shahid Beheshti Tehran University h.jabal(a)Mail.sbu.ac.ir hamideh.jabalameli(a)gmail.com

1 0

Re: [mpich-discuss] Maximum number of communicators
by Nichols A. Romero 25 Feb '13

25 Feb '13

I cannot reproduce this in a simple test code. I am convinced that this is an application error. I linked the application against mpiP and I see that the calls to Comm_create >> Comm_free by two to three orders of magnitude. The application is template C++ and I suspect that the C++ class destructor is not properly calling Comm_free. ----- Original Message ----- > From: "Dave Goodell" <goodell(a)mcs.anl.gov> > To: discuss(a)mpich.org > Sent: Sunday, February 24, 2013 3:36:54 PM > Subject: Re: [mpich-discuss] Maximum number of communicators > > Test code indeed would be helpful. > > You may get better warnings about context ID and/or communicator > leaks if you configure a stock MPICH with "--enable-g=all". > > -Dave > > On Feb 23, 2013, at 7:05 PM CST, Jeff Hammond wrote: > > > It's 2048 in all MPICH2-derived implementations that I know about, > > including the ones used on BGP and BGQ. > > > > Since I know what your issue is already, I think the problem has > > something to do with context id leaking. You should post your test > > code. > > > > Jeff > > > > On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero > > <naromero(a)alcf.anl.gov> wrote: > >> Hi, > >> > >> What is the maximum number of communicators in the MPICH that is > >> used on BG/P vs. that used on BG/Q? > >> > >> Sent from my iPhone > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > -- > > Jeff Hammond > > Argonne Leadership Computing Facility > > University of Chicago Computation Institute > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > http://www.linkedin.com/in/jeffhammond > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441

1 0

get the process rank in the bash script
by Kirill Lykov 25 Feb '13

25 Feb '13

Dear mpich users, I'm looking for an environmental variable which can be used in a bash script to identify the process rank. I launch a bash script using mpi: mpirun ./my-script.sh In my-script I want to use the rank of the process to identify which input file to use. In OpenMPI, I used OMPI_COMM_WORLD_RANK and my-script.sh looked like the following: inputName="MyInput_$OMPI_COMM_WORLD_RANK"./myexe $inputName What can be used in MPICH instead of OMPI_COMM_WORLD_RANK? I found somewhere that $PMI_RANK is the same but I found that it is empty. Kirill Lykov, Mr. Doctoral Assistant Institute of Computational Science Faculty of Informatics Università della Svizzera italiana, USI Via G. Buffi 13 CH - 6904 Lugano (e) kirill.lykov(a)usi.ch (t) +41 (0) 788413392

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jim Dinan 25 Feb '13

25 Feb '13

On 2/25/13 9:30 AM, Jeff Hammond wrote: >> It depends on the op. We start processing the op as soon as the packet >> header arrives. For ops that send data to the target, if the data (payload >> and datatype) has not all arrived at the time we enter the progress engine, >> the op will be enqueued to finish later using an internal/device-level >> request. Ops that send a response to the origin process, always enqueue a >> request that completes when we finish sending the response message. Some >> ops (notably, CAS and FOP and also Acc in some cases) embed the data payload >> in the packet header, allowing us to handle them completely with a single >> iteration through the progress engine. > > Is it possible to add an option to force the progress engine to finish > RMA ops once they start? That might help NWChem since it does larger > transfers, particularly Acc. Could this be an MPICH-specific info key > on the window? I'm not sure if that mode of operation would be compatible with the basic CH3 architecture. We handle the data that has been received and then re-enter the progress engine, presumably because we could also receive data from other processes. I'm not sure that we have the option to block on communication with a particular process at that level. ~Jim.

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jeff Hammond 25 Feb '13

25 Feb '13

On Mon, Feb 25, 2013 at 9:22 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > On 2/25/13 9:08 AM, Jeff Hammond wrote: >> >> Is it the case that entering into MPI will lead to processing of all >> packets in an incoming RMA op? Does the receipt of the first packet >> from an MPI_Accumulate cause the target to sit inside of the progress >> engine until all packages in the message have arrived or does the >> passive target merely process the packets that arrive while it is >> inside of MPI and then return. > > > Hi Jeff, > > It depends on the op. We start processing the op as soon as the packet > header arrives. For ops that send data to the target, if the data (payload > and datatype) has not all arrived at the time we enter the progress engine, > the op will be enqueued to finish later using an internal/device-level > request. Ops that send a response to the origin process, always enqueue a > request that completes when we finish sending the response message. Some > ops (notably, CAS and FOP and also Acc in some cases) embed the data payload > in the packet header, allowing us to handle them completely with a single > iteration through the progress engine. Is it possible to add an option to force the progress engine to finish RMA ops once they start? That might help NWChem since it does larger transfers, particularly Acc. Could this be an MPICH-specific info key on the window? Thanks, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jim Dinan 25 Feb '13

25 Feb '13

On 2/25/13 9:08 AM, Jeff Hammond wrote: > Is it the case that entering into MPI will lead to processing of all > packets in an incoming RMA op? Does the receipt of the first packet > from an MPI_Accumulate cause the target to sit inside of the progress > engine until all packages in the message have arrived or does the > passive target merely process the packets that arrive while it is > inside of MPI and then return. Hi Jeff, It depends on the op. We start processing the op as soon as the packet header arrives. For ops that send data to the target, if the data (payload and datatype) has not all arrived at the time we enter the progress engine, the op will be enqueued to finish later using an internal/device-level request. Ops that send a response to the origin process, always enqueue a request that completes when we finish sending the response message. Some ops (notably, CAS and FOP and also Acc in some cases) embed the data payload in the packet header, allowing us to handle them completely with a single iteration through the progress engine. ~Jim.

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jeff Hammond 25 Feb '13

25 Feb '13

>> I think the only way for async progress to work well is to have >> fine-grain locking since of MPI, as is done in PAMID. Any >> implementation that resorts to fat-locking is probably better off >> without async progress unless the application is doing something >> really silly (like never calling MPI on a rank that is the target of >> MPI RMA). > > If processes are entering the MPI progress engine frequently enough that > they are competing with the comm thread to get the (per-process, not > per-node) library lock, the additional comm thread is probably not needed. Yep. That's exactly what I see with NWChem. I've never profiled it with enough detail to prove that this is why NWChem runs faster without comm threads, but it is entirely logical to me that this is the reason. If there was a way to instrument RMA to measure time spent waiting on remote progress vs. time spent actually moving data, that would help, but I imagine that is far from trivial to implement. Is it the case that entering into MPI will lead to processing of all packets in an incoming RMA op? Does the receipt of the first packet from an MPI_Accumulate cause the target to sit inside of the progress engine until all packages in the message have arrived or does the passive target merely process the packets that arrive while it is inside of MPI and then return. I'm sorry if this is a poorly posed question. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jim Dinan 25 Feb '13

25 Feb '13

On 2/23/13 8:21 PM, Jeff Hammond wrote: > I think the only way for async progress to work well is to have > fine-grain locking since of MPI, as is done in PAMID. Any > implementation that resorts to fat-locking is probably better off > without async progress unless the application is doing something > really silly (like never calling MPI on a rank that is the target of > MPI RMA). Jeff, If processes are entering the MPI progress engine frequently enough that they are competing with the comm thread to get the (per-process, not per-node) library lock, the additional comm thread is probably not needed. ~Jim.

1 0