September 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Comm_spawn_multiple Segment fault
by myself 27 Sep '13

27 Sep '13

Thanks. I found on our 16 nodes cluster, when I run another application which has more initialization parameters, hydra can only spawn 49 process (3 per node). And the segment fault happened in PMI_Spawn_multiple. So I tried to use smpd, and it seems to be able to spawn more. However, when child processes increase to 150+ (almost 10 per node), it seems to run out of resources. My question is whether there is a upper limit of child processes one parent process can spawn and how can I determine the upper limit? If I want to use one parent process to spawn more processes, using its child processes to spawn descendant is the only method ( Suppose we want to spawn 160 process on 160 nodes )? At 2013-09-27 21:43:43,"Wesley Bland" <wbland(a)mcs.anl.gov> wrote: You're probably just running out of resources. 90 processes (not threads) on one node is a lot. A common use of MPI is to have one (or a few) processes per node and if you need more local instances, you investigate using additional threads (such as OpenMP). Thanks, Wesley On Sep 26, 2013, at 9:45 PM, myself <chcdlf(a)126.com> wrote: I'm trying to spawn several processes on two node. When the number of child processes is small, it works fine, such as less than 80. However, when I tried to spawn 90+ processes, mpirun told me "Segmentation fault (core dumped)". Is something I did wrong or I didn't do? My MPICH compile command: $ ./configure --prefix=/opt/mpich3 --with-device=ch3:nemesis --with-pm=hydra --enable-fast=none --enable-g=dbg CFLAGS=-fPIC --disable-f77 --disable-fc Here is my source code: #include "mpi.h" #include <stdio.h> #include <stdlib.h> #include <unistd.h> int main( int argc, char *argv[] ) { int HLEN=2; char *host[]={"node1","node2"}; char name[65]; int count; if(argc==2) count=atoi(argv[1]); else count=2; int root=0; char* array_of_commands[count]; int array_of_maxprocs[count]; int errcodes[count]; MPI_Info array_of_info[count]; MPI_Comm parentcomm, intercomm; int i; MPI_Init( &argc, &argv ); for(i=0; i< count; i++){ array_of_commands[i]="/home/mpitest/spawn"; array_of_maxprocs[i]=1; MPI_Info_create(&array_of_info[i]); MPI_Info_set(array_of_info[i],"host",host[i % HLEN]); } MPI_Comm_get_parent( &parentcomm ); if (parentcomm == MPI_COMM_NULL) { MPI_Comm_spawn_multiple( count, array_of_commands, MPI_ARGVS_NULL /*array_of_argv*/, array_of_maxprocs, array_of_info, root, MPI_COMM_WORLD, &intercomm, errcodes ); printf("I'm the parent.\n"); } else { gethostname(name, sizeof(name)); printf("I'm the spawned at %s.\n",name); } fflush(stdout); MPI_Finalize(); return 0; } _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Comm_spawn_multiple Segment fault
by Wesley Bland 27 Sep '13

27 Sep '13

You're probably just running out of resources. 90 processes (not threads) on one node is a lot. A common use of MPI is to have one (or a few) processes per node and if you need more local instances, you investigate using additional threads (such as OpenMP). Thanks, Wesley On Sep 26, 2013, at 9:45 PM, myself <chcdlf(a)126.com> wrote: > I'm trying to spawn several processes on two node. When the number of child processes is small, it works fine, such as less than 80. However, when I tried to spawn 90+ processes, mpirun told me "Segmentation fault (core dumped)". Is something I did wrong or I didn't do? > > My MPICH compile command: > > $ ./configure --prefix=/opt/mpich3 --with-device=ch3:nemesis --with-pm=hydra --enable-fast=none --enable-g=dbg CFLAGS=-fPIC --disable-f77 --disable-fc > > Here is my source code: > > #include "mpi.h" > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > > int main( int argc, char *argv[] ) { > > int HLEN=2; > char *host[]={"node1","node2"}; > char name[65]; > > int count; > if(argc==2) > count=atoi(argv[1]); > else > count=2; > > int root=0; > char* array_of_commands[count]; > int array_of_maxprocs[count]; > int errcodes[count]; > MPI_Info array_of_info[count]; > MPI_Comm parentcomm, intercomm; > > int i; > MPI_Init( &argc, &argv ); > for(i=0; i< count; i++){ > array_of_commands[i]="/home/mpitest/spawn"; > array_of_maxprocs[i]=1; > MPI_Info_create(&array_of_info[i]); > MPI_Info_set(array_of_info[i],"host",host[i % HLEN]); > } > > MPI_Comm_get_parent( &parentcomm ); > if (parentcomm == MPI_COMM_NULL) { > MPI_Comm_spawn_multiple( count, array_of_commands, MPI_ARGVS_NULL /*array_of_argv*/, > array_of_maxprocs, array_of_info, root, MPI_COMM_WORLD, &intercomm, errcodes ); > printf("I'm the parent.\n"); > } else { > gethostname(name, sizeof(name)); > printf("I'm the spawned at %s.\n",name); > } > fflush(stdout); > MPI_Finalize(); > return 0; > } > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Comm_spawn_multiple Segment fault
by myself 26 Sep '13

26 Sep '13

I'm trying to spawn several processes on two node. When the number of child processes is small, it works fine, such as less than 80. However, when I tried to spawn 90+ processes, mpirun told me "Segmentation fault (core dumped)". Is something I did wrong or I didn't do? My MPICH compile command: $ ./configure --prefix=/opt/mpich3 --with-device=ch3:nemesis --with-pm=hydra --enable-fast=none --enable-g=dbg CFLAGS=-fPIC --disable-f77 --disable-fc Here is my source code: #include "mpi.h" #include <stdio.h> #include <stdlib.h> #include <unistd.h> int main( int argc, char *argv[] ) { int HLEN=2; char *host[]={"node1","node2"}; char name[65]; int count; if(argc==2) count=atoi(argv[1]); else count=2; int root=0; char* array_of_commands[count]; int array_of_maxprocs[count]; int errcodes[count]; MPI_Info array_of_info[count]; MPI_Comm parentcomm, intercomm; int i; MPI_Init( &argc, &argv ); for(i=0; i< count; i++){ array_of_commands[i]="/home/mpitest/spawn"; array_of_maxprocs[i]=1; MPI_Info_create(&array_of_info[i]); MPI_Info_set(array_of_info[i],"host",host[i % HLEN]); } MPI_Comm_get_parent( &parentcomm ); if (parentcomm == MPI_COMM_NULL) { MPI_Comm_spawn_multiple( count, array_of_commands, MPI_ARGVS_NULL /*array_of_argv*/, array_of_maxprocs, array_of_info, root, MPI_COMM_WORLD, &intercomm, errcodes ); printf("I'm the parent.\n"); } else { gethostname(name, sizeof(name)); printf("I'm the spawned at %s.\n",name); } fflush(stdout); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] Building MPICH-3.0.4 to create libmpich.so
by John S. Rivas M. 26 Sep '13

26 Sep '13

Thanks for your answer Ken, After your confirmation that *--enable-shared=yes* should has worked, I decided to backtrack and start again from scratch. Fortunately, I got it working properly. Thanks a lot for your help. On Wed, Sep 25, 2013 at 5:51 PM, Ken Raffenetti <raffenet(a)mcs.anl.gov>wrote: > John, > > Configuring with --enable-shared or --enable-shared=yes should have worked > for this. Can you send your config.log so we can inspect and see what went > wrong? > > Ken > > ----- Original Message ----- > > From: "John S. Rivas M." <jsrivas(a)gmail.com> > > To: discuss(a)mpich.org > > Sent: Wednesday, September 25, 2013 4:12:24 PM > > Subject: [mpich-discuss] Building MPICH-3.0.4 to create libmpich.so > > > > > > > > > > > > > > > > > > > > > > > > Greetings to all, > > > > I have successfully built mpich-3.0.4 to work with some applications > > that require parallel support. But, I face a problem that I haven't > > been able to solve: > > > > I need to setup boost libraries (from boost.org ) in my system and an > > error has popped up. The error indicates that, during the attempt to > > setup boost, the library libmpich.a cannot be used when making > > shared objects. It is my understanding that I should use a library > > called libmpich.so (or something similar, that allows the creation > > of shared objects), however such library is not created when I build > > mpich-3.0.4 . > > > > Can anybody indicate to me whether I don't get the libmpich.so (or > > similar) because I'm missing something during the building of > > mpich-3.0.4 or because it's simply not created and I need to find > > another flavor of mpi for this? I appreciate any inputs. > > > > Some extra info: > > - system: Ubuntu 12.04 (64-bit) > > - compilers used: GNU, including gcc, g++, cc, c++, cpp, f95, and > > gfortran > > - libraries built in /usr/local/lib/: libmpich.a, libmpich.la , > > libmpichcxx.a, libmpichcxx.la , libmpichf90.a, libmpichf90.la > > - attempts to run ./configure script with options: <none> and > > --enable-shared=yes. Both produce the same libraries mentioned above > > and not the desired libmpich.so (or similar) > > > > > > Thanks a lot and best regards. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > JSR > > "Montani Semper Liberi" > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- *JSR * *"Montani Semper Liberi"*

1 0

Re: [mpich-discuss] Building MPICH-3.0.4 to create libmpich.so
by Ken Raffenetti 25 Sep '13

25 Sep '13

John, Configuring with --enable-shared or --enable-shared=yes should have worked for this. Can you send your config.log so we can inspect and see what went wrong? Ken ----- Original Message ----- > From: "John S. Rivas M." <jsrivas(a)gmail.com> > To: discuss(a)mpich.org > Sent: Wednesday, September 25, 2013 4:12:24 PM > Subject: [mpich-discuss] Building MPICH-3.0.4 to create libmpich.so > > > > > > > > > > > > Greetings to all, > > I have successfully built mpich-3.0.4 to work with some applications > that require parallel support. But, I face a problem that I haven't > been able to solve: > > I need to setup boost libraries (from boost.org ) in my system and an > error has popped up. The error indicates that, during the attempt to > setup boost, the library libmpich.a cannot be used when making > shared objects. It is my understanding that I should use a library > called libmpich.so (or something similar, that allows the creation > of shared objects), however such library is not created when I build > mpich-3.0.4 . > > Can anybody indicate to me whether I don't get the libmpich.so (or > similar) because I'm missing something during the building of > mpich-3.0.4 or because it's simply not created and I need to find > another flavor of mpi for this? I appreciate any inputs. > > Some extra info: > - system: Ubuntu 12.04 (64-bit) > - compilers used: GNU, including gcc, g++, cc, c++, cpp, f95, and > gfortran > - libraries built in /usr/local/lib/: libmpich.a, libmpich.la , > libmpichcxx.a, libmpichcxx.la , libmpichf90.a, libmpichf90.la > - attempts to run ./configure script with options: <none> and > --enable-shared=yes. Both produce the same libraries mentioned above > and not the desired libmpich.so (or similar) > > > Thanks a lot and best regards. > > > > > > > > > > > > -- > > > JSR > "Montani Semper Liberi" > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Building MPICH-3.0.4 to create libmpich.so
by John S. Rivas M. 25 Sep '13

25 Sep '13

Greetings to all, I have successfully built *mpich-3.0.4* to work with some applications that require parallel support. But, I face a problem that I haven't been able to solve: I need to setup boost libraries (from boost.org) in my system and an error has popped up. The error indicates that, during the attempt to setup boost, the library *libmpich.a* cannot be used when making shared objects. It is my understanding that I should use a library called *libmpich.so* (or something similar, that allows the creation of shared objects), however such library is not created when I build *mpich-3.0.4*. Can anybody indicate to me whether I don't get the *libmpich.so* (or similar) because I'm missing something during the building of *mpich-3.0.4*or because it's simply not created and I need to find another flavor of mpi for this? I appreciate any inputs. Some extra info: - system: Ubuntu 12.04 (64-bit) - compilers used: GNU, including gcc, g++, cc, c++, cpp, f95, and gfortran - libraries built in /usr/local/lib/: libmpich.a, libmpich.la, libmpichcxx.a, libmpichcxx.la, libmpichf90.a, libmpichf90.la - attempts to run ./configure script with options: <none> and --enable-shared=yes. Both produce the same libraries mentioned above and not the desired libmpich.so (or similar) Thanks a lot and best regards. -- *JSR * *"Montani Semper Liberi"*

1 0

Re: [mpich-discuss] Missing MPICH2ib.dll in the Windows version mpich2-1.4.1p1-win-x86-64
by Pavan Balaji 25 Sep '13

25 Sep '13

Hello, MPICH does not support windows anymore. http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_can… -- Pavan On Sep 25, 2013, at 4:04 AM, Carron, Thierry wrote: > Hello > > When I add -env MPICH2_CHANNEL ib to the mpiexec launcher in order to run over Infiniband fabric I have the following dll load error > > “Unable to load 'mpich2ib.dll', error 126” > > and after checking I can’t find the dll on the system. > > Is this MPICH2 version supporting to run over WinOFF driver ? > If not do you have any other version suitable for running IB on a Windows server 2008 R2 HPC cluster or should I have to used MS-MPI ? > > Thanks > > Thierry Carron > HPC Senior Consultant > HPC EMEA Competency Center > CC20 > 5 Avenue Raymond Chanas > 38053 Grenoble > Tel: +33 4 7614 4382 > Mob: +33 6 7299 1385 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Reg: mpdtrace nodes get out of cluster mpirun
by Pavan Balaji 25 Sep '13

25 Sep '13

Hello, Please don't use mpd or such an old version of mpich2. The latest version of mpich can be downloaded here. http://www.mpich.org -- Pavan On Sep 23, 2013, at 7:59 AM, Murugayen M wrote: > > > i ran mpdboot for 10 nodes, mpirun it working fine first time, after some time i had check mpdtrace some nodes out of state in mpdboot. > > Please help to solve the problem. > > Here i am using rhel 5, ethernet connectivity, mpich2., intel compiler > > > > > -- > Thanks & Regards, > Murugayen M > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Missing MPICH2ib.dll in the Windows version mpich2-1.4.1p1-win-x86-64
by Carron, Thierry 25 Sep '13

25 Sep '13

Hello When I add -env MPICH2_CHANNEL ib to the mpiexec launcher in order to run over Infiniband fabric I have the following dll load error "Unable to load 'mpich2ib.dll', error 126" and after checking I can't find the dll on the system. Is this MPICH2 version supporting to run over WinOFF driver ? If not do you have any other version suitable for running IB on a Windows server 2008 R2 HPC cluster or should I have to used MS-MPI ? Thanks Thierry Carron HPC Senior Consultant HPC EMEA Competency Center CC20 5 Avenue Raymond Chanas 38053 Grenoble Tel: +33 4 7614 4382 Mob: +33 6 7299 1385

1 0

Fwd: Reg: mpdtrace nodes get out of cluster mpirun
by Murugayen M 23 Sep '13

23 Sep '13

i ran mpdboot for 10 nodes, mpirun it working fine first time, after some time i had check mpdtrace some nodes out of state in mpdboot. Please help to solve the problem. Here i am using rhel 5, ethernet connectivity, mpich2., intel compiler -- Thanks & Regards, Murugayen M

1 0