October 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] BLCR linker option typo?
by Halim Amer 14 Oct '15

14 Oct '15

Hello Torquil, This bug got propagated. Your fix is reasonable, so we pushed it upstream. It should be available in the 3.2 series. Thank you for reporting the problem. --Halim www.mcs.anl.gov/~aamer On 10/2/15 10:49 AM, Torquil Macdonald Sørensen wrote: > Hi! > > When building MPICH 3.1.4 with ./configure options > --enable-checkpointing --with-hydra-ckpointlib=blcr, as specified at: > > https://wiki.mpich.org/mpich/index.php/Checkpointing > > I'm get the error: > > CC src/pmi/simple/lib_libmpi_la-simple_pmiutil.lo > CC src/pmi/simple/lib_libmpi_la-simple_pmi.lo > GEN lib/libmpi.la > /usr/bin/ld: cannot find -lblcr > collect2: error: ld returned 1 exit status > Makefile:16217: recipe for target 'lib/libmpi.la' failed > make[2]: *** [lib/libmpi.la] Error 1 > make[2]: Leaving directory '/tmp/tmp.stowinstall.kvTp' > Makefile:40727: recipe for target 'install-recursive' failed > make[1]: *** [install-recursive] Error 1 > > Looking at ./configure.ac, on line 5515, I find the string -lblcr. > Shoudn't that be -lcr. As far as I know, the BLCR shared library > filename is libcr.so, on my Debian system as well as in a BLCR RPM file > I found at pkgs.org. > > The -lblcr string was introduced in configure.ac on GIT revision > 51db1a05720c3d1a51eff58084a02b6b6c66e80f > > http://trac.mpich.org/projects/mpich/changeset/51db1a05720c3d1a51eff58084a0… > > If I change it to -lcr in configure.ac and run autoconf, I can build > successfully, and mpiexec.hydra ends up correctly linked to libcr.so.0. > > I don't think I have an MPICH TRAC account, so I couldn't submit this as > a bug report. > > Best regards, > Torquil Macdonald Sørensen > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi Cluster configuration
by Halim Amer 13 Oct '15

13 Oct '15

Sorry, we do not support Windows (https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_ca…) --Halim www.mcs.anl.gov/~aamer On 10/13/15 5:28 AM, Buonomo Veglia srl - Architettura Ingegneria wrote: > See attached file for Cluster configuration problem > > Thank you for any help > > Best regards > > ***Descrizione: BVLOGOdiritto2005_letterarid Buonomo Veglia Srl* > /Corso Novara 99 10154 Torino /***T*/ 39 011 2481365 /***F*/ 39 011 > 0674473 /***E*/studio(a)buonomoveglia.com > <mailto:[email protected]> /*C*/certificata(a)pec.buonomoveglia.com > <mailto:[email protected]> /*W*/www.buonomoveglia.com > <http://www.buonomoveglia.com>/ > > *C.F. e P.I.*/08968900012 Capitale Sociale interamente versato 10.000 > euro /***CCIAA*/di Torino 08968900012 /***REA*/1015323/ > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpi Cluster configuration
by Buonomo Veglia srl - Architettura Ingegneria 13 Oct '15

13 Oct '15

See attached file for Cluster configuration problem Thank you for any help Best regards Descrizione: BVLOGOdiritto2005_letterarid Buonomo Veglia Srl Corso Novara 99 10154 Torino T 39 011 2481365 F 39 011 0674473 E <mailto:[email protected]> studio(a)buonomoveglia.com C <mailto:[email protected]> certificata(a)pec.buonomoveglia.com W <http://www.buonomoveglia.com> www.buonomoveglia.com C.F. e P.I. 08968900012 Capitale Sociale interamente versato 10.000 euro CCIAA di Torino 08968900012 REA 1015323 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue with MPI_Comm_spawn_multiple
by Antonio J. Peña 13 Oct '15

13 Oct '15

Thanks a lot for looking into this. Toni On 10/10/2015 05:47 AM, Seo, Sangmin wrote: > Hi Toni, > > We figured out the reason (the spawned processes hang because when the > "hosts" info keys for two commands have the same value (e.g., > localhost) two processes get the same rank.), but it seems it will > take some time to fix this. Pavan will rewrite the problematic part in > hydra. > > Iï¿½ve created a ticket > (https://trac.mpich.org/projects/mpich/ticket/2308) for this issue and > ccï¿½ed you in the ticket. > > Regards, > Sangmin > > >> On Oct 8, 2015, at 10:17 AM, Antonio J. Peï¿½a <antonio.pena(a)bsc.es >> <mailto:[email protected]>> wrote: >> >> >> Thanks a lot Sangmin! >> >> >> On 10/08/2015 03:43 PM, Seo, Sangmin wrote: >>> Hi Toni, >>> >>> I confirmed the problem and am looking into it. I will get back to >>> you after I resolve it. >>> >>> Regards, >>> Sangmin >>> >>> >>>> On Oct 6, 2015, at 7:23 AM, Antonio J. Peï¿½a <antonio.pena(a)bsc.es >>>> <mailto:[email protected]>> wrote: >>>> >>>> >>>> Right, but I do need to use the hosts key to get control of where >>>> the processes are spawned. Any clue? >>>> >>>> >>>> On 10/02/2015 07:32 PM, Thakur, Rajeev wrote: >>>>> Looks like it is related to the hosts key. Works without that. >>>>> >>>>> Rajeev >>>>> >>>>>> On Oct 2, 2015, at 10:34 AM, Antonio J. Peï¿½a <antonio.pena(a)bsc.es >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hi folks! >>>>>> >>>>>> I've been facing an issue with MPI_Comm_spawn_multiple when all the >>>>>> following conditions are met: >>>>>> >>>>>> 1. 'count' is > 1 >>>>>> 2. Any value of 'array_of_maxprocs' is > 1 >>>>>> 3. Setting the "hosts" info key >>>>>> >>>>>> What I'm finding is that the spawned processes hang at MPI_Init. Find >>>>>> attached a test case. You can reproduce my issue with -np 1. >>>>>> >>>>>> Is there anything I'm doing wrong? I'm using MPICH master at >>>>>> 1e6c4d8. I >>>>>> haven't found a pending ticket, xfail test, or test case covering >>>>>> this >>>>>> situation. >>>>>> >>>>>> All the best, >>>>>> Toni >>>>>> >>>>>> [P.S.: It still feels weird to be on the other side of >>>>>> discuss(a)mpich.org <mailto:[email protected]>] >>>>>> >>>>>> -- >>>>>> Antonio J. Peï¿½a >>>>>> Senior Researcher >>>>>> Barcelona Supercomputing Center >>>>>> >>>>>> >>>>>> >>>>>> WARNING / LEGAL TEXT: This message is intended only for the use >>>>>> of the >>>>>> individual or entity to which it is addressed and may contain >>>>>> information which is privileged, confidential, proprietary, or exempt >>>>>> from disclosure under applicable law. If you are not the intended >>>>>> recipient or the person responsible for delivering the message to the >>>>>> intended recipient, you are strictly prohibited from disclosing, >>>>>> distributing, copying, or in any way using this message. If you have >>>>>> received this communication in error, please notify the sender and >>>>>> destroy and delete any copies you may have received. >>>>>> >>>>>> http://www.bsc.es/disclaimer<spawn_multiple.c>__ >>>>>> <http://www.bsc.es/disclaimer%3Cspawn_multiple.c%3E__>_____________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> -- >>>> Antonio J. Peï¿½a >>>> Senior Researcher >>>> Barcelona Supercomputing Center >>>> >>>> >>>> WARNING / LEGAL TEXT: This message is intended only for the use of the >>>> individual or entity to which it is addressed and may contain >>>> information which is privileged, confidential, proprietary, or exempt >>>> from disclosure under applicable law. If you are not the intended >>>> recipient or the person responsible for delivering the message to the >>>> intended recipient, you are strictly prohibited from disclosing, >>>> distributing, copying, or in any way using this message. If you have >>>> received this communication in error, please notify the sender and >>>> destroy and delete any copies you may have received. >>>> >>>> http://www.bsc.es/disclaimer >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Antonio J. Peï¿½a >> Senior Researcher >> Barcelona Supercomputing Center >> >> >> WARNING / LEGAL TEXT: This message is intended only for the use of the >> individual or entity to which it is addressed and may contain >> information which is privileged, confidential, proprietary, or exempt >> from disclosure under applicable law. If you are not the intended >> recipient or the person responsible for delivering the message to the >> intended recipient, you are strictly prohibited from disclosing, >> distributing, copying, or in any way using this message. If you have >> received this communication in error, please notify the sender and >> destroy and delete any copies you may have received. >> >> http://www.bsc.es/disclaimer >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peï¿½a Senior Researcher Barcelona Supercomputing Center WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] error spawning processes in mpich-3.2rc1
by Min Si 12 Oct '15

12 Oct '15

Sure. Please send to msi(a)il.is.s.u-tokyo.ac.jp. Min On 10/12/15 9:24 AM, Siegmar Gross wrote: > Hi Min, > >> It seems you already enabled the most detailed error outputs. We could >> not think out any clue for now. If you can give us access to your >> machine, we are glad to help you debug on it. > > Can you send me your email address because I don't want to send > login data to this list. > > > Kind regards > > Siegmar > > >> >> Min >> >> On 10/8/15 12:02 AM, Siegmar Gross wrote: >>> Hi Min, >>> >>> thank you very much for your answer. >>> >>>> We cannot reproduce this error on our test machines (Solaris i386, >>>> Ubuntu x86_64) by using your programs. And unfortunately we do not >>>> have >>>> Solaris Sparc machine thus could not verify it. >>> >>> The programs work fine on my Solaris x86_64 and Linux machines >>> as well. I only have a problem on Solaris Sparc. >>> >>> >>>> Sometime, it can happen that you need to add "./" in front of the >>>> program path, could you try it ? >>>> >>>> For example, in spawn_master.c MPI: A Message-Passing Interface >>>> Standard >>>>> #define SLAVE_PROG "./spawn_slave" >>> >>> No, it wil not work, because the programs are stored in a >>> different directory ($HOME/{SunOS, Linux}/{sparc, x86_64}/bin) >>> which is part of PATH (as well as "."). >>> >>> Can I do anything to track the source of the error? >>> >>> >>> Kind regards >>> >>> Siegmar >>> >>>> >>>> Min >>>> >>>> On 10/7/15 5:03 AM, Siegmar Gross wrote: >>>>> Hi, >>>>> >>>>> today I've built mpich-3.2rc1 on my machines (Solaris 10 Sparc, >>>>> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-5.1.0 >>>>> and Sun C 5.13. I still get the following errors on my Sparc machine >>>>> which I'd already reported September 8th. "mpiexec" is aliased to >>>>> 'mpiexec -genvnone'. It still doesn't matter if I use my cc- or >>>>> gcc-version of MPICH. >>>>> >>>>> >>>>> tyr spawn 119 mpichversion >>>>> MPICH Version: 3.2rc1 >>>>> MPICH Release date: Wed Oct 7 00:00:33 CDT 2015 >>>>> MPICH Device: ch3:nemesis >>>>> MPICH configure: --prefix=/usr/local/mpich-3.2_64_cc >>>>> --libdir=/usr/local/mpich-3.2_64_cc/lib64 >>>>> --includedir=/usr/local/mpich-3.2_64_cc/include64 CC=cc CXX=CC >>>>> F77=f77 >>>>> FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 >>>>> LDFLAGS=-m64 >>>>> -L/usr/lib/sparcv9 -R/usr/lib/sparcv9 --enable-fortran=yes >>>>> --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll >>>>> --enable-threads=multiple --with-thread-package=posix --enable-shared >>>>> MPICH CC: cc -m64 -O2 >>>>> MPICH CXX: CC -m64 -O2 >>>>> MPICH F77: f77 -m64 >>>>> MPICH FC: f95 -m64 -O2 >>>>> tyr spawn 120 >>>>> >>>>> >>>>> >>>>> tyr spawn 111 mpiexec -np 1 spawn_master >>>>> >>>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>>> I create 4 slave processes >>>>> >>>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_slave", >>>>> argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>>> intercomm=ffffffff7fffde50, errors=0) failed >>>>> MPIDI_Comm_spawn_multiple(274): >>>>> MPID_Comm_accept(153).........: >>>>> MPIDI_Comm_accept(1057).......: >>>>> MPIR_Bcast_intra(1287)........: >>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>> >>>>> >>>>> >>>>> >>>>> tyr spawn 112 mpiexec -np 1 spawn_multiple_master >>>>> >>>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>>> I create 3 slave processes. >>>>> >>>>> Fatal error in MPI_Comm_spawn_multiple: Unknown error class, error >>>>> stack: >>>>> MPI_Comm_spawn_multiple(162)..: MPI_Comm_spawn_multiple(count=2, >>>>> cmds=ffffffff7fffde08, argvs=ffffffff7fffddf8, >>>>> maxprocs=ffffffff7fffddf0, infos=ffffffff7fffdde8, root=0, >>>>> MPI_COMM_WORLD, intercomm=ffffffff7fffdde4, errors=0) failed >>>>> MPIDI_Comm_spawn_multiple(274): >>>>> MPID_Comm_accept(153).........: >>>>> MPIDI_Comm_accept(1057).......: >>>>> MPIR_Bcast_intra(1287)........: >>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>> >>>>> >>>>> >>>>> >>>>> tyr spawn 113 mpiexec -np 1 spawn_intra_comm >>>>> Parent process 0: I create 2 slave processes >>>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>>> MPI_Comm_spawn(144)...........: >>>>> MPI_Comm_spawn(cmd="spawn_intra_comm", >>>>> argv=0, maxprocs=2, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>>> intercomm=ffffffff7fffded4, errors=0) failed >>>>> MPIDI_Comm_spawn_multiple(274): >>>>> MPID_Comm_accept(153).........: >>>>> MPIDI_Comm_accept(1057).......: >>>>> MPIR_Bcast_intra(1287)........: >>>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>>> tyr spawn 114 >>>>> >>>>> >>>>> I would be grateful if somebody can fix the problem. Thank you very >>>>> much for any help in advance. I've attached my programs. Please let >>>>> me know if you need anything else. >>>>> >>>>> >>>>> Kind regards >>>>> >>>>> Siegmar >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing listdiscuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] error spawning processes in mpich-3.2rc1
by Siegmar Gross 12 Oct '15

12 Oct '15

Hi Min, > It seems you already enabled the most detailed error outputs. We could > not think out any clue for now. If you can give us access to your > machine, we are glad to help you debug on it. Can you send me your email address because I don't want to send login data to this list. Kind regards Siegmar > > Min > > On 10/8/15 12:02 AM, Siegmar Gross wrote: >> Hi Min, >> >> thank you very much for your answer. >> >>> We cannot reproduce this error on our test machines (Solaris i386, >>> Ubuntu x86_64) by using your programs. And unfortunately we do not have >>> Solaris Sparc machine thus could not verify it. >> >> The programs work fine on my Solaris x86_64 and Linux machines >> as well. I only have a problem on Solaris Sparc. >> >> >>> Sometime, it can happen that you need to add "./" in front of the >>> program path, could you try it ? >>> >>> For example, in spawn_master.c MPI: A Message-Passing Interface Standard >>>> #define SLAVE_PROG "./spawn_slave" >> >> No, it wil not work, because the programs are stored in a >> different directory ($HOME/{SunOS, Linux}/{sparc, x86_64}/bin) >> which is part of PATH (as well as "."). >> >> Can I do anything to track the source of the error? >> >> >> Kind regards >> >> Siegmar >> >>> >>> Min >>> >>> On 10/7/15 5:03 AM, Siegmar Gross wrote: >>>> Hi, >>>> >>>> today I've built mpich-3.2rc1 on my machines (Solaris 10 Sparc, >>>> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-5.1.0 >>>> and Sun C 5.13. I still get the following errors on my Sparc machine >>>> which I'd already reported September 8th. "mpiexec" is aliased to >>>> 'mpiexec -genvnone'. It still doesn't matter if I use my cc- or >>>> gcc-version of MPICH. >>>> >>>> >>>> tyr spawn 119 mpichversion >>>> MPICH Version: 3.2rc1 >>>> MPICH Release date: Wed Oct 7 00:00:33 CDT 2015 >>>> MPICH Device: ch3:nemesis >>>> MPICH configure: --prefix=/usr/local/mpich-3.2_64_cc >>>> --libdir=/usr/local/mpich-3.2_64_cc/lib64 >>>> --includedir=/usr/local/mpich-3.2_64_cc/include64 CC=cc CXX=CC F77=f77 >>>> FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 LDFLAGS=-m64 >>>> -L/usr/lib/sparcv9 -R/usr/lib/sparcv9 --enable-fortran=yes >>>> --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll >>>> --enable-threads=multiple --with-thread-package=posix --enable-shared >>>> MPICH CC: cc -m64 -O2 >>>> MPICH CXX: CC -m64 -O2 >>>> MPICH F77: f77 -m64 >>>> MPICH FC: f95 -m64 -O2 >>>> tyr spawn 120 >>>> >>>> >>>> >>>> tyr spawn 111 mpiexec -np 1 spawn_master >>>> >>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>> I create 4 slave processes >>>> >>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_slave", >>>> argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>> intercomm=ffffffff7fffde50, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> >>>> >>>> >>>> >>>> tyr spawn 112 mpiexec -np 1 spawn_multiple_master >>>> >>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>> I create 3 slave processes. >>>> >>>> Fatal error in MPI_Comm_spawn_multiple: Unknown error class, error >>>> stack: >>>> MPI_Comm_spawn_multiple(162)..: MPI_Comm_spawn_multiple(count=2, >>>> cmds=ffffffff7fffde08, argvs=ffffffff7fffddf8, >>>> maxprocs=ffffffff7fffddf0, infos=ffffffff7fffdde8, root=0, >>>> MPI_COMM_WORLD, intercomm=ffffffff7fffdde4, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> >>>> >>>> >>>> >>>> tyr spawn 113 mpiexec -np 1 spawn_intra_comm >>>> Parent process 0: I create 2 slave processes >>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_intra_comm", >>>> argv=0, maxprocs=2, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>> intercomm=ffffffff7fffded4, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> tyr spawn 114 >>>> >>>> >>>> I would be grateful if somebody can fix the problem. Thank you very >>>> much for any help in advance. I've attached my programs. Please let >>>> me know if you need anything else. >>>> >>>> >>>> Kind regards >>>> >>>> Siegmar >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing listdiscuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Can the MPICH job pause and resumed
by Jeff Hammond 10 Oct '15

10 Oct '15

Yeah, I think checkpoint-restart is probably the best way to do what this user wants. Jeff On Thursday, October 8, 2015, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 10/08/2015 08:18 AM, Jeff Hammond wrote: > >> Nodes and cores are *hardware* concepts. If you need to add hardware >> resources to your job, that is a resource manager issue. If you want to >> create new MPI processes _that will run on those new nodes/cores_, you >> can use MPI_Comm_spawn for that. >> >> As for pausing a job, I'm not sure why you want to do that. Are you >> trying to suspect the job until the new hardware resources become >> available? >> > > Probably not what Wahi has in mind, but back on the old Crays (old > vector-era crays, I mean), when checkpoint/restart was an os-level service, > the job scheduler used that feature: new job comes in with higher > priority, checkpoint the old job. when new job finishes, restart old job. > > ==rob > > >> On Wed, Oct 7, 2015 at 10:08 PM, wahi <wahi(a)sci.am <mailto:[email protected]>> >> wrote: >> >> Hi Jeff, >> >> Thank you for your reply, I think the MPI_COMM_Spawn is for >> generating new processes but I need to add more node or cores after >> pausing the job, am I right ? >> >> >> On 10/08/2015 08:11 AM, Jeff Hammond wrote: >> >>> See MPI_Comm_spawn(_multiple). >>> >>> Jeff >>> >>> On Wednesday, October 7, 2015, Zhao, Xin >>> <<mailto:[email protected]>xinzhao3(a)illinois.edu >>> <mailto:[email protected]>> wrote: >>> >>> Hi Wahi, >>> >>> Do you mean you want to change number of hosts during one MPI >>> execution? If so, we think that is not possible. >>> >>> Xin >>> ________________________________________ >>> From: wahi [wahi(a)sci.am] >>> Sent: Wednesday, October 07, 2015 2:43 AM >>> To: discuss(a)mpich.org >>> Subject: [mpich-discuss] Can the MPICH job pause and resumed >>> >>> Hi, >>> >>> I would like to know if there is possibility to pause the >>> MPICH job and >>> restart it with more node numbers ? >>> >>> >>> >>> Thanks in advance for any help or suggestion. >>> >>> >>> >>> >>> >>> Regards, >>> Wahi >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com <mailto:[email protected]> >>> http://jeffhammond.github.io/ >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue with MPI_Comm_spawn_multiple
by Seo, Sangmin 09 Oct '15

09 Oct '15

Hi Toni, We figured out the reason (the spawned processes hang because when the "hosts" info keys for two commands have the same value (e.g., localhost) two processes get the same rank.), but it seems it will take some time to fix this. Pavan will rewrite the problematic part in hydra. I’ve created a ticket (https://trac.mpich.org/projects/mpich/ticket/2308) for this issue and cc’ed you in the ticket. Regards, Sangmin On Oct 8, 2015, at 10:17 AM, Antonio J. Peña <antonio.pena(a)bsc.es<mailto:[email protected]>> wrote: Thanks a lot Sangmin! On 10/08/2015 03:43 PM, Seo, Sangmin wrote: Hi Toni, I confirmed the problem and am looking into it. I will get back to you after I resolve it. Regards, Sangmin On Oct 6, 2015, at 7:23 AM, Antonio J. Peña <antonio.pena(a)bsc.es<mailto:[email protected]>> wrote: Right, but I do need to use the hosts key to get control of where the processes are spawned. Any clue? On 10/02/2015 07:32 PM, Thakur, Rajeev wrote: Looks like it is related to the hosts key. Works without that. Rajeev On Oct 2, 2015, at 10:34 AM, Antonio J. Peña <antonio.pena(a)bsc.es<mailto:[email protected]>> wrote: Hi folks! I've been facing an issue with MPI_Comm_spawn_multiple when all the following conditions are met: 1. 'count' is > 1 2. Any value of 'array_of_maxprocs' is > 1 3. Setting the "hosts" info key What I'm finding is that the spawned processes hang at MPI_Init. Find attached a test case. You can reproduce my issue with -np 1. Is there anything I'm doing wrong? I'm using MPICH master at 1e6c4d8. I haven't found a pending ticket, xfail test, or test case covering this situation. All the best, Toni [P.S.: It still feels weird to be on the other side of discuss(a)mpich.org<mailto:[email protected]>] -- Antonio J. Peña Senior Researcher Barcelona Supercomputing Center WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer<spawn_multiple.c>_______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Senior Researcher Barcelona Supercomputing Center WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Senior Researcher Barcelona Supercomputing Center WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] error spawning processes in mpich-3.2rc1
by Siegmar Gross 09 Oct '15

09 Oct '15

Hi Min, thank you very much for your answer. > It seems you already enabled the most detailed error outputs. We could > not think out any clue for now. If you can give us access to your > machine, we are glad to help you debug on it. I'll check on Monday if our admins allow it and let you know the answer as soon as possible. Kind regards Siegmar > > Min > > On 10/8/15 12:02 AM, Siegmar Gross wrote: >> Hi Min, >> >> thank you very much for your answer. >> >>> We cannot reproduce this error on our test machines (Solaris i386, >>> Ubuntu x86_64) by using your programs. And unfortunately we do not have >>> Solaris Sparc machine thus could not verify it. >> >> The programs work fine on my Solaris x86_64 and Linux machines >> as well. I only have a problem on Solaris Sparc. >> >> >>> Sometime, it can happen that you need to add "./" in front of the >>> program path, could you try it ? >>> >>> For example, in spawn_master.c MPI: A Message-Passing Interface Standard >>>> #define SLAVE_PROG "./spawn_slave" >> >> No, it wil not work, because the programs are stored in a >> different directory ($HOME/{SunOS, Linux}/{sparc, x86_64}/bin) >> which is part of PATH (as well as "."). >> >> Can I do anything to track the source of the error? >> >> >> Kind regards >> >> Siegmar >> >>> >>> Min >>> >>> On 10/7/15 5:03 AM, Siegmar Gross wrote: >>>> Hi, >>>> >>>> today I've built mpich-3.2rc1 on my machines (Solaris 10 Sparc, >>>> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-5.1.0 >>>> and Sun C 5.13. I still get the following errors on my Sparc machine >>>> which I'd already reported September 8th. "mpiexec" is aliased to >>>> 'mpiexec -genvnone'. It still doesn't matter if I use my cc- or >>>> gcc-version of MPICH. >>>> >>>> >>>> tyr spawn 119 mpichversion >>>> MPICH Version: 3.2rc1 >>>> MPICH Release date: Wed Oct 7 00:00:33 CDT 2015 >>>> MPICH Device: ch3:nemesis >>>> MPICH configure: --prefix=/usr/local/mpich-3.2_64_cc >>>> --libdir=/usr/local/mpich-3.2_64_cc/lib64 >>>> --includedir=/usr/local/mpich-3.2_64_cc/include64 CC=cc CXX=CC F77=f77 >>>> FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 LDFLAGS=-m64 >>>> -L/usr/lib/sparcv9 -R/usr/lib/sparcv9 --enable-fortran=yes >>>> --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll >>>> --enable-threads=multiple --with-thread-package=posix --enable-shared >>>> MPICH CC: cc -m64 -O2 >>>> MPICH CXX: CC -m64 -O2 >>>> MPICH F77: f77 -m64 >>>> MPICH FC: f95 -m64 -O2 >>>> tyr spawn 120 >>>> >>>> >>>> >>>> tyr spawn 111 mpiexec -np 1 spawn_master >>>> >>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>> I create 4 slave processes >>>> >>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_slave", >>>> argv=0, maxprocs=4, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>> intercomm=ffffffff7fffde50, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> >>>> >>>> >>>> >>>> tyr spawn 112 mpiexec -np 1 spawn_multiple_master >>>> >>>> Parent process 0 running on tyr.informatik.hs-fulda.de >>>> I create 3 slave processes. >>>> >>>> Fatal error in MPI_Comm_spawn_multiple: Unknown error class, error >>>> stack: >>>> MPI_Comm_spawn_multiple(162)..: MPI_Comm_spawn_multiple(count=2, >>>> cmds=ffffffff7fffde08, argvs=ffffffff7fffddf8, >>>> maxprocs=ffffffff7fffddf0, infos=ffffffff7fffdde8, root=0, >>>> MPI_COMM_WORLD, intercomm=ffffffff7fffdde4, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> >>>> >>>> >>>> >>>> tyr spawn 113 mpiexec -np 1 spawn_intra_comm >>>> Parent process 0: I create 2 slave processes >>>> Fatal error in MPI_Comm_spawn: Unknown error class, error stack: >>>> MPI_Comm_spawn(144)...........: MPI_Comm_spawn(cmd="spawn_intra_comm", >>>> argv=0, maxprocs=2, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, >>>> intercomm=ffffffff7fffded4, errors=0) failed >>>> MPIDI_Comm_spawn_multiple(274): >>>> MPID_Comm_accept(153).........: >>>> MPIDI_Comm_accept(1057).......: >>>> MPIR_Bcast_intra(1287)........: >>>> MPIR_Bcast_binomial(310)......: Failure during collective >>>> tyr spawn 114 >>>> >>>> >>>> I would be grateful if somebody can fix the problem. Thank you very >>>> much for any help in advance. I've attached my programs. Please let >>>> me know if you need anything else. >>>> >>>> >>>> Kind regards >>>> >>>> Siegmar >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing listdiscuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] I do not succeed to correctly setup password-less ssh
by William Scullin 09 Oct '15

09 Oct '15

Hi Moshe, Your question may be a bit off-topic for this list, but others can correct me either way. I think the tutorial you found is missing a couple of steps and needs correction. SSH relies on public key cryptography for authentication. The id_rsa file is a user's private key. The id_rsa.pub file is the corresponding public key. The authorized_keys file is a listing of public user keys which are trusted to access an account and should be readable by the account. The known_hosts file is a listing of hosts' public keys that allow ssh clients to authenticate server identity. My short instructions for setting up user public key authentication might read like: 1. On node00, create passphraseless ssh keys that you won't use outside of this cluster, the -f option allows you to specify the name of the keys: $ cd ~/.ssh/ $ ssh-keygen -f cluster_only_key Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in cluster. Your public key has been saved in cluster.pub. The key fingerprint is: f1:a4:56:b7:31:ae:e9:22:53:47:70:28:b1:d0:40:55 wscullin(a)alcfwl197.alcf.anl.gov The key's randomart image is: +--[ RSA 2048]----+ | .++ooE. | | .o.o . | | ...oo + | | *.o + | | S.. o | | .. .o | | . .o | | o .. | | o .. | +-----------------+ 2. Still on node00 in the ~/.ssh directory set up your ssh_config and authorized_keys files, for more information about the .ssh/config file look at the ssh_config man page: $ echo "IdentityFile ~/.ssh/cluster_only_key" >> config $ cat cluster_only_key >> authorized_keys $ chmod 600 config authorized_keys 3. Test functionality by sshing to the current host: $ ssh -o StrictHostKeyChecking=no localhost node00 date if it fails, adding the -v option more or one times will provide debugging output. The option: -o StrictHostKeyChecking=no localhost will automatically add the host's public key to your known_hosts file without querying you. Use this only when you are sure about the security and identity of hosts. If you're lucky whoever setup the cluster already put all the nodes in the system-wide known_hosts file. 4. Propagate your authorized_keys, cluster_only_key, cluster_only_key.pub, and config file to every node in the cluster. IMPORTANT NOTE: Most clusters have home directories mounted on all nodes over the network via NFS or some parallel file system which makes this automatic. If you do not have home directories and other shared storage mounted globally over the network to all nodes, this will make running MPI applications very difficult later as as binaries and files needed to run will need to be staged on all nodes. If you don't have shared storage, ask your systems administrator to rectify this post-haste. For the manually minded: $ cd ~/.ssh $ scp -rp ~/.ssh/* node01:~/.ssh/ $ scp -rp ~/.ssh/* node02:~/.ssh/ ... 5. Once all nodes have been ssh'ed into, propagate your known_hosts to all nodes in the cluster. 6. Test all to all: for i in miralac{1..4}; \ do \ ssh $i 'echo "testing from $(hostname)"; \ for k in miralac{1..4}; \ do\ ssh $k "echo \$(hostname) connected";\ done'; \ done I hope this helps, William On Fri, Oct 9, 2015 at 2:57 AM, Moshe Goldstein <goldmosh(a)g.jct.ac.il> wrote: > > Hi! > > I am new with MPI. > First I tried an MPI program in a multicore Linux computer, and it worked > OK. > My current step is to try to run the same program in a "cluster" composed by > two multicore Linux computers, when my final goal is to have a cluster > composed by 16 computers (actually, a 64-cores cluster). > I learned that in order to run such a program I need to prevent ssh password > requirement when nodes send and/or receive messages among each other. I > found the "How to set up SSH Keys" tutorial and setup the two-computer > "cluster" according to what is explained in the tutorial. > > The result of the 4-steps tutorial procedure is as follows: > > the .ssh directory of the first computer has the following files: id_rsa, > id_rsa.sub, and known_hosts > the .ssh directory of the second computer has the following files: > authorized_keys, and known_hosts > If I do by-hand ssh from the first computer to the second, the second > computer does not require password. > If I do by-hand ssh from the second computer to the first, the first > computer requires password. > > Because of this, I guess, when I run the MPI program, it gets stuck when an > ssh connection is tried between the second computer to the first one. Right? > > Thus, I request your help to correctly setup password-less ssh capability in > the first computer, in order to allow password-less ssh among all the nodes > in the cluster (currently, among only two computers). Actually, the question > is how I allow password-less ssh connection to the first computer. > > I thank you all of you in advance for your help. > Sincerely, > -- Moshe Goldstein > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0