October 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich-master: final link failed: nonrepresentable section on output
by Madhu, Kavitha Tiptur 31 Oct '18

31 Oct '18

Hi Siegmar Sorry I was away for some time. I have created an issue(https://github.com/pmodels/mpich/issues/3358) and am looking into it. I will get back to you with updates as soon as possible. Sorry about the delays. On Oct 31, 2018, at 2:57 AM, Siegmar Gross via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I tried to build mpich-master-v3.3b3-357-gdcd523e and the latest version of mpich-master from github on my "SUSE Linux Enterprise Server 12.3 (x86_64)" with Sun C 5.15 (Oracle Developer Studio 12.6). Unfortunately, I get the following error for both versions. loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 138 tail -30 log.make.Linux.x86_64.64_cc | grep -v NOTICE "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 122: warning: null dimension: indirect CC src/pmi/simple/lib_libmpi_la-simple_pmiutil.lo CC src/pmi/simple/lib_libmpi_la-simple_pmi.lo CC src/mpi/debugger/src_mpi_debugger_libdbginitdummy_la-dbginit.lo "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 98: warning: parameter in inline asm statement unused: %2 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 105: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 112: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 121: warning: parameter in inline asm statement unused: %2 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 128: warning: parameter in inline asm statement unused: %3 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 140: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 148: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 155: warning: parameter in inline asm statement unused: %3 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 162: warning: parameter in inline asm statement unused: %3 "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 61: warning: null dimension: indirect "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 122: warning: null dimension: indirect CCLD src/mpi/debugger/libdbginitdummy.la<http://libdbginitdummy.la> GEN lib/libmpi.la<http://libmpi.la> f90: Warning: Option -shared passed to ld, if ld is invoked, ignored otherwise /usr/bin/ld: /opt/sun/developerstudio12.6/lib/compilers/amd64/crt1x.o: relocation R_X86_64_32 against hidden symbol `__dso_handle' can not be used when making a shared object /usr/bin/ld: final link failed: nonrepresentable section on output postopt: error: ld failed to link the binary Makefile:17007: recipe for target 'lib/libmpi.la<http://libmpi.la>' failed make[2]: *** [lib/libmpi.la<http://libmpi.la>] Error 2 make[2]: Leaving directory '/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc' Makefile:40517: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc' Makefile:10583: recipe for target 'all' failed make: *** [all] Error 2 loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 138 loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 141 head -7 config.log | tail -1 $ ../mpich-master-v3.3b3-357-gdcd523e/configure --prefix=/usr/local/mpich-3.3_64_cc --libdir=/usr/local/mpich-3.3_64_cc/lib64 --includedir=/usr/local/mpich-3.3_64_cc/include64 CC=cc CXX=CC F77=f77 FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 LDFLAGS=-m64 -L/usr/lib64 -R/usr/lib64 --enable-fortran=yes --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll --enable-threads=multiple --with-thread-package=posix --enable-shared loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 142 I would be grateful, if somebody can fix the problem. Do you need anything else? Thank you very much for any help in advance. Kind regards Siegmar _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich-master: final link failed: nonrepresentable section on output
by Siegmar Gross 31 Oct '18

31 Oct '18

Hi, I tried to build mpich-master-v3.3b3-357-gdcd523e and the latest version of mpich-master from github on my "SUSE Linux Enterprise Server 12.3 (x86_64)" with Sun C 5.15 (Oracle Developer Studio 12.6). Unfortunately, I get the following error for both versions. loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 138 tail -30 log.make.Linux.x86_64.64_cc | grep -v NOTICE "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 122: warning: null dimension: indirect CC src/pmi/simple/lib_libmpi_la-simple_pmiutil.lo CC src/pmi/simple/lib_libmpi_la-simple_pmi.lo CC src/mpi/debugger/src_mpi_debugger_libdbginitdummy_la-dbginit.lo "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 98: warning: parameter in inline asm statement unused: %2 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 105: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 112: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 121: warning: parameter in inline asm statement unused: %2 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 128: warning: parameter in inline asm statement unused: %3 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 140: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 148: warning: parameter in inline asm statement unused: %1 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 155: warning: parameter in inline asm statement unused: %3 "/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e/src/openpa/src/primitives/opa_gcc_intel_32_64_ops.h", line 162: warning: parameter in inline asm statement unused: %3 "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 61: warning: null dimension: indirect "../mpich-master-v3.3b3-357-gdcd523e/src/include/mpir_handlemem.h", line 122: warning: null dimension: indirect CCLD src/mpi/debugger/libdbginitdummy.la GEN lib/libmpi.la f90: Warning: Option -shared passed to ld, if ld is invoked, ignored otherwise /usr/bin/ld: /opt/sun/developerstudio12.6/lib/compilers/amd64/crt1x.o: relocation R_X86_64_32 against hidden symbol `__dso_handle' can not be used when making a shared object /usr/bin/ld: final link failed: nonrepresentable section on output postopt: error: ld failed to link the binary Makefile:17007: recipe for target 'lib/libmpi.la' failed make[2]: *** [lib/libmpi.la] Error 2 make[2]: Leaving directory '/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc' Makefile:40517: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/export2/src/mpich-3.3/mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc' Makefile:10583: recipe for target 'all' failed make: *** [all] Error 2 loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 138 loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 141 head -7 config.log | tail -1 $ ../mpich-master-v3.3b3-357-gdcd523e/configure --prefix=/usr/local/mpich-3.3_64_cc --libdir=/usr/local/mpich-3.3_64_cc/lib64 --includedir=/usr/local/mpich-3.3_64_cc/include64 CC=cc CXX=CC F77=f77 FC=f95 CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 LDFLAGS=-m64 -L/usr/lib64 -R/usr/lib64 --enable-fortran=yes --enable-cxx --enable-romio --enable-debuginfo --enable-smpcoll --enable-threads=multiple --with-thread-package=posix --enable-shared loki mpich-master-v3.3b3-357-gdcd523e-Linux.x86_64.64_cc 142 I would be grateful, if somebody can fix the problem. Do you need anything else? Thank you very much for any help in advance. Kind regards Siegmar

1 0

[BUG REPORT] /usr/local/lib/libmpicxx.so.12: undefined reference to `MPIR_Keyval_set_proxy'
by Yuri 31 Oct '18

31 Oct '18

The mpich-3.2.1 built in the FreeBSD port has this problem of missing function when other project tries to link with it: /usr/bin/c++ -O2 -pipe -fno-omit-frame-pointer -fstack-protector -fno-strict-aliasing -fno-omit-frame-pointer -ffast-math -pthread -std=c++11 -O2 -pipe -fno-omit-frame-pointer -fstack-protector -fno-strict-aliasing -fno-omit-frame-pointer /usr/local/lib/libmpir.so -Wl,-rpath=/usr/local/lib/gcc7 -L/usr/local/lib/gcc7 -B/usr/local/bin -fstack-protector CMakeFiles/ssages.dir/src/main.cpp.o CMakeFiles/ssages.dir/hooks/lammps/Driver.cpp.o -o ssages -L/usr/local/mpi/openmpi/lib/libmpi_cxx.so -L/usr/local/mpi/openmpi/lib/libmpi.so -L/usr/ports/science/ssages/work/SSAGES-public-0.8.3-alpha/_lammps_/src -Wl,-rpath,/usr/local/mpi/openmpi/lib/libmpi_cxx.so:/usr/local/mpi/openmpi/lib/libmpi.so:/usr/ports/science/ssages/work/SSAGES-public-0.8.3-alpha/_lammps_/src:/usr/local/mpi/openmpi/lib libssages.a -llammps_mpi /usr/local/mpi/openmpi/lib/libmpi_cxx.so /usr/local/mpi/openmpi/lib/libmpi.so CMakeFiles/ssages.dir/hooks/lammps/Driver.cpp.o: In function `SSAGES::Driver::Run()': Driver.cpp:(.text+0x3f): undefined reference to `LAMMPS_NS::LAMMPS::LAMMPS(int, char**, ompi_communicator_t*)' //usr/local/lib/libmpicxx.so.12: undefined reference to `MPIR_Keyval_set_proxy' //usr/local/lib/libmpicxx.so.12: undefined reference to `MPIR_Errhandler_set_cxx' //usr/local/lib/libmpicxx.so.12: undefined reference to `MPIR_Op_set_cxx' These functions are either missing from this library, or it isn't linked to some ther library. Yuri

1 0

Re: [mpich-discuss] Maximum number of processes in compute farm
by Guo, Yanfei 29 Oct '18

29 Oct '18

Hi Shuwei, The MPICH library does not implement a hard limit on the number of ranks. That being said, the rank is stored as a 32bit integer, therefore subject to the limit of MAX_INT. In practice, it is more likely that the library will run out-of-memory before hitting that limit. MPICH is optimized for large scale use cases than contains hundreds of thousands processes, so it should be fine for your use cases. Best, Yanfei Guo Assistant Computer Scientist MCS, ANL From: Shuwei Zhao via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, October 29, 2018 at 3:43 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Shuwei Zhao <shuweizhao1991(a)gmail.com> Subject: [mpich-discuss] Maximum number of processes in compute farm Hi, all I have several questions about parallel scale of mpich: Is there a hard limit on the number of processes that mpich can start in a compute farm? As the number of process going up , for a large-scale(1000-10000) loose-communication application(the communication overhead is minimal compared to the runtime), is it possible that the performance of the mpich will go down because the message passing will take longer time? Thank you very much, Shuwei

1 0

Re: [mpich-discuss] MPICH interact with RTDA
by Shuwei Zhao 29 Oct '18

29 Oct '18

Hi hui Thank you very much for the detailed answer. I learned a lot. Br Shuwei On Mon, Oct 29, 2018 at 14:51 Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Shuwei, > > Once the proxy is started, it will be communicating to the control > (mpiexec) and all pmi interface works as usual — i.e. nothing special. > > With manual mode, the responsibility of detecting launch failures is on > you (as implied by manual mode). Your custom solution need to detect launch > failures, then either re-launch or abort. With re-launch, as long as the > proxy eventually start to run, hydra won’t care and function the same. If > you decide to abort, then I guess you can simply kill hydra. > > — > Hui Zhou > > > > > > > On Oct 29, 2018, at 12:09 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com> > wrote: > > Hi hui > > thanks for your response. I tried the manual mode on a simply hello world > mpi application, it works as expected. > You are pretty mind-reading. I have two follow up questions, one is that > as you said how can we detect launch failures? > another one is that except the launching part, all other things like fault > tolerance, recovery, etc will be all the same under manual mode as other > modes, nothing special under manual mode right? > > Thanks > Shuwei > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Maximum number of processes in compute farm
by Shuwei Zhao 29 Oct '18

29 Oct '18

Hi, all I have several questions about parallel scale of mpich: Is there a hard limit on the number of processes that mpich can start in a compute farm? As the number of process going up , for a large-scale(1000-10000) loose-communication application(the communication overhead is minimal compared to the runtime), is it possible that the performance of the mpich will go down because the message passing will take longer time? Thank you very much, Shuwei

1 0

Re: [mpich-discuss] MPICH interact with RTDA
by Zhou, Hui 29 Oct '18

29 Oct '18

Hi Shuwei, Once the proxy is started, it will be communicating to the control (mpiexec) and all pmi interface works as usual — i.e. nothing special. With manual mode, the responsibility of detecting launch failures is on you (as implied by manual mode). Your custom solution need to detect launch failures, then either re-launch or abort. With re-launch, as long as the proxy eventually start to run, hydra won’t care and function the same. If you decide to abort, then I guess you can simply kill hydra. — Hui Zhou On Oct 29, 2018, at 12:09 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com<mailto:[email protected]>> wrote: Hi hui thanks for your response. I tried the manual mode on a simply hello world mpi application, it works as expected. You are pretty mind-reading. I have two follow up questions, one is that as you said how can we detect launch failures? another one is that except the launching part, all other things like fault tolerance, recovery, etc will be all the same under manual mode as other modes, nothing special under manual mode right? Thanks Shuwei

1 0

Re: [mpich-discuss] MPICH interact with RTDA
by Zhou, Hui 29 Oct '18

29 Oct '18

Hi Shuwei, Once the proxy is started, it will be communicating to the control (mpiexec) and all pmi interface works as usual — i.e. nothing special. With manual mode, the responsibility of detecting launch failures is on you (as implied by manual mode). Your custom solution need to detect launch failures, then either re-launch or abort. With re-launch, as long as the proxy eventually start to run, hydra won’t care and function the same. If you decide to abort, then I guess you can simply kill hydra. — Hui Zhou On Oct 29, 2018, at 12:09 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com<mailto:[email protected]>> wrote: Hi hui thanks for your response. I tried the manual mode on a simply hello world mpi application, it works as expected. You are pretty mind-reading. I have two follow up questions, one is that as you said how can we detect launch failures? another one is that except the launching part, all other things like fault tolerance, recovery, etc will be all the same under manual mode as other modes, nothing special under manual mode right? Thanks Shuwei

1 0

Re: [mpich-discuss] MPICH interact with RTDA
by Shuwei Zhao 29 Oct '18

29 Oct '18

Hi hui thanks for your response. I tried the manual mode on a simply hello world mpi application, it works as expected. You are pretty mind-reading. I have two follow up questions, one is that as you said how can we detect launch failures? another one is that except the launching part, all other things like fault tolerance, recovery, etc will be all the same under manual mode as other modes, nothing special under manual mode right? Thanks Shuwei On Fri, Oct 26, 2018 at 11:29 Zhou, Hui <zhouh(a)anl.gov> wrote: > In fact there is a manual mode in hydra. Try this: > > mpiexec -n 2 -launcher=manual -host=A,B date > > It will print out the launch command lines and wait (very patiently). Then > if you paste the proxy command line on your local host or any host on your > network, it will continue. > > I guess that is all the pipelines that you can use to create your own > wrapper scripts to work for custom environment. > > — > Hui > > On Oct 26, 2018, at 8:42 AM, Zhou, Hui <zhouh(a)anl.gov> wrote: > > Having a manual mode of launching proxies sounds very interesting. I don’t > think currently hydra supports it though. > > The manually launched proxies need establish communication back to > mpiexec.hydra — I suspect the mechanism already exist. I guess the next > question is how to reliably detect launch failures other than having > mpiexec.hydra hanging forever — maybe that is a valid option. > > Hui Zhou > > On Oct 23, 2018, at 11:44 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com> > wrote: > > Hi, > > I'm trying integrate MPICH with network submission tool called > RTDA(RunTime Design Automation - www.rtda.com) I use the mpich-3.2.1a > but looks like that mpich doesn't detect the resource manager and launcher, > mpich cannot distribute jobs as expected. > > Does mpich support interact with RTDA resource manager and launcher? > > Since the hydra process manager did tight integration with SGE, LSF, > SLURM, PBS, etc. I was thinking for platform it doesn't support - if there > is way to submit jobs with loose integration? > ( loose integration means that we do the job submission self and run > hydra_pmi_proxy on each allocated node manually, instead of mpich finished > everything under the hood. > ) > > Thank you very much, > Shuwei > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH interact with RTDA
by Zhou, Hui 26 Oct '18

26 Oct '18

In fact there is a manual mode in hydra. Try this: mpiexec -n 2 -launcher=manual -host=A,B date It will print out the launch command lines and wait (very patiently). Then if you paste the proxy command line on your local host or any host on your network, it will continue. I guess that is all the pipelines that you can use to create your own wrapper scripts to work for custom environment. — Hui On Oct 26, 2018, at 8:42 AM, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Having a manual mode of launching proxies sounds very interesting. I don’t think currently hydra supports it though. The manually launched proxies need establish communication back to mpiexec.hydra — I suspect the mechanism already exist. I guess the next question is how to reliably detect launch failures other than having mpiexec.hydra hanging forever — maybe that is a valid option. Hui Zhou On Oct 23, 2018, at 11:44 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com<mailto:[email protected]>> wrote: Hi, I'm trying integrate MPICH with network submission tool called RTDA(RunTime Design Automation - www.rtda.com<http://www.rtda.com/>), I use the mpich-3.2.1a but looks like that mpich doesn't detect the resource manager and launcher, mpich cannot distribute jobs as expected. Does mpich support interact with RTDA resource manager and launcher? Since the hydra process manager did tight integration with SGE, LSF, SLURM, PBS, etc. I was thinking for platform it doesn't support - if there is way to submit jobs with loose integration? ( loose integration means that we do the job submission self and run hydra_pmi_proxy on each allocated node manually, instead of mpich finished everything under the hood. ) Thank you very much, Shuwei _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0