- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] success and failure report for mpich-3.0.2
by Pavan Balaji 28 Apr '13

28 Apr '13

This is not an architectural problem. You need to have mpich installed on all machines. It looks like some of your machines don't have it installed. -- Pavan On 03/22/2013 10:22 AM US Central Time, Siegmar Gross wrote: > Hi > > today I tried to install mpich-3.0.2. I could compile and install the > package for the following architectures. > > openSuSE Linux 12.1, x86_64, Sun C 5.12, 64-bit > Solaris 10, x86_64, Sun C 5.12, 64-bit > Solaris 10, sparc, Sun C 5.12, 32-bit > Solaris 10, sparc, Sun C 5.12, 64-bit > > As described in the RELEASE_NOTES a communication between little-endian > and big-endian machines is not possible (broken pipe). Unfortunately > you don't even allow different operating systems (it seems that you > don't evaluate PATH to find a program). > > sunpc1 fd1026 103 mpiexec -np 2 -host sunpc0,sunpc1 init_finalize > Hello! > Hello! > > sunpc1 fd1026 104 mpiexec -np 2 -host sunpc0,linpc1 init_finalize > [proxy:0:1@linpc1] HYDU_create_process > (../../../../mpich-3.0.2/src/pm/hydra/utils/launch/launch.c:74): > execvp error on file init_finalize (No such file or directory) > > ======================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 255 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > ======================================================================== > [proxy:0:0@sunpc0] HYD_pmcd_pmip_control_cmd_cb > (../../../../mpich-3.0.2/src/pm/hydra/pm/pmiserv/pmip_cb.c:886): > assert (!closed) failed > [proxy:0:0@sunpc0] HYDT_dmxu_poll_wait_for_event > (../../../../mpich-3.0.2/src/pm/hydra/tools/demux/demux_poll.c:77): > callback returned error status > [proxy:0:0@sunpc0] main > (../../../../mpich-3.0.2/src/pm/hydra/pm/pmiserv/pmip.c:206): > demux engine error waiting for event > > > > The programs are available on both machines. > > sunpc1 fd1026 105 which init_finalize > /home/fd1026/SunOS/x86_64/bin/init_finalize > sunpc1 fd1026 106 ssh linpc1 which init_finalize > /home/fd1026/Linux/x86_64/bin/init_finalize > sunpc1 fd1026 107 > > > > > I wasn't able to build the package on the following architectures. > > > openSuSE Linux 12.1, x86_64, Sun C 5.12, 32-bit > openSuSE Linux 12.1, x86_64, GNU C 4.7.1, 32-bit > ------------------------------------------------ > > tyr mpich-3.0.2-Linux.x86_64.32_cc 158 tail log.make.Linux.x86_64.32_cc > CC mplstr.lo > CC mpltrmem.lo > CC mplenv.lo > CCLD libmpl.la > /usr/lib/libxml2.so: could not read symbols: File in wrong format > make[2]: *** [libmpl.la] Error 2 > make[2]: Leaving directory `.../src/mpl' > make[1]: *** [all-recursive] Error 1 > > > > openSuSE Linux 12.1, x86_64, GNU C 4.7.1, 64-bit > ------------------------------------------------ > > tyr mpich-3.0.2-Linux.x86_64.64_gcc 171 tail log.make.Linux.x86_64.64_gcc > CC mpltrmem.lo > CC mplenv.lo > CCLD libmpl.la > /usr/local/bin/ld: libmpl.so.1: No such file: No such file or directory > collect2: error: ld returned 1 exit status > make[2]: *** [libmpl.la] Error 1 > make[2]: Leaving directory `.../src/mpl' > > > > Solaris 10, x86_64, Sun C 5.12, 32-bit > -------------------------------------- > > hangs > > > > Solaris 10, x86_64, GNU C 4.7.1, 32-bit > --------------------------------------- > > tyr mpich-3.0.2-SunOS.x86_64.32_gcc 193 tail -12 log.make.SunOS.x86_64.32_gcc > CC src/mpi/coll/barrier_group.lo > CC src/mpi/coll/helper_fns.lo > Assembler: helper_fns.c > "/var/tmp//ccIdXgrh.s", line 912 : Syntax error > Near line: "mov $PREFETCHBLOCK/16, %eax" > "/var/tmp//ccIdXgrh.s", line 920 : Syntax error > Near line: "mov $PREFETCHBLOCK/8, %eax" > make[2]: *** [src/mpi/coll/helper_fns.lo] Error 1 > > > > Solaris 10, x86_64, GNU C 4.7.1, 64-bit > Solaris 10, sparc, GNU C 4.7.1, 32-bit > Solaris 10, sparc, GNU C 4.7.1, 64-bit > --------------------------------------- > > tyr mpich-3.0.2-SunOS.x86_64.64_gcc 195 tail log.make.SunOS.x86_64.64_gcc > CXX src/binding/cxx/initcxx.lo > CXXLD lib/libmpichcxx.la > ld: fatal: file libmpichcxx.so.10: open failed: No such file or directory > ld: fatal: file processing errors. No output written to > lib/.libs/libmpichcxx.so.10.0.2 > collect2: error: ld returned 1 exit status > make[2]: *** [lib/libmpichcxx.la] Error 1 > > > Kind regards > > Siegmar > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] mpich2 support processor affinity by default
by Pavan Balaji 28 Apr '13

28 Apr '13

By default, Hydra does not do process binding to cpu cores. However, you can ask it to do binding using an mpiexec argument. See mpiexec -help for more information. -- Pavan On 03/22/2013 05:23 AM US Central Time, benzhi cao wrote: > > > 2013/3/22 benzhi cao <caobenzhi0915(a)gmail.com > <mailto:[email protected]>> > > Hi~ > when you run a program with multi-threads.(For instance ,you > just type "mpiexec -n 4 ./app" to run an app with four threads.) > ,obviously the processor manager will fork several processes to > execute the app.What I am wondering is the MPICH2 support processor > affinity by default? Does the MPI process manager(MPICH2 hydra) > will bind each forked process to a cpu automatically or just do > nothing after forking the processes and waithing for the os to > dispatch this? > I am appreciate for any reply~Thanks in advance~ > Best~ > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Hydra fills in the MPIR_proctable[] incorrectly with multiple processes per node
by Pavan Balaji 28 Apr '13

28 Apr '13

John, I've created a ticket for this so we don't forget it: https://trac.mpich.org/projects/mpich/ticket/1821 However, there's currently no ETA for this work. -- Pavan On 03/18/2013 02:23 PM US Central Time, John DelSignore wrote: > Pavan Balaji wrote: >> On 03/18/2013 11:56 AM US Central Time, John DelSignore wrote: >>> But the bigger win is that tool would have to read only 513 separate >>> strings using ptrace() or /proc, which represents a big time >>> savings. >> >> Doesn't the tool still have to read all the strings? > > No. > >> Or are you >> planning to compare pointers to check if they are the same and not read >> the string again if they are the same? > > It could be done by comparing pointers, but the way TotalView implements it is through a "target address space data cache". Before TV reads from the target address space for the process (in any target read situation, not just MPIR), it looks in a data cache associated with the target address space, and if the data is already in the cache, there is no reason to reread it from the target address space using ptrace() or /proc. > > Cheers, John D. > >> >> -- Pavan >> -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] suspend/resume an MPI process using SIGSTOP and SIGCONT
by Pavan Balaji 28 Apr '13

28 Apr '13

Not that we know of. All signals that Hydra doesn't trap are forwarded directly to the MPI processes. -- Pavan On 02/26/2013 03:15 PM US Central Time, Ali Bouaricha wrote: > Hello, > > > > I wanted to suspend a process in MPICH2 by using kill(procid, SIGSTOP) > > > > But it seems that when I do that, the whole run gets suspended. > > > > Is there any implication of kill(procid,SIGSTOP) on the remainder of > > processes? > > > > Please help. > > > > Thank you and Regards, > > > > Ali Bouaricha > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] get the process rank in the bash script
by Pavan Balaji 28 Apr '13

28 Apr '13

You can use PMI_RANK. I just tried it and it works as expected. Which version of MPICH are you using? -- Pavan On 02/25/2013 10:06 AM US Central Time, Kirill Lykov wrote: > Dear mpich users, > > I'm looking for an environmental variable which can be used in a bash script to identify the process rank. > I launch a bash script using mpi: > mpirun ./my-script.sh > > In my-script I want to use the rank of the process to identify which input file to use. In OpenMPI, I used OMPI_COMM_WORLD_RANK and my-script.sh looked like the following: > > inputName="MyInput_$OMPI_COMM_WORLD_RANK"./myexe $inputName > > What can be used in MPICH instead of OMPI_COMM_WORLD_RANK? I found somewhere that $PMI_RANK is the same but I found that it is empty. > > Kirill Lykov, Mr. > > Doctoral Assistant > Institute of > Computational Science > Faculty of Informatics > Universitï¿½ della Svizzera italiana, USI > Via G. Buffi 13 > CH - 6904 Lugano > (e) kirill.lykov(a)usi.ch > (t) +41 (0) 788413392 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] src/pm/hydra/tools/topo/hwloc/hwloc/src/topology-xml-libxml.c:224: undefined reference to `xmlReadMemory'
by Pavan Balaji 28 Apr '13

28 Apr '13

Hello, I can't reproduce this error. Can you try the latest version of mpich to see if you are still seeing this problem? -- Pavan On 02/01/2013 09:50 AM US Central Time, Sree wrote: > > hi > I got the error while running make 2nd time, > i am running CENTOS 4 x86_64 architecture kernel version 2.6.9-34.0.2 > > > -- > ======================= > Sreejith M > ======================= > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] BLCR kernel module not present
by Michael Bane 26 Apr '13

26 Apr '13

Thanks for that Raghu On 26 Apr 2013, at 21:56, Raghunath wrote: > Michael, > > The BLCR support in MVAPICH works fine as well. MVAPICH implements > its own Checkpoint-Restart mechanism for the CH3-IB and Nemesis-IB > channels. The MPICH design for the Nemesis-TCP channel is left > untouched, as Pavan indicated. > > -- > Raghu > > > On Fri, Apr 26, 2013 at 4:25 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> I can't speak for mvapich, of course. I'm only speaking about mpich. >> However, most of our derivatives don't destroy the features that are in >> stock mpich. So I'd think it'll work fine with mvapich as well. >> >> -- Pavan >> >> On 04/26/2013 03:20 PM US Central Time, michael wrote: >>> Thanks Pavan >>> >>> Just to be clear, you're saying if I use mvapich with blcr then a >>> running multi-node MPI job when killed (eg out of time) by a batch >>> scheduler can be restarted (from which checkpoint?) presuming it doesn't >>> have open files? >>> >>> Many thanks, M >>> >>> On Fri, 2013-04-26 at 15:12 -0500, Pavan Balaji wrote: >>>> Michael, >>>> >>>> BLCR support for mpich should work fine; if something is broken, please >>>> let us know. >>>> >>>> However, the core BLCR group itself hadn't released updates in a while, >>>> primarily because they didn't have direct funding for it. I believe >>>> that's fixed now and they are working on newer releases. >>>> >>>> -- Pavan >>>> >>>> On 04/26/2013 03:07 PM US Central Time, michael wrote: >>>>> Hi folks >>>>> I was wondering what the state of BLCR for mpich/mvapich is? eg how >>>>> reliably can one presume it to be? >>>>> Thanks, Michael >>>>> >>>>> >>>>> On Fri, 2013-04-26 at 14:33 -0500, Wesley Bland wrote: >>>>>> It looks like you might have missed installing the kernel module for >>>>>> BLCR. What is the output of `lsmod`? >>>>>> >>>>>> >>>>>> Alternatively, if you installed BLCR by using apt-get in Ubuntu, you >>>>>> should be able to use dkms to manage your kernel modules >>>>>> automatically. Make sure you have the package 'blcr-dkms' installed >>>>>> (you should be able to check this by typing `dims status`. >>>>>> >>>>>> >>>>>> Do either of those solutions solve your issue? >>>>>> >>>>>> >>>>>> Wesley >>>>>> >>>>>> On Apr 26, 2013, at 1:55 PM, basma a.azeem >>>>>> <basmaabdelazeem(a)hotmail.com <mailto:[email protected]> <mailto:[email protected]>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> Thank you for your help >>>>>>> >>>>>>> >>>>>>> i installed BLCR 0.8.5 on my ubuntu 12.10 to be used for MPICH -3.0.3 >>>>>>> this version of blcr should support to kernels through 3.7.1 >>>>>>> >>>>>>> when i run the command : >>>>>>> basma@basma-Satellite-A500:~$ mpiexec --info >>>>>>> >>>>>>> results: >>>>>>> >>>>>>> HYDRA build details: >>>>>>> Version: 3.0.3 >>>>>>> Release Date: Thu Mar 28 16:01:21 CDT 2013 >>>>>>> CC: gcc >>>>>>> CXX: c++ >>>>>>> F77: no >>>>>>> F90: no >>>>>>> Configure options: >>>>>>> '--disable-option-checking' '--prefix=/home/basma/mpich2-install' >>>>>>> '--disable-f77' '--disable-fc' '--enable-checkpointing' >>>>>>> '--with-hydra-ckpointlib=blcr' '--cache-file=/dev/null' '--srcdir=.' >>>>>>> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= ' 'LIBS=-lrt -lcr -lpthread ' >>>>>>> 'CPPFLAGS= -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >>>>>>> -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >>>>>>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >>>>>>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >>>>>>> -I/home/basma/libraries/mpich-3.0.3/src/mpi/romio/include' >>>>>>> Process Manager: pmi >>>>>>> Launchers available: ssh rsh fork slurm ll >>>>>>> lsf sge manual persist >>>>>>> Topology libraries available: hwloc >>>>>>> Resource management kernels available: user slurm ll lsf sge >>>>>>> pbs cobalt >>>>>>> Checkpointing libraries available: blcr >>>>>>> Demux engines available: poll select >>>>>>> >>>>>>> so i thought that every thing is ok but when i try to rum mpiexec it >>>>>>> failed: >>>>>>> >>>>>>> basma@basma-Satellite-A500:~$ mpiexec -ckpointlib blcr >>>>>>> -ckpoint-prefix /home/business/ckpts/app.ckpoint -ckpoint-interval >>>>>>> 3600 -n 4 /home/basma/libraries/mpich-3.0.3/examples/cpi >>>>>>> >>>>>>> results: >>>>>>> >>>>>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>>>>> MPIR_Init_thread(433)...: >>>>>>> MPID_Init(151)..........: channel initialization failed >>>>>>> MPIDI_CH3_Init(70)......: >>>>>>> MPID_nem_init(379)......: >>>>>>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>>>>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>>>>> MPIR_Init_thread(433)...: >>>>>>> MPID_Init(151)..........: channel initialization failed >>>>>>> MPIDI_CH3_Init(70)......: >>>>>>> MPID_nem_init(379)......: >>>>>>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>>>>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>>>>> MPIR_Init_thread(433)...: >>>>>>> MPID_Init(151)..........: channel initialization failed >>>>>>> MPIDI_CH3_Init(70)......: >>>>>>> MPID_nem_init(379)......: >>>>>>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>>>>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>>>>> MPIR_Init_thread(433)...: >>>>>>> MPID_Init(151)..........: channel initialization failed >>>>>>> MPIDI_CH3_Init(70)......: >>>>>>> MPID_nem_init(379)......: >>>>>>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>>>>>> >>>>>>> =================================================================================== >>>>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>>>> = EXIT CODE: 1 >>>>>>> = CLEANING UP REMAINING PROCESSES >>>>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>>>> =================================================================================== >>>>>>> >>>>>>> >>>>>>> >>>>>>> i am a Linux and parallel programming beginner >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BLCR kernel module not present
by Raghunath 26 Apr '13

26 Apr '13

Michael, The BLCR support in MVAPICH works fine as well. MVAPICH implements its own Checkpoint-Restart mechanism for the CH3-IB and Nemesis-IB channels. The MPICH design for the Nemesis-TCP channel is left untouched, as Pavan indicated. -- Raghu On Fri, Apr 26, 2013 at 4:25 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > I can't speak for mvapich, of course. I'm only speaking about mpich. > However, most of our derivatives don't destroy the features that are in > stock mpich. So I'd think it'll work fine with mvapich as well. > > -- Pavan > > On 04/26/2013 03:20 PM US Central Time, michael wrote: >> Thanks Pavan >> >> Just to be clear, you're saying if I use mvapich with blcr then a >> running multi-node MPI job when killed (eg out of time) by a batch >> scheduler can be restarted (from which checkpoint?) presuming it doesn't >> have open files? >> >> Many thanks, M >> >> On Fri, 2013-04-26 at 15:12 -0500, Pavan Balaji wrote: >>> Michael, >>> >>> BLCR support for mpich should work fine; if something is broken, please >>> let us know. >>> >>> However, the core BLCR group itself hadn't released updates in a while, >>> primarily because they didn't have direct funding for it. I believe >>> that's fixed now and they are working on newer releases. >>> >>> -- Pavan >>> >>> On 04/26/2013 03:07 PM US Central Time, michael wrote: >>> > Hi folks >>> > I was wondering what the state of BLCR for mpich/mvapich is? eg how >>> > reliably can one presume it to be? >>> > Thanks, Michael >>> > >>> > >>> > On Fri, 2013-04-26 at 14:33 -0500, Wesley Bland wrote: >>> >> It looks like you might have missed installing the kernel module for >>> >> BLCR. What is the output of `lsmod`? >>> >> >>> >> >>> >> Alternatively, if you installed BLCR by using apt-get in Ubuntu, you >>> >> should be able to use dkms to manage your kernel modules >>> >> automatically. Make sure you have the package 'blcr-dkms' installed >>> >> (you should be able to check this by typing `dims status`. >>> >> >>> >> >>> >> Do either of those solutions solve your issue? >>> >> >>> >> >>> >> Wesley >>> >> >>> >> On Apr 26, 2013, at 1:55 PM, basma a.azeem >>> >> <basmaabdelazeem(a)hotmail.com <mailto:[email protected]> <mailto:[email protected]>> wrote: >>> >> >>> >>> >>> >>> >>> >>> Thank you for your help >>> >>> >>> >>> >>> >>> i installed BLCR 0.8.5 on my ubuntu 12.10 to be used for MPICH -3.0.3 >>> >>> this version of blcr should support to kernels through 3.7.1 >>> >>> >>> >>> when i run the command : >>> >>> basma@basma-Satellite-A500:~$ mpiexec --info >>> >>> >>> >>> results: >>> >>> >>> >>> HYDRA build details: >>> >>> Version: 3.0.3 >>> >>> Release Date: Thu Mar 28 16:01:21 CDT 2013 >>> >>> CC: gcc >>> >>> CXX: c++ >>> >>> F77: no >>> >>> F90: no >>> >>> Configure options: >>> >>> '--disable-option-checking' '--prefix=/home/basma/mpich2-install' >>> >>> '--disable-f77' '--disable-fc' '--enable-checkpointing' >>> >>> '--with-hydra-ckpointlib=blcr' '--cache-file=/dev/null' '--srcdir=.' >>> >>> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= ' 'LIBS=-lrt -lcr -lpthread ' >>> >>> 'CPPFLAGS= -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >>> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >>> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >>> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >>> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpi/romio/include' >>> >>> Process Manager: pmi >>> >>> Launchers available: ssh rsh fork slurm ll >>> >>> lsf sge manual persist >>> >>> Topology libraries available: hwloc >>> >>> Resource management kernels available: user slurm ll lsf sge >>> >>> pbs cobalt >>> >>> Checkpointing libraries available: blcr >>> >>> Demux engines available: poll select >>> >>> >>> >>> so i thought that every thing is ok but when i try to rum mpiexec it >>> >>> failed: >>> >>> >>> >>> basma@basma-Satellite-A500:~$ mpiexec -ckpointlib blcr >>> >>> -ckpoint-prefix /home/business/ckpts/app.ckpoint -ckpoint-interval >>> >>> 3600 -n 4 /home/basma/libraries/mpich-3.0.3/examples/cpi >>> >>> >>> >>> results: >>> >>> >>> >>> Fatal error in MPI_Init: Other MPI error, error stack: >>> >>> MPIR_Init_thread(433)...: >>> >>> MPID_Init(151)..........: channel initialization failed >>> >>> MPIDI_CH3_Init(70)......: >>> >>> MPID_nem_init(379)......: >>> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>> >>> Fatal error in MPI_Init: Other MPI error, error stack: >>> >>> MPIR_Init_thread(433)...: >>> >>> MPID_Init(151)..........: channel initialization failed >>> >>> MPIDI_CH3_Init(70)......: >>> >>> MPID_nem_init(379)......: >>> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>> >>> Fatal error in MPI_Init: Other MPI error, error stack: >>> >>> MPIR_Init_thread(433)...: >>> >>> MPID_Init(151)..........: channel initialization failed >>> >>> MPIDI_CH3_Init(70)......: >>> >>> MPID_nem_init(379)......: >>> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>> >>> Fatal error in MPI_Init: Other MPI error, error stack: >>> >>> MPIR_Init_thread(433)...: >>> >>> MPID_Init(151)..........: channel initialization failed >>> >>> MPIDI_CH3_Init(70)......: >>> >>> MPID_nem_init(379)......: >>> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >>> >>> >>> >>> =================================================================================== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = EXIT CODE: 1 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> =================================================================================== >>> >>> >>> >>> >>> >>> >>> >>> i am a Linux and parallel programming beginner >>> >>> >>> >>> Thank you >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >>> >>> To manage subscription options or unsubscribe: >>> >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >>> >> >>> >> _______________________________________________ >>> >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >>> >> To manage subscription options or unsubscribe: >>> >> https://lists.mpich.org/mailman/listinfo/discuss >>> > >>> > >>> > _______________________________________________ >>> > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> > To manage subscription options or unsubscribe: >>> > https://lists.mpich.org/mailman/listinfo/discuss >>> > >>> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BLCR kernel module not present
by michael 26 Apr '13

26 Apr '13

Many thanks, Pavan I shall have to try... I'd previously been led to believe that there was no way of checkpointing an MPI code under say SGE Yours, MIchael On Fri, 2013-04-26 at 15:25 -0500, Pavan Balaji wrote: > I can't speak for mvapich, of course. I'm only speaking about mpich. > However, most of our derivatives don't destroy the features that are in > stock mpich. So I'd think it'll work fine with mvapich as well. > > -- Pavan > > On 04/26/2013 03:20 PM US Central Time, michael wrote: > > Thanks Pavan > > > > Just to be clear, you're saying if I use mvapich with blcr then a > > running multi-node MPI job when killed (eg out of time) by a batch > > scheduler can be restarted (from which checkpoint?) presuming it doesn't > > have open files? > > > > Many thanks, M > > > > On Fri, 2013-04-26 at 15:12 -0500, Pavan Balaji wrote: > >> Michael, > >> > >> BLCR support for mpich should work fine; if something is broken, please > >> let us know. > >> > >> However, the core BLCR group itself hadn't released updates in a while, > >> primarily because they didn't have direct funding for it. I believe > >> that's fixed now and they are working on newer releases. > >> > >> -- Pavan > >> > >> On 04/26/2013 03:07 PM US Central Time, michael wrote: > >> > Hi folks > >> > I was wondering what the state of BLCR for mpich/mvapich is? eg how > >> > reliably can one presume it to be? > >> > Thanks, Michael > >> > > >> > > >> > On Fri, 2013-04-26 at 14:33 -0500, Wesley Bland wrote: > >> >> It looks like you might have missed installing the kernel module for > >> >> BLCR. What is the output of `lsmod`? > >> >> > >> >> > >> >> Alternatively, if you installed BLCR by using apt-get in Ubuntu, you > >> >> should be able to use dkms to manage your kernel modules > >> >> automatically. Make sure you have the package 'blcr-dkms' installed > >> >> (you should be able to check this by typing `dims status`. > >> >> > >> >> > >> >> Do either of those solutions solve your issue? > >> >> > >> >> > >> >> Wesley > >> >> > >> >> On Apr 26, 2013, at 1:55 PM, basma a.azeem > >> >> <basmaabdelazeem(a)hotmail.com <mailto:[email protected]> <mailto:[email protected]>> wrote: > >> >> > >> >>> > >> >>> > >> >>> Thank you for your help > >> >>> > >> >>> > >> >>> i installed BLCR 0.8.5 on my ubuntu 12.10 to be used for MPICH -3.0.3 > >> >>> this version of blcr should support to kernels through 3.7.1 > >> >>> > >> >>> when i run the command : > >> >>> basma@basma-Satellite-A500:~$ mpiexec --info > >> >>> > >> >>> results: > >> >>> > >> >>> HYDRA build details: > >> >>> Version: 3.0.3 > >> >>> Release Date: Thu Mar 28 16:01:21 CDT 2013 > >> >>> CC: gcc > >> >>> CXX: c++ > >> >>> F77: no > >> >>> F90: no > >> >>> Configure options: > >> >>> '--disable-option-checking' '--prefix=/home/basma/mpich2-install' > >> >>> '--disable-f77' '--disable-fc' '--enable-checkpointing' > >> >>> '--with-hydra-ckpointlib=blcr' '--cache-file=/dev/null' '--srcdir=.' > >> >>> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= ' 'LIBS=-lrt -lcr -lpthread ' > >> >>> 'CPPFLAGS= -I/home/basma/libraries/mpich-3.0.3/src/mpl/include > >> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpl/include > >> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src > >> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src > >> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpi/romio/include' > >> >>> Process Manager: pmi > >> >>> Launchers available: ssh rsh fork slurm ll > >> >>> lsf sge manual persist > >> >>> Topology libraries available: hwloc > >> >>> Resource management kernels available: user slurm ll lsf sge > >> >>> pbs cobalt > >> >>> Checkpointing libraries available: blcr > >> >>> Demux engines available: poll select > >> >>> > >> >>> so i thought that every thing is ok but when i try to rum mpiexec it > >> >>> failed: > >> >>> > >> >>> basma@basma-Satellite-A500:~$ mpiexec -ckpointlib blcr > >> >>> -ckpoint-prefix /home/business/ckpts/app.ckpoint -ckpoint-interval > >> >>> 3600 -n 4 /home/basma/libraries/mpich-3.0.3/examples/cpi > >> >>> > >> >>> results: > >> >>> > >> >>> Fatal error in MPI_Init: Other MPI error, error stack: > >> >>> MPIR_Init_thread(433)...: > >> >>> MPID_Init(151)..........: channel initialization failed > >> >>> MPIDI_CH3_Init(70)......: > >> >>> MPID_nem_init(379)......: > >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present > >> >>> Fatal error in MPI_Init: Other MPI error, error stack: > >> >>> MPIR_Init_thread(433)...: > >> >>> MPID_Init(151)..........: channel initialization failed > >> >>> MPIDI_CH3_Init(70)......: > >> >>> MPID_nem_init(379)......: > >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present > >> >>> Fatal error in MPI_Init: Other MPI error, error stack: > >> >>> MPIR_Init_thread(433)...: > >> >>> MPID_Init(151)..........: channel initialization failed > >> >>> MPIDI_CH3_Init(70)......: > >> >>> MPID_nem_init(379)......: > >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present > >> >>> Fatal error in MPI_Init: Other MPI error, error stack: > >> >>> MPIR_Init_thread(433)...: > >> >>> MPID_Init(151)..........: channel initialization failed > >> >>> MPIDI_CH3_Init(70)......: > >> >>> MPID_nem_init(379)......: > >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present > >> >>> > >> >>> =================================================================================== > >> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > >> >>> = EXIT CODE: 1 > >> >>> = CLEANING UP REMAINING PROCESSES > >> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > >> >>> =================================================================================== > >> >>> > >> >>> > >> >>> > >> >>> i am a Linux and parallel programming beginner > >> >>> > >> >>> Thank you > >> >>> > >> >>> > >> >>> > >> >>> _______________________________________________ > >> >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> > >> >>> To manage subscription options or unsubscribe: > >> >>> https://lists.mpich.org/mailman/listinfo/discuss > >> >> > >> >> > >> >> _______________________________________________ > >> >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> > >> >> To manage subscription options or unsubscribe: > >> >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > >

1 0

Re: [mpich-discuss] BLCR kernel module not present
by Pavan Balaji 26 Apr '13

26 Apr '13

I can't speak for mvapich, of course. I'm only speaking about mpich. However, most of our derivatives don't destroy the features that are in stock mpich. So I'd think it'll work fine with mvapich as well. -- Pavan On 04/26/2013 03:20 PM US Central Time, michael wrote: > Thanks Pavan > > Just to be clear, you're saying if I use mvapich with blcr then a > running multi-node MPI job when killed (eg out of time) by a batch > scheduler can be restarted (from which checkpoint?) presuming it doesn't > have open files? > > Many thanks, M > > On Fri, 2013-04-26 at 15:12 -0500, Pavan Balaji wrote: >> Michael, >> >> BLCR support for mpich should work fine; if something is broken, please >> let us know. >> >> However, the core BLCR group itself hadn't released updates in a while, >> primarily because they didn't have direct funding for it. I believe >> that's fixed now and they are working on newer releases. >> >> -- Pavan >> >> On 04/26/2013 03:07 PM US Central Time, michael wrote: >> > Hi folks >> > I was wondering what the state of BLCR for mpich/mvapich is? eg how >> > reliably can one presume it to be? >> > Thanks, Michael >> > >> > >> > On Fri, 2013-04-26 at 14:33 -0500, Wesley Bland wrote: >> >> It looks like you might have missed installing the kernel module for >> >> BLCR. What is the output of `lsmod`? >> >> >> >> >> >> Alternatively, if you installed BLCR by using apt-get in Ubuntu, you >> >> should be able to use dkms to manage your kernel modules >> >> automatically. Make sure you have the package 'blcr-dkms' installed >> >> (you should be able to check this by typing `dims status`. >> >> >> >> >> >> Do either of those solutions solve your issue? >> >> >> >> >> >> Wesley >> >> >> >> On Apr 26, 2013, at 1:55 PM, basma a.azeem >> >> <basmaabdelazeem(a)hotmail.com <mailto:[email protected]> <mailto:[email protected]>> wrote: >> >> >> >>> >> >>> >> >>> Thank you for your help >> >>> >> >>> >> >>> i installed BLCR 0.8.5 on my ubuntu 12.10 to be used for MPICH -3.0.3 >> >>> this version of blcr should support to kernels through 3.7.1 >> >>> >> >>> when i run the command : >> >>> basma@basma-Satellite-A500:~$ mpiexec --info >> >>> >> >>> results: >> >>> >> >>> HYDRA build details: >> >>> Version: 3.0.3 >> >>> Release Date: Thu Mar 28 16:01:21 CDT 2013 >> >>> CC: gcc >> >>> CXX: c++ >> >>> F77: no >> >>> F90: no >> >>> Configure options: >> >>> '--disable-option-checking' '--prefix=/home/basma/mpich2-install' >> >>> '--disable-f77' '--disable-fc' '--enable-checkpointing' >> >>> '--with-hydra-ckpointlib=blcr' '--cache-file=/dev/null' '--srcdir=.' >> >>> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= ' 'LIBS=-lrt -lcr -lpthread ' >> >>> 'CPPFLAGS= -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpl/include >> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >> >>> -I/home/basma/libraries/mpich-3.0.3/src/openpa/src >> >>> -I/home/basma/libraries/mpich-3.0.3/src/mpi/romio/include' >> >>> Process Manager: pmi >> >>> Launchers available: ssh rsh fork slurm ll >> >>> lsf sge manual persist >> >>> Topology libraries available: hwloc >> >>> Resource management kernels available: user slurm ll lsf sge >> >>> pbs cobalt >> >>> Checkpointing libraries available: blcr >> >>> Demux engines available: poll select >> >>> >> >>> so i thought that every thing is ok but when i try to rum mpiexec it >> >>> failed: >> >>> >> >>> basma@basma-Satellite-A500:~$ mpiexec -ckpointlib blcr >> >>> -ckpoint-prefix /home/business/ckpts/app.ckpoint -ckpoint-interval >> >>> 3600 -n 4 /home/basma/libraries/mpich-3.0.3/examples/cpi >> >>> >> >>> results: >> >>> >> >>> Fatal error in MPI_Init: Other MPI error, error stack: >> >>> MPIR_Init_thread(433)...: >> >>> MPID_Init(151)..........: channel initialization failed >> >>> MPIDI_CH3_Init(70)......: >> >>> MPID_nem_init(379)......: >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >> >>> Fatal error in MPI_Init: Other MPI error, error stack: >> >>> MPIR_Init_thread(433)...: >> >>> MPID_Init(151)..........: channel initialization failed >> >>> MPIDI_CH3_Init(70)......: >> >>> MPID_nem_init(379)......: >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >> >>> Fatal error in MPI_Init: Other MPI error, error stack: >> >>> MPIR_Init_thread(433)...: >> >>> MPID_Init(151)..........: channel initialization failed >> >>> MPIDI_CH3_Init(70)......: >> >>> MPID_nem_init(379)......: >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >> >>> Fatal error in MPI_Init: Other MPI error, error stack: >> >>> MPIR_Init_thread(433)...: >> >>> MPID_Init(151)..........: channel initialization failed >> >>> MPIDI_CH3_Init(70)......: >> >>> MPID_nem_init(379)......: >> >>> MPIDI_nem_ckpt_init(153): BLCR kernel module not present >> >>> >> >>> =================================================================================== >> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >>> = EXIT CODE: 1 >> >>> = CLEANING UP REMAINING PROCESSES >> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >>> =================================================================================== >> >>> >> >>> >> >>> >> >>> i am a Linux and parallel programming beginner >> >>> >> >>> Thank you >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected]> >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0