October 2013 - discuss - mailman3.cels.anl.gov

Can MPICH2 support IPv6?
by shenwei9008 30 Oct '13

30 Oct '13

Currently,I hava deployed MPICH2 in Ubuntu with ipv4 environment. And it can enable in WAN environment. But now I'd like to know whether MPICH2 can support IPV6 enviroment or not. This point does not write in the MPICH2 offical website. If you have deployed in the IPV6 environment, please tell how to config it?

1 0

Re: [mpich-discuss] Unable to load 'mpich2nemesis.dll', error 126 (or 193)
by Wesley Bland 29 Oct '13

29 Oct '13

To be clear, you can download the newer version, but it also won’t work on Windows. You’ll need to look at the MPICH download page for alternatives for your platform (per the FAQ page cited by Huiwei). Thanks, Wesley On Oct 29, 2013, at 8:30 AM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > Hi, Wangmeng, > > Sorry we do not support windows any more. > http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_can… > > Also, we suggest you download the latest MPICH from http://www.mpich.org/ instead of other sources because it is better supported and has many new features. > > — > Huiwei Lu > http://www.mcs.anl.gov/~huiweilu/ > > On Oct 29, 2013, at 3:31 AM, 王萌(HUST) <wmconan(a)gmail.com> wrote: > >> I got an error message " Unable to load 'mpich2nemesis.dll', error 126 ", when I found the file 'mpich2nemesis.dll', it send an other error message " Unable to load 'mpich2nemesis.dll', error 193 ". How to solve this problem? My OS is windows server 2008, and MPICH package is Jayesh Krishna's 1.4.1p1[x86 64] for windows version. >> >> Thanks! >> >> 2013-10-29 >> wangmeng(HUST) China >> >> >> >> 此电子邮件不含病毒和恶意软件，因为 avast! 杀毒软件保护处于活动状态。 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec crash
by Pavan Balaji 29 Oct '13

29 Oct '13

These are typically cleanup messages, meaning something went wrong in the application. Basically, the application terminated abruptly without cleaning up resources with hydra correctly. So hydra went berserk and cleaned up all the remaining processes and associated resources. Perhaps there is room to add some more diagnostic messages to improve the error reporting. But, for the time being, can you run your application through a debugger to see what’s going on? —- Pavan On Oct 29, 2013, at 12:56 AM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > Unfortunately, there is no other indication of what went wrong. > > When do we expect these hydra messages to show up? > Does it point to application problem or issue in hydra itself? > > Regards, > Rohit > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji > Sent: Monday, October 28, 2013 7:57 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] mpiexec crash > > Hi Rohit, > > In the cleanup message, did Hydra say what went wrong? Maybe it said the application had a segmentation fault? > > -- Pavan > > On Oct 28, 2013, at 9:43 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > >> Hi Pavan, >> >> I don't see any other messages from application or hydra. Run goes for a while and then ends abruptly with these messages. >> >> Regards, >> Rohit >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Pavan Balaji >> Sent: Monday, October 28, 2013 7:32 PM >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] mpiexec crash >> >> Rohit, >> >> Can you send us the error messages? What you sent are just the cleanup messages when Hydra noticed that your application died and cleaned up the remaining processes. That doesn't tell us any information we need. >> >> Thanks, >> >> -- Pavan >> >> On Oct 28, 2013, at 7:48 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: >> >>> Pavan, >>> >>> We retried the runs again. There is no ENOENT error now. But, MPI is still failing consistently with same error: >>> >>>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>>> callback returned error status [proxy:0:0@gretel] main >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): >>>> demux engine error waiting for event [mpiexec@gretel] control_cb >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>>> assert (!closed) failed [mpiexec@gretel] >>>> HYDT_dmxu_poll_wait_for_event >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>>> callback returned error status [mpiexec@gretel] >>>> HYD_pmci_wait_for_completion >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:19 >>>> 6 >>>> ): error waiting for event [mpiexec@gretel] main >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>>> process manager error waiting for completion >>> >>> We are running it on same machine as: >>> mpiexec -n 1 <exec> : -n 1 <exec> :..... >>> >>> What would cause such error to appear? How do we debug such issues? >>> >>> Regards, >>> Rohit >>> >>> >>> -----Original Message----- >>> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >>> Behalf Of Pavan Balaji >>> Sent: Wednesday, October 23, 2013 7:22 PM >>> To: discuss(a)mpich.org >>> Subject: Re: [mpich-discuss] mpiexec crash >>> >>> >>> On Oct 23, 2013, at 5:29 PM, Cherukumilli, Vasu <Vasu_Cherukumilli(a)mentor.com> wrote: >>>> Crash that we are seeing: >>>> >>>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>>> callback returned error status [proxy:0:0@gretel] main >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): >>>> demux engine error waiting for event [mpiexec@gretel] control_cb >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>>> assert (!closed) failed [mpiexec@gretel] >>>> HYDT_dmxu_poll_wait_for_event >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>>> callback returned error status [mpiexec@gretel] >>>> HYD_pmci_wait_for_completion >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:19 >>>> 6 >>>> ): error waiting for event [mpiexec@gretel] main >>>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>>> process manager error waiting for completion >>> >>> These are cleanup messages. You should have gotten an output which says so. >>> >>>> No such file or directory. (errno = ENOENT) >>> >>> This is the real error message. Did you make sure your executables are located on all the nodes in the same location? >>> >>> -- Pavan >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Unable to load 'mpich2nemesis.dll', error 126 (or 193)
by Huiwei Lu 29 Oct '13

29 Oct '13

Hi, Wangmeng, Sorry we do not support windows any more. http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_can… Also, we suggest you download the latest MPICH from http://www.mpich.org/ instead of other sources because it is better supported and has many new features. — Huiwei Lu http://www.mcs.anl.gov/~huiweilu/ On Oct 29, 2013, at 3:31 AM, 王萌(HUST) <wmconan(a)gmail.com> wrote: > I got an error message " Unable to load 'mpich2nemesis.dll', error 126 ", when I found the file 'mpich2nemesis.dll', it send an other error message " Unable to load 'mpich2nemesis.dll', error 193 ". How to solve this problem? My OS is windows server 2008, and MPICH package is Jayesh Krishna's 1.4.1p1[x86 64] for windows version. > > Thanks! > > 2013-10-29 > wangmeng(HUST) China > > > > 此电子邮件不含病毒和恶意软件，因为 avast! 杀毒软件保护处于活动状态。 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Unable to load 'mpich2nemesis.dll', error 126 (or 193)
by 王萌(HUST) 29 Oct '13

29 Oct '13

I got an error message " Unable to load 'mpich2nemesis.dll', error 126 ", when I found the file 'mpich2nemesis.dll', it send an other error message " Unable to load 'mpich2nemesis.dll', error 193 ". How to solve this problem? My OS is windows server 2008, and MPICH package is Jayesh Krishna's 1.4.1p1[x86 64] for windows version. Thanks! 2013-10-29 wangmeng(HUST) China --- 此电子邮件没有病毒和恶意软件，因为 avast! 防病毒保护处于活动状态。 http://www.avast.com

1 0

Re: [mpich-discuss] coupling
by fereshteh komijani 29 Oct '13

29 Oct '13

Dear Gus and Dear Huiwei I would like to offer my heartfelt appreciation for your kind consideration. After Huiwei 's last post I am examining his suggestions. I have installed current version of mpich2 but it could not solve problem. utilizing same of various values for Ntilei and Ntilej can not solve problem too. Then I check swan model again. Now I am sure it is the reason of errors. I am trying to sort out what cause those errors. I find this model mailing list. Thanks again Cheers fereshte On Mon, Oct 28, 2013 at 8:03 PM, Gus Correa <gus(a)ldeo.columbia.edu> wrote: > Hi Fereshteh > > 1) I would look for log or error messages in the model > output *before* the final one ("MPI_Abort ..."). > > Although it aborted with 2 processors, > that was a graceful termination by the program. > So what is before the last error message may shed some light > on why the model aborted, > and what it requires to run correctly. > > ** > > 2) MCT is also used for coupling climate models that we use > here. Some of these models it run in SPMD mode (i.e. a single > executable is launched by mpiexec). > Others run in MPMD mode (i.e. several executables > are launched by mpiexec, which requires a mpiexec command > line with more parameters). > > Your mpiexec suggests that the model relies on a > single executable (oceanG). > However, this may/may not be the case, as you mentioned > three separate items also: ROMS, SWAN, and the coupler. > > Is it a single executable or more than one that your model > uses? > [If you describe how you compile the model it may help.] > > ** > > 3) Are you trying to run it in hybrid mode, i.e. > using both MPI and OpenMP? > > I am not familiar to this model, > so I am just guessing. > > The conventional wisdom is to use parameters names like > "Nthreads" for OpenMP threads, although parameter names are elusive. > OpenMP would add another twist to your processor configuration, > as you would need to provision additional processors > for OpenMP, besides those for MPI. > > ** > > 4) Does the domain decomposition (Ntilei and Ntilej) have to > be the same for both models? > > > Have you tried something like: > Ntilei=2 > Ntilej=2 > i.e. 2*2=4 processors for the ocean > plus one processor for SWANN, a total of 5 processors, > then: > > mpiexec -np 5 ... > > In some models we run here (e.g. MITgcm) you need to match precisely > the number of processors to the tiles used to decompose your domain > (and possibly also the OpenMP threads). > > When you tried mpiexec -np 2, did you choose Ntilei=Ntilej=Nthreads=1 for > both models? > > Do you have perhaps to add one processor (or more) > specifically for the coupler? > > ** > > 5) Do your input files (describing the ocean initial state, > and perhaps the wave state), have to be organized based on the > "tile" configuration, or can they span the whole domain? > This could be another source of error. > > 6) Does ROMS and SWANN have mailing lists that can perhaps help > you more than the generic MPICH list? > > > I hope this helps, > Gus Correa > > > > On 10/27/2013 06:33 AM, fereshteh komijani wrote: > >> Sure. >> >> ROMS and SWAN models are models for ocean and wave models, >> respectively. Coupling of them require 3 input files: coupling_test.in >> <http://coupling_test.in>, swan_test.in <http://swan_test.in> and >> roms_test.in <http://roms_test.in> which first one is coupled input file >> >> and also one build.bash file which in it user called requested >> libraries, cpp optins, header file, compilers (for me gcc and gfortran). >> For example in build.bash file I set >> >> USE_MPI=on = on >> USE_MPIF90 =on >> >> WHICH_MPI=mpich2 >> >> FORT = gfortran >> >> also with regard that for coupling model, MCT (model coupling toolkit) >> is necessary I have installed it and set its include and lib >> directories in build.bash file. >> >> In coupling_test.in <http://coupling_test.in> user call swan_test.in >> <http://swan_test.in>and roms_test.in <http://roms_test.in> as wave and >> ocean model's inputs. >> >> In roms_test.in <http://roms_test.in> some coefficients, solving >> >> technique and some input forcing file determined. >> >> for choosing number of nodes for each model there are NtileI and Ntilej >> in roms_test.in <http://roms_test.in> and Nthreads (ocean) and Nthreads >> (wave) in coupling_test.in <http://coupling_test.in> file ( Nthreads >> >> (ocean)=NtileI *Ntilej ) and total nodes are equal to Nthreads >> (ocean)+Nthreads (wave). >> >> whenever i set one node for wave model (SWAN) (Nthreads (wave)=1) after >> running by >> >> mpirun -np 2 ./oceanG coupling_inlet-test.in >> <http://coupling_inlet-test.**in/ <http://coupling_inlet-test.in/> >> >>mpi.log >> >> >> it replies: >> >> application called MPI_Abort(comm=0x84000002, 4) - process 0 >> >> But when setNthreads (wave)>1 (and everything for Nthreads (ocean)) for >> >> example >> >> mpirun -np 8 ./oceanG coupling_inlet-test.in >> <http://coupling_inlet-test.**in/ <http://coupling_inlet-test.in/> >> >>mpi.log >> >> >> mpi.log file (attaché file) shows that roms model does not have problems >> and its nodes are active but nothing happen for swan model after 2 weeks >> than running. >> >> I hope that this information be sufficient. >> >> All the best >> >> fereshte >> >> >> >> ______________________________**_________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/**mailman/listinfo/discuss<https://lists.mpich.org/mailman/listinfo/discuss> >> > > ______________________________**_________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/**mailman/listinfo/discuss<https://lists.mpich.org/mailman/listinfo/discuss> > -- ***Angel*** **

1 0

Re: [mpich-discuss] mpiexec crash
by Jain, Rohit 29 Oct '13

29 Oct '13

Unfortunately, there is no other indication of what went wrong. When do we expect these hydra messages to show up? Does it point to application problem or issue in hydra itself? Regards, Rohit -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Monday, October 28, 2013 7:57 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] mpiexec crash Hi Rohit, In the cleanup message, did Hydra say what went wrong? Maybe it said the application had a segmentation fault? -- Pavan On Oct 28, 2013, at 9:43 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > Hi Pavan, > > I don't see any other messages from application or hydra. Run goes for a while and then ends abruptly with these messages. > > Regards, > Rohit > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Pavan Balaji > Sent: Monday, October 28, 2013 7:32 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] mpiexec crash > > Rohit, > > Can you send us the error messages? What you sent are just the cleanup messages when Hydra noticed that your application died and cleaned up the remaining processes. That doesn't tell us any information we need. > > Thanks, > > -- Pavan > > On Oct 28, 2013, at 7:48 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > >> Pavan, >> >> We retried the runs again. There is no ENOENT error now. But, MPI is still failing consistently with same error: >> >>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [proxy:0:0@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): >>> demux engine error waiting for event [mpiexec@gretel] control_cb >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>> assert (!closed) failed [mpiexec@gretel] >>> HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [mpiexec@gretel] >>> HYD_pmci_wait_for_completion >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:19 >>> 6 >>> ): error waiting for event [mpiexec@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>> process manager error waiting for completion >> >> We are running it on same machine as: >> mpiexec -n 1 <exec> : -n 1 <exec> :..... >> >> What would cause such error to appear? How do we debug such issues? >> >> Regards, >> Rohit >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Pavan Balaji >> Sent: Wednesday, October 23, 2013 7:22 PM >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] mpiexec crash >> >> >> On Oct 23, 2013, at 5:29 PM, Cherukumilli, Vasu <Vasu_Cherukumilli(a)mentor.com> wrote: >>> Crash that we are seeing: >>> >>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [proxy:0:0@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): >>> demux engine error waiting for event [mpiexec@gretel] control_cb >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>> assert (!closed) failed [mpiexec@gretel] >>> HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [mpiexec@gretel] >>> HYD_pmci_wait_for_completion >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:19 >>> 6 >>> ): error waiting for event [mpiexec@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>> process manager error waiting for completion >> >> These are cleanup messages. You should have gotten an output which says so. >> >>> No such file or directory. (errno = ENOENT) >> >> This is the real error message. Did you make sure your executables are located on all the nodes in the same location? >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec crash
by Pavan Balaji 28 Oct '13

28 Oct '13

Hi Rohit, In the cleanup message, did Hydra say what went wrong? Maybe it said the application had a segmentation fault? —- Pavan On Oct 28, 2013, at 9:43 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > Hi Pavan, > > I don't see any other messages from application or hydra. Run goes for a while and then ends abruptly with these messages. > > Regards, > Rohit > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji > Sent: Monday, October 28, 2013 7:32 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] mpiexec crash > > Rohit, > > Can you send us the error messages? What you sent are just the cleanup messages when Hydra noticed that your application died and cleaned up the remaining processes. That doesn't tell us any information we need. > > Thanks, > > -- Pavan > > On Oct 28, 2013, at 7:48 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > >> Pavan, >> >> We retried the runs again. There is no ENOENT error now. But, MPI is still failing consistently with same error: >> >>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [proxy:0:0@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux >>> engine error waiting for event [mpiexec@gretel] control_cb >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>> assert (!closed) failed [mpiexec@gretel] >>> HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [mpiexec@gretel] >>> HYD_pmci_wait_for_completion >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196 >>> ): error waiting for event [mpiexec@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>> process manager error waiting for completion >> >> We are running it on same machine as: >> mpiexec -n 1 <exec> : -n 1 <exec> :..... >> >> What would cause such error to appear? How do we debug such issues? >> >> Regards, >> Rohit >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Pavan Balaji >> Sent: Wednesday, October 23, 2013 7:22 PM >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] mpiexec crash >> >> >> On Oct 23, 2013, at 5:29 PM, Cherukumilli, Vasu <Vasu_Cherukumilli(a)mentor.com> wrote: >>> Crash that we are seeing: >>> >>> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >>> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >>> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [proxy:0:0@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux >>> engine error waiting for event [mpiexec@gretel] control_cb >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >>> assert (!closed) failed [mpiexec@gretel] >>> HYDT_dmxu_poll_wait_for_event >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >>> callback returned error status [mpiexec@gretel] >>> HYD_pmci_wait_for_completion >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196 >>> ): error waiting for event [mpiexec@gretel] main >>> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >>> process manager error waiting for completion >> >> These are cleanup messages. You should have gotten an output which says so. >> >>> No such file or directory. (errno = ENOENT) >> >> This is the real error message. Did you make sure your executables are located on all the nodes in the same location? >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] mpiexec crash
by Jain, Rohit 28 Oct '13

28 Oct '13

Hi Pavan, I don't see any other messages from application or hydra. Run goes for a while and then ends abruptly with these messages. Regards, Rohit -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Monday, October 28, 2013 7:32 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] mpiexec crash Rohit, Can you send us the error messages? What you sent are just the cleanup messages when Hydra noticed that your application died and cleaned up the remaining processes. That doesn't tell us any information we need. Thanks, -- Pavan On Oct 28, 2013, at 7:48 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > Pavan, > > We retried the runs again. There is no ENOENT error now. But, MPI is still failing consistently with same error: > >> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >> callback returned error status [proxy:0:0@gretel] main >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux >> engine error waiting for event [mpiexec@gretel] control_cb >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >> assert (!closed) failed [mpiexec@gretel] >> HYDT_dmxu_poll_wait_for_event >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >> callback returned error status [mpiexec@gretel] >> HYD_pmci_wait_for_completion >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196 >> ): error waiting for event [mpiexec@gretel] main >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >> process manager error waiting for completion > > We are running it on same machine as: > mpiexec -n 1 <exec> : -n 1 <exec> :..... > > What would cause such error to appear? How do we debug such issues? > > Regards, > Rohit > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Pavan Balaji > Sent: Wednesday, October 23, 2013 7:22 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] mpiexec crash > > > On Oct 23, 2013, at 5:29 PM, Cherukumilli, Vasu <Vasu_Cherukumilli(a)mentor.com> wrote: >> Crash that we are seeing: >> >> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb >> (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) >> failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >> callback returned error status [proxy:0:0@gretel] main >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux >> engine error waiting for event [mpiexec@gretel] control_cb >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): >> assert (!closed) failed [mpiexec@gretel] >> HYDT_dmxu_poll_wait_for_event >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): >> callback returned error status [mpiexec@gretel] >> HYD_pmci_wait_for_completion >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196 >> ): error waiting for event [mpiexec@gretel] main >> (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): >> process manager error waiting for completion > > These are cleanup messages. You should have gotten an output which says so. > >> No such file or directory. (errno = ENOENT) > > This is the real error message. Did you make sure your executables are located on all the nodes in the same location? > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec crash
by Pavan Balaji 28 Oct '13

28 Oct '13

Rohit, Can you send us the error messages? What you sent are just the cleanup messages when Hydra noticed that your application died and cleaned up the remaining processes. That doesn’t tell us any information we need. Thanks, —- Pavan On Oct 28, 2013, at 7:48 PM, Jain, Rohit <Rohit_Jain(a)mentor.com> wrote: > Pavan, > > We retried the runs again. There is no ENOENT error now. But, MPI is still failing consistently with same error: > >> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) failed >> [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status >> [proxy:0:0@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux engine error waiting for event >> [mpiexec@gretel] control_cb (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): assert (!closed) failed >> [mpiexec@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status >> [mpiexec@gretel] HYD_pmci_wait_for_completion (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196): error waiting for event >> [mpiexec@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion > > We are running it on same machine as: > mpiexec -n 1 <exec> : -n 1 <exec> :..... > > What would cause such error to appear? How do we debug such issues? > > Regards, > Rohit > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji > Sent: Wednesday, October 23, 2013 7:22 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] mpiexec crash > > > On Oct 23, 2013, at 5:29 PM, Cherukumilli, Vasu <Vasu_Cherukumilli(a)mentor.com> wrote: >> Crash that we are seeing: >> >> [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) failed >> [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status >> [proxy:0:0@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux engine error waiting for event >> [mpiexec@gretel] control_cb (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): assert (!closed) failed >> [mpiexec@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status >> [mpiexec@gretel] HYD_pmci_wait_for_completion (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196): error waiting for event >> [mpiexec@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion > > These are cleanup messages. You should have gotten an output which says so. > >> No such file or directory. (errno = ENOENT) > > This is the real error message. Did you make sure your executables are located on all the nodes in the same location? > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0