February 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Warning at the End of the Simulation
by Jeff Hammond 05 Feb '15

05 Feb '15

Indeed, see also http://www.pgroup.com/userforum/viewtopic.php?t=2943&view=next&sid=e7381c43…, particularly this suggestion: 'Try adding the flag "-Ktrap=fp".' Jeff On Thu, Feb 5, 2015 at 7:59 AM, "Antonio J. Peña" <apenya(a)mcs.anl.gov> wrote: > > Hi Abhishek, > > Those problems are likely to come from your code, not from the MPICH > library. They seem to reveal computation issues like division by zero, so > your results might not be accurate. There's not much we can help from here, > though. You may contact the application developer if it's not you or look at > PGI's documentation to debug the places where these happen (a starting point > may be here: > http://www.pgroup.com/userforum/viewtopic.php?t=2928&postdays=0&postorder=a…) > > Best, > Antonio > > > > On 02/05/2015 09:20 AM, Abhishek Bhat wrote: > > All, > > > > I am running MPICH3 with program compiled with PGI Fortran. At the end of > (successful) completion of the simulation, I am getting the following > warnings for each node > > > > Warning: ieee_invalid is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_invalid is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_underflow is signaling > > Warning: ieee_inexact is signaling > > > > Any suggestion if these warnings are harmful or on how to turn them off? > > > > Thank You > > Abhishek > > …………………………………………………………………………………………………. > > Abhishek Bhat, PhD, EPI, > Senior Consultant > > > _________________________________________________________________________ > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. > _________________________________________________________________________ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Warning at the End of the Simulation
by "Antonio J. Peña" 05 Feb '15

05 Feb '15

Hi Abhishek, Those problems are likely to come from your code, not from the MPICH library. They seem to reveal computation issues like division by zero, so your results might not be accurate. There's not much we can help from here, though. You may contact the application developer if it's not you or look at PGI's documentation to debug the places where these happen (a starting point may be here: http://www.pgroup.com/userforum/viewtopic.php?t=2928&postdays=0&postorder=a…) Best, Antonio On 02/05/2015 09:20 AM, Abhishek Bhat wrote: > > All, > > I am running MPICH3 with program compiled with PGI Fortran. At the > end of (successful) completion of the simulation, I am getting the > following warnings for each node > > Warning: ieee_invalid is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_invalid is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_denorm is signaling > > Warning: ieee_divide_by_zero is signaling > > Warning: ieee_underflow is signaling > > Warning: ieee_inexact is signaling > > Any suggestion if these warnings are harmful or on how to turn them off? > > Thank You > > Abhishek > > …………………………………………………………………………………………………. > > *Abhishek Bhat, PhD, EPI, > *Senior Consultant > > > _________________________________________________________________________ > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. > _________________________________________________________________________ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Warning at the End of the Simulation
by Abhishek Bhat 05 Feb '15

05 Feb '15

All, I am running MPICH3 with program compiled with PGI Fortran. At the end of (successful) completion of the simulation, I am getting the following warnings for each node Warning: ieee_invalid is signaling Warning: ieee_denorm is signaling Warning: ieee_invalid is signaling Warning: ieee_denorm is signaling Warning: ieee_denorm is signaling Warning: ieee_denorm is signaling Warning: ieee_denorm is signaling Warning: ieee_divide_by_zero is signaling Warning: ieee_divide_by_zero is signaling Warning: ieee_denorm is signaling Warning: ieee_divide_by_zero is signaling Warning: ieee_underflow is signaling Warning: ieee_inexact is signaling Any suggestion if these warnings are harmful or on how to turn them off? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in running parallel if two programs merged
by Huiwei Lu 05 Feb '15

05 Feb '15

I didn't see program_off is used anywhere. Just compile them together does not mean the function in that file will be called. -- Huiwei On Thu, Feb 5, 2015 at 1:51 AM, Muhammad Ashfaqur Rahman < ashfaq226(a)gmail.com> wrote: > Dear Huiwei Lu, > > Thank you for your suggestion. I tried for ifort with mpich wrappers (But > one set of programs is works only on PGI ). It is still not moving > parallel. The 'merged' means I have gathered two sets of programs into one > executable and try to run in mpi. Both the programs have their own mpi tags > declared, and need to be tailored, i.e., Only one MPI_Initialize, > MPI_Finalize etc. > > Please let me know what to do. > > Regards > Ashfaq > > On Tue, Feb 3, 2015 at 5:43 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > >> Can you change the following to use mpi fortran compiler and try again? >> >> FC = mpifort >> CLMF = mpifort >> >> >> Also, what do you mean by "merge" these two programs? I didn't see any >> interaction between program_off.F90 and aadmn.F. >> >> -- >> Huiwei >> >> On Tue, Feb 3, 2015 at 7:42 AM, Muhammad Ashfaqur Rahman < >> ashfaq226(a)gmail.com> wrote: >> >>> Dear Huiwei Lu, >>> Many thanks for your email. I have attached here the source code of my >>> programs. These are two models. >>> The Makefile should be looked at. >>> >>> Regards >>> Ashfaq >>> >>> On Tue, Feb 3, 2015 at 4:32 PM, Muhammad Ashfaqur Rahman < >>> ashfaq226(a)gmail.com> wrote: >>> >>>> Dear Huiwei Lu, >>>> Many thanks for your email. I have attached here the source code of my >>>> programs. The folder contain two models. >>>> The Makefile should be looked at. >>>> >>>> Regards >>>> Ashfaq >>>> >>>> On Tue, Feb 3, 2015 at 3:20 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: >>>> >>>>> Hi Ashfaq, >>>>> >>>>> If you program already has started with separate parallel instances, >>>>> how can they becomes serial again? >>>>> >>>>> Can you give us a sample of minimum code to explain your problem? >>>>> >>>>> -- >>>>> Huiwei Lu >>>>> Postdoc Appointee >>>>> Mathematics and Computer Science Division >>>>> Argonne National Laboratory >>>>> http://www.mcs.anl.gov/~huiweilu/ >>>>> >>>>> On Tue, Feb 3, 2015 at 1:22 AM, Muhammad Ashfaqur Rahman < >>>>> ashfaq226(a)gmail.com> wrote: >>>>> >>>>>> Dear All, >>>>>> Take my greetings. I am new in mpich usage. I have problems in >>>>>> parallel run, when two fortran mpi programs are merged to one executable. >>>>>> If these two are separate, then they are running parallel. >>>>>> >>>>>> One program has used spmd and another one has used mpich header >>>>>> directly. >>>>>> >>>>>> Other issue is that while trying to run the above mentioned merged >>>>>> program in mpich, it's first started with separate parallel instances of >>>>>> same step and then after some steps it becomes serial. >>>>>> >>>>>> Please help me in this regards >>>>>> >>>>>> Ashfaq >>>>>> Ph.D Student >>>>>> Dept. of Meteorology >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem in running parallel if two programs merged
by Muhammad Ashfaqur Rahman 05 Feb '15

05 Feb '15

Dear Huiwei Lu, Thank you for your suggestion. I tried for ifort with mpich wrappers (But one set of programs is works only on PGI ). It is still not moving parallel. The 'merged' means I have gathered two sets of programs into one executable and try to run in mpi. Both the programs have their own mpi tags declared, and need to be tailored, i.e., Only one MPI_Initialize, MPI_Finalize etc. Please let me know what to do. Regards Ashfaq On Tue, Feb 3, 2015 at 5:43 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > Can you change the following to use mpi fortran compiler and try again? > > FC = mpifort > CLMF = mpifort > > > Also, what do you mean by "merge" these two programs? I didn't see any > interaction between program_off.F90 and aadmn.F. > > -- > Huiwei > > On Tue, Feb 3, 2015 at 7:42 AM, Muhammad Ashfaqur Rahman < > ashfaq226(a)gmail.com> wrote: > >> Dear Huiwei Lu, >> Many thanks for your email. I have attached here the source code of my >> programs. These are two models. >> The Makefile should be looked at. >> >> Regards >> Ashfaq >> >> On Tue, Feb 3, 2015 at 4:32 PM, Muhammad Ashfaqur Rahman < >> ashfaq226(a)gmail.com> wrote: >> >>> Dear Huiwei Lu, >>> Many thanks for your email. I have attached here the source code of my >>> programs. The folder contain two models. >>> The Makefile should be looked at. >>> >>> Regards >>> Ashfaq >>> >>> On Tue, Feb 3, 2015 at 3:20 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: >>> >>>> Hi Ashfaq, >>>> >>>> If you program already has started with separate parallel instances, >>>> how can they becomes serial again? >>>> >>>> Can you give us a sample of minimum code to explain your problem? >>>> >>>> -- >>>> Huiwei Lu >>>> Postdoc Appointee >>>> Mathematics and Computer Science Division >>>> Argonne National Laboratory >>>> http://www.mcs.anl.gov/~huiweilu/ >>>> >>>> On Tue, Feb 3, 2015 at 1:22 AM, Muhammad Ashfaqur Rahman < >>>> ashfaq226(a)gmail.com> wrote: >>>> >>>>> Dear All, >>>>> Take my greetings. I am new in mpich usage. I have problems in >>>>> parallel run, when two fortran mpi programs are merged to one executable. >>>>> If these two are separate, then they are running parallel. >>>>> >>>>> One program has used spmd and another one has used mpich header >>>>> directly. >>>>> >>>>> Other issue is that while trying to run the above mentioned merged >>>>> program in mpich, it's first started with separate parallel instances of >>>>> same step and then after some steps it becomes serial. >>>>> >>>>> Please help me in this regards >>>>> >>>>> Ashfaq >>>>> Ph.D Student >>>>> Dept. of Meteorology >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Two problems about MPI_Publish_name/MPI_Loopup_name?
by Kenneth Raffenetti 04 Feb '15

04 Feb '15

Haozi, I re-ran your example with mpich 3.1.3 and saw the behavior you described in #2. The current mpich/master does not behave like this. Can you try a nightly tarball to confirm? http://www.mpich.org/static/downloads/nightly/master/mpich/ Ken On 02/02/2015 08:02 PM, haozi wrote: > Hi, Ken, > > which version is your MPICH? > > My MPICH is 3.1.3. > > > > At 2015-01-30 04:34:51, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: >>Haozi, >> >>I'm not sure I understand the issue you are seeing. Can you include >>output from this code that illustrates the issue? I tried running with >>strings both less than and greater than 7 characters with no issue. The >>port_name is the same length in all cases on my laptop. >> >>Ken >> >>On 01/28/2015 01:56 AM, haozi wrote: >>> Hi, guys. >>> My example code is the attachment. >>> >>> 1. What is the naming convention of service_name ? >>> The hydra_nameserver CAN'T accept those strings whose length is >>> less than 7. >>> What's the exact naming convention? >>> >>> 2. The length of port_name by MPI_Loopup_name is different >>> with MPI_Publish_name! >>> You can test my code. >>> >>> With best regards. >>> Haozi >>> >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>_______________________________________________ >>discuss mailing list discuss(a)mpich.org >>To manage subscription options or unsubscribe: >>https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Seo, Sangmin 04 Feb '15

04 Feb '15

No, you don’t need to install MPICH on all nodes if it is shared and LD_LIBRARY_PATH on all nodes is set correctly for the MPICH installation. — Sangmin On Feb 4, 2015, at 3:32 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, All the firewalls are off. This might be a very basic question but do I need to install MPICH libraries on all nodes even if I have installed MPICH (including libraries) on the master in a shared file between all nodes? Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Wednesday, February 04, 2015 1:41 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Can you also check whether the firewalls are turned off on all nodes, as described in https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI… — Sangmin On Feb 4, 2015, at 12:55 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got [12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi Process 0 of 5 is on Earth Process 1 of 5 is on Earth Process 4 of 5 is on Earth Process 2 of 5 is on node1 Process 3 of 5 is on node1 Fatal error in PMPI_Reduce: Unknown error class, error stack: PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(826)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(188).......: MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2 MPIR_Reduce_intra(846)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(250).......: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 11705 RUNNING AT Earth = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 6:26 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error? — Sangmin On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error – Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. — Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn’t have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error – /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Abhishek Bhat 04 Feb '15

04 Feb '15

Sangmin, All the firewalls are off. This might be a very basic question but do I need to install MPICH libraries on all nodes even if I have installed MPICH (including libraries) on the master in a shared file between all nodes? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Wednesday, February 04, 2015 1:41 PM To: <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH3 Problem Can you also check whether the firewalls are turned off on all nodes, as described in https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI… - Sangmin On Feb 4, 2015, at 12:55 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got [12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi Process 0 of 5 is on Earth Process 1 of 5 is on Earth Process 4 of 5 is on Earth Process 2 of 5 is on node1 Process 3 of 5 is on node1 Fatal error in PMPI_Reduce: Unknown error class, error stack: PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(826)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(188).......: MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2 MPIR_Reduce_intra(846)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(250).......: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 11705 RUNNING AT Earth = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 6:26 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error? - Sangmin On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error - Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. - Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn't have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error - /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question in MPI_Info_get
by Thakur, Rajeev 04 Feb '15

04 Feb '15

Yes. On Feb 4, 2015, at 11:29 AM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: > Thanks Rajeev and Bill, > > So you think this is a bug in MPICH (so as in OpenMPI) with respect to the standard, right? > I’ll keep watching the ticket, and for the moment I’ll workaround in my code by passing the length including null character. > > > Best, > Hajime > >> On Feb 3, 2015, at 5:42 PM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: >> >> This bug is tracked in ticket #2225. >> https://trac.mpich.org/projects/mpich/ticket/2225 >> >> Rajeev >> >> On Feb 3, 2015, at 4:26 PM, William Gropp <wgropp(a)illinois.edu> wrote: >> >>> Thanks. Yes, I believe this is a bug. The implementation of MPI_Info_get in MPICH passes valuelen to the internal copy routine, yet it is clear that the definition in the standard for MPI_Info_get, it should pass valuelen+1 (it should also check for the error return code from the internal routine and return a value MPI error code if the buffer is too small. >>> >>> Bill >>> >>> On Feb 1, 2015, at 12:13 AM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: >>> >>>> Hello, >>>> >>>> I came across some question about a behavior of MPI_Info_get in MPICH. >>>> >>>> If I knew the size of the value, what should I pass to MPI_Info_get as `valuelen`? >>>> Suppose I have 3-byte value in the info object, and I allocate 4-byte buffer including a space for null-termination. Then I think I should specify 3 as valuelen to retrieve the whole value, as the MPI-3 specifications says >>>> "In C, valuelen should be one less than the amount of allocated space to allow for the null terminator.” >>>> >>>> However, when I run the attached program in MPICH 3.1.1, I get: >>>> $ ./test >>>> key found: vaule=va # The last character ‘l’ is missing >>>> >>>> If I specify `sizeof(buff)` as valuelen, it shows “val” correctly. >>>> >>>> So apparently MPICH thinks valuelen includes null terminator. >>>> Am I missing anything, or any misunderstanding here? >>>> >>>> >>>> Thank you in advance, >>>> Hajime >>>> >>>> -- >>>> Hajime Fujita >>>> Postdoctoral Scholar, Large-Scale Systems Group >>>> Department of Computer Science, The University of Chicago >>>> http://www.cs.uchicago.edu/people/hfujita >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> <test.c> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Seo, Sangmin 04 Feb '15

04 Feb '15

Can you also check whether the firewalls are turned off on all nodes, as described in https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI… — Sangmin On Feb 4, 2015, at 12:55 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got [12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi Process 0 of 5 is on Earth Process 1 of 5 is on Earth Process 4 of 5 is on Earth Process 2 of 5 is on node1 Process 3 of 5 is on node1 Fatal error in PMPI_Reduce: Unknown error class, error stack: PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(826)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(188).......: MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2 MPIR_Reduce_intra(846)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(250).......: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 11705 RUNNING AT Earth = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 6:26 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error? — Sangmin On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error – Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. — Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn’t have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error – /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0