February 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH3 Problem
by Abhishek Bhat 04 Feb '15

04 Feb '15

Sangmin, All nodes can communicate with each other with ssh. I ran the MPI example program and here is the error I got [12:52:46] [clusteruser@Earth examples]$ mpiexec -n 5 -f machinefile ./cpi Process 0 of 5 is on Earth Process 1 of 5 is on Earth Process 4 of 5 is on Earth Process 2 of 5 is on node1 Process 3 of 5 is on node1 Fatal error in PMPI_Reduce: Unknown error class, error stack: PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0x7fff3bea9148, rbuf=0x7fff3bea9140, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(826)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(188).......: MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 2 MPIR_Reduce_intra(846)..........: MPIR_Reduce_impl(1075)..........: MPIR_Reduce_intra(881)..........: MPIR_Reduce_binomial(250).......: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 11705 RUNNING AT Earth = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@node1] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@node1] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@node1] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 6:26 PM To: <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH3 Problem Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error? - Sangmin On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error - Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. - Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn't have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error - /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question in MPI_Info_get
by Hajime Fujita 04 Feb '15

04 Feb '15

Thanks Rajeev and Bill, So you think this is a bug in MPICH (so as in OpenMPI) with respect to the standard, right? I’ll keep watching the ticket, and for the moment I’ll workaround in my code by passing the length including null character. Best, Hajime > On Feb 3, 2015, at 5:42 PM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > > This bug is tracked in ticket #2225. > https://trac.mpich.org/projects/mpich/ticket/2225 > > Rajeev > > On Feb 3, 2015, at 4:26 PM, William Gropp <wgropp(a)illinois.edu> wrote: > >> Thanks. Yes, I believe this is a bug. The implementation of MPI_Info_get in MPICH passes valuelen to the internal copy routine, yet it is clear that the definition in the standard for MPI_Info_get, it should pass valuelen+1 (it should also check for the error return code from the internal routine and return a value MPI error code if the buffer is too small. >> >> Bill >> >> On Feb 1, 2015, at 12:13 AM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: >> >>> Hello, >>> >>> I came across some question about a behavior of MPI_Info_get in MPICH. >>> >>> If I knew the size of the value, what should I pass to MPI_Info_get as `valuelen`? >>> Suppose I have 3-byte value in the info object, and I allocate 4-byte buffer including a space for null-termination. Then I think I should specify 3 as valuelen to retrieve the whole value, as the MPI-3 specifications says >>> "In C, valuelen should be one less than the amount of allocated space to allow for the null terminator.” >>> >>> However, when I run the attached program in MPICH 3.1.1, I get: >>> $ ./test >>> key found: vaule=va # The last character ‘l’ is missing >>> >>> If I specify `sizeof(buff)` as valuelen, it shows “val” correctly. >>> >>> So apparently MPICH thinks valuelen includes null terminator. >>> Am I missing anything, or any misunderstanding here? >>> >>> >>> Thank you in advance, >>> Hajime >>> >>> -- >>> Hajime Fujita >>> Postdoctoral Scholar, Large-Scale Systems Group >>> Department of Computer Science, The University of Chicago >>> http://www.cs.uchicago.edu/people/hfujita >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> <test.c> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question in MPI_Info_get
by William Gropp 04 Feb '15

04 Feb '15

I don’t agree that the standard obscures the case (at least in this instance). The language is clear. You are objecting to the choice that was made in the standard, and I might agree with you that it is unfortunate. But that doesn’t change the fact that (a) the standard is clear and (b) any change would not be backward compatible. Bill On Feb 3, 2015, at 10:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > I still feel the standard obscures the case. Passing a buffer (value) and the length of the buffer (valuelen) is easier for programmers. > The Fortran interface "CHARACTER(LEN=valuelen), INTENT(OUT) :: value" is clearer in this regard. > > --Junchao Zhang > > On Tue, Feb 3, 2015 at 5:33 PM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > I guess it may have been defined that way because in Fortran strings are not null-terminated. The definition in the standard allows the same value to be returned/passed in valuelen in both C and Fortran. > > Rajeev > > On Feb 3, 2015, at 5:28 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > > OK, I guess MPI standard wants to have this kind of code. > > > > int valuelen; > > MPI_Info_get_valuelen(..., &valuelen,...); > > int buflen = valuelen + 1; > > char* value = (char *)malloc(buflen); > > MPI_Info_get(..., valuelen, value, ...); > > > > Though, from a user's point of view, I fell better if it was MPI_Info_get(..., buflen, value, ...), that is, using buflen to specify the length of buffer I pass to MPI. > > > > > > --Junchao Zhang > > > > On Tue, Feb 3, 2015 at 4:26 PM, William Gropp <wgropp(a)illinois.edu> wrote: > > Thanks. Yes, I believe this is a bug. The implementation of MPI_Info_get in MPICH passes valuelen to the internal copy routine, yet it is clear that the definition in the standard for MPI_Info_get, it should pass valuelen+1 (it should also check for the error return code from the internal routine and return a value MPI error code if the buffer is too small. > > > > Bill > > > > On Feb 1, 2015, at 12:13 AM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: > > > >> Hello, > >> > >> I came across some question about a behavior of MPI_Info_get in MPICH. > >> > >> If I knew the size of the value, what should I pass to MPI_Info_get as `valuelen`? > >> Suppose I have 3-byte value in the info object, and I allocate 4-byte buffer including a space for null-termination. Then I think I should specify 3 as valuelen to retrieve the whole value, as the MPI-3 specifications says > >> "In C, valuelen should be one less than the amount of allocated space to allow for the null terminator.” > >> > >> However, when I run the attached program in MPICH 3.1.1, I get: > >> $ ./test > >> key found: vaule=va # The last character ‘l’ is missing > >> > >> If I specify `sizeof(buff)` as valuelen, it shows “val” correctly. > >> > >> So apparently MPICH thinks valuelen includes null terminator. > >> Am I missing anything, or any misunderstanding here? > >> > >> > >> Thank you in advance, > >> Hajime > >> > >> -- > >> Hajime Fujita > >> Postdoctoral Scholar, Large-Scale Systems Group > >> Department of Computer Science, The University of Chicago > >> http://www.cs.uchicago.edu/people/hfujita > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> <test.c> > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question in MPI_Info_get
by Junchao Zhang 03 Feb '15

03 Feb '15

I still feel the standard obscures the case. Passing a buffer (value) and the length of the buffer (valuelen) is easier for programmers. The Fortran interface "CHARACTER(LEN=valuelen), INTENT(OUT) :: value" is clearer in this regard. --Junchao Zhang On Tue, Feb 3, 2015 at 5:33 PM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > I guess it may have been defined that way because in Fortran strings are > not null-terminated. The definition in the standard allows the same value > to be returned/passed in valuelen in both C and Fortran. > > Rajeev > > On Feb 3, 2015, at 5:28 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > > OK, I guess MPI standard wants to have this kind of code. > > > > int valuelen; > > MPI_Info_get_valuelen(..., &valuelen,...); > > int buflen = valuelen + 1; > > char* value = (char *)malloc(buflen); > > MPI_Info_get(..., valuelen, value, ...); > > > > Though, from a user's point of view, I fell better if it was > MPI_Info_get(..., buflen, value, ...), that is, using buflen to specify the > length of buffer I pass to MPI. > > > > > > --Junchao Zhang > > > > On Tue, Feb 3, 2015 at 4:26 PM, William Gropp <wgropp(a)illinois.edu> > wrote: > > Thanks. Yes, I believe this is a bug. The implementation of > MPI_Info_get in MPICH passes valuelen to the internal copy routine, yet it > is clear that the definition in the standard for MPI_Info_get, it should > pass valuelen+1 (it should also check for the error return code from the > internal routine and return a value MPI error code if the buffer is too > small. > > > > Bill > > > > On Feb 1, 2015, at 12:13 AM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: > > > >> Hello, > >> > >> I came across some question about a behavior of MPI_Info_get in MPICH. > >> > >> If I knew the size of the value, what should I pass to MPI_Info_get as > `valuelen`? > >> Suppose I have 3-byte value in the info object, and I allocate 4-byte > buffer including a space for null-termination. Then I think I should > specify 3 as valuelen to retrieve the whole value, as the MPI-3 > specifications says > >> "In C, valuelen should be one less than the amount of allocated space > to allow for the null terminator.” > >> > >> However, when I run the attached program in MPICH 3.1.1, I get: > >> $ ./test > >> key found: vaule=va # The last character ‘l’ is missing > >> > >> If I specify `sizeof(buff)` as valuelen, it shows “val” correctly. > >> > >> So apparently MPICH thinks valuelen includes null terminator. > >> Am I missing anything, or any misunderstanding here? > >> > >> > >> Thank you in advance, > >> Hajime > >> > >> -- > >> Hajime Fujita > >> Postdoctoral Scholar, Large-Scale Systems Group > >> Department of Computer Science, The University of Chicago > >> http://www.cs.uchicago.edu/people/hfujita > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> <test.c> > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] (no subject)
by Huiwei Lu 03 Feb '15

03 Feb '15

The error is probably caused by your "parallel code" section. -- Huiwei On Tue, Feb 3, 2015 at 6:39 PM, Chafik sanaa <san.chafik(a)gmail.com> wrote: > Hi, > when i run my code i have this error "process 1 exited without calling > finalize" what is the cause? although I put a barrier before the end of my > code > > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &rang_proc); > MPI_Comm_size(MPI_COMM_WORLD, &nbr_proc); > //parallel code > MPI_Barrier(MPI_COMM_WORLD); > MPI_Finalize(); > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

(no subject)
by Chafik sanaa 03 Feb '15

03 Feb '15

Hi, when i run my code i have this error "process 1 exited without calling finalize" what is the cause? although I put a barrier before the end of my code MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rang_proc); MPI_Comm_size(MPI_COMM_WORLD, &nbr_proc); //parallel code MPI_Barrier(MPI_COMM_WORLD); MPI_Finalize(); _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Seo, Sangmin 03 Feb '15

03 Feb '15

Can all nodes that you are using do ssh each other? And, can you also try to run an MPICH example, cpi, located in <mpich_top_dir>/examples with the same nodes to see if you run into the same error? — Sangmin On Feb 3, 2015, at 6:06 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error – Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. — Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn’t have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error – /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Abhishek Bhat 03 Feb '15

03 Feb '15

So, after installing the PGI fortran on the shared drive, I re-ran and now I am getting communication error - Model startup ......Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x1983bef8, count=1, MPI_INTEGER, dest=2, tag=1, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 2: Connection timed out ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:57 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] MPICH3 Problem Correct. libpgmp.so should be from your PGI compiler installation. - Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn't have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error - /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Seo, Sangmin 03 Feb '15

03 Feb '15

Correct. libpgmp.so should be from your PGI compiler installation. — Sangmin On Feb 3, 2015, at 5:53 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn’t have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error – /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek …………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 Problem
by Abhishek Bhat 03 Feb '15

03 Feb '15

Sangmin, I have installed MPICH on shared drive but PGI fortran is installed on /opt/pgi which node do not have access. I am assuming that is the issue currently. I am trying to re-install PGI on the shared drive to see if that will fix the problem. Just to confirm libpgmp.so is not mpich file correct? Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant From: Seo, Sangmin [mailto:[email protected]] Sent: Tuesday, February 03, 2015 5:50 PM To: <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH3 Problem Hi Abhishek, As the error message says, it looks that the node running the application doesn't have libpgmp.so. Can you confirm whether the node has libpgmp.so and whether LD_LIBRARY_PATH is correctly set on the node in case that it has libpgmp.so? Best regards, Sangmin On Feb 3, 2015, at 5:31 PM, Abhishek Bhat <abhat(a)trinityconsultants.com<mailto:[email protected]>> wrote: HI All, I installed MPICH3 on a master terminal along with PGI Fortran. Then used MPICH3 and PGI to compile my software. When I run the program on the master terminal only I do not have any error messages but when I am trying to run them on one of the nodes, I am getting following message error - /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory /home/Earth/MODELS/camx/src_611/CAMx.v6.11.MPICH3.pgfomp: error while loading shared libraries: libpgmp.so: cannot open shared object file: No such file or directory =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5200 RUNNING AT node3 = EXIT CODE: 127 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@Earth] HYD_pmcd_pmip_control_cmd_cb (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0@Earth] HYDT_dmxu_poll_wait_for_event (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@Earth] HYDT_bscu_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@Earth] HYDT_bsci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@Earth] HYD_pmci_wait_for_completion (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@Earth] main (/home/Earth/MODELS/mpi/mpich-3.1.3/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion The /home/Earth is shared and mapped on all nodes Any help is much appreciated. Thank You Abhishek ................................................................................................................ Abhishek Bhat, PhD, EPI, Senior Consultant _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- _________________________________________________________________________ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0