November 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] code from different version of LINUX
by Kenneth Raffenetti 26 Nov '14

26 Nov '14

It sounds like MPI is not being properly initialized and all of your processes are running as if they are the only rank. I would suggest making sure your are not mixing an mpiexec/mpirun from another MPI package with your MPICH library. A safer bet would be to copy over the mpiexec your built with MPICH and ensure it is the first one found in your PATH. Ken On 11/26/2014 11:28 AM, Shuchi Yang wrote: > Thanks for your reply. I am trying it in the way you mentioned. > But I met one problem is that on my original machine, I can run the code > with 20 CPUs so that each CPU works on part of the job. But at the new > machine, it starts the process with 20 CPUS, but every CPU works on all > the data, so that it looks like it is running 20 times the job at same > time. Is that because of MPI problem? > Thanks, > > Shuchi > > On Wed, Nov 26, 2014 at 9:50 AM, Junchao Zhang <jczhang(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > You can copy MPI libraries installed on the Ubuntu machine to the > Suse machine, then add that path to LD_LIBRARY_PATH on the Suse. > > --Junchao Zhang > > On Wed, Nov 26, 2014 at 9:55 AM, Shuchi Yang <yang.shuchi(a)gmail.com > <mailto:[email protected]>> wrote: > > I met some problem. > The question is that > I compile a fortran code at ubuntu, but I need run the code at > Suse Linux, I was always told > / error while loading shared libraries: libmpifort.so.12: > cannot open shared object file: No such file or directory/ > > Furthermore, will this be a problem, I mean, if I compile the > code with mpich-gcc and run the code at another type of Linux? > > Thanks, > > Shuchi > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] code from different version of LINUX
by Junchao Zhang 26 Nov '14

26 Nov '14

I guess it is not an MPI problem. When you say "every CPU works on all the data", you need a clear idea of what is the data decomposition in your code. --Junchao Zhang On Wed, Nov 26, 2014 at 11:28 AM, Shuchi Yang <yang.shuchi(a)gmail.com> wrote: > Thanks for your reply. I am trying it in the way you mentioned. > But I met one problem is that on my original machine, I can run the code > with 20 CPUs so that each CPU works on part of the job. But at the new > machine, it starts the process with 20 CPUS, but every CPU works on all the > data, so that it looks like it is running 20 times the job at same time. Is > that because of MPI problem? > Thanks, > > Shuchi > > On Wed, Nov 26, 2014 at 9:50 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > >> You can copy MPI libraries installed on the Ubuntu machine to the Suse >> machine, then add that path to LD_LIBRARY_PATH on the Suse. >> >> --Junchao Zhang >> >> On Wed, Nov 26, 2014 at 9:55 AM, Shuchi Yang <yang.shuchi(a)gmail.com> >> wrote: >> >>> I met some problem. >>> The question is that >>> I compile a fortran code at ubuntu, but I need run the code at Suse >>> Linux, I was always told >>> * error while loading shared libraries: libmpifort.so.12: cannot open >>> shared object file: No such file or directory* >>> >>> Furthermore, will this be a problem, I mean, if I compile the code with >>> mpich-gcc and run the code at another type of Linux? >>> >>> Thanks, >>> >>> Shuchi >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] code from different version of LINUX
by Shuchi Yang 26 Nov '14

26 Nov '14

Thanks for your reply. I am trying it in the way you mentioned. But I met one problem is that on my original machine, I can run the code with 20 CPUs so that each CPU works on part of the job. But at the new machine, it starts the process with 20 CPUS, but every CPU works on all the data, so that it looks like it is running 20 times the job at same time. Is that because of MPI problem? Thanks, Shuchi On Wed, Nov 26, 2014 at 9:50 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > You can copy MPI libraries installed on the Ubuntu machine to the Suse > machine, then add that path to LD_LIBRARY_PATH on the Suse. > > --Junchao Zhang > > On Wed, Nov 26, 2014 at 9:55 AM, Shuchi Yang <yang.shuchi(a)gmail.com> > wrote: > >> I met some problem. >> The question is that >> I compile a fortran code at ubuntu, but I need run the code at Suse >> Linux, I was always told >> * error while loading shared libraries: libmpifort.so.12: cannot open >> shared object file: No such file or directory* >> >> Furthermore, will this be a problem, I mean, if I compile the code with >> mpich-gcc and run the code at another type of Linux? >> >> Thanks, >> >> Shuchi >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] code from different version of LINUX
by Junchao Zhang 26 Nov '14

26 Nov '14

You can copy MPI libraries installed on the Ubuntu machine to the Suse machine, then add that path to LD_LIBRARY_PATH on the Suse. --Junchao Zhang On Wed, Nov 26, 2014 at 9:55 AM, Shuchi Yang <yang.shuchi(a)gmail.com> wrote: > I met some problem. > The question is that > I compile a fortran code at ubuntu, but I need run the code at Suse Linux, > I was always told > * error while loading shared libraries: libmpifort.so.12: cannot open > shared object file: No such file or directory* > > Furthermore, will this be a problem, I mean, if I compile the code with > mpich-gcc and run the code at another type of Linux? > > Thanks, > > Shuchi > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

code from different version of LINUX
by Shuchi Yang 26 Nov '14

26 Nov '14

I met some problem. The question is that I compile a fortran code at ubuntu, but I need run the code at Suse Linux, I was always told * error while loading shared libraries: libmpifort.so.12: cannot open shared object file: No such file or directory* Furthermore, will this be a problem, I mean, if I compile the code with mpich-gcc and run the code at another type of Linux? Thanks, Shuchi _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Kenneth Raffenetti 26 Nov '14

26 Nov '14

The connection refused makes me think a firewall is getting in the way. Is TCP communication limited to specific ports on the cluster? If so, you can use this envvar to enforce a range of ports in MPICH. MPIR_CVAR_CH3_PORT_RANGE Description: The MPIR_CVAR_CH3_PORT_RANGE environment variable allows you to specify the range of TCP ports to be used by the process manager and the MPICH library. The format of this variable is <low>:<high>. To specify any available port, use 0:0. Default: {0,0} On 11/25/2014 11:50 PM, Amin Hassani wrote: > Tried with the new configure too. same problem :( > > $ mpirun -hostfile hosts-hydra -np 2 test_dup > Fatal error in MPI_Send: Unknown error class, error stack: > MPI_Send(174)..............: MPI_Send(buf=0x7fffd90c76c8, count=1, > MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed > MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connection > refused > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 5459 RUNNING AT oakmnt-0-a > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > [proxy:0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb > (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed > [proxy:0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback > returned error status > [proxy:0:1@oakmnt-0-b] main > (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error > waiting for event > [mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion > (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of > the processes terminated badly; aborting > [mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion > (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher > returned error waiting for completion > [mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher > returned error waiting for completion > [mpiexec@oakmnt-0-a] main > (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error > waiting for completion > > > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > On Tue, Nov 25, 2014 at 11:44 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > So the error only happens when there is communication. > > It may be caused by IB as your guessed before. Could you try to > reconfigure MPICH using "./configure --with-device=ch3:nemesis:tcp” > and try again? > > — > Huiwei > > > On Nov 25, 2014, at 11:23 PM, Amin Hassani <ahassani(a)cis.uab.edu > <mailto:[email protected]>> wrote: > > > > Yes it works. > > output: > > > > $ mpirun -hostfile hosts-hydra -np 2 test > > rank 1 > > rank 0 > > > > > > Amin Hassani, > > CIS department at UAB, > > Birmingham, AL, USA. > > > > On Tue, Nov 25, 2014 at 11:20 PM, Lu, Huiwei > <huiweilu(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > Could you try to run the following simple code to see if it works? > > > > #include <mpi.h> > > #include <stdio.h> > > int main(int argc, char** argv) > > { > > int rank, size; > > MPI_Init(&argc, &argv); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > printf("rank %d\n", rank); > > MPI_Finalize(); > > return 0; > > } > > > > — > > Huiwei > > > > > On Nov 25, 2014, at 11:11 PM, Amin Hassani > <ahassani(a)cis.uab.edu <mailto:[email protected]>> wrote: > > > > > > No, I checked. Also I always install my MPI's in > /nethome/students/ahassani/usr/mpi. I never install them in > /nethome/students/ahassani/usr. So MPI files will never get there. > Even if put the /usr/mpi/bin in front of /usr/bin, it won't affect > anything. There has never been any mpi installed in /usr/bin. > > > > > > Thank you. > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich error
by Bland, Wesley B. 26 Nov '14

26 Nov '14

I think the error description itself it pretty accurate. MPICH can’t find the executable in the location you specified. You need to make sure that the executable is in the correct place. On Nov 26, 2014, at 1:51 AM, نازنین <nazanin.mirshokraei(a)gmail.com<mailto:[email protected]>> wrote: hi i am using mpich 3.0.4 and it is my first time running it and i do my run like : mpirun -np 2 ./Projects/g/swan_only.in<http://swan_only.in/> and i receive this error . will u please help me how to solve this error? [proxy:0:0@nazanin-VirtualBox] HYDU_create_process (./utils/launch/launch.c:75): execvp error on file ./Projects/g/swan_only.in<http://swan_only.in/> (No such file or directory) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 255 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich error
by نازنین 26 Nov '14

26 Nov '14

hi i am using mpich 3.0.4 and it is my first time running it and i do my run like : mpirun -np 2 ./Projects/g/swan_only.in and i receive this error . will u please help me how to solve this error? [proxy:0:0@nazanin-VirtualBox] HYDU_create_process (./utils/launch/launch.c:75): execvp error on file ./Projects/g/swan_only.in (No such file or directory) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 255 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Amin Hassani 25 Nov '14

25 Nov '14

Tried with the new configure too. same problem :( $ mpirun -hostfile hosts-hydra -np 2 test_dup Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x7fffd90c76c8, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connection refused =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5459 RUNNING AT oakmnt-0-a = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@oakmnt-0-b] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@oakmnt-0-a] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Amin Hassani, CIS department at UAB, Birmingham, AL, USA. On Tue, Nov 25, 2014 at 11:44 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > So the error only happens when there is communication. > > It may be caused by IB as your guessed before. Could you try to > reconfigure MPICH using "./configure --with-device=ch3:nemesis:tcp” and try > again? > > — > Huiwei > > > On Nov 25, 2014, at 11:23 PM, Amin Hassani <ahassani(a)cis.uab.edu> wrote: > > > > Yes it works. > > output: > > > > $ mpirun -hostfile hosts-hydra -np 2 test > > rank 1 > > rank 0 > > > > > > Amin Hassani, > > CIS department at UAB, > > Birmingham, AL, USA. > > > > On Tue, Nov 25, 2014 at 11:20 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> > wrote: > > Could you try to run the following simple code to see if it works? > > > > #include <mpi.h> > > #include <stdio.h> > > int main(int argc, char** argv) > > { > > int rank, size; > > MPI_Init(&argc, &argv); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > printf("rank %d\n", rank); > > MPI_Finalize(); > > return 0; > > } > > > > — > > Huiwei > > > > > On Nov 25, 2014, at 11:11 PM, Amin Hassani <ahassani(a)cis.uab.edu> > wrote: > > > > > > No, I checked. Also I always install my MPI's in > /nethome/students/ahassani/usr/mpi. I never install them in > /nethome/students/ahassani/usr. So MPI files will never get there. Even if > put the /usr/mpi/bin in front of /usr/bin, it won't affect anything. There > has never been any mpi installed in /usr/bin. > > > > > > Thank you. > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] having problem running MPICH on multiple nodes
by Lu, Huiwei 25 Nov '14

25 Nov '14

So the error only happens when there is communication. It may be caused by IB as your guessed before. Could you try to reconfigure MPICH using "./configure --with-device=ch3:nemesis:tcp” and try again? — Huiwei > On Nov 25, 2014, at 11:23 PM, Amin Hassani <ahassani(a)cis.uab.edu> wrote: > > Yes it works. > output: > > $ mpirun -hostfile hosts-hydra -np 2 test > rank 1 > rank 0 > > > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > On Tue, Nov 25, 2014 at 11:20 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > Could you try to run the following simple code to see if it works? > > #include <mpi.h> > #include <stdio.h> > int main(int argc, char** argv) > { > int rank, size; > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > printf("rank %d\n", rank); > MPI_Finalize(); > return 0; > } > > — > Huiwei > > > On Nov 25, 2014, at 11:11 PM, Amin Hassani <ahassani(a)cis.uab.edu> wrote: > > > > No, I checked. Also I always install my MPI's in /nethome/students/ahassani/usr/mpi. I never install them in /nethome/students/ahassani/usr. So MPI files will never get there. Even if put the /usr/mpi/bin in front of /usr/bin, it won't affect anything. There has never been any mpi installed in /usr/bin. > > > > Thank you. > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0