November 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhou, Hui 26 Nov '18

26 Nov '18

Hi Pavan, Is there difference between mpirun and mpiexec. I understand the official standards only specifies mpiexec, and mpirun often (is it always?) is a link to mpiexec. Is there `unofficial` convention for mpirun, such as always present, always a link, no difference to mpiexec at all? It appears to me that mpirun is more wide spread than mpiexec (I always have been using mpirun). — Hui Zhou On Nov 21, 2018, at 5:34 PM, Balaji, Pavan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I should clarify one piece of how mpiexec works: % ./real.exe and % mpirun -np 1 ./real.exe are equivalent. % mpirun ./real.exe uses some number of processes depending on the environment. For unmanaged clusters, that's typically 1. For clusters that have some job management system (such as slurm or pbs), mpiexec will figure out how many nodes you allocated and use all of the cores allocated to that job. My guess is that real.exe has some dependencies that are met on the local machine, but not on other machines. So when mpiexec tries to launch real.exe on other nodes, it's throwing an error. This is not an mpiexec problem, but you might want to use the -prepend-pattern option in mpiexec to figure out where the error is coming from. Something like this: mpiexec -prepend-pattern %h ./real.exe -- Pavan On Nov 4, 2018, at 10:27 AM, Zhifeng Yang via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi After I installed mpich and used it in a FORTRAN code. There is an error while running this FORTRAN code called real.exe by using the following command $./real.exe or $mpirun ./real.exe real.exe: error: _get_addr: No such file or directory real.exe: error: _get_addr: No such file or directory real.exe: error: _get_addr: No such file or directory Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(784).....: MPID_Init(1323)...........: channel initialization failed MPIDI_CH3_Init(120).......: MPID_nem_init_ckpt(852)...: MPIDI_CH3I_Seg_commit(364): PMI_Barrier returned -1 But while using mpirun with specifying number of processors. as follows $mpirun -np 1 ./real.exe There is no error any more. I am not sure why? do you have any explanation? Thank you very much Best regards Zhifeng _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Balaji, Pavan 21 Nov '18

21 Nov '18

I should clarify one piece of how mpiexec works: % ./real.exe and % mpirun -np 1 ./real.exe are equivalent. % mpirun ./real.exe uses some number of processes depending on the environment. For unmanaged clusters, that's typically 1. For clusters that have some job management system (such as slurm or pbs), mpiexec will figure out how many nodes you allocated and use all of the cores allocated to that job. My guess is that real.exe has some dependencies that are met on the local machine, but not on other machines. So when mpiexec tries to launch real.exe on other nodes, it's throwing an error. This is not an mpiexec problem, but you might want to use the -prepend-pattern option in mpiexec to figure out where the error is coming from. Something like this: mpiexec -prepend-pattern %h ./real.exe -- Pavan > On Nov 4, 2018, at 10:27 AM, Zhifeng Yang via discuss <discuss(a)mpich.org> wrote: > > Hi > > After I installed mpich and used it in a FORTRAN code. There is an error while running this FORTRAN code called real.exe by using the following command > > $./real.exe > or > $mpirun ./real.exe > real.exe: error: _get_addr: No such file or directory > real.exe: error: _get_addr: No such file or directory > real.exe: error: _get_addr: No such file or directory > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(784).....: > MPID_Init(1323)...........: channel initialization failed > MPIDI_CH3_Init(120).......: > MPID_nem_init_ckpt(852)...: > MPIDI_CH3I_Seg_commit(364): PMI_Barrier returned -1 > > But while using mpirun with specifying number of processors. as follows > $mpirun -np 1 ./real.exe > There is no error any more. I am not sure why? do you have any explanation? Thank you very much > > Best regards > Zhifeng > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhifeng Yang 21 Nov '18

21 Nov '18

Hi Joseph, I have not tried gdb yet. But I found a description in the initialization module. Here it says ! <DESCRIPTION> ! This routine USES the modules in WRF and then calls the init routines ! they provide to perform module specific initializations at the ! beginning of a run. Note, this is only once per run, not once per ! domain; domain specific initializations should be handled elsewhere, ! such as in <a href=start_domain.html>start_domain</a>. ! ! Certain framework specific module initializations in this file are ! dependent on order they are called. For example, since the quilt module ! relies on internal I/O, the init routine for internal I/O must be ! called first. In the case of DM_PARALLEL compiles, the quilt module ! calls MPI_INIT as part of setting up and dividing communicators between ! compute and I/O server tasks. Therefore, it must be called prior to ! module_dm, which will <em>also</em> try to call MPI_INIT if it sees ! that MPI has not be initialized yet (implementations of module_dm ! should in fact behave this way by first calling MPI_INITIALIZED before ! they try to call MPI_INIT). If MPI is already initialized before the ! the quilting module is called, quilting will not work. ! ! The phase argument is used to allow other superstructures like ESMF to ! place their initialization calls following the WRF initialization call ! that calls MPI_INIT(). When used with ESMF, ESMF will call wrf_init() ! which in turn will call phase 2 of this routine. Phase 1 will be called ! earlier. ! ! </DESCRIPTION> INTEGER, INTENT(IN) :: phase ! phase==1 means return after MPI_INIT() ! phase==2 means resume after MPI_INIT() It mentions something about MPI_INIT, but I can not understand its meaning. It may help you to understand. Best, Zhifeng On Tue, Nov 20, 2018 at 10:17 AM Joseph Schuchart via discuss < discuss(a)mpich.org> wrote: > Zhifeng, > > Another way to approach this is to start the application under gdb and > set a breakpoint to MPICH's internal abort function (MPID_Abort iirc). > Once you hit this you can walk up the stack and try to find out where > _get_addr was found to be faulty. Since you are running with a single > process starting under GDB should be straight forward: > > $ gdb -ex "b MPID_Abort" -ex r ./real.exe > > (If you pass arguments to real.exe you have to pass --args to gdb) > > Cheers > Joseph > > On 11/19/18 1:49 PM, Zhifeng Yang via discuss wrote: > > Hi Hui, > > > > I just searched the whole code. There is no MPI_T_* name in the code. I > > may tried the newer version later on. Thank you very much > > > > Zhifeng > > > > > > On Mon, Nov 19, 2018 at 12:14 PM Zhou, Hui <zhouh(a)anl.gov > > <mailto:[email protected]>> wrote: > > > > Hi Zhifeng, > > > > We just had a new mpich release: mpich-3.3rc1. You may try that > > release see if you still have the same error. > > > > That aside, does your code uses MPI_T_ interfaces? You may try search > > MPI_T_ prefixes in your code base. In particular, I am interested in > > any MPI_T_ calls before MPI_Init call. > > > > -- > > Hui Zhou > > > > On Mon, Nov 19, 2018 at 10:39:20AM -0500, Zhifeng Yang wrote: > > >Hi Hui, > > >Here are the outputs. I tried the following commands > > >mpirun --version > > >./cpi > > >mpirun ./cpi > > >mpirun -np 1 ./cpi > > > > > >[vy57456@maya-usr1 em_real]$mpirun --version > > >HYDRA build details: > > > Version: 3.2.1 > > > Release Date: Fri Nov 10 20:21:01 > > CST 2017 > > > CC: gcc > > > CXX: g++ > > > F77: gfortran > > > F90: gfortran > > > Configure options: > > '--disable-option-checking' > > > >'--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' > > >'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' > > '--cache-file=/dev/null' > > >'--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > > >-D_REENTRANT > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' > > >'MPLLIBNAME=mpl' > > > Process Manager: pmi > > > Launchers available: ssh rsh fork slurm ll > > lsf sge > > >manual persist > > > Topology libraries available: hwloc > > > Resource management kernels available: user slurm ll lsf sge > pbs > > >cobalt > > > Checkpointing libraries available: > > > Demux engines available: poll select > > > > > > > > >[vy57456@maya-usr1 examples]$./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000066 > > > > > > > > >[vy57456@maya-usr1 examples]$mpirun ./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000095 > > > > > >[vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000093 > > > > > >There is no error. > > > > > >Zhifeng > > > > > > > > >On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov > > <mailto:[email protected]>> wrote: > > > > > >> On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: > > >> >Thank you for helping me on this error. Actually, real.exe is a > > portion of > > >> >a very large weather model. It is very difficult to extract it > or > > >> duplicate > > >> >the error in a simple fortran code, since I am not sure where > > the problem > > >> >is. From your discussion, I barely can understand them, in > > fact. Even I do > > >> >not know what is "_get_addr". Is it related to MPI? > > >> > > >> It is difficult to pin-point the problem without reproducing it. > > >> > > >> Anyway, let's start with mpirun. What is your output if you try: > > >> > > >> mpirun --version > > >> > > >> Next, what is your mpich version? If you built mpich, locate the > > `cpi` > > >> program in the examples folder and try `./cpi` and `mpirun > > ./cpi`. Do > > >> you have error? > > >> > > >> -- > > >> Hui Zhou > > >> > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Joseph Schuchart 20 Nov '18

20 Nov '18

Zhifeng, Another way to approach this is to start the application under gdb and set a breakpoint to MPICH's internal abort function (MPID_Abort iirc). Once you hit this you can walk up the stack and try to find out where _get_addr was found to be faulty. Since you are running with a single process starting under GDB should be straight forward: $ gdb -ex "b MPID_Abort" -ex r ./real.exe (If you pass arguments to real.exe you have to pass --args to gdb) Cheers Joseph On 11/19/18 1:49 PM, Zhifeng Yang via discuss wrote: > Hi Hui, > > I just searched the whole code. There is no MPI_T_* name in the code. I > may tried the newer version later on. Thank you very much > > Zhifeng > > > On Mon, Nov 19, 2018 at 12:14 PM Zhou, Hui <zhouh(a)anl.gov > <mailto:[email protected]>> wrote: > > Hi Zhifeng, > > We just had a new mpich release: mpich-3.3rc1. You may try that > release see if you still have the same error. > > That aside, does your code uses MPI_T_ interfaces? You may try search > MPI_T_ prefixes in your code base. In particular, I am interested in > any MPI_T_ calls before MPI_Init call. > > -- > Hui Zhou > > On Mon, Nov 19, 2018 at 10:39:20AM -0500, Zhifeng Yang wrote: > >Hi Hui, > >Here are the outputs. I tried the following commands > >mpirun --version > >./cpi > >mpirun ./cpi > >mpirun -np 1 ./cpi > > > >[vy57456@maya-usr1 em_real]$mpirun --version > >HYDRA build details: > > Version: 3.2.1 > > Release Date: Fri Nov 10 20:21:01 > CST 2017 > > CC: gcc > > CXX: g++ > > F77: gfortran > > F90: gfortran > > Configure options: > '--disable-option-checking' > >'--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' > >'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' > '--cache-file=/dev/null' > >'--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > >-D_REENTRANT > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' > >'MPLLIBNAME=mpl' > > Process Manager: pmi > > Launchers available: ssh rsh fork slurm ll > lsf sge > >manual persist > > Topology libraries available: hwloc > > Resource management kernels available: user slurm ll lsf sge pbs > >cobalt > > Checkpointing libraries available: > > Demux engines available: poll select > > > > > >[vy57456@maya-usr1 examples]$./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000066 > > > > > >[vy57456@maya-usr1 examples]$mpirun ./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000095 > > > >[vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000093 > > > >There is no error. > > > >Zhifeng > > > > > >On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov > <mailto:[email protected]>> wrote: > > > >> On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: > >> >Thank you for helping me on this error. Actually, real.exe is a > portion of > >> >a very large weather model. It is very difficult to extract it or > >> duplicate > >> >the error in a simple fortran code, since I am not sure where > the problem > >> >is. From your discussion, I barely can understand them, in > fact. Even I do > >> >not know what is "_get_addr". Is it related to MPI? > >> > >> It is difficult to pin-point the problem without reproducing it. > >> > >> Anyway, let's start with mpirun. What is your output if you try: > >> > >> mpirun --version > >> > >> Next, what is your mpich version? If you built mpich, locate the > `cpi` > >> program in the examples folder and try `./cpi` and `mpirun > ./cpi`. Do > >> you have error? > >> > >> -- > >> Hui Zhou > >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhou, Hui 20 Nov '18

20 Nov '18

Hi Zhifeng, I searched the mpich-3.2.1 codebase and there is no `get_addr` symbol, so it appears it is a function in your codebase. Try search that see if it yields any clues. The error is during MPI_INIT, so try identify the codes before MPI_INIT, hopefully (as it should), it comes quite early. Then you may try a smaller code that copies the code up to MPI_INIT and see if you can reproduce the errror. In the case that MPI_INIT is burried in your codebase, could you try put your code in a debugger and obtain a backtrace at the point of MPI_INIT? Also in your original email, was there some texts inside those eclipses? >> MPIR_Init_thread(784).....: >> MPID_Init(1323)...........: channel initialization failed >> MPIDI_CH3_Init(120).......: >> MPID_nem_init_ckpt(852)...: >> MPIDI_CH3I_Seg_commit(364): PMI_Barrier returned -1 -- Hui Zhou On Mon, Nov 19, 2018 at 01:49:04PM -0500, Zhifeng Yang wrote: >Hi Hui, > >I just searched the whole code. There is no MPI_T_* name in the code. I may >tried the newer version later on. Thank you very much > >Zhifeng >

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhifeng Yang 19 Nov '18

19 Nov '18

Hi Hui, I just searched the whole code. There is no MPI_T_* name in the code. I may tried the newer version later on. Thank you very much Zhifeng On Mon, Nov 19, 2018 at 12:14 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Zhifeng, > > We just had a new mpich release: mpich-3.3rc1. You may try that > release see if you still have the same error. > > That aside, does your code uses MPI_T_ interfaces? You may try search > MPI_T_ prefixes in your code base. In particular, I am interested in > any MPI_T_ calls before MPI_Init call. > > -- > Hui Zhou > > On Mon, Nov 19, 2018 at 10:39:20AM -0500, Zhifeng Yang wrote: > >Hi Hui, > >Here are the outputs. I tried the following commands > >mpirun --version > >./cpi > >mpirun ./cpi > >mpirun -np 1 ./cpi > > > >[vy57456@maya-usr1 em_real]$mpirun --version > >HYDRA build details: > > Version: 3.2.1 > > Release Date: Fri Nov 10 20:21:01 CST 2017 > > CC: gcc > > CXX: g++ > > F77: gfortran > > F90: gfortran > > Configure options: '--disable-option-checking' > > >'--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' > >'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' '--cache-file=/dev/null' > >'--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > >-D_REENTRANT > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' > >'MPLLIBNAME=mpl' > > Process Manager: pmi > > Launchers available: ssh rsh fork slurm ll lsf sge > >manual persist > > Topology libraries available: hwloc > > Resource management kernels available: user slurm ll lsf sge pbs > >cobalt > > Checkpointing libraries available: > > Demux engines available: poll select > > > > > >[vy57456@maya-usr1 examples]$./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000066 > > > > > >[vy57456@maya-usr1 examples]$mpirun ./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000095 > > > >[vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi > >Process 0 of 1 is on maya-usr1 > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > >wall clock time = 0.000093 > > > >There is no error. > > > >Zhifeng > > > > > >On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > > >> On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: > >> >Thank you for helping me on this error. Actually, real.exe is a > portion of > >> >a very large weather model. It is very difficult to extract it or > >> duplicate > >> >the error in a simple fortran code, since I am not sure where the > problem > >> >is. From your discussion, I barely can understand them, in fact. Even > I do > >> >not know what is "_get_addr". Is it related to MPI? > >> > >> It is difficult to pin-point the problem without reproducing it. > >> > >> Anyway, let's start with mpirun. What is your output if you try: > >> > >> mpirun --version > >> > >> Next, what is your mpich version? If you built mpich, locate the `cpi` > >> program in the examples folder and try `./cpi` and `mpirun ./cpi`. Do > >> you have error? > >> > >> -- > >> Hui Zhou > >> >

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhou, Hui 19 Nov '18

19 Nov '18

Hi Zhifeng, We just had a new mpich release: mpich-3.3rc1. You may try that release see if you still have the same error. That aside, does your code uses MPI_T_ interfaces? You may try search MPI_T_ prefixes in your code base. In particular, I am interested in any MPI_T_ calls before MPI_Init call. -- Hui Zhou On Mon, Nov 19, 2018 at 10:39:20AM -0500, Zhifeng Yang wrote: >Hi Hui, >Here are the outputs. I tried the following commands >mpirun --version >./cpi >mpirun ./cpi >mpirun -np 1 ./cpi > >[vy57456@maya-usr1 em_real]$mpirun --version >HYDRA build details: > Version: 3.2.1 > Release Date: Fri Nov 10 20:21:01 CST 2017 > CC: gcc > CXX: g++ > F77: gfortran > F90: gfortran > Configure options: '--disable-option-checking' >'--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' >'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' '--cache-file=/dev/null' >'--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src >-D_REENTRANT >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' >'MPLLIBNAME=mpl' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll lsf sge >manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge pbs >cobalt > Checkpointing libraries available: > Demux engines available: poll select > > >[vy57456@maya-usr1 examples]$./cpi >Process 0 of 1 is on maya-usr1 >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 >wall clock time = 0.000066 > > >[vy57456@maya-usr1 examples]$mpirun ./cpi >Process 0 of 1 is on maya-usr1 >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 >wall clock time = 0.000095 > >[vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi >Process 0 of 1 is on maya-usr1 >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 >wall clock time = 0.000093 > >There is no error. > >Zhifeng > > >On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > >> On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: >> >Thank you for helping me on this error. Actually, real.exe is a portion of >> >a very large weather model. It is very difficult to extract it or >> duplicate >> >the error in a simple fortran code, since I am not sure where the problem >> >is. From your discussion, I barely can understand them, in fact. Even I do >> >not know what is "_get_addr". Is it related to MPI? >> >> It is difficult to pin-point the problem without reproducing it. >> >> Anyway, let's start with mpirun. What is your output if you try: >> >> mpirun --version >> >> Next, what is your mpich version? If you built mpich, locate the `cpi` >> program in the examples folder and try `./cpi` and `mpirun ./cpi`. Do >> you have error? >> >> -- >> Hui Zhou >>

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhifeng Yang 19 Nov '18

19 Nov '18

Hi Hui, Here are the outputs. I tried the following commands mpirun --version ./cpi mpirun ./cpi mpirun -np 1 ./cpi [vy57456@maya-usr1 em_real]$mpirun --version HYDRA build details: Version: 3.2.1 Release Date: Fri Nov 10 20:21:01 CST 2017 CC: gcc CXX: g++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' 'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= -I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include -I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include -I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src -I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src -D_REENTRANT -I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' 'MPLLIBNAME=mpl' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select [vy57456@maya-usr1 examples]$./cpi Process 0 of 1 is on maya-usr1 pi is approximately 3.1415926544231341, Error is 0.0000000008333410 wall clock time = 0.000066 [vy57456@maya-usr1 examples]$mpirun ./cpi Process 0 of 1 is on maya-usr1 pi is approximately 3.1415926544231341, Error is 0.0000000008333410 wall clock time = 0.000095 [vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi Process 0 of 1 is on maya-usr1 pi is approximately 3.1415926544231341, Error is 0.0000000008333410 wall clock time = 0.000093 There is no error. Zhifeng On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: > >Thank you for helping me on this error. Actually, real.exe is a portion of > >a very large weather model. It is very difficult to extract it or > duplicate > >the error in a simple fortran code, since I am not sure where the problem > >is. From your discussion, I barely can understand them, in fact. Even I do > >not know what is "_get_addr". Is it related to MPI? > > It is difficult to pin-point the problem without reproducing it. > > Anyway, let's start with mpirun. What is your output if you try: > > mpirun --version > > Next, what is your mpich version? If you built mpich, locate the `cpi` > program in the examples folder and try `./cpi` and `mpirun ./cpi`. Do > you have error? > > -- > Hui Zhou >

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhou, Hui 19 Nov '18

19 Nov '18

On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: >Thank you for helping me on this error. Actually, real.exe is a portion of >a very large weather model. It is very difficult to extract it or duplicate >the error in a simple fortran code, since I am not sure where the problem >is. From your discussion, I barely can understand them, in fact. Even I do >not know what is "_get_addr". Is it related to MPI? It is difficult to pin-point the problem without reproducing it. Anyway, let's start with mpirun. What is your output if you try: mpirun --version Next, what is your mpich version? If you built mpich, locate the `cpi` program in the examples folder and try `./cpi` and `mpirun ./cpi`. Do you have error? -- Hui Zhou

1 0

Re: [mpich-discuss] how to dynamically schedule process in mpich
by Zhou, Hui 19 Nov '18

19 Nov '18

On Sun, Nov 18, 2018 at 10:33:48PM -0600, Shuwei Zhao wrote: >2. About the MPI_Info object, I tried to use mpi_comm_spawn on the same >machine, it can work. But I don't know how to spawn processes cross >machines. For example, if I have 4 processes on machine 1, and I want to >spawn 4 more processes on other machines (resource managed by qsub), how do >I need to declare in the MPI_Info object? Did you try setting the host parameter in the info object? For example: MPI_Info info; MPI_Info_create(&info, "host", "A,B,C"); MPI_Comm_spawn("worker", MPI_ARGV_NULL, n_hosts, info, 0, MPI_COMM_SELF, &comm_workers, MPI_ERRCODES_IGNORE)? MPI_Info_free(&info); Replace "A,B,C" with host name list that you want to spawn process on. -- Hui Zhou

1 0