- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] package issue on fedora
by bruno.guerraz＠orange.com 29 Nov '18

29 Nov '18

The answer is in the link provided by Zbyszek : " MPI implementation specific files MUST be installed in the directories used by the used MPI compiler ($MPI_BIN, $MPI_LIB and so on)." With my binaries installed under /usr/lib64/mpich/bin, I have the good requires -----Message d'origine----- De : Zbigniew Jędrzejewski-Szmek [mailto:[email protected]] Envoyé : mardi 27 novembre 2018 15:01 À : bruno via discuss Cc : GUERRAZ Bruno TGI/OLS Objet : Re: [mpich-discuss] package issue on fedora On Tue, Nov 27, 2018 at 01:40:32PM +0000, bruno via discuss wrote: > Hi all, > I am packaging a software that use MPICH on fedora. When I try to install the package, dnf tells me : > conflicting requests > - nothing provides libmpi.so.12()(64bit) needed by khiops-9.0.x86_64 > - nothing provides libmpicxx.so.12()(64bit) needed by > khiops-9.0.x86_64 > > And when I run "rpm -q --provides mpich" I have : > libmpi.so.12()(64bit)(mpich-x86_64) > libmpicxx.so.12()(64bit)(mpich-x86_64) The mpi libraries are not installed in the normal library search path in Fedora, hence the provides/requires tags use a suffix. This suffix also allows multiple mpi implementations to provide libmpi. See https://docs.fedoraproject.org/en-US/packaging-guidelines/MPI/. If you post your .spec file somewhere, I can take a look. Zbyszek _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] how to dynamically schedule process in mpich
by Zhou, Hui 28 Nov '18

28 Nov '18

Hi Shuwei, Were you able to run over multiple hosts directly, e.g.: ``` $ mpiexec -n 2 -hosts “hostname_1,hostname_2” ./cpi ``` ? — Hui Zhou On Nov 28, 2018, at 3:33 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com<mailto:[email protected]>> wrote: Hi hui I tried your suggestion but looks like the newly spawned jobs are using ssh to distribute on the new hosts. Howver In my machine farm, ssh & rsh connections are disabled. have to submit through qsub network interface. Is using qsub possible in mpi_comm_spawn? Thanks a lot Shuwei On Mon, Nov 19, 2018 at 09:21 Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: On Sun, Nov 18, 2018 at 10:33:48PM -0600, Shuwei Zhao wrote: >2. About the MPI_Info object, I tried to use mpi_comm_spawn on the same >machine, it can work. But I don't know how to spawn processes cross >machines. For example, if I have 4 processes on machine 1, and I want to >spawn 4 more processes on other machines (resource managed by qsub), how do >I need to declare in the MPI_Info object? Did you try setting the host parameter in the info object? For example: MPI_Info info; MPI_Info_create(&info, "host", "A,B,C"); MPI_Comm_spawn("worker", MPI_ARGV_NULL, n_hosts, info, 0, MPI_COMM_SELF, &comm_workers, MPI_ERRCODES_IGNORE)? MPI_Info_free(&info); Replace "A,B,C" with host name list that you want to spawn process on. -- Hui Zhou

1 0

Re: [mpich-discuss] how to dynamically schedule process in mpich
by Zhou, Hui 28 Nov '18

28 Nov '18

Hi Shuwei, Were you able to run over multiple hosts directly, e.g.: ``` $ mpiexec -n 2 -hosts “hostname_1,hostname_2” ./cpi ``` ? — Hui Zhou On Nov 28, 2018, at 3:33 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com<mailto:[email protected]>> wrote: Hi hui I tried your suggestion but looks like the newly spawned jobs are using ssh to distribute on the new hosts. Howver In my machine farm, ssh & rsh connections are disabled. have to submit through qsub network interface. Is using qsub possible in mpi_comm_spawn? Thanks a lot Shuwei On Mon, Nov 19, 2018 at 09:21 Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: On Sun, Nov 18, 2018 at 10:33:48PM -0600, Shuwei Zhao wrote: >2. About the MPI_Info object, I tried to use mpi_comm_spawn on the same >machine, it can work. But I don't know how to spawn processes cross >machines. For example, if I have 4 processes on machine 1, and I want to >spawn 4 more processes on other machines (resource managed by qsub), how do >I need to declare in the MPI_Info object? Did you try setting the host parameter in the info object? For example: MPI_Info info; MPI_Info_create(&info, "host", "A,B,C"); MPI_Comm_spawn("worker", MPI_ARGV_NULL, n_hosts, info, 0, MPI_COMM_SELF, &comm_workers, MPI_ERRCODES_IGNORE)? MPI_Info_free(&info); Replace "A,B,C" with host name list that you want to spawn process on. -- Hui Zhou

1 0

Re: [mpich-discuss] how to dynamically schedule process in mpich
by Shuwei Zhao 28 Nov '18

28 Nov '18

Hi hui I tried your suggestion but looks like the newly spawned jobs are using ssh to distribute on the new hosts. Howver In my machine farm, ssh & rsh connections are disabled. have to submit through qsub network interface. Is using qsub possible in mpi_comm_spawn? Thanks a lot Shuwei On Mon, Nov 19, 2018 at 09:21 Zhou, Hui <zhouh(a)anl.gov> wrote: > On Sun, Nov 18, 2018 at 10:33:48PM -0600, Shuwei Zhao wrote: > >2. About the MPI_Info object, I tried to use mpi_comm_spawn on the same > >machine, it can work. But I don't know how to spawn processes cross > >machines. For example, if I have 4 processes on machine 1, and I want to > >spawn 4 more processes on other machines (resource managed by qsub), how > do > >I need to declare in the MPI_Info object? > > Did you try setting the host parameter in the info object? For > example: > > MPI_Info info; > MPI_Info_create(&info, "host", "A,B,C"); > > MPI_Comm_spawn("worker", MPI_ARGV_NULL, n_hosts, info, 0, > MPI_COMM_SELF, &comm_workers, MPI_ERRCODES_IGNORE)? > > MPI_Info_free(&info); > > Replace "A,B,C" with host name list that you want to spawn process on. > > -- > Hui Zhou >

1 0

Re: [mpich-discuss] package issue on fedora
by Zbigniew Jędrzejewski-Szmek 27 Nov '18

27 Nov '18

On Tue, Nov 27, 2018 at 01:40:32PM +0000, bruno via discuss wrote: > Hi all, > I am packaging a software that use MPICH on fedora. When I try to install the package, dnf tells me : > conflicting requests > - nothing provides libmpi.so.12()(64bit) needed by khiops-9.0.x86_64 > - nothing provides libmpicxx.so.12()(64bit) needed by khiops-9.0.x86_64 > > And when I run "rpm -q --provides mpich" I have : > libmpi.so.12()(64bit)(mpich-x86_64) > libmpicxx.so.12()(64bit)(mpich-x86_64) The mpi libraries are not installed in the normal library search path in Fedora, hence the provides/requires tags use a suffix. This suffix also allows multiple mpi implementations to provide libmpi. See https://docs.fedoraproject.org/en-US/packaging-guidelines/MPI/. If you post your .spec file somewhere, I can take a look. Zbyszek

1 0

package issue on fedora
by bruno.guerraz＠orange.com 27 Nov '18

27 Nov '18

Hi all, I am packaging a software that use MPICH on fedora. When I try to install the package, dnf tells me : conflicting requests - nothing provides libmpi.so.12()(64bit) needed by khiops-9.0.x86_64 - nothing provides libmpicxx.so.12()(64bit) needed by khiops-9.0.x86_64 And when I run "rpm -q --provides mpich" I have : libmpi.so.12()(64bit)(mpich-x86_64) libmpicxx.so.12()(64bit)(mpich-x86_64) there is not an exact match between Requires and Provides... how can I specify that libmpi.so.12()(64bit)(mpich-x86_64) is libmpi.so.12()(64bit) ? For the record, with the same spec file I have built packages for redhat and centOS with no issue. Thank you for your help Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Balaji, Pavan 26 Nov '18

26 Nov '18

> On Nov 26, 2018, at 10:12 AM, Zhou, Hui <zhouh(a)anl.gov> wrote: > Is there difference between mpirun and mpiexec. I understand the official standards only specifies mpiexec, and mpirun often (is it always?) is a link to mpiexec. Is there `unofficial` convention for mpirun, such as always present, always a link, no difference to mpiexec at all? It appears to me that mpirun is more wide spread than mpiexec (I always have been using mpirun). They are the same. The history is that MPI-1 didn't specify anything, so implementations started using "mpirun". When MPI-2 specified a launcher, they wanted to use a different name and came up with "mpiexec". In current implementations there is no difference. In mpich, they are both symlinks to mpiexec.hydra. -- Pavan

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhou, Hui 26 Nov '18

26 Nov '18

Hi Pavan, Is there difference between mpirun and mpiexec. I understand the official standards only specifies mpiexec, and mpirun often (is it always?) is a link to mpiexec. Is there `unofficial` convention for mpirun, such as always present, always a link, no difference to mpiexec at all? It appears to me that mpirun is more wide spread than mpiexec (I always have been using mpirun). — Hui Zhou On Nov 21, 2018, at 5:34 PM, Balaji, Pavan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I should clarify one piece of how mpiexec works: % ./real.exe and % mpirun -np 1 ./real.exe are equivalent. % mpirun ./real.exe uses some number of processes depending on the environment. For unmanaged clusters, that's typically 1. For clusters that have some job management system (such as slurm or pbs), mpiexec will figure out how many nodes you allocated and use all of the cores allocated to that job. My guess is that real.exe has some dependencies that are met on the local machine, but not on other machines. So when mpiexec tries to launch real.exe on other nodes, it's throwing an error. This is not an mpiexec problem, but you might want to use the -prepend-pattern option in mpiexec to figure out where the error is coming from. Something like this: mpiexec -prepend-pattern %h ./real.exe -- Pavan On Nov 4, 2018, at 10:27 AM, Zhifeng Yang via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi After I installed mpich and used it in a FORTRAN code. There is an error while running this FORTRAN code called real.exe by using the following command $./real.exe or $mpirun ./real.exe real.exe: error: _get_addr: No such file or directory real.exe: error: _get_addr: No such file or directory real.exe: error: _get_addr: No such file or directory Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(784).....: MPID_Init(1323)...........: channel initialization failed MPIDI_CH3_Init(120).......: MPID_nem_init_ckpt(852)...: MPIDI_CH3I_Seg_commit(364): PMI_Barrier returned -1 But while using mpirun with specifying number of processors. as follows $mpirun -np 1 ./real.exe There is no error any more. I am not sure why? do you have any explanation? Thank you very much Best regards Zhifeng _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Balaji, Pavan 21 Nov '18

21 Nov '18

I should clarify one piece of how mpiexec works: % ./real.exe and % mpirun -np 1 ./real.exe are equivalent. % mpirun ./real.exe uses some number of processes depending on the environment. For unmanaged clusters, that's typically 1. For clusters that have some job management system (such as slurm or pbs), mpiexec will figure out how many nodes you allocated and use all of the cores allocated to that job. My guess is that real.exe has some dependencies that are met on the local machine, but not on other machines. So when mpiexec tries to launch real.exe on other nodes, it's throwing an error. This is not an mpiexec problem, but you might want to use the -prepend-pattern option in mpiexec to figure out where the error is coming from. Something like this: mpiexec -prepend-pattern %h ./real.exe -- Pavan > On Nov 4, 2018, at 10:27 AM, Zhifeng Yang via discuss <discuss(a)mpich.org> wrote: > > Hi > > After I installed mpich and used it in a FORTRAN code. There is an error while running this FORTRAN code called real.exe by using the following command > > $./real.exe > or > $mpirun ./real.exe > real.exe: error: _get_addr: No such file or directory > real.exe: error: _get_addr: No such file or directory > real.exe: error: _get_addr: No such file or directory > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(784).....: > MPID_Init(1323)...........: channel initialization failed > MPIDI_CH3_Init(120).......: > MPID_nem_init_ckpt(852)...: > MPIDI_CH3I_Seg_commit(364): PMI_Barrier returned -1 > > But while using mpirun with specifying number of processors. as follows > $mpirun -np 1 ./real.exe > There is no error any more. I am not sure why? do you have any explanation? Thank you very much > > Best regards > Zhifeng > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] _get_addr error while running application using MPICH
by Zhifeng Yang 21 Nov '18

21 Nov '18

Hi Joseph, I have not tried gdb yet. But I found a description in the initialization module. Here it says ! <DESCRIPTION> ! This routine USES the modules in WRF and then calls the init routines ! they provide to perform module specific initializations at the ! beginning of a run. Note, this is only once per run, not once per ! domain; domain specific initializations should be handled elsewhere, ! such as in <a href=start_domain.html>start_domain</a>. ! ! Certain framework specific module initializations in this file are ! dependent on order they are called. For example, since the quilt module ! relies on internal I/O, the init routine for internal I/O must be ! called first. In the case of DM_PARALLEL compiles, the quilt module ! calls MPI_INIT as part of setting up and dividing communicators between ! compute and I/O server tasks. Therefore, it must be called prior to ! module_dm, which will <em>also</em> try to call MPI_INIT if it sees ! that MPI has not be initialized yet (implementations of module_dm ! should in fact behave this way by first calling MPI_INITIALIZED before ! they try to call MPI_INIT). If MPI is already initialized before the ! the quilting module is called, quilting will not work. ! ! The phase argument is used to allow other superstructures like ESMF to ! place their initialization calls following the WRF initialization call ! that calls MPI_INIT(). When used with ESMF, ESMF will call wrf_init() ! which in turn will call phase 2 of this routine. Phase 1 will be called ! earlier. ! ! </DESCRIPTION> INTEGER, INTENT(IN) :: phase ! phase==1 means return after MPI_INIT() ! phase==2 means resume after MPI_INIT() It mentions something about MPI_INIT, but I can not understand its meaning. It may help you to understand. Best, Zhifeng On Tue, Nov 20, 2018 at 10:17 AM Joseph Schuchart via discuss < discuss(a)mpich.org> wrote: > Zhifeng, > > Another way to approach this is to start the application under gdb and > set a breakpoint to MPICH's internal abort function (MPID_Abort iirc). > Once you hit this you can walk up the stack and try to find out where > _get_addr was found to be faulty. Since you are running with a single > process starting under GDB should be straight forward: > > $ gdb -ex "b MPID_Abort" -ex r ./real.exe > > (If you pass arguments to real.exe you have to pass --args to gdb) > > Cheers > Joseph > > On 11/19/18 1:49 PM, Zhifeng Yang via discuss wrote: > > Hi Hui, > > > > I just searched the whole code. There is no MPI_T_* name in the code. I > > may tried the newer version later on. Thank you very much > > > > Zhifeng > > > > > > On Mon, Nov 19, 2018 at 12:14 PM Zhou, Hui <zhouh(a)anl.gov > > <mailto:[email protected]>> wrote: > > > > Hi Zhifeng, > > > > We just had a new mpich release: mpich-3.3rc1. You may try that > > release see if you still have the same error. > > > > That aside, does your code uses MPI_T_ interfaces? You may try search > > MPI_T_ prefixes in your code base. In particular, I am interested in > > any MPI_T_ calls before MPI_Init call. > > > > -- > > Hui Zhou > > > > On Mon, Nov 19, 2018 at 10:39:20AM -0500, Zhifeng Yang wrote: > > >Hi Hui, > > >Here are the outputs. I tried the following commands > > >mpirun --version > > >./cpi > > >mpirun ./cpi > > >mpirun -np 1 ./cpi > > > > > >[vy57456@maya-usr1 em_real]$mpirun --version > > >HYDRA build details: > > > Version: 3.2.1 > > > Release Date: Fri Nov 10 20:21:01 > > CST 2017 > > > CC: gcc > > > CXX: g++ > > > F77: gfortran > > > F90: gfortran > > > Configure options: > > '--disable-option-checking' > > > >'--prefix=/umbc/xfs1/zzbatmos/users/vy57456/application/gfortran/mpich-3.2.1' > > >'CC=gcc' 'CXX=g++' 'FC=gfortran' 'F77=gfortran' > > '--cache-file=/dev/null' > > >'--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpl/include > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/openpa/src > > >-D_REENTRANT > > > >-I/home/vy57456/zzbatmos_user/application/gfortran/source_code/mpich-3.2.1/src/mpi/romio/include' > > >'MPLLIBNAME=mpl' > > > Process Manager: pmi > > > Launchers available: ssh rsh fork slurm ll > > lsf sge > > >manual persist > > > Topology libraries available: hwloc > > > Resource management kernels available: user slurm ll lsf sge > pbs > > >cobalt > > > Checkpointing libraries available: > > > Demux engines available: poll select > > > > > > > > >[vy57456@maya-usr1 examples]$./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000066 > > > > > > > > >[vy57456@maya-usr1 examples]$mpirun ./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000095 > > > > > >[vy57456@maya-usr1 examples]$mpirun -np 1 ./cpi > > >Process 0 of 1 is on maya-usr1 > > >pi is approximately 3.1415926544231341, Error is 0.0000000008333410 > > >wall clock time = 0.000093 > > > > > >There is no error. > > > > > >Zhifeng > > > > > > > > >On Mon, Nov 19, 2018 at 10:33 AM Zhou, Hui <zhouh(a)anl.gov > > <mailto:[email protected]>> wrote: > > > > > >> On Mon, Nov 19, 2018 at 10:14:54AM -0500, Zhifeng Yang wrote: > > >> >Thank you for helping me on this error. Actually, real.exe is a > > portion of > > >> >a very large weather model. It is very difficult to extract it > or > > >> duplicate > > >> >the error in a simple fortran code, since I am not sure where > > the problem > > >> >is. From your discussion, I barely can understand them, in > > fact. Even I do > > >> >not know what is "_get_addr". Is it related to MPI? > > >> > > >> It is difficult to pin-point the problem without reproducing it. > > >> > > >> Anyway, let's start with mpirun. What is your output if you try: > > >> > > >> mpirun --version > > >> > > >> Next, what is your mpich version? If you built mpich, locate the > > `cpi` > > >> program in the examples folder and try `./cpi` and `mpirun > > ./cpi`. Do > > >> you have error? > > >> > > >> -- > > >> Hui Zhou > > >> > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0