March 2015 - discuss - mailman3.cels.anl.gov

Troubles compiling mpich 3.1.3 and 3.1.4 with MX
by Caroline Papadopoulos 12 Mar '15

12 Mar '15

Dear list, I am having issues compiling the following versions of mpich: 3.1.3 3.1.4 but 3.1.2 and earlier versions compile just fine. I am using the same compile options except MYINSTALLDIR is updated to the correct version: > export MYINSTALLDIR=/export/home/root/bin/mpich-3.1.3_pgi-15.3 > export CC=pgcc > export CXX=pgcpp > export F77=pgfortran > export FC=pgf90 > unset F90 > export CFLAGS='' > export CXXFLAGS=${CFLAGS} > export FFLAGS=${CFLAGS} > export FCFLAGS=${FFLAGS} > unset F90FLAGS > ./configure --prefix=${MYINSTALLDIR} --with-device=ch3:nemesis:mx --with-mx-include=/opt/mx/include --with-mx-lib=/opt/mx/lib >& config.txt > make >& make.txt > make install >& make_install.txt The error that I am getting for 3.1.3 and 3.1.4 is: > CC src/pmi/simple/lib_libmpi_la-simple_pmiutil.lo > CC src/pmi/simple/lib_libmpi_la-simple_pmi.lo > GEN lib/libmpi.la > /usr/bin/ld: cannot find -lmx > make[2]: *** [lib/libmpi.la] Error 2 > make[2]: Leaving directory `/export/home/root/src/mpich-3.1.3' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/export/home/root/src/mpich-3.1.3' > make: *** [all] Error 2 In fact the mx library is called: libmyriexpress.a libmyriexpress.so rather than libmx.a and libmx.so So is there a configure option to have it use -lmyriexpress instead of -lmx ?? This all works fine up to 3.1.3, so for now I am just going to use 3.1.2 but did want to update to the newest version 3.1.4. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra and oversubscription on a TORQUE/PBS cluster
by Balaji, Pavan 11 Mar '15

11 Mar '15

Arjen, I'm not sure I understand exactly what you are doing. Suppose you allocated 4 nodes, each with 4 cores -- * If you just do "mpiexec ./foo", it should automatically pick the number of cores allocated and run a 16 process job. * If you do "mpiexec -n 100 ./foo", it'll run 100 processes on 4 nodes (ranks 0-3 on node 0, ranks 1-4 on node 1, ..., ranks 16-19 on node 0, ...). But, well, you did ask mpiexec to run 100 processes, so I'm not sure it's doing anything wrong here. Is this behavior not what you are seeing? Or are you expecting a different behavior? -- Pavan > On Mar 11, 2015, at 9:48 AM, Arjen van Elteren <info(a)arjenvanelteren.com> wrote: > > Hello, > > I would like to use mpiexec.hydra on a cluster managed with torque. I > have compiled mpiexec.hydra with support for pbs as the resource manager > and launcher. All of this works fine and I can submit and run jobs just > fine on the cluster. > > Unfortunately, mpiexec.hydra does not respect the available nodes gived > to it via the PBS_NODEFILE environment variable. Instead it happily > oversubscribes even if I give it the -ppn variable. > > Apparently the torque system expects the codes to abide by the rules and > keep within their boundaries, but does not enforce this. > > The normal mpiexec provided with PBS > (https://www.osc.edu/~djohnson/mpiexec/index.php) has accounting to do > exactly so and will fail if you oversubscribe. (But development seems to > have stalled) > > There are cases where oversubscription is a useful option, but I would > like mpiexec to fail when this happens (BTW it would also be nice if it > failed when you give it a host file, this seems to be the same logic in > the code) > > This problem seems to be located src/pm/hydra/utils/alloc/alloc.c in > HYDU_create_proxy_list, where the nodes are simply oversubscribed > instead of failing. Maybe the processor binding logic could also limit > this, but I could not find any good reference for that. > > I'm I right in assuming the place to change this is in the > HYDU_create_proxy_list function and would it be possible to have a > compile option that disables oversubscription? (which would make system > management on this cluster happier). I'm a bit unsure if and how this > would work for hierarchical proxy's, does the launch node do all the > accounting? > > Kind regards, > > Arjen > > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpiexec.hydra and oversubscription on a TORQUE/PBS cluster
by Arjen van Elteren 11 Mar '15

11 Mar '15

Hello, I would like to use mpiexec.hydra on a cluster managed with torque. I have compiled mpiexec.hydra with support for pbs as the resource manager and launcher. All of this works fine and I can submit and run jobs just fine on the cluster. Unfortunately, mpiexec.hydra does not respect the available nodes gived to it via the PBS_NODEFILE environment variable. Instead it happily oversubscribes even if I give it the -ppn variable. Apparently the torque system expects the codes to abide by the rules and keep within their boundaries, but does not enforce this. The normal mpiexec provided with PBS (https://www.osc.edu/~djohnson/mpiexec/index.php) has accounting to do exactly so and will fail if you oversubscribe. (But development seems to have stalled) There are cases where oversubscription is a useful option, but I would like mpiexec to fail when this happens (BTW it would also be nice if it failed when you give it a host file, this seems to be the same logic in the code) This problem seems to be located src/pm/hydra/utils/alloc/alloc.c in HYDU_create_proxy_list, where the nodes are simply oversubscribed instead of failing. Maybe the processor binding logic could also limit this, but I could not find any good reference for that. I'm I right in assuming the place to change this is in the HYDU_create_proxy_list function and would it be possible to have a compile option that disables oversubscription? (which would make system management on this cluster happier). I'm a bit unsure if and how this would work for hierarchical proxy's, does the launch node do all the accounting? Kind regards, Arjen _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] discuss Digest, Vol 29, Issue 7
by Zhao, Xin 09 Mar '15

09 Mar '15

Hi Sylvain, You can find the related details in README file under MPICH directory. Thanks, Xin ________________________________ From: Korzennik, Sylvain [skorzennik(a)cfa.harvard.edu] Sent: Monday, March 09, 2015 6:25 PM To: Zhao, Xin Subject: Fwd: discuss Digest, Vol 29, Issue 7 Thanks, this worked (build and install) I have yet to test it. Is there a documentation on the values that can be used for --with-device and more details on enabling IB? ---------- Forwarded message ---------- From: <discuss-request(a)mpich.org<mailto:[email protected]>> Date: Mon, Mar 9, 2015 at 4:39 PM Subject: discuss Digest, Vol 29, Issue 7 To: discuss(a)mpich.org<mailto:[email protected]> [...] ---------------------------------------------------------------------- Message: 1 Date: Sat, 7 Mar 2015 23:46:47 +0000 From: "Zhao, Xin" <xinzhao3(a)illinois.edu<mailto:[email protected]>> To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] Building mpich 1.3.4 w/ IB enabled Message-ID: <0A407957589BAB4F924824150C4293EF5887668C(a)CITESMBX3.ad.uillinois.edu<mailto:0A407957589BAB4F924824150C4293EF5887668C@CITESMBX3.ad.uillinois.edu>> Content-Type: text/plain; charset="utf-8" Hi Sylvain, IB netmod is contributed by RIKEN and they are working on cleaning up this netmod. At the meantime, I would recommend using MXM netmod instead by specifying ??with-device=ch3:nemesis:mxm?. MXM is a netmod for IB networks using the MXM API instead of Verbs to interact with the HCA. Note that it is only for Mellanox hardware. Thanks, Xin ________________________________ From: Korzennik, Sylvain [skorzennik(a)cfa.harvard.edu<mailto:[email protected]>] Sent: Saturday, March 07, 2015 5:18 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: [mpich-discuss] Building mpich 1.3.4 w/ IB enabled I downloaded mpich-1.3.4.tar.gz and was able to build it with GCC 4.9.1 and the following configure flags: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest If instead I try to build w/ IB support, namely: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis:ib \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest I get the following (slew) of errors when building (make): [...] Cheers, Sylvain -- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Non-constant extent for derived data types
by mhgardner 09 Mar '15

09 Mar '15

Thanks, and sorry about that. Will keep future questions related to MPICH. On Mon, Mar 9, 2015 at 1:36 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > No, datatypes are meant to be persistent for a reasonably long period. > > Also, for general MPI questions, please use stack overflow or equivalent. > This list is for MPICH related questions. > > -- Pavan > > > On Mar 9, 2015, at 3:30 PM, mhgardner <mhgardner(a)berkeley.edu> wrote: > > > > Hi, > > > > I'm rather new to MPI, but I'm trying to define a derived MPI struct > data type. The data type is made up of int, float and vector types. The > number of entries in the vector elements are not constant throughout the > analysis, so is there a way to define an MPI data type that can deal with > vectors of changing size? Or is it better to create vectors of fixed extent > and just add enough padding so they don't overflow? > > > > Thanks, > > > > Mike > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Non-constant extent for derived data types
by Balaji, Pavan 09 Mar '15

09 Mar '15

No, datatypes are meant to be persistent for a reasonably long period. Also, for general MPI questions, please use stack overflow or equivalent. This list is for MPICH related questions. -- Pavan > On Mar 9, 2015, at 3:30 PM, mhgardner <mhgardner(a)berkeley.edu> wrote: > > Hi, > > I'm rather new to MPI, but I'm trying to define a derived MPI struct data type. The data type is made up of int, float and vector types. The number of entries in the vector elements are not constant throughout the analysis, so is there a way to define an MPI data type that can deal with vectors of changing size? Or is it better to create vectors of fixed extent and just add enough padding so they don't overflow? > > Thanks, > > Mike > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Non-constant extent for derived data types
by mhgardner 09 Mar '15

09 Mar '15

Hi, I'm rather new to MPI, but I'm trying to define a derived MPI struct data type. The data type is made up of int, float and vector types. The number of entries in the vector elements are not constant throughout the analysis, so is there a way to define an MPI data type that can deal with vectors of changing size? Or is it better to create vectors of fixed extent and just add enough padding so they don't overflow? Thanks, Mike

1 0

Re: [mpich-discuss] (no subject)
by Huiwei Lu 07 Mar '15

07 Mar '15

Please use this mailing list for discussion MPICH related questions only. If you have general parallel computing questions, please ask it on stack overflow or other generic discussion Forums. -- Huiwei Lu On Sat, Mar 7, 2015 at 5:56 AM, Chafik sanaa <san.chafik(a)gmail.com> wrote: > when I execute my program nothing appears, it must use a parallelism even > for reading our text file !? > this is my code: > #include <stdio.h> > #include <stdlib.h> > #include "mpi.h" > #define MAX_LIGNE 40 > #define MAX_COL 40 > #define Matrice_adjacence "fichier22.txt" > #define Matrice_General "fichier111.txt" > typedef int Matrice[MAX_LIGNE][MAX_COL]; > int main(int argc, char** argv) > { > int nbr_proc, rang_proc; > Matrice E; > int m1, n1; > FILE * aLire = fopen(Matrice_General, "r"); > if (aLire == NULL) > { > printf("No file"); > } > else > { > fscanf(aLire, "%d%d", &n1, &m1); > for (int i = 0; i < n1; i++) { > for (int j = 0; j < m1; j++) > fscanf(aLire, "%d", &E[i][j]); > fscanf(aLire, "\n"); > } > } > fclose(aLire); > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &rang_proc); > MPI_Comm_size(MPI_COMM_WORLD, &nbr_proc); > > printf("Hi from proc %d ", rang_proc); > > MPI_Barrier(MPI_COMM_WORLD); > MPI_Finalize(); > } > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building mpich 1.3.4 w/ IB enabled
by Zhao, Xin 07 Mar '15

07 Mar '15

Hi Sylvain, IB netmod is contributed by RIKEN and they are working on cleaning up this netmod. At the meantime, I would recommend using MXM netmod instead by specifying “—with-device=ch3:nemesis:mxm”. MXM is a netmod for IB networks using the MXM API instead of Verbs to interact with the HCA. Note that it is only for Mellanox hardware. Thanks, Xin ________________________________ From: Korzennik, Sylvain [skorzennik(a)cfa.harvard.edu] Sent: Saturday, March 07, 2015 5:18 PM To: discuss(a)mpich.org Subject: [mpich-discuss] Building mpich 1.3.4 w/ IB enabled I downloaded mpich-1.3.4.tar.gz and was able to build it with GCC 4.9.1 and the following configure flags: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest If instead I try to build w/ IB support, namely: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis:ib \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest I get the following (slew) of errors when building (make): CC src/mpid/ch3/channels/nemesis/netmod/ib/lib_libmpi_la-ib_poll.lo src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c: In function 'MPID_nem_ib_Pkt\ Handler_Put': src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1839:24: error: 'MPIDI_CH3_Re\ qHandler_PutRecvComplete' undeclared (first use in this function) req->dev.OnFinal = MPIDI_CH3_ReqHandler_PutRecvComplete; ^ src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1839:24: note: each undeclared identifier is reported only once for each function it appears in src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1847:20: error: 'MPIDI_CH3_Pkt_put_t' has no member named 'immed_len' if (put_pkt->immed_len > 0) { ^ In file included from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_memdefs.h:10:0, from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h:12, from ./src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h:9, from ./src/mpid/ch3/include/mpidpre.h:49, from ./src/include/mpiimpl.h:172, from ./src/mpid/ch3/include/mpidimpl.h:22, from ./src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h:10, from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h:11, from src/mpid/ch3/channels/nemesis/netmod/ib/ib_impl.h:11, from src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:8: src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1849:66: error: 'MPIDI_CH3_Pkt_put_t' has no member named 'immed_len' MPIU_Memcpy(req->dev.user_buf, put_pkt->data, put_pkt->immed_len); [...] What I am doing wrong? Is there a documentation on the values that can be used for --with-device and how to build mpich w/ InfiniBand support. Cheers, Sylvain -- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Building mpich 1.3.4 w/ IB enabled
by Korzennik, Sylvain 07 Mar '15

07 Mar '15

I downloaded mpich-1.3.4.tar.gz and was able to build it with GCC 4.9.1 and the following configure flags: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest If instead I try to build w/ IB support, namely: ./configure \ CC=/opt/gnu/gcc/bin/gcc \ FC=/opt/gnu/gcc/bin/gfortran \ CXX=/opt/gnu/gcc/bin/g++ \ --enable-static \ --with-device=ch3:nemesis:ib \ --with-thread-package=posix \ --with-pm=hydra \ --enable-shared \ --enable-debuginfo \ --prefix=$dest I get the following (slew) of errors when building (make): CC src/mpid/ch3/channels/nemesis/netmod/ib/lib_libmpi_la-ib_poll.lo src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c: In function 'MPID_nem_ib_Pkt\ Handler_Put': src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1839:24: error: 'MPIDI_CH3_Re\ qHandler_PutRecvComplete' undeclared (first use in this function) req->dev.OnFinal = MPIDI_CH3_ReqHandler_PutRecvComplete; ^ src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1839:24: note: each undeclared identifier is reported only once for each function it appears in src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1847:20: error: 'MPIDI_CH3_Pkt_put_t' has no member named 'immed_len' if (put_pkt->immed_len > 0) { ^ In file included from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_memdefs.h:10:0, from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h:12, from ./src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h:9, from ./src/mpid/ch3/include/mpidpre.h:49, from ./src/include/mpiimpl.h:172, from ./src/mpid/ch3/include/mpidimpl.h:22, from ./src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h:10, from ./src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h:11, from src/mpid/ch3/channels/nemesis/netmod/ib/ib_impl.h:11, from src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:8: src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c:1849:66: error: 'MPIDI_CH3_Pkt_put_t' has no member named 'immed_len' MPIU_Memcpy(req->dev.user_buf, put_pkt->data, put_pkt->immed_len); [...] What I am doing wrong? Is there a documentation on the values that can be used for --with-device and how to build mpich w/ InfiniBand support. Cheers, Sylvain -- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0