February 2015 - discuss - mailman3.cels.anl.gov

mpich-3.1.4 error in mx_init (error no mx device entry in /dev
by Khalid Hasanov 25 Feb '15

25 Feb '15

Hello, I have an issue with mpich-3.1.4 when I run mpirun on a cluster which has Infiniband-20G and Gigabit Ethernet networks: error in mx_init (error no mx device entry in /dev As far as I know in Open MPI exactly the same error message is considered as a warning which may come from collisions between IB and MYRINET technologies and these warnings can be disabled by MCA parameters. Is there any similar option in mpich3 to disable/solve the error? Best regards, Khalid _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Is there any optimization of collective calls (MPI_Allreduce) for 2^n ranks?
by Junchao Zhang 25 Feb '15

25 Feb '15

Yes. Many collectives have optimizations for power-of-two processes. In MPICH's source code allreduce.c, you can find the following comments. /* This is the default implementation of allreduce. The algorithm is: Algorithm: MPI_Allreduce For the heterogeneous case, we call MPI_Reduce followed by MPI_Bcast in order to meet the requirement that all processes must have the same result. For the homogeneous case, we use the following algorithms. For long messages and for builtin ops and if count >= pof2 (where pof2 is the nearest power-of-two less than or equal to the number of processes), we use Rabenseifner's algorithm (see http://www.hlrs.de/mpi/myreduce.html) This algorithm implements the allreduce in two steps: first a reduce-scatter, followed by an allgather. A recursive-halving algorithm (beginning with processes that are distance 1 apart) is used for the reduce-scatter, and a recursive doubling algorithm is used for the allgather. The non-power-of-two case is handled by dropping to the nearest lower power-of-two: the first few even-numbered processes send their data to their right neighbors (rank+1), and the reduce-scatter and allgather happen among the remaining power-of-two processes. At the end, the first few even-numbered processes get the result from their right neighbors. For the power-of-two case, the cost for the reduce-scatter is lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the allgather lgp.alpha + n.((p-1)/p).beta. Therefore, the total cost is: Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma For the non-power-of-two case, Cost = (2.floor(lgp)+2).alpha + (2.((p-1)/p) + 2).n.beta + n.(1+(p-1)/p).gamma For short messages, for user-defined ops, and for count < pof2 we use a recursive doubling algorithm (similar to the one in MPI_Allgather). We use this algorithm in the case of user-defined ops because in this case derived datatypes are allowed, and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky. Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma Possible improvements: End Algorithm: MPI_Allreduce */ --Junchao Zhang On Wed, Feb 25, 2015 at 2:59 PM, Aiman Fang <aimanf(a)cs.uchicago.edu> wrote: > Hi, > > I came across a problem in experiments that makes me wondering if there is > any optimization of collective calls, such as MPI_Allreduce, for 2^n number > of ranks? > > We did some experiments on Argonne Vesta system to measure the time of > MPI_Allreduce calls using 511, 512 and 513 processes. (one process per > node). In each run, the synthetic benchmark first did some computation and > then called MPI_Allreduce 30 times, for total 100 loops. We measured the > total time spent on communication. > > We found that 512-process run gives the best performance. The time for > 511, 512 and 513 processes are 0.1492, 0.1449 and 0.1547 seconds > respectively. 512-proc outperforms 511-proc by 3.7%, and 513-proc by 6.7%. > > The mpich version we used is as follows. > > $ mpichversion > MPICH Version: 3.1.2 > MPICH Release date: Mon Jul 21 16:00:21 CDT 2014 > MPICH Device: pamid > MPICH configure: --prefix=/home/fujita/soft/mpich-3.1.2 > --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs:BGQ > --disable-wrapper-rpath > MPICH CC: powerpc64-bgq-linux-gcc -O2 > MPICH CXX: powerpc64-bgq-linux-g++ -O2 > MPICH F77: powerpc64-bgq-linux-gfortran -O2 > MPICH FC: powerpc64-bgq-linux-gfortran -O2 > > Thanks! > > Best, > Aiman > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Is there any optimization of collective calls (MPI_Allreduce) for 2^n ranks?
by Aiman Fang 25 Feb '15

25 Feb '15

Hi, I came across a problem in experiments that makes me wondering if there is any optimization of collective calls, such as MPI_Allreduce, for 2^n number of ranks? We did some experiments on Argonne Vesta system to measure the time of MPI_Allreduce calls using 511, 512 and 513 processes. (one process per node). In each run, the synthetic benchmark first did some computation and then called MPI_Allreduce 30 times, for total 100 loops. We measured the total time spent on communication. We found that 512-process run gives the best performance. The time for 511, 512 and 513 processes are 0.1492, 0.1449 and 0.1547 seconds respectively. 512-proc outperforms 511-proc by 3.7%, and 513-proc by 6.7%. The mpich version we used is as follows. $ mpichversion MPICH Version: 3.1.2 MPICH Release date: Mon Jul 21 16:00:21 CDT 2014 MPICH Device: pamid MPICH configure: --prefix=/home/fujita/soft/mpich-3.1.2 --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs:BGQ --disable-wrapper-rpath MPICH CC: powerpc64-bgq-linux-gcc -O2 MPICH CXX: powerpc64-bgq-linux-g++ -O2 MPICH F77: powerpc64-bgq-linux-gfortran -O2 MPICH FC: powerpc64-bgq-linux-gfortran -O2 Thanks! Best, Aiman _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] query ABI version
by Jeff Hammond 23 Feb '15

23 Feb '15

Thanks for creating ticket 2231 for this. I am making the explicit reference here in case anyone else wants to follow it. Best, Jeff On Fri, Feb 20, 2015 at 11:32 AM, Wesley Bland <wbland(a)anl.gov> wrote: > I don’t think there’s a good answer for this. The file maint/version.m4 is where we keep the ABI string: > > m4_define([libmpi_so_version_m4],[0:0:0]) > > Obviously, that’s pretty terrible to try to parse for a user. Also, in unreleased versions, those numbers will remain 0. It’s just in tarballs that the number will be updated. > > Otherwise, you can get the name at the end of your shared library string that libtool appends. If this is important, we’ll have to figure out something for future versions. > > Thanks, > Wesley > >> On Feb 19, 2015, at 2:52 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> How do I query the MPICH ABI information at compile and run time? I >> grepped the source and it isn't obvious. >> >> Are there preprocess symbols in mpi.h? I didn't see them. >> >> Is the ABI version encoded in the output of MPI_Get_library_version? >> I have not tested this myself, for which I am deeply sorry. If I am >> adverse to string parsing, is there a way to get this information more >> directly from the tools interface, or perhaps as a non-standard >> attribute on MPI_COMM_WORLD? >> >> Is there a way to encode the ABI version in the binary itself via >> something like MPICH_ABI_SYMBOL_1_1? >> >> I do not want to rely upon any mechanism that is associated with >> shared libraries, because I want a solution that works with CrayMPI >> when linking static binaries. >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] query ABI version
by Jeff Hammond 23 Feb '15

23 Feb '15

Why not keep the ABI defined in the devel version? Shouldn't you be able to keep the ABI well-defined and change it when it needs changing? In any case, I do not really care. This is about querying the ABI of well-defined releases of MPICH and its derivatives. For example, I want to be able to download a release of MPICH, build a library against it and then be able to redistribute that library to users that might build their application with Cray MPI or Intel MPI. As a responsible library developer, I want to detect any ABI problems as early as possible rather than have potentially silent errors. Right now, it seems rather difficult for me to do this. Jeff On Sun, Feb 22, 2015 at 4:35 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> On Feb 20, 2015, at 1:32 PM, Wesley Bland <wbland(a)anl.gov> wrote: >> Obviously, that’s pretty terrible to try to parse for a user. Also, in unreleased versions, those numbers will remain 0. It’s just in tarballs that the number will be updated. > > It's updated only in stable releases. Alpha, beta, RC releases all keep it at 0:0:0. > > -- Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] query ABI version
by Balaji, Pavan 22 Feb '15

22 Feb '15

> On Feb 20, 2015, at 1:32 PM, Wesley Bland <wbland(a)anl.gov> wrote: > Obviously, that’s pretty terrible to try to parse for a user. Also, in unreleased versions, those numbers will remain 0. It’s just in tarballs that the number will be updated. It's updated only in stable releases. Alpha, beta, RC releases all keep it at 0:0:0. -- Pavan -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] query ABI version
by Wesley Bland 20 Feb '15

20 Feb '15

I don’t think there’s a good answer for this. The file maint/version.m4 is where we keep the ABI string: m4_define([libmpi_so_version_m4],[0:0:0]) Obviously, that’s pretty terrible to try to parse for a user. Also, in unreleased versions, those numbers will remain 0. It’s just in tarballs that the number will be updated. Otherwise, you can get the name at the end of your shared library string that libtool appends. If this is important, we’ll have to figure out something for future versions. Thanks, Wesley > On Feb 19, 2015, at 2:52 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > How do I query the MPICH ABI information at compile and run time? I > grepped the source and it isn't obvious. > > Are there preprocess symbols in mpi.h? I didn't see them. > > Is the ABI version encoded in the output of MPI_Get_library_version? > I have not tested this myself, for which I am deeply sorry. If I am > adverse to string parsing, is there a way to get this information more > directly from the tools interface, or perhaps as a non-standard > attribute on MPI_COMM_WORLD? > > Is there a way to encode the ABI version in the binary itself via > something like MPICH_ABI_SYMBOL_1_1? > > I do not want to rely upon any mechanism that is associated with > shared libraries, because I want a solution that works with CrayMPI > when linking static binaries. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

query ABI version
by Jeff Hammond 19 Feb '15

19 Feb '15

How do I query the MPICH ABI information at compile and run time? I grepped the source and it isn't obvious. Are there preprocess symbols in mpi.h? I didn't see them. Is the ABI version encoded in the output of MPI_Get_library_version? I have not tested this myself, for which I am deeply sorry. If I am adverse to string parsing, is there a way to get this information more directly from the tools interface, or perhaps as a non-standard attribute on MPI_COMM_WORLD? Is there a way to encode the ABI version in the binary itself via something like MPICH_ABI_SYMBOL_1_1? I do not want to rely upon any mechanism that is associated with shared libraries, because I want a solution that works with CrayMPI when linking static binaries. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Finalize hangs in dynamic connection in case of failed process
by Roy, Hirak 19 Feb '15

19 Feb '15

Hi All, I am using MPICH with sock connection. I also setup processes using dynamic connection method (MPI_Comm_connect/MPI_Comm_accept). It's a master-slave architecture where master accepts the connections from slaves. Now if one of the process dies (or get killed), I can still recover from this (without using checkpoint/restore method). For the particular process in master, I do not call MPI_disconnect (it hangs and does not complete). As a result, my MPI_Finalize in master hangs and does not complete. Do you have a workaround to forcefully complete MPI_Finalize or MPI_disconnect? I tried MPI_Comm_free on the failed connection. However, it does not solve the hang in finalize. Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Memory alignment with MPI_Alloc_mem
by Marcin Zalewski 19 Feb '15

19 Feb '15

On Wed, Feb 18, 2015 at 10:01 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > On Mon, Feb 16, 2015 at 6:23 AM, Marcin Zalewski > <marcin.zalewski(a)gmail.com> wrote: >> Thanks for your answer Jeff. >> >> On Mon, Feb 16, 2015 at 1:42 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> If you are going to suballocate from a slab yourself, you can handle >>> alignment yourself easy enough, no? Do I not understand what you mean >>> here. And what sort of alignment do you want? Are you trying to >>> align to 32/64 bytes because of AVX or some other x86 feature on Cray >>> XC30 or do you want page alignment? >> >> We have an implementation of a global heap using a custom memory >> allocator that depends on the heap being aligned to the maximum size >> of a block, which we want to be 32 bits where possible. We are able to >> achieve this with mmap by trying to allocate at addresses that are >> right for our heap. So, trying to allocate random memory and align to >> 32 bits may be difficult. > > I would recommend posix_memalign and then let the CrayMPI registration > cache do the rest. Yes, I think I will go with that. I just wanted to experiment with MPI_Mem_alloc to empirically verify that it does not make (much) difference, but getting the address I need may be just too much trouble. > Best, > > Jeff > >>> But what do you really want to achieve? While it is usually >>> beneficial to use pre-registered buffers on RDMA networks, good MPI >>> implementations have a page-registration cache. If, as you say, you >>> are suballocating from a slab, Cray MPI should have the backing pages >>> in the registration cache after you use them as MPI buffers. >> >> That's true, I did not think of registration cache. As you say below, >> if I use huge pages for our heap, there should not be a problem with >> the whole heap being cached. So probably the complexity of dealing >> with MPI_Alloc_mem is not worth it at all. >> >>> You can maximize the efficiency of the page registration cache by >>> using large pages. Search for intro_hugepages using 'man' or on the >>> Internet to learn the specifics of this. I suspect that using large >>> pages will induce much of the benefit you hoped to achieve with an >>> explicitly-registering MPI_Alloc_mem. >>> >>> If you really want to max out RDMA on Cray networks, you need to use >>> DMAPP. I have some simple examples and pointers to docs here: >>> https://github.com/jeffhammond/HPCInfo/tree/master/dmapp. I have more >>> examples other places that I'll migrate to that location if requested. >>> >>> If you're interested in portability, MPI-3 RMA is a good abstraction >>> for RDMA networks. Some implementations do a better job than others >>> at exposing this relationship. Cray MPI has a DMAPP back-end for RMA >>> now, although it is not active by default. You could also try Torsten >>> Hoefler's foMPI >>> [http://spcl.inf.ethz.ch/Research/Parallel_Programming/foMPI/] >>> >>> Best, >>> >>> Jeff >>> >>> On Sat, Feb 14, 2015 at 2:37 PM, Marcin Zalewski >>> <marcin.zalewski(a)gmail.com> wrote: >>>> I am using Cray MPT, and I would like to allocate a large region of >>>> memory from which, in turn, I will allocate buffers to be used with >>>> MPI. I am wondering if there is any benefit from allocating that heap >>>> with MPI_Alloc_mem. I would hope that it could be pre-registered for >>>> RDMA, speeding things up. However, I need this memory to have a >>>> specific alignment. Is there a general way in MPICH or maybe a >>>> specific way for MPT to request alignment with MPI_Alloc_mem? >>>> >>>> Thanks, >>>> Marcin >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0