On Wed, Feb 18, 2015 at 9:48 AM, Jeff Hammond <[email protected]> wrote:
I have a hard time imagining that Cray doesn't do what's necessary to ensure proper utilization of their network with the system software they package.
Oh, I am sure that they do something. However, I know that my large heap will be the source of buffers used with MPI, so, in theory, I could cut down on overhead by just preregistering it once and for all rather than in small chunks. Probably with hugepages the overhead gets close to 0 though.
The only place MPI does atomics on user buffers is in MPI-3 RMA, and CrayMPI uses the software implementation in Ch3 by default (they have a DMAPP implementation as an option, but I don't know the details).
The easiest way to deal with the NIC alignment requirements is for Cray to patch glibc so that malloc always returns an 8-byte aligned address if it doesn't already. And stack alignment is always to basic datatype granularity, so the only case where it could be an issue is for atomics on 32b types on the stack, no?
Actually, I need a huge alignment, as in an address divisible by 2^32. This is due to some implementation requirements, but I never have a problem getting such address with mmap. Thank you, Marcin
Best,
Jeff
On Mon, Feb 16, 2015 at 4:53 AM, Atchley, Scott <[email protected]> wrote:
Jeff,
I expect that he is concerned with GNI's 4-byte alignment requirement for both the address and length for RDMA Reads and the 8-byte alignment for atomics.
Scott
On Feb 16, 2015, at 1:42 AM, Jeff Hammond <[email protected]> wrote:
If you are going to suballocate from a slab yourself, you can handle alignment yourself easy enough, no? Do I not understand what you mean here. And what sort of alignment do you want? Are you trying to align to 32/64 bytes because of AVX or some other x86 feature on Cray XC30 or do you want page alignment?
But what do you really want to achieve? While it is usually beneficial to use pre-registered buffers on RDMA networks, good MPI implementations have a page-registration cache. If, as you say, you are suballocating from a slab, Cray MPI should have the backing pages in the registration cache after you use them as MPI buffers.
You can maximize the efficiency of the page registration cache by using large pages. Search for intro_hugepages using 'man' or on the Internet to learn the specifics of this. I suspect that using large pages will induce much of the benefit you hoped to achieve with an explicitly-registering MPI_Alloc_mem.
If you really want to max out RDMA on Cray networks, you need to use DMAPP. I have some simple examples and pointers to docs here: https://github.com/jeffhammond/HPCInfo/tree/master/dmapp. I have more examples other places that I'll migrate to that location if requested.
If you're interested in portability, MPI-3 RMA is a good abstraction for RDMA networks. Some implementations do a better job than others at exposing this relationship. Cray MPI has a DMAPP back-end for RMA now, although it is not active by default. You could also try Torsten Hoefler's foMPI [http://spcl.inf.ethz.ch/Research/Parallel_Programming/foMPI/].
Best,
Jeff
On Sat, Feb 14, 2015 at 2:37 PM, Marcin Zalewski <[email protected]> wrote:
I am using Cray MPT, and I would like to allocate a large region of memory from which, in turn, I will allocate buffers to be used with MPI. I am wondering if there is any benefit from allocating that heap with MPI_Alloc_mem. I would hope that it could be pre-registered for RDMA, speeding things up. However, I need this memory to have a specific alignment. Is there a general way in MPICH or maybe a specific way for MPT to request alignment with MPI_Alloc_mem?
Thanks, Marcin _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss