Hi Steffen, Thank you very much for your contribution. We pushed your patch into MPICH (03d4c77b2148769436b8ce0b682b9b7a662baeef) with some modification. You can directly check it from our repo: git.mpich.org/mpich.git. Thanks, Xin ________________________________________ From: Steffen Christgau [[email protected]] Sent: Tuesday, January 20, 2015 7:57 AM To: [email protected] Subject: [mpich-devel] patch for ticket #1526 (MPI_Alloc_mem registered memory) Hi, I'm currently going to improve the RMA operations (and only those) of an existing CH3 channel for the underlying hardware in a local development branch of MPICH. I appreciate the feature to override the window creation functions in the channel implementation, and the possibility to override the RMA functions via the function table associated to each window. These together simplify development a lot. As one can override the OSC/RMA functions, I really wonder why it is not possible to do the same with the implementation of MPI_Alloc_mem at a CH3 channel level. I found that there was an attempt to introduce this in ticket #1526. Although the original ticket points at a slightly different spot, the discussion is about the issue I'm currently facing. Anyways, it seems to be dormant for a long time and I was not able to find traces of the OSC branch mentioned in the comments (likely that is has been merged into master or it is lost in history) where a solution for this issue was presented according to the comment. That said, I'd like to propose a patch to allow a channel implementation to provide a custom implementation of MPI_Alloc_mem resp. MPI_Free_mem. The idea is borrowed from MPIDI_CH3_Channel_close() which is declared if the MPIDI_CH3_HAS_CHANNEL_CLOSE macro is defined or falls back to the default preprocessor macro "implementation". Same thing here: if according macros are defined, the channel provides an implementation. If not, the defaults are used, i.e. the current MPIU-calls. The CH3 device implementation had to be adjusted to account these changes. As the solution falls back to macro substitution by default, I don't expect any performance impacts, broken code or test failures for existing channel devices. The RMA test cases succeeded both for the my channel device as they did before. The patch (see attachment as well) is based on the v3.1.3 tag. Maybe it helps other implementers (if there's any ;-)) to integrate the allocation of special memory into their channel. diff --git a/src/mpid/ch3/include/mpidimpl.h b/src/mpid/ch3/include/mpidimpl.h index 6a9db2d..1b0ae17 100644 --- a/src/mpid/ch3/include/mpidimpl.h +++ b/src/mpid/ch3/include/mpidimpl.h @@ -1222,6 +1222,19 @@ int MPIDI_Win_sync(MPID_Win *win); void *MPIDI_Alloc_mem(size_t size, MPID_Info *info_ptr); int MPIDI_Free_mem(void *ptr); +#ifdef MPIDI_CH3_HAS_ALLOC_MEM +void* MPIDI_CH3_Alloc_mem(size_t size, MPID_Info *info_ptr); +/* fallback to MPIU_Malloc if channel does not have its own RMA memory allocator */ +#else +#define MPIDI_CH3_Alloc_mem(size, info_ptr) MPIU_Malloc(size) +#endif + +#ifdef MPIDI_CH3_HAS_FREE_MEM +int MPIDI_Free_mem(void *ptr); +#else +#define MPIDI_CH3_Free_mem(ptr) MPIU_Free(ptr); +#endif + /* Pvars */ void MPIDI_CH3_RMA_Init_Pvars(void); diff --git a/src/mpid/ch3/src/ch3u_rma_ops.c b/src/mpid/ch3/src/ch3u_rma_ops.c index f30c464..7a3e22b 100644 --- a/src/mpid/ch3/src/ch3u_rma_ops.c +++ b/src/mpid/ch3/src/ch3u_rma_ops.c @@ -472,7 +472,7 @@ void *MPIDI_Alloc_mem( size_t size, MPID_Info *info_ptr ) MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_ALLOC_MEM); - ap = MPIU_Malloc(size); + ap = MPIDI_CH3_Alloc_mem(size, info_ptr); MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_ALLOC_MEM); return ap; @@ -490,7 +490,7 @@ int MPIDI_Free_mem( void *ptr ) MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_FREE_MEM); - MPIU_Free(ptr); + MPIDI_CH3_Free_mem(ptr); MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_FREE_MEM); return mpi_errno; Regards, Steffen