I think the atomic ops need to be independent of the channel/devices. If channels and devices are dynamically loaded, then the atomics will be dynamically loaded and can't be inlined. I don't think that is acceptable from a performance perspective. Furthermore, which hardware architecture you use is really orthogonal to which channel or device you use. Look at any of our channels. MPICH2 implementations like BG and Cray just happen to be targeted to one kind of hardware. I suggest we have one include file for atomics for each compiler-architecture, e.g.: mpiatomic-gcc-x86.h, mpiatomic-gcc-sparc.h, mpiatomic-sunc-sparc.h etc. There would be one mpiatomic.h that would include the appropriate compiler/arch specific file which would be chosen at configure time. With this method, we could provide a way for a device or channel to override this by allowing it to have its own mpiatomic-*-*.h in an internal directory on the include path, and somehow get configure to choose that. However this would not work for dynamically loaded devices or channels. An alternative method would be to again have one file per compiler/arch but have mpiatomic.h include all of them. Each file would wrap the functions in HAVE_GCC_AND_X86_64_ASM-like macros. Right now, we're using the HAVE_GCC_AND_X86_64_ASM-like macros. Support for a new compiler/arch can added by adding an additional mpiatomic-*-*.h file and an include line to mpiatomic.h. I actually prefer the former. -d On 08/29/2007 12:27 PM, Pavan Balaji wrote:

Dave,

During our discussion yesterday, I forgot to mention something --

I was taking to someone from the IBM BG team at the Hot Interconnects conference last week and this aspect of portable atomic operations came up. In fact, BG has its own atomic operations for maintaining lock-less queues, etc. One way to support BG would be to all its code inside src/include/mpiatomic.h as well, but that'll make the code very architecture specific and might not be a good way of doing it. In that case, allowing different devices to override these atomic operation functions using function pointers to support their own architecture-specific atomic operations might be a better idea. However, that might add an extra redirection and lose a little performance (do we care about this?).

That is something that we need to think about especially given our new relationship with the IBM BG team. Probably others can comment?

-- Pavan

On 08/28/2007 05:51 PM, Dave Goodell wrote:

...

I'm currently working on the "Generic and portable atomics" bullet for the mpich2 1.1 release. Since this exposes a new API to our users, it seems best to run this change by all of you mpich-core@ folks who have quite a bit more mpich context than I do.

Nemesis currently contains several routines for performing atomic operations on a limited set of platforms. They are defined in "src/mpid/ch3/channels/nemesis/nemesis/include/mpid_nem-atomics.h". Their function signatures are given below: static inline void *MPID_NEM_SWAP (volatile void *ptr, void *val) static inline void *MPID_NEM_CAS (volatile void *ptr, void *oldv, void *newv) static inline int MPID_NEM_CAS_INT (volatile int *ptr, int oldv, int newv) static inline int MPID_NEM_FETCH_AND_ADD (volatile int *ptr, int val) static inline void MPID_NEM_ATOMIC_ADD (int *ptr, int val) static inline int MPID_NEM_FETCH_AND_INC (volatile int *ptr) static inline int MPID_NEM_FETCH_AND_DEC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_INC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_DEC (volatile int *ptr)

The following similarly named routines already exist in mpich2: % grep -ri mpi._atomic_ /sandbox/mpi/src/mpich2/src | perl -ne 's/^.*(mpi._atomic\w+).*$/$1/i and print' | sort | uniq MPID_Atomic_decr_flag MPID_Atomic_fetch_and_incr MPID_Atomic_incr MPID_ATOMIC_SET_IF_ZERO MPID_Atomic_write_barrier

I propose that we extend the Nemesis functions to provide implementations for additional platforms as well as a slower fallback implemented via locking. We would then make these improved functions available to the rest of MPI as the following routines: static inline void *MPID_Atomic_swap (volatile void *ptr, void *val) static inline void *MPID_Atomic_cas (volatile void *ptr, void *oldv, void *newv) static inline int MPID_Atomic_cas_int (volatile int *ptr, int oldv, int newv) static inline int MPID_Atomic_fetch_and_add (volatile int *ptr, int val) static inline void MPID_Atomic_add (int *ptr, int val) + static inline int MPID_Atomic_fetch_and_incr (volatile int *ptr) static inline int MPID_Atomic_fetch_and_decr (volatile int *ptr) + static inline void MPID_Atomic_incr (volatile int *ptr) static inline void MPID_Atomic_decr (volatile int *ptr)

The functions prepended with a '+' already exist as cpp macros and will be replaced by the Nemesis versions. The two existing uses of MPID_Atomic_fetch_and_decr will be altered to take the old value as a normal return value instead of as a "parameter return" value.

Question: do we have any reason to believe that any MPICH derivatives might be using this macro as it currently exists? And if so, do we need to maintain backwards compatibility?

These functions would all be defined in "src/include/mpiatomic.h" in order to make them available to the entirety of the mpich code. The original Nemesis versions would eventually be removed and all uses would be replaced by the new routines.

I would appreciate any comments you might have, since I am not very familiar with the history of the existing MPID_Atomic_* routines.

Thanks, -Dave

Darius, Rajeev, Pavan, Jayesh, and I just talked about these atomic functions for a while this afternoon. There were a few takeaways from the discussion: - The ability for DLLs to override one or more of the primitive atomic functions is not really needed. - Investigation of the HP Atomic Operations code has shown that it will probably work pretty well for us. It has an MIT-style license, so we could take the code, modify it, and distribute it as part of mpich2 if we wanted to. The AO code will likely need some minor modifications and additions in order to be suitable for our use. The compare_and_swap variants that are available in AO do not entirely meet our needs. - The general consensus was that all atomic primitives (as opposed to only higher level operations) should be usable by all portions of the MPI codebase. -Dave

I think access to atomic primitives is important for lower layers. E.g., in the nemesis channel there are several places where primitives are used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock-free algorithm where both processes decide which request to transfer next.

In some of these cases, a generic utility function can be used, e.g., barrier, queue/dequeue, other cases a custom algorithm is needed. So I'd agree that a two-level hierarchy is needed, I would still like to expose the primitives to the lower layers.

If code in a lower layer does use atomic primitives, it would have to provide an alternative algorithm that doesn't require atomics and probably would use locks. (Of course all atomics can be implemented as lock; op; unlock;, but the algorithm may be more efficiently implemented using non-atomic ops and wrapping the whole thing in a lock.

For dynamically loaded libraries, if the main library used one version of atomics and the channel used another, there would only be an issue if the both the channel and main library were accessing the same data with atomics. So, e.g., for message queues in shared memory, the channel can use any version it wants, even a version incompatible with what the main library is using. So only the atomics uses between channel and main library would need to be wrapped in the dllchan macros, e.g., when incrementing a ref count.

-d

On 08/30/2007 03:46 AM, William Gropp wrote:

...

Thanks, Dave. Here's a little more background. The top-level MPI atomic routines were designed to implement the necessary semantics rather than specific operations. This was done to avoid forcing the emulation of an operation on a platform that doesn't directly support it. For example, the pair of functions Atomic_incr and Atomic_decr_flag provide the semantics needed for reference counting - atomically increment by one, and atomically decrement by one and indicate if the count is now zero. This is more limited that fetch-and-increment, but has the advantage that earlier IA32 architectures could implement it with a single instruction. This is important when defining operations for RISC systems - they usually provide a load-reservation/store-conditional construct that allows very general atomic operations rather than some specific ops such as fetch-and-increment. So what I'd like to see (as usual :) ) is a two-level hierarchy: Portable, Atomic, high-level operations for things like reference count update and queue operations. The implementation of these depends on what low-level operations are available. A fall-back implementation using lock/unlock routines should be available. Portable, Atomic, low-level operations in terms of processor features, with a fall-back to lock/op/unlock (with routine calls for lock/unlock). One source for these is the libatomic_ops http://www.hpl.hp.com/research/linux/atomic_ops/ . These routines should normally *not* be used directly in MPICH2 code (even for channels). Instead, the higher-level operations should be used. Those should be extended if new, high-level operations are required. In fact, the code style checker could be told to look for the low-level operations and flag them as potential problems (except when used to implement the high-level operations). Without that check, we're likely to see portability and performance problems. Finally, I agree that normally, these should be inlined without any routine calls. Partly that's because once you add the cost of function calls, you might as well use the lock/unlock calls for the implementation. However, if we want to provide runtime-dynamic interface option, we can use the same trick used by the routines in ch3/src to implement the dllchan - a macro that turns into either a direct reference, thus inlining, or into a function pointer reference. Bill On Aug 28, 2007, at 5:51 PM, Dave Goodell wrote:

...

I'm currently working on the "Generic and portable atomics" bullet for the mpich2 1.1 release. Since this exposes a new API to our users, it seems best to run this change by all of you mpich-core@ folks who have quite a bit more mpich context than I do. Nemesis currently contains several routines for performing atomic operations on a limited set of platforms. They are defined in "src/mpid/ch3/channels/nemesis/nemesis/include/mpid_nem-atomics.h". Their function signatures are given below: static inline void *MPID_NEM_SWAP (volatile void *ptr, void *val) static inline void *MPID_NEM_CAS (volatile void *ptr, void *oldv, void *newv) static inline int MPID_NEM_CAS_INT (volatile int *ptr, int oldv, int newv) static inline int MPID_NEM_FETCH_AND_ADD (volatile int *ptr, int val) static inline void MPID_NEM_ATOMIC_ADD (int *ptr, int val) static inline int MPID_NEM_FETCH_AND_INC (volatile int *ptr) static inline int MPID_NEM_FETCH_AND_DEC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_INC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_DEC (volatile int *ptr) The following similarly named routines already exist in mpich2: % grep -ri mpi._atomic_ /sandbox/mpi/src/mpich2/src | perl -ne 's/^.*(mpi._atomic\w+).*$/$1/i and print' | sort | uniq MPID_Atomic_decr_flag MPID_Atomic_fetch_and_incr MPID_Atomic_incr MPID_ATOMIC_SET_IF_ZERO MPID_Atomic_write_barrier I propose that we extend the Nemesis functions to provide implementations for additional platforms as well as a slower fallback implemented via locking. We would then make these improved functions available to the rest of MPI as the following routines: static inline void *MPID_Atomic_swap (volatile void *ptr, void *val) static inline void *MPID_Atomic_cas (volatile void *ptr, void *oldv, void *newv) static inline int MPID_Atomic_cas_int (volatile int *ptr, int oldv, int newv) static inline int MPID_Atomic_fetch_and_add (volatile int *ptr, int val) static inline void MPID_Atomic_add (int *ptr, int val) + static inline int MPID_Atomic_fetch_and_incr (volatile int *ptr) static inline int MPID_Atomic_fetch_and_decr (volatile int *ptr) + static inline void MPID_Atomic_incr (volatile int *ptr) static inline void MPID_Atomic_decr (volatile int *ptr) The functions prepended with a '+' already exist as cpp macros and will be replaced by the Nemesis versions. The two existing uses of MPID_Atomic_fetch_and_decr will be altered to take the old value as a normal return value instead of as a "parameter return" value. Question: do we have any reason to believe that any MPICH derivatives might be using this macro as it currently exists? And if so, do we need to maintain backwards compatibility? These functions would all be defined in "src/include/mpiatomic.h" in order to make them available to the entirety of the mpich code. The original Nemesis versions would eventually be removed and all uses would be replaced by the new routines. I would appreciate any comments you might have, since I am not very familiar with the history of the existing MPID_Atomic_* routines. Thanks, -Dave

There are a few examples that Darius specified:

...

used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock-free algorithm where both processes decide which request to transfer next.

Even if we provide high-level functions for each of these operations, later if we want to build something new, should we just go and add new high-level functions? Will that be acceptable or do we want to keep this interface reasonably fixed? -- Pavan On 09/08/2007 05:52 PM, William Gropp wrote:

Thanks. I'm still a little worried about the use of low-level operations in the general code if there is a possibility that the low-level operation will have to be emulated (at the consequent performance cost), or if a different choice of atomic operations might be a better fit to the needs of the code. There is a significant difference between the philosophies of atomic operations in the CISC and RISC processor architecture, and it seems to me that putting low-level code into the MPI code base is making an unnecessary selection of a preferred architecture. E.g., if you have a load-link/store-conditional architecture, you have more options than just emulating atomic single location memory update instructions.

Is there some reason why it isn't possible to define the higher level abstractions? I don't see the advantage of inserting portability hazards into the code :)

Bill

On Sep 7, 2007, at 3:26 PM, Dave Goodell wrote:

...

Darius, Rajeev, Pavan, Jayesh, and I just talked about these atomic functions for a while this afternoon. There were a few takeaways from the discussion:

- The ability for DLLs to override one or more of the primitive atomic functions is not really needed. - Investigation of the HP Atomic Operations code has shown that it will probably work pretty well for us. It has an MIT-style license, so we could take the code, modify it, and distribute it as part of mpich2 if we wanted to. The AO code will likely need some minor modifications and additions in order to be suitable for our use. The compare_and_swap variants that are available in AO do not entirely meet our needs. - The general consensus was that all atomic primitives (as opposed to only higher level operations) should be usable by all portions of the MPI codebase.

-Dave

...

I think access to atomic primitives is important for lower layers. E.g., in the nemesis channel there are several places where primitives are used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock-free algorithm where both processes decide which request to transfer next.

In some of these cases, a generic utility function can be used, e.g., barrier, queue/dequeue, other cases a custom algorithm is needed. So I'd agree that a two-level hierarchy is needed, I would still like to expose the primitives to the lower layers.

If code in a lower layer does use atomic primitives, it would have to provide an alternative algorithm that doesn't require atomics and probably would use locks. (Of course all atomics can be implemented as lock; op; unlock;, but the algorithm may be more efficiently implemented using non-atomic ops and wrapping the whole thing in a lock.

For dynamically loaded libraries, if the main library used one version of atomics and the channel used another, there would only be an issue if the both the channel and main library were accessing the same data with atomics. So, e.g., for message queues in shared memory, the channel can use any version it wants, even a version incompatible with what the main library is using. So only the atomics uses between channel and main library would need to be wrapped in the dllchan macros, e.g., when incrementing a ref count.

-d

On 08/30/2007 03:46 AM, William Gropp wrote:

...

Thanks, Dave. Here's a little more background. The top-level MPI atomic routines were designed to implement the necessary semantics rather than specific operations. This was done to avoid forcing the emulation of an operation on a platform that doesn't directly support it. For example, the pair of functions Atomic_incr and Atomic_decr_flag provide the semantics needed for reference counting - atomically increment by one, and atomically decrement by one and indicate if the count is now zero. This is more limited that fetch-and-increment, but has the advantage that earlier IA32 architectures could implement it with a single instruction. This is important when defining operations for RISC systems - they usually provide a load-reservation/store-conditional construct that allows very general atomic operations rather than some specific ops such as fetch-and-increment. So what I'd like to see (as usual :) ) is a two-level hierarchy: Portable, Atomic, high-level operations for things like reference count update and queue operations. The implementation of these depends on what low-level operations are available. A fall-back implementation using lock/unlock routines should be available. Portable, Atomic, low-level operations in terms of processor features, with a fall-back to lock/op/unlock (with routine calls for lock/unlock). One source for these is the libatomic_ops http://www.hpl.hp.com/research/linux/atomic_ops/ . These routines should normally *not* be used directly in MPICH2 code (even for channels). Instead, the higher-level operations should be used. Those should be extended if new, high-level operations are required. In fact, the code style checker could be told to look for the low-level operations and flag them as potential problems (except when used to implement the high-level operations). Without that check, we're likely to see portability and performance problems. Finally, I agree that normally, these should be inlined without any routine calls. Partly that's because once you add the cost of function calls, you might as well use the lock/unlock calls for the implementation. However, if we want to provide runtime-dynamic interface option, we can use the same trick used by the routines in ch3/src to implement the dllchan - a macro that turns into either a direct reference, thus inlining, or into a function pointer reference. Bill On Aug 28, 2007, at 5:51 PM, Dave Goodell wrote:

...

I'm currently working on the "Generic and portable atomics" bullet for the mpich2 1.1 release. Since this exposes a new API to our users, it seems best to run this change by all of you mpich-core@ folks who have quite a bit more mpich context than I do. Nemesis currently contains several routines for performing atomic operations on a limited set of platforms. They are defined in "src/mpid/ch3/channels/nemesis/nemesis/include/mpid_nem-atomics.h". Their function signatures are given below: static inline void *MPID_NEM_SWAP (volatile void *ptr, void *val) static inline void *MPID_NEM_CAS (volatile void *ptr, void *oldv, void *newv) static inline int MPID_NEM_CAS_INT (volatile int *ptr, int oldv, int newv) static inline int MPID_NEM_FETCH_AND_ADD (volatile int *ptr, int val) static inline void MPID_NEM_ATOMIC_ADD (int *ptr, int val) static inline int MPID_NEM_FETCH_AND_INC (volatile int *ptr) static inline int MPID_NEM_FETCH_AND_DEC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_INC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_DEC (volatile int *ptr) The following similarly named routines already exist in mpich2: % grep -ri mpi._atomic_ /sandbox/mpi/src/mpich2/src | perl -ne 's/^.*(mpi._atomic\w+).*$/$1/i and print' | sort | uniq MPID_Atomic_decr_flag MPID_Atomic_fetch_and_incr MPID_Atomic_incr MPID_ATOMIC_SET_IF_ZERO MPID_Atomic_write_barrier I propose that we extend the Nemesis functions to provide implementations for additional platforms as well as a slower fallback implemented via locking. We would then make these improved functions available to the rest of MPI as the following routines: static inline void *MPID_Atomic_swap (volatile void *ptr, void *val) static inline void *MPID_Atomic_cas (volatile void *ptr, void *oldv, void *newv) static inline int MPID_Atomic_cas_int (volatile int *ptr, int oldv, int newv) static inline int MPID_Atomic_fetch_and_add (volatile int *ptr, int val) static inline void MPID_Atomic_add (int *ptr, int val) + static inline int MPID_Atomic_fetch_and_incr (volatile int *ptr) static inline int MPID_Atomic_fetch_and_decr (volatile int *ptr) + static inline void MPID_Atomic_incr (volatile int *ptr) static inline void MPID_Atomic_decr (volatile int *ptr) The functions prepended with a '+' already exist as cpp macros and will be replaced by the Nemesis versions. The two existing uses of MPID_Atomic_fetch_and_decr will be altered to take the old value as a normal return value instead of as a "parameter return" value. Question: do we have any reason to believe that any MPICH derivatives might be using this macro as it currently exists? And if so, do we need to maintain backwards compatibility? These functions would all be defined in "src/include/mpiatomic.h" in order to make them available to the entirety of the mpich code. The original Nemesis versions would eventually be removed and all uses would be replaced by the new routines. I would appreciate any comments you might have, since I am not very familiar with the history of the existing MPID_Atomic_* routines. Thanks, -Dave

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

Let's meet in person to talk about this tomorrow or day after when Bill is in. Rajeev

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Monday, September 10, 2007 9:16 AM To: [email protected] Subject: Re: [mpich2-core] MPID_Atomic functions

Its always been the philosophy of MPICH2 to enhance the design by adding new abstractions as we discover them. So asking for only the high-level interface doesn't reduce the ability to add new and better options later. In other words, I'm not looking for a minimalist, high-level interface. Rather, if there's a need for a barrier, for example, I'd like to avoid having a particular choice of implementation embedded in the code. If there's a need for a doubly- linked list atomic update, that's what I'd like to see in the code.

We will probably need a way for individual methods to add their own, higher-level, atomic ops. These need not be exported to the top level unless they have general benefit.

Bill

On Sep 9, 2007, at 5:50 AM, Pavan Balaji wrote:

...

There are a few examples that Darius specified:

...

...

used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a

lock-free

...

...

...

algorithm where both processes decide which request to transfer next.

Even if we provide high-level functions for each of these operations, later if we want to build something new, should we just go and add new high-level functions? Will that be acceptable or do we want to keep this interface reasonably fixed?

-- Pavan

On 09/08/2007 05:52 PM, William Gropp wrote:

...

Thanks. I'm still a little worried about the use of low-level operations in the general code if there is a possibility that the low-level operation will have to be emulated (at the consequent performance cost), or if a different choice of atomic operations might be a better fit to the needs of the code. There is a significant difference between the philosophies of atomic operations in the CISC and RISC processor architecture, and it seems to me that putting low-level code into the MPI code base is making an unnecessary selection of a preferred architecture. E.g., if you have a load-link/store-conditional architecture, you have more options than just emulating atomic single location memory update instructions. Is there some reason why it isn't possible to define the higher level abstractions? I don't see the advantage of inserting portability hazards into the code :) Bill On Sep 7, 2007, at 3:26 PM, Dave Goodell wrote:

...

Darius, Rajeev, Pavan, Jayesh, and I just talked about these atomic functions for a while this afternoon. There were a few takeaways from the discussion:

- The ability for DLLs to override one or more of the primitive atomic functions is not really needed. - Investigation of the HP Atomic Operations code has shown that it will probably work pretty well for us. It has an MIT-style license, so we could take the code, modify it, and distribute it as part of mpich2 if we wanted to. The AO code will likely need some minor modifications and additions in order to be suitable for our use. The compare_and_swap variants that are available in AO do not entirely meet our needs. - The general consensus was that all atomic primitives (as opposed to only higher level operations) should be usable by all portions of the MPI codebase.

-Dave

...

I think access to atomic primitives is important for lower layers. E.g., in the nemesis channel there are several places where primitives are used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock-free algorithm where both processes decide which request to transfer next.

In some of these cases, a generic utility function can be used, e.g., barrier, queue/dequeue, other cases a custom algorithm is needed. So I'd agree that a two-level hierarchy is needed, I would still like to expose the primitives to the lower layers.

If code in a lower layer does use atomic primitives, it would have to provide an alternative algorithm that doesn't require atomics and probably would use locks. (Of course all atomics can be implemented as lock; op; unlock;, but the algorithm may be more efficiently implemented using non-atomic ops and wrapping the whole thing in a lock.

For dynamically loaded libraries, if the main library used one version of atomics and the channel used another, there would only be an issue if the both the channel and main library were accessing the same data with atomics. So, e.g., for message queues in shared memory, the channel can use any version it wants, even a version incompatible with what the main library is using. So only the atomics uses between channel and main library would need to be wrapped in the dllchan macros, e.g., when incrementing a ref count.

-d

On 08/30/2007 03:46 AM, William Gropp wrote:

...

Thanks, Dave. Here's a little more background. The top-level MPI atomic routines were designed to implement the necessary semantics rather than specific operations. This was done to avoid forcing the emulation of an operation on a platform that doesn't directly support it. For example, the pair of functions Atomic_incr and Atomic_decr_flag provide the semantics needed for reference counting - atomically increment by one, and atomically decrement by one and indicate if the count is now zero. This is more limited that fetch-and-increment, but has the advantage that earlier IA32 architectures could implement it with a single instruction. This is important when defining operations for RISC systems - they usually provide a load-reservation/store-conditional construct that allows very general atomic operations rather than some specific ops such as fetch-and-increment. So what I'd like to see (as usual :) ) is a two-level hierarchy: Portable, Atomic, high-level operations for things like reference count update and queue operations. The implementation of these depends on what low-level operations are available. A fall-back implementation using lock/unlock routines should be available. Portable, Atomic, low-level operations in terms of processor features, with a fall-back to lock/op/unlock (with routine calls for lock/unlock). One source for these is the libatomic_ops http://www.hpl.hp.com/research/linux/atomic_ops/ . These routines should normally *not* be used directly in MPICH2 code (even for channels). Instead, the higher-level operations should be used. Those should be extended if new, high-level operations are required. In fact, the code style checker could be told to look for the low-level operations and flag them as potential problems (except when used to implement the high-level operations). Without that check, we're likely to see portability and performance problems. Finally, I agree that normally, these should be inlined without any routine calls. Partly that's because once you add the cost of function calls, you might as well use the lock/unlock calls for the implementation. However, if we want to provide runtime-dynamic interface option, we can use the same trick used by the routines in ch3/src to implement the dllchan - a macro that turns into either a direct reference, thus inlining, or into a function pointer reference. Bill On Aug 28, 2007, at 5:51 PM, Dave Goodell wrote:

...

I'm currently working on the "Generic and portable atomics" bullet for the mpich2 1.1 release. Since this exposes a new API to our users, it seems best to run this change by all of you mpich-core@ folks who have quite a bit more mpich context than I do. Nemesis currently contains several routines for performing atomic operations on a limited set of platforms. They are defined in

"src/mpid/ch3/channels/nemesis/nemesis/include/mpid_nem-atomics.h".

...

Their

...

...

...

function signatures are given below: static inline void *MPID_NEM_SWAP (volatile void *ptr, void *val) static inline void *MPID_NEM_CAS (volatile void *ptr, void *oldv, void *newv) static inline int MPID_NEM_CAS_INT (volatile int *ptr, int oldv, int newv) static inline int MPID_NEM_FETCH_AND_ADD (volatile int *ptr, int val) static inline void MPID_NEM_ATOMIC_ADD (int *ptr, int val) static inline int MPID_NEM_FETCH_AND_INC (volatile int *ptr) static inline int MPID_NEM_FETCH_AND_DEC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_INC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_DEC (volatile int *ptr) The following similarly named routines already exist in mpich2: % grep -ri mpi._atomic_ /sandbox/mpi/src/mpich2/src | perl -ne 's/^.*(mpi._atomic\w+).*$/$1/i and print' | sort | uniq MPID_Atomic_decr_flag MPID_Atomic_fetch_and_incr MPID_Atomic_incr MPID_ATOMIC_SET_IF_ZERO MPID_Atomic_write_barrier I propose that we extend the Nemesis functions to provide implementations for additional platforms as well as a slower fallback implemented via locking. We would then make these improved functions available to the rest of MPI as the following routines: static inline void *MPID_Atomic_swap (volatile void *ptr, void *val) static inline void *MPID_Atomic_cas (volatile void *ptr, void *oldv, void *newv) static inline int MPID_Atomic_cas_int (volatile int *ptr, int oldv, int newv) static inline int MPID_Atomic_fetch_and_add (volatile int *ptr, int val) static inline void MPID_Atomic_add (int *ptr, int val) + static inline int MPID_Atomic_fetch_and_incr (volatile int *ptr) static inline int MPID_Atomic_fetch_and_decr (volatile int *ptr) + static inline void MPID_Atomic_incr (volatile int *ptr) static inline void MPID_Atomic_decr (volatile int *ptr) The functions prepended with a '+' already exist as cpp macros and will be replaced by the Nemesis versions. The two existing uses of MPID_Atomic_fetch_and_decr will be altered to take the old value as a normal return value instead of as a "parameter return" value. Question: do we have any reason to believe that any MPICH derivatives might be using this macro as it currently exists? And if so, do we need to maintain backwards compatibility? These functions would all be defined in "src/include/ mpiatomic.h" in order to make them available to the entirety of the mpich code. The original Nemesis versions would eventually be removed and all uses would be replaced by the new routines. I would appreciate any comments you might have, since I am not very familiar with the history of the existing MPID_Atomic_* routines. Thanks, -Dave

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

We did discuss this, though there is no specific design. Darius is now happy with the high-level abstraction approach. Bill On Sep 16, 2007, at 11:40 AM, Pavan Balaji wrote:

Did this meeting already happen? If not, I can call in from Austin.

-- Pavan

On 09/10/2007 01:26 PM, Rajeev Thakur wrote:

...

Let's meet in person to talk about this tomorrow or day after when Bill is in. Rajeev

...

-----Original Message----- From: [email protected] [mailto:owner-mpich2- [email protected]] On Behalf Of William Gropp Sent: Monday, September 10, 2007 9:16 AM To: [email protected] Subject: Re: [mpich2-core] MPID_Atomic functions

Its always been the philosophy of MPICH2 to enhance the design by adding new abstractions as we discover them. So asking for only the high-level interface doesn't reduce the ability to add new and better options later. In other words, I'm not looking for a minimalist, high-level interface. Rather, if there's a need for a barrier, for example, I'd like to avoid having a particular choice of implementation embedded in the code. If there's a need for a doubly- linked list atomic update, that's what I'd like to see in the code.

We will probably need a way for individual methods to add their own, higher-level, atomic ops. These need not be exported to the top level unless they have general benefit.

Bill

On Sep 9, 2007, at 5:50 AM, Pavan Balaji wrote:

...

There are a few examples that Darius specified:

...

...

used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock-free algorithm where both processes decide which request to transfer next.

Even if we provide high-level functions for each of these operations, later if we want to build something new, should we just go and add new high-level functions? Will that be acceptable or do we want to keep this interface reasonably fixed?

-- Pavan

On 09/08/2007 05:52 PM, William Gropp wrote:

...

Thanks. I'm still a little worried about the use of low-level operations in the general code if there is a possibility that the low-level operation will have to be emulated (at the consequent performance cost), or if a different choice of atomic operations might be a better fit to the needs of the code. There is a significant difference between the philosophies of atomic operations in the CISC and RISC processor architecture, and it seems to me that putting low- level code into the MPI code base is making an unnecessary selection of a preferred architecture. E.g., if you have a load-link/store-conditional architecture, you have more options than just emulating atomic single location memory update instructions. Is there some reason why it isn't possible to define the higher level abstractions? I don't see the advantage of inserting portability hazards into the code :) Bill On Sep 7, 2007, at 3:26 PM, Dave Goodell wrote:

...

Darius, Rajeev, Pavan, Jayesh, and I just talked about these atomic functions for a while this afternoon. There were a few takeaways from the discussion:

- The ability for DLLs to override one or more of the primitive atomic functions is not really needed. - Investigation of the HP Atomic Operations code has shown that it will probably work pretty well for us. It has an MIT-style license, so we could take the code, modify it, and distribute it as part of mpich2 if we wanted to. The AO code will likely need some minor modifications and additions in order to be suitable for our use. The compare_and_swap variants that are available in AO do not entirely meet our needs. - The general consensus was that all atomic primitives (as opposed to only higher level operations) should be usable by all portions of the MPI codebase.

-Dave

...

I think access to atomic primitives is important for lower layers. E.g., in the nemesis channel there are several places where primitives are used: the shared-memory barrier algorithm; a lock-free algorithm for allocating memory region for barriers; in LMT, there's a lock- free algorithm where both processes decide which request to transfer next. In some of these cases, a generic utility function can be used, e.g., barrier, queue/dequeue, other cases a custom algorithm is needed. So I'd agree that a two-level hierarchy is needed, I would still like to expose the primitives to the lower layers. If code in a lower layer does use atomic primitives, it would have to provide an alternative algorithm that doesn't require atomics and probably would use locks. (Of course all atomics can be implemented as lock; op; unlock;, but the algorithm may be more efficiently implemented using non-atomic ops and wrapping the whole thing in a lock. For dynamically loaded libraries, if the main library used one version of atomics and the channel used another, there would only be an issue if the both the channel and main library were accessing the same data with atomics. So, e.g., for message queues in shared memory, the channel can use any version it wants, even a version incompatible with what the main library is using. So only the atomics uses between channel and main library would need to be wrapped in the dllchan macros, e.g., when incrementing a ref count. -d

On 08/30/2007 03:46 AM, William Gropp wrote:

...

Thanks, Dave. Here's a little more background. The top-level MPI atomic routines were designed to implement the necessary semantics rather than specific operations. This was done to avoid forcing the emulation of an operation on a platform that doesn't directly support it. For example, the pair of functions Atomic_incr and Atomic_decr_flag provide the semantics needed for reference counting - atomically increment by one, and atomically decrement by one and indicate if the count is now zero. This is more limited that fetch-and-increment, but has the advantage that earlier IA32 architectures could implement it with a single instruction. This is important when defining operations for RISC systems - they usually provide a load-reservation/store-conditional construct that allows very general atomic operations rather than some specific ops such as fetch-and-increment. So what I'd like to see (as usual :) ) is a two-level hierarchy: Portable, Atomic, high-level operations for things like reference count update and queue operations. The implementation of these depends on what low-level operations are available. A fall-back implementation using lock/unlock routines should be available. Portable, Atomic, low-level operations in terms of processor features, with a fall-back to lock/op/unlock (with routine calls for lock/unlock). One source for these is the libatomic_ops http://www.hpl.hp.com/research/linux/atomic_ops/ . These routines should normally *not* be used directly in MPICH2 code (even for channels). Instead, the higher-level operations should be used. Those should be extended if new, high-level operations are required. In fact, the code style checker could be told to look for the low-level operations and flag them as potential problems (except when used to implement the high-level operations). Without that check, we're likely to see portability and performance problems. Finally, I agree that normally, these should be inlined without any routine calls. Partly that's because once you add the cost of function calls, you might as well use the lock/unlock calls for the implementation. However, if we want to provide runtime-dynamic interface option, we can use the same trick used by the routines in ch3/src to implement the dllchan - a macro that turns into either a direct reference, thus inlining, or into a function pointer reference. Bill On Aug 28, 2007, at 5:51 PM, Dave Goodell wrote:

...

I'm currently working on the "Generic and portable atomics" bullet for the mpich2 1.1 release. Since this exposes a new API to our users, it seems best to run this change by all of you mpich-core@ folks who have quite a bit more mpich context than I do. Nemesis currently contains several routines for performing atomic operations on a limited set of platforms. They are defined in

"src/mpid/ch3/channels/nemesis/nemesis/include/mpid_nem-atomics.h".

...

Their

...

...

...

function signatures are given below: static inline void *MPID_NEM_SWAP (volatile void *ptr, void *val) static inline void *MPID_NEM_CAS (volatile void *ptr, void *oldv, void *newv) static inline int MPID_NEM_CAS_INT (volatile int *ptr, int oldv, int newv) static inline int MPID_NEM_FETCH_AND_ADD (volatile int *ptr, int val) static inline void MPID_NEM_ATOMIC_ADD (int *ptr, int val) static inline int MPID_NEM_FETCH_AND_INC (volatile int *ptr) static inline int MPID_NEM_FETCH_AND_DEC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_INC (volatile int *ptr) static inline void MPID_NEM_ATOMIC_DEC (volatile int *ptr) The following similarly named routines already exist in mpich2: % grep -ri mpi._atomic_ /sandbox/mpi/src/mpich2/src | perl -ne 's/^.*(mpi._atomic\w+).*$/$1/i and print' | sort | uniq MPID_Atomic_decr_flag MPID_Atomic_fetch_and_incr MPID_Atomic_incr MPID_ATOMIC_SET_IF_ZERO MPID_Atomic_write_barrier I propose that we extend the Nemesis functions to provide implementations for additional platforms as well as a slower fallback implemented via locking. We would then make these improved functions available to the rest of MPI as the following routines: static inline void *MPID_Atomic_swap (volatile void *ptr, void *val) static inline void *MPID_Atomic_cas (volatile void *ptr, void *oldv, void *newv) static inline int MPID_Atomic_cas_int (volatile int *ptr, int oldv, int newv) static inline int MPID_Atomic_fetch_and_add (volatile int *ptr, int val) static inline void MPID_Atomic_add (int *ptr, int val) + static inline int MPID_Atomic_fetch_and_incr (volatile int *ptr) static inline int MPID_Atomic_fetch_and_decr (volatile int *ptr) + static inline void MPID_Atomic_incr (volatile int *ptr) static inline void MPID_Atomic_decr (volatile int *ptr) The functions prepended with a '+' already exist as cpp macros and will be replaced by the Nemesis versions. The two existing uses of MPID_Atomic_fetch_and_decr will be altered to take the old value as a normal return value instead of as a "parameter return" value. Question: do we have any reason to believe that any MPICH derivatives might be using this macro as it currently exists? And if so, do we need to maintain backwards compatibility? These functions would all be defined in "src/include/ mpiatomic.h" in order to make them available to the entirety of the mpich code. The original Nemesis versions would eventually be removed and all uses would be replaced by the new routines. I would appreciate any comments you might have, since I am not very familiar with the history of the existing MPID_Atomic_* routines. Thanks, -Dave

-- Pavan Balaji http://www.mcs.anl.gov/~balaji

-- Pavan Balaji http://www.mcs.anl.gov/~balaji