- mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] RMA WG Telecon
by Jeff Hammond 12 Sep '18

12 Sep '18

On the topic of accumulate info keys, https://github.com/mpi-forum/mpi-issues/issues/36 is the necessary first step to make the default sufficiently general for RMA to support all usage models. https://github.com/mpi-forum/mpi-issues/issues/46 was my idea for the next step. It is not very specific, but the intent was to allow the user to describe their use of RMA in detail. This could go all the way to a database of (op,basic_datatype,complete_datatype,max_count). I suspect a more pragmatic approach is to define additional keys we think implementations will actually use, such as: * no_noncontiguous_datatypes - exclude e.g. MPI_Type_vector with stride!=1 and anything else that might lead to pack/unpack. * max_count (or should it be max_target_count) - specify upper bound on message size. Jeff On Wed, Sep 12, 2018 at 1:26 AM, Joseph Schuchart via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > Pavan, > > I listened in for the MPI-RMA WG telco yesterday and found some > interesting points in it. In particular, I am interested in the discussion > on atomics and same_op. I support the notion that MPI should choose a > conservative default to make sure users do not run into surprising UB > because the implementation expects or supports only same_op, which the user > may not be aware of. > > As a developer of a framework built on top of MPI RMA I would also be > interested in getting information from the MPI implementation on which > atomic operations are actually supported in hardware on the current system, > which would allow me to pick different implementations of a specific > feature to fully exploit the available hardware capabilities (similar to > C++ `std::atomic_is_lock_free`). Are there any plans to provide such an > interface? This could be used in combination with an info key (say > `native_op`) that promises that the user will only use a mix of operations > that are supported in hardware, which would then avoid the required > fall-back to active messages discussed yesterday. > > Last but not least, I am also interested in the shared memory optimization > for collectives (iirc, MPI_DISCARD?). I couldn't find an issue on Github on > this. Is there any publicly available information you can share? > > Any input would be much appreciated. > > Many thanks in advance, > Joseph > > > On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >> >> _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Jim Dinan 12 Sep '18

12 Sep '18

In addition to op, hardware acceleration can also be tied to datatype. For example, MPI_SUM may be accelerated for integer types, but not floating point types. I think it's possible to support such a query using the recent tools interfaces. I created a ticket, so we remember to discuss: https://github.com/mpiwg-rma/rma-issues/issues/6 ~Jim. On Wed, Sep 12, 2018 at 4:26 AM Joseph Schuchart via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > Pavan, > > I listened in for the MPI-RMA WG telco yesterday and found some > interesting points in it. In particular, I am interested in the > discussion on atomics and same_op. I support the notion that MPI should > choose a conservative default to make sure users do not run into > surprising UB because the implementation expects or supports only > same_op, which the user may not be aware of. > > As a developer of a framework built on top of MPI RMA I would also be > interested in getting information from the MPI implementation on which > atomic operations are actually supported in hardware on the current > system, which would allow me to pick different implementations of a > specific feature to fully exploit the available hardware capabilities > (similar to C++ `std::atomic_is_lock_free`). Are there any plans to > provide such an interface? This could be used in combination with an > info key (say `native_op`) that promises that the user will only use a > mix of operations that are supported in hardware, which would then avoid > the required fall-back to active messages discussed yesterday. > > Last but not least, I am also interested in the shared memory > optimization for collectives (iirc, MPI_DISCARD?). I couldn't find an > issue on Github on this. Is there any publicly available information you > can share? > > Any input would be much appreciated. > > Many thanks in advance, > Joseph > > On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Joseph Schuchart 12 Sep '18

12 Sep '18

Pavan, I listened in for the MPI-RMA WG telco yesterday and found some interesting points in it. In particular, I am interested in the discussion on atomics and same_op. I support the notion that MPI should choose a conservative default to make sure users do not run into surprising UB because the implementation expects or supports only same_op, which the user may not be aware of. As a developer of a framework built on top of MPI RMA I would also be interested in getting information from the MPI implementation on which atomic operations are actually supported in hardware on the current system, which would allow me to pick different implementations of a specific feature to fully exploit the available hardware capabilities (similar to C++ `std::atomic_is_lock_free`). Are there any plans to provide such an interface? This could be used in combination with an info key (say `native_op`) that promises that the user will only use a mix of operations that are supported in hardware, which would then avoid the required fall-back to active messages discussed yesterday. Last but not least, I am also interested in the shared memory optimization for collectives (iirc, MPI_DISCARD?). I couldn't find an issue on Github on this. Is there any publicly available information you can share? Any input would be much appreciated. Many thanks in advance, Joseph On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Jeff Hammond 12 Sep '18

12 Sep '18

Noncoherent shared memory was discussed in MPI-3 days but we have no hardware to motivate it besides Intel SCC and that was not widely available. Now that things like NVIDIA UM exist, there is a nontrivial hardware platform that think about if we try to define such features. However, I’m not aware of NVIDIA publishing a formal memory model for CUDA or any generation of hardware. Furthermore, one cannot - AFAIK - run an real MPI process on a GPU, so one would only be able to reason about noncoherent window memory living on GPU(s) but accessed only by CPUs. That’s much less interesting. Anyways, Jeff who works for Intel defers to literally anybody else who lays claim to superior understanding of NVIDIA stuff. Another angle would be to try to align MPI shared memory with OpenCL 2.2 SVM variants. I know exactly who to talk to about that topic. Jeff Sent from my iPhone > On Sep 11, 2018, at 10:26 PM, Mark Hoemmen via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > I’m sorry to have missed this — I’m very much interested in C++ binding and interoperability issues. I know Jeff at least is involved in our weekly C++ heterogeneity phone meetings — we’re trying to move the Standard to support multiple memory spaces in a sane way, and I think it helps to have RDMA represented there as well. We’ve also thought a bit about wrapping MPI RDMA in Kokkos, and about making the mdspan proposal flexible enough that we could perhaps support RDMA with it as well. The most interesting bit might be actually introducing semantics for noncoherent shared memory into the Standard — I think we want to go there eventually but it is really hard, and I think y’all are well equipped to help. > > Best, > Mark Hoemmen > >> On Mon, Sep 10, 2018 at 1:21 PM Balaji, Pavan via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: >> All, >> >> We are restarting the RMA WG telecons. A subset of the following proposals will be discussed during this call. >> >> - Interoperability of MPI shared memory with C11, C++11 language semantics. Lead: Bill Gropp >> - MPI Generalized atomics. Lead: Pavan Balaji >> - Neighborhood communication in RMA. Lead: Nathan Hjelm >> - Nonblocking RMA synchronization. Lead: Pavan Balaji >> - RMA Notify. Leads: Jim and Torsten >> - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan Balaji >> - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond >> - <Unnamed proposal>. Lead: Nathan Hjelm_______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Mark Hoemmen 12 Sep '18

12 Sep '18

I’m sorry to have missed this — I’m very much interested in C++ binding and interoperability issues. I know Jeff at least is involved in our weekly C++ heterogeneity phone meetings — we’re trying to move the Standard to support multiple memory spaces in a sane way, and I think it helps to have RDMA represented there as well. We’ve also thought a bit about wrapping MPI RDMA in Kokkos, and about making the mdspan proposal flexible enough that we could perhaps support RDMA with it as well. The most interesting bit might be actually introducing semantics for noncoherent shared memory into the Standard — I think we want to go there eventually but it is really hard, and I think y’all are well equipped to help. Best, Mark Hoemmen On Mon, Sep 10, 2018 at 1:21 PM Balaji, Pavan via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > All, > > We are restarting the RMA WG telecons. A subset of the following > proposals will be discussed during this call. > > - Interoperability of MPI shared memory with C11, C++11 language > semantics. Lead: Bill Gropp > - MPI Generalized atomics. Lead: Pavan Balaji > - Neighborhood communication in RMA. Lead: Nathan Hjelm > - Nonblocking RMA synchronization. Lead: Pavan Balaji > - RMA Notify. Leads: Jim and Torsten > - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan > Balaji > - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond > - <Unnamed proposal>. Lead: Nathan > Hjelm_______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >

1 0

RMA WG telecon info
by Balaji, Pavan 11 Sep '18

11 Sep '18

Folks, Sorry, I forgot to send out the call-in information yesterday. We'll use this: http://argonne.bluejeans.com/balaji.anl.gov I'll add it to the calendar invite as well. -- Pavan

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Jim Dinan 11 Sep '18

11 Sep '18

Hi Pavan, I don't see a link to the blue jeans connection info. Where can I find that? Thanks, ~Jim. On Mon, Sep 10, 2018 at 3:21 PM Balaji, Pavan via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > All, > > We are restarting the RMA WG telecons. A subset of the following > proposals will be discussed during this call. > > - Interoperability of MPI shared memory with C11, C++11 language > semantics. Lead: Bill Gropp > - MPI Generalized atomics. Lead: Pavan Balaji > - Neighborhood communication in RMA. Lead: Nathan Hjelm > - Nonblocking RMA synchronization. Lead: Pavan Balaji > - RMA Notify. Leads: Jim and Torsten > - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan > Balaji > - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond > - <Unnamed proposal>. Lead: Nathan > Hjelm_______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >

1 0

RMA WG Telecon
by Balaji, Pavan 10 Sep '18

10 Sep '18

All, We are restarting the RMA WG telecons. A subset of the following proposals will be discussed during this call. - Interoperability of MPI shared memory with C11, C++11 language semantics. Lead: Bill Gropp - MPI Generalized atomics. Lead: Pavan Balaji - Neighborhood communication in RMA. Lead: Nathan Hjelm - Nonblocking RMA synchronization. Lead: Pavan Balaji - RMA Notify. Leads: Jim and Torsten - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan Balaji - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond - <Unnamed proposal>. Lead: Nathan Hjelm

1 0

Re: [mpiwg-rma] MPI_Win_create on unmapped memory?
by maik peterson 04 Jan '18

04 Jan '18

jeff, in mpi you can put in what you like, the result is always the same: unexpected fiddle-faddle. mp 2018-01-04 0:23 GMT+01:00 Jeff Hammond <jeff.science(a)gmail.com>: > > > On Wed, Jan 3, 2018 at 1:07 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > > > > On 01/02/18 20:52, Jeff Hammond wrote: > >> > >> If a user passes a (base,size) pair that corresponds to a range of > addresses that are not allocated and/or mapped, is that a correct MPI > program? > > > > > > Such a thing wouldn't be a valid object pointer in C/C++ or a variable > in Fortran, i.e. both languages prohibit this. > > Which of the following MPI_Win_create calls are not using a valid object > pointer according to ISO C and/or C++? > > char a = '\0'; > > int main() { > .. > char * x = malloc(1); > MPI_Win_create(x,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); > ... > MPI_Win_create(&a,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); > .. > foo(); // compiler cannot prove this function does not call > // mmap(0x10000, 1, PROT_READ|PROT_WRITE, > MAP_FIXED|MAP_ANONYMOUS, -1, 4096); > .. > MPI_Win_create(0x10000,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); > .. > } > > >> The only text I can find that implies that buffers passed to > MPI_Win_create have to be allocated is the following, assuming that one > interprets "existing memory" this way. > >> > >> "Each process specifies a window of existing memory that it exposes to > RMA accesses by the processes in the group of comm." (MPI 3.1 11.2.1) > > > > > > Since MPI is a library standard I would still expect all the guarantees > of the language standards involved to hold and the above to only be an > additional clarification? > > It's only a clarification. However, I don't think it's a good idea for > the MPI standard to punt this off on ISO C, because not every > implementation of MPI is an ISO C language lawyer, and very few users are. > > >> The Fortran-specific text is similarly suggestive, but it only says > that a user can pass a simply contiguous array, not that such a thing is > required. > >> > >> "In Fortran, one can pass the first element of a memory region or a > whole array, which must be ‘simply contiguous’" > > > > > > That's most probably because of the Fortran storage association > "feature" which is more about how arrays were passed in pre-Fortran 90 > times, i.e. it's legal for a caller C to pass some array element A(i) to a > subprogram P accepting an array argument B(*) provided the effective size > (that is the indices of B used in P match to valid indices of A). > > The widespread use of MPI in the context of non-strict Fortran is a > particularly good reason to clarify this ourselves rather than rely on ISO > Fortran. As has been discussed many times in the passed, MPI relies on > extensions and/or implementation-specific behavior with Fortran through the > latest standard. Only with Fortran 2018 will this situation disappear. > > Jeff > > > Regards, Thomas > > -- > > Thomas Jahns > > HD(CP)^2 > > Abteilung Anwendungssoftware > > > > Deutsches Klimarechenzentrum GmbH > > Bundesstraße 45a • D-20146 Hamburg • Germany > > > > Phone: +49 40 460094-151 <+49%2040%20460094151> > > Fax: +49 40 460094-270 <+49%2040%20460094270> > > Email: Thomas Jahns <jahns(a)dkrz.de> > > URL: www.dkrz.de > > > > Geschäftsführer: Prof. Dr. Thomas Ludwig > > Sitz der Gesellschaft: Hamburg > > Amtsgericht Hamburg HRB 39784 > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >

1 0

Re: [mpiwg-rma] MPI_Win_create on unmapped memory?
by Jeff Hammond 03 Jan '18

03 Jan '18

On Wed, Jan 3, 2018 at 1:07 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > > On 01/02/18 20:52, Jeff Hammond wrote: >> >> If a user passes a (base,size) pair that corresponds to a range of addresses that are not allocated and/or mapped, is that a correct MPI program? > > > Such a thing wouldn't be a valid object pointer in C/C++ or a variable in Fortran, i.e. both languages prohibit this. Which of the following MPI_Win_create calls are not using a valid object pointer according to ISO C and/or C++? char a = '\0'; int main() { .. char * x = malloc(1); MPI_Win_create(x,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); ... MPI_Win_create(&a,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); .. foo(); // compiler cannot prove this function does not call // mmap(0x10000, 1, PROT_READ|PROT_WRITE, MAP_FIXED|MAP_ANONYMOUS, -1, 4096); .. MPI_Win_create(0x10000,1000,1,MPI_INFO_NULL,MPI_COMM_WORLD,&win); .. } >> The only text I can find that implies that buffers passed to MPI_Win_create have to be allocated is the following, assuming that one interprets "existing memory" this way. >> >> "Each process specifies a window of existing memory that it exposes to RMA accesses by the processes in the group of comm." (MPI 3.1 11.2.1) > > > Since MPI is a library standard I would still expect all the guarantees of the language standards involved to hold and the above to only be an additional clarification? It's only a clarification. However, I don't think it's a good idea for the MPI standard to punt this off on ISO C, because not every implementation of MPI is an ISO C language lawyer, and very few users are. >> The Fortran-specific text is similarly suggestive, but it only says that a user can pass a simply contiguous array, not that such a thing is required. >> >> "In Fortran, one can pass the first element of a memory region or a whole array, which must be ‘simply contiguous’" > > > That's most probably because of the Fortran storage association "feature" which is more about how arrays were passed in pre-Fortran 90 times, i.e. it's legal for a caller C to pass some array element A(i) to a subprogram P accepting an array argument B(*) provided the effective size (that is the indices of B used in P match to valid indices of A). The widespread use of MPI in the context of non-strict Fortran is a particularly good reason to clarify this ourselves rather than rely on ISO Fortran. As has been discussed many times in the passed, MPI relies on extensions and/or implementation-specific behavior with Fortran through the latest standard. Only with Fortran 2018 will this situation disappear. Jeff > Regards, Thomas > -- > Thomas Jahns > HD(CP)^2 > Abteilung Anwendungssoftware > > Deutsches Klimarechenzentrum GmbH > Bundesstraße 45a • D-20146 Hamburg • Germany > > Phone: +49 40 460094-151 > Fax: +49 40 460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > URL: www.dkrz.de > > Geschäftsführer: Prof. Dr. Thomas Ludwig > Sitz der Gesellschaft: Hamburg > Amtsgericht Hamburg HRB 39784 > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0