- devel - mailman3.cels.anl.gov

Re: [mpich-devel] code path for non-blocking operations that are deferred until synchronization
by Jeff Hammond 07 Jan '23

07 Jan '23

On Fri, Jan 6, 2023 at 4:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Requests are performance-critical objects and thus are allocated and > managed internally by MPICH. You may not have access to it. It is possible > to have user pre-allocated the request objects, but it will add much > complications that we'd very much like to avoid. > > The implementation impact is negligible. As noted on https://github.com/mpi-forum/mpi-issues/issues/664, request attributes require adding a single pointer value to the internal request object, which does not change its size noticeably, given what is already in there. struct MPIR_Request { ... struct MPIR_Attribute * attributes; ... } > Anyway, I pretty much support that we not to bother with the cases of > allreduce on different input/output layout. I don't think it is working > with any implementations, so technically it is not even breaking anything. > If we want to support this, the straightforward approach is to offer an > extension that allows allreduce with different in/out datatypes and assume > per-element operations. > Nothing from the standard is implemented until somebody does it, and MPICH is almost always first. I am trying to figure out whether it is feasible to implement all of the Fortran 2018 features in MPI-3 to know if we should amend the standard. Jeff > -- > Hui > ------------------------------ > *From:* Jeff Hammond <jeff.science(a)gmail.com> > *Sent:* Friday, January 6, 2023 2:17 AM > *To:* Zhou, Hui <zhouh(a)anl.gov> > *Cc:* MPICH <devel(a)mpich.org> > *Subject:* Re: [mpich-devel] code path for non-blocking operations that > are deferred until synchronization > > Thanks. This is helpful. > > In order to avoid ABI issues with the status object, I was thinking > instead we could define attributes on requests. That is sufficient to > solve all sorts of problems, because one can just get the attributes off > the request and decide to clean up as appropriate. > > Do you think adding attributes to requests causes any implementation > headaches? My expectation is that it does not slow down any critical path > because the only time the attribute fields are touched are when the user > accesses them. The internal object behind a request would need an > additional field, but that's a small change to MPIR_Request. > > However, if we end up standardizing a 256-bit status object in MPI-5 > (hypothetically), then we have an option to put hidden fields for this in > status. It's debatable if MPI_Status_get_buffer_address(MPI_Status * > status) is significantly better than a request attribute lookup. > > In any case, I figured out that I can do the deferred implementation using > generalized requests. It requires Pthreads and MPI_THREAD_MULTIPLE but I > can live with that. > > Jeff > > On Fri, Jan 6, 2023 at 12:39 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Jeff, > > The nonblocking cases can also do the same support by copying to and from > contiguous buffers if we stick the temporary buffer pointer in the request > object and free it upon completion. But if you are looking at this from the > Fortran binding layer, which does not have access to MPICH internals, then > it is difficult to do. The straight-forward solution would be ask MPICH to > add an extension, so the user can stick the buffer pointer somewhere, for > example, the status object. > > I wonder if there is a code path in MPICH associated with nonblocking > communication that is deferred until the request is synchronized. For > example, is there code that merely copies all the arguments passed to the > call into some object and then processes the whole operation when the > request is synchronized? > > I am not exactly sure what do you mean here, but the persistent request > most resembles your idea of merely hold all the arguments and only process > it at start time. But again, if you are not allowed to access MPICH > internals, then that is out-of-reach. > > -- > Hui > > ------------------------------ > *From:* Jeff Hammond via devel <devel(a)mpich.org> > *Sent:* Thursday, January 5, 2023 4:50 AM > *To:* MPICH <devel(a)mpich.org> > *Cc:* Jeff Hammond <jeff.science(a)gmail.com> > *Subject:* [mpich-devel] code path for non-blocking operations that are > deferred until synchronization > > This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but > there are other scenarios where nonblocking communication with > non-contiguous Fortran subarrays is extremely difficult to implement. The > blocking cases are easy to support by copying to and from contiguous > buffers. > > I wonder if there is a code path in MPICH associated with nonblocking > communication that is deferred until the request is synchronized. For > example, is there code that merely copies all the arguments passed to the > call into some object and then processes the whole operation when the > request is synchronized? > > Critically for my case, the Fortran arguments would need to be copied, > meaning the CFI_cdesc_t pointer not just the associated base_addr, which I > doubt is there based on what I've read so far. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] code path for non-blocking operations that are deferred until synchronization
by Zhou, Hui 06 Jan '23

06 Jan '23

Requests are performance-critical objects and thus are allocated and managed internally by MPICH. You may not have access to it. It is possible to have user pre-allocated the request objects, but it will add much complications that we'd very much like to avoid. Anyway, I pretty much support that we not to bother with the cases of allreduce on different input/output layout. I don't think it is working with any implementations, so technically it is not even breaking anything. If we want to support this, the straightforward approach is to offer an extension that allows allreduce with different in/out datatypes and assume per-element operations. -- Hui ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com> Sent: Friday, January 6, 2023 2:17 AM To: Zhou, Hui <zhouh(a)anl.gov> Cc: MPICH <devel(a)mpich.org> Subject: Re: [mpich-devel] code path for non-blocking operations that are deferred until synchronization Thanks. This is helpful. In order to avoid ABI issues with the status object, I was thinking instead we could define attributes on requests. That is sufficient to solve all sorts of problems, because one can just get the attributes off the request and decide to clean up as appropriate. Do you think adding attributes to requests causes any implementation headaches? My expectation is that it does not slow down any critical path because the only time the attribute fields are touched are when the user accesses them. The internal object behind a request would need an additional field, but that's a small change to MPIR_Request. However, if we end up standardizing a 256-bit status object in MPI-5 (hypothetically), then we have an option to put hidden fields for this in status. It's debatable if MPI_Status_get_buffer_address(MPI_Status * status) is significantly better than a request attribute lookup. In any case, I figured out that I can do the deferred implementation using generalized requests. It requires Pthreads and MPI_THREAD_MULTIPLE but I can live with that. Jeff On Fri, Jan 6, 2023 at 12:39 AM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Jeff, The nonblocking cases can also do the same support by copying to and from contiguous buffers if we stick the temporary buffer pointer in the request object and free it upon completion. But if you are looking at this from the Fortran binding layer, which does not have access to MPICH internals, then it is difficult to do. The straight-forward solution would be ask MPICH to add an extension, so the user can stick the buffer pointer somewhere, for example, the status object. I wonder if there is a code path in MPICH associated with nonblocking communication that is deferred until the request is synchronized. For example, is there code that merely copies all the arguments passed to the call into some object and then processes the whole operation when the request is synchronized? I am not exactly sure what do you mean here, but the persistent request most resembles your idea of merely hold all the arguments and only process it at start time. But again, if you are not allowed to access MPICH internals, then that is out-of-reach. -- Hui ________________________________ From: Jeff Hammond via devel <devel(a)mpich.org<mailto:[email protected]>> Sent: Thursday, January 5, 2023 4:50 AM To: MPICH <devel(a)mpich.org<mailto:[email protected]>> Cc: Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Subject: [mpich-devel] code path for non-blocking operations that are deferred until synchronization This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but there are other scenarios where nonblocking communication with non-contiguous Fortran subarrays is extremely difficult to implement. The blocking cases are easy to support by copying to and from contiguous buffers. I wonder if there is a code path in MPICH associated with nonblocking communication that is deferred until the request is synchronized. For example, is there code that merely copies all the arguments passed to the call into some object and then processes the whole operation when the request is synchronized? Critically for my case, the Fortran arguments would need to be copied, meaning the CFI_cdesc_t pointer not just the associated base_addr, which I doubt is there based on what I've read so far. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpich-devel] code path for non-blocking operations that are deferred until synchronization
by Jeff Hammond 06 Jan '23

06 Jan '23

Thanks. This is helpful. In order to avoid ABI issues with the status object, I was thinking instead we could define attributes on requests. That is sufficient to solve all sorts of problems, because one can just get the attributes off the request and decide to clean up as appropriate. Do you think adding attributes to requests causes any implementation headaches? My expectation is that it does not slow down any critical path because the only time the attribute fields are touched are when the user accesses them. The internal object behind a request would need an additional field, but that's a small change to MPIR_Request. However, if we end up standardizing a 256-bit status object in MPI-5 (hypothetically), then we have an option to put hidden fields for this in status. It's debatable if MPI_Status_get_buffer_address(MPI_Status * status) is significantly better than a request attribute lookup. In any case, I figured out that I can do the deferred implementation using generalized requests. It requires Pthreads and MPI_THREAD_MULTIPLE but I can live with that. Jeff On Fri, Jan 6, 2023 at 12:39 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Jeff, > > The nonblocking cases can also do the same support by copying to and from > contiguous buffers if we stick the temporary buffer pointer in the request > object and free it upon completion. But if you are looking at this from the > Fortran binding layer, which does not have access to MPICH internals, then > it is difficult to do. The straight-forward solution would be ask MPICH to > add an extension, so the user can stick the buffer pointer somewhere, for > example, the status object. > > I wonder if there is a code path in MPICH associated with nonblocking > communication that is deferred until the request is synchronized. For > example, is there code that merely copies all the arguments passed to the > call into some object and then processes the whole operation when the > request is synchronized? > > I am not exactly sure what do you mean here, but the persistent request > most resembles your idea of merely hold all the arguments and only process > it at start time. But again, if you are not allowed to access MPICH > internals, then that is out-of-reach. > > -- > Hui > > ------------------------------ > *From:* Jeff Hammond via devel <devel(a)mpich.org> > *Sent:* Thursday, January 5, 2023 4:50 AM > *To:* MPICH <devel(a)mpich.org> > *Cc:* Jeff Hammond <jeff.science(a)gmail.com> > *Subject:* [mpich-devel] code path for non-blocking operations that are > deferred until synchronization > > This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but > there are other scenarios where nonblocking communication with > non-contiguous Fortran subarrays is extremely difficult to implement. The > blocking cases are easy to support by copying to and from contiguous > buffers. > > I wonder if there is a code path in MPICH associated with nonblocking > communication that is deferred until the request is synchronized. For > example, is there code that merely copies all the arguments passed to the > call into some object and then processes the whole operation when the > request is synchronized? > > Critically for my case, the Fortran arguments would need to be copied, > meaning the CFI_cdesc_t pointer not just the associated base_addr, which I > doubt is there based on what I've read so far. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] code path for non-blocking operations that are deferred until synchronization
by Zhou, Hui 05 Jan '23

05 Jan '23

Hi Jeff, The nonblocking cases can also do the same support by copying to and from contiguous buffers if we stick the temporary buffer pointer in the request object and free it upon completion. But if you are looking at this from the Fortran binding layer, which does not have access to MPICH internals, then it is difficult to do. The straight-forward solution would be ask MPICH to add an extension, so the user can stick the buffer pointer somewhere, for example, the status object. I wonder if there is a code path in MPICH associated with nonblocking communication that is deferred until the request is synchronized. For example, is there code that merely copies all the arguments passed to the call into some object and then processes the whole operation when the request is synchronized? I am not exactly sure what do you mean here, but the persistent request most resembles your idea of merely hold all the arguments and only process it at start time. But again, if you are not allowed to access MPICH internals, then that is out-of-reach. -- Hui ________________________________ From: Jeff Hammond via devel <devel(a)mpich.org> Sent: Thursday, January 5, 2023 4:50 AM To: MPICH <devel(a)mpich.org> Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: [mpich-devel] code path for non-blocking operations that are deferred until synchronization This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but there are other scenarios where nonblocking communication with non-contiguous Fortran subarrays is extremely difficult to implement. The blocking cases are easy to support by copying to and from contiguous buffers. I wonder if there is a code path in MPICH associated with nonblocking communication that is deferred until the request is synchronized. For example, is there code that merely copies all the arguments passed to the call into some object and then processes the whole operation when the request is synchronized? Critically for my case, the Fortran arguments would need to be copied, meaning the CFI_cdesc_t pointer not just the associated base_addr, which I doubt is there based on what I've read so far. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

code path for non-blocking operations that are deferred until synchronization
by Jeff Hammond 05 Jan '23

05 Jan '23

This is related to https://github.com/mpi-forum/mpi-issues/issues/663 but there are other scenarios where nonblocking communication with non-contiguous Fortran subarrays is extremely difficult to implement. The blocking cases are easy to support by copying to and from contiguous buffers. I wonder if there is a code path in MPICH associated with nonblocking communication that is deferred until the request is synchronized. For example, is there code that merely copies all the arguments passed to the call into some object and then processes the whole operation when the request is synchronized? Critically for my case, the Fortran arguments would need to be copied, meaning the CFI_cdesc_t pointer not just the associated base_addr, which I doubt is there based on what I've read so far. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Slides from SC22 MPICH BoF posted
by Raffenetti, Ken 15 Dec '22

15 Dec '22

Hi all, With permission from the presenters, we have posted slides from the SC22 MPICH BoF to the MPICH Github wiki. https://github.com/pmodels/mpich/wiki/2022-Supercomputing-BoF#presentations Thanks to all the presenters and the MPICH community for participating! Cheers, The MPICH team

1 0

Join the MPICH BoF this Wednesday evening at SC22
by Raffenetti, Ken 14 Nov '22

14 Nov '22

All, The MPICH team will host a Birds-of-a-Feather session this Wednesday evening at 5:15pm CST as part of the SC22 proceedings. We hope to see you there to discuss the latest developments in MPICH, and to hear from partners Intel, HPE, ParTec, MVAPICH, NVIDIA, and Cornelis Networks. Visit the link below for more info. https://github.com/pmodels/mpich/wiki/2022-Supercomputing-BoF Cheers, The MPICH team

1 0

Re: [mpich-devel] is there a way to speedup build related to yaksuri?
by Zhou, Hui 17 Aug '22

17 Aug '22

Yes! Try ./autogen.sh -yaksa-depth=2. You can try -yaksa-depth=1 if 2 is still too slow. -- Hui ________________________________ From: Jeff Hammond via devel <devel(a)mpich.org> Sent: Wednesday, August 17, 2022 1:02 AM To: MPICH <devel(a)mpich.org> Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: [mpich-devel] is there a way to speedup build related to yaksuri? Is there a configure option that would allow me to avoid the compilation bottleneck in `src/backend/seq/pup/yaksuri_seqi_pup*`? I want to rebuild MPICH frequently for development purposes, and I do not care about datatype performance in this context. Some of the yaksuri objects take seconds to compile with GCC 11 at -O2 on Tiger Lake core, and there are hundreds of them. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

is there a way to speedup build related to yaksuri?
by Jeff Hammond 17 Aug '22

17 Aug '22

Is there a configure option that would allow me to avoid the compilation bottleneck in `src/backend/seq/pup/yaksuri_seqi_pup*`? I want to rebuild MPICH frequently for development purposes, and I do not care about datatype performance in this context. Some of the yaksuri objects take seconds to compile with GCC 11 at -O2 on Tiger Lake core, and there are hundreds of them. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

Quick update to the agenda. Krishna Kandalla will give an update for HPE. Agenda (subject to change): [5 mins] Intro - Ken Raffenetti ANL [30 mins] MPICH Update - Yanfei Guo & Hui Zhou (ANL) [12 mins] Intel Update - Gengbin Zheng (Intel) [12 mins] MVAPICH2 Update - Hari Subramoni (OSU) [12 mins] HPE Update - Krishna Kandalla (HPE) [12 mins] Pilgrim Update - Chen Wang (UIUC) [~5 mins] Q&A Cheers, The MPICH team On 5/6/22, 2:12 PM, "Raffenetti, Ken" <raffenet(a)anl.gov> wrote: All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0