August 2016 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] O(N^p) data
by Jeff Hammond 24 Aug '16

24 Aug '16

It's been a few years. I don't remember. Probably 82B/rank was the bug that caused me to measure the prefactor in the first place, and the bug-free value was ~8B/rank. Anyways, we could run MPI with 64 ppn on 48Ki nodes with 16 GB per node, which means MPI state was a lot less than 200 MB. I recall MPI footprint was ~20 MB, not including the 32 MB of POSIX shared memory that MPI required. Jeff On Wed, Aug 24, 2016 at 12:16 PM, Dan Ibanez <dan.a.ibanez(a)gmail.com> wrote: > Just a slight discrepancy: > > the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M > ranks) > 82 * 3M = 246MB > Did you mean 8.2 bytes per rank ? > > On Wed, Aug 24, 2016 at 2:05 PM, Dan Ibanez <dan.a.ibanez(a)gmail.com> > wrote: > >> Thanks Jeff ! >> >> Yea, I've been able to write scalable MPI-based code >> that doesn't use MPI_All* functions, and the >> MPI_Neighbor_all* variants are just perfect; they have >> replaced lots of low-level send/recv systems. >> >> I was interested in the theoretical scalability of the >> implementation, and your answer is pretty comprehensive >> so I'll go read those papers. >> >> On Wed, Aug 24, 2016 at 1:55 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> It depends on where you look in MPICH. I analyzed memory consumption of >>> MPI on Blue Gene/Q, which was based on MPICH (and is OSS, so you can read >>> all of it). There was O(nproc) memory usage at every node, but I recall it >>> the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M >>> ranks). I don't know if the O(nproc) data was in MPICH itself or the >>> underlying layer (PAMI), or both, but it doesn't really matter from a user >>> perspective. >>> >>> Some _networks_ might make it hard not to have O(nproc) eager buffers on >>> every rank, and there are other "features" of network HW/SW that may >>> require O(nproc) data. Obviously, since this sort of thing is not >>> scalable, networks that historically had such properties have evolved to >>> support more scalable designs. Some of the low-level issues are addressed >>> in https://www.open-mpi.org/papers/ipdps-2006/ipdps-2006-ope >>> nmpi-ib-scalability.pdf. >>> >>> User buffers are a separate issue. MPI_Alltoall and MPI_Allgather acts >>> on O(nproc) user storage. MPI_Allgatherv, MPI_Alltoallv and MPI_Alltoallw >>> have O(nproc) input vectors. MPI experts often refer to the vector >>> collectives as unscalable, but of course this may not matter in practice >>> for many users. And in some of the cases where MPI_Alltoallv is used, one >>> can replace with a carefully written loop over Send-Recv calls that does >>> not require the user to allocate O(nproc) vectors specifically for MPI. >>> >>> There's a paper by Argonne+IBM that addresses this topic in more detail: >>> http://www.mcs.anl.gov/~thakur/papers/mpi-million.pdf >>> >>> Jeff >>> >>> >>> On Wed, Aug 24, 2016 at 10:28 AM, Dan Ibanez <dan.a.ibanez(a)gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> This may be a silly question, but the reason >>>> I'm asking is to obtain a fairly definitive answer. >>>> Basically, does MPICH have data structures >>>> which are of size: >>>> 1) O(N) >>>> 2) O(N^2) >>>> Where N is the size of MPI_COMM_WORLD ? >>>> My initial guess would be no, because there >>>> exist machines (Mira) for which it is not >>>> possible to store N^2 bytes, and even N bytes >>>> becomes an issue. >>>> I understand there are MPI functions (MPI_alltoall) one can >>>> call that by definition will require at least O(N) memory, >>>> but supposing one does not use these, would the internal >>>> MPICH systems still have this memory complexity ? >>>> >>>> Thank you for looking at this anyway >>>> >>>> _______________________________________________ >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/devel >>>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >>> >> >> > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] O(N^p) data
by Dan Ibanez 24 Aug '16

24 Aug '16

Just a slight discrepancy: > the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M ranks) 82 * 3M = 246MB Did you mean 8.2 bytes per rank ? On Wed, Aug 24, 2016 at 2:05 PM, Dan Ibanez <dan.a.ibanez(a)gmail.com> wrote: > Thanks Jeff ! > > Yea, I've been able to write scalable MPI-based code > that doesn't use MPI_All* functions, and the > MPI_Neighbor_all* variants are just perfect; they have > replaced lots of low-level send/recv systems. > > I was interested in the theoretical scalability of the > implementation, and your answer is pretty comprehensive > so I'll go read those papers. > > On Wed, Aug 24, 2016 at 1:55 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> It depends on where you look in MPICH. I analyzed memory consumption of >> MPI on Blue Gene/Q, which was based on MPICH (and is OSS, so you can read >> all of it). There was O(nproc) memory usage at every node, but I recall it >> the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M >> ranks). I don't know if the O(nproc) data was in MPICH itself or the >> underlying layer (PAMI), or both, but it doesn't really matter from a user >> perspective. >> >> Some _networks_ might make it hard not to have O(nproc) eager buffers on >> every rank, and there are other "features" of network HW/SW that may >> require O(nproc) data. Obviously, since this sort of thing is not >> scalable, networks that historically had such properties have evolved to >> support more scalable designs. Some of the low-level issues are addressed >> in https://www.open-mpi.org/papers/ipdps-2006/ipdps-2006-ope >> nmpi-ib-scalability.pdf. >> >> User buffers are a separate issue. MPI_Alltoall and MPI_Allgather acts >> on O(nproc) user storage. MPI_Allgatherv, MPI_Alltoallv and MPI_Alltoallw >> have O(nproc) input vectors. MPI experts often refer to the vector >> collectives as unscalable, but of course this may not matter in practice >> for many users. And in some of the cases where MPI_Alltoallv is used, one >> can replace with a carefully written loop over Send-Recv calls that does >> not require the user to allocate O(nproc) vectors specifically for MPI. >> >> There's a paper by Argonne+IBM that addresses this topic in more detail: >> http://www.mcs.anl.gov/~thakur/papers/mpi-million.pdf >> >> Jeff >> >> >> On Wed, Aug 24, 2016 at 10:28 AM, Dan Ibanez <dan.a.ibanez(a)gmail.com> >> wrote: >> >>> Hello, >>> >>> This may be a silly question, but the reason >>> I'm asking is to obtain a fairly definitive answer. >>> Basically, does MPICH have data structures >>> which are of size: >>> 1) O(N) >>> 2) O(N^2) >>> Where N is the size of MPI_COMM_WORLD ? >>> My initial guess would be no, because there >>> exist machines (Mira) for which it is not >>> possible to store N^2 bytes, and even N bytes >>> becomes an issue. >>> I understand there are MPI functions (MPI_alltoall) one can >>> call that by definition will require at least O(N) memory, >>> but supposing one does not use these, would the internal >>> MPICH systems still have this memory complexity ? >>> >>> Thank you for looking at this anyway >>> >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >>> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel >> > >

1 0

Re: [mpich-devel] O(N^p) data
by Dan Ibanez 24 Aug '16

24 Aug '16

Thanks Jeff ! Yea, I've been able to write scalable MPI-based code that doesn't use MPI_All* functions, and the MPI_Neighbor_all* variants are just perfect; they have replaced lots of low-level send/recv systems. I was interested in the theoretical scalability of the implementation, and your answer is pretty comprehensive so I'll go read those papers. On Wed, Aug 24, 2016 at 1:55 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > It depends on where you look in MPICH. I analyzed memory consumption of > MPI on Blue Gene/Q, which was based on MPICH (and is OSS, so you can read > all of it). There was O(nproc) memory usage at every node, but I recall it > the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M > ranks). I don't know if the O(nproc) data was in MPICH itself or the > underlying layer (PAMI), or both, but it doesn't really matter from a user > perspective. > > Some _networks_ might make it hard not to have O(nproc) eager buffers on > every rank, and there are other "features" of network HW/SW that may > require O(nproc) data. Obviously, since this sort of thing is not > scalable, networks that historically had such properties have evolved to > support more scalable designs. Some of the low-level issues are addressed > in https://www.open-mpi.org/papers/ipdps-2006/ipdps-2006- > openmpi-ib-scalability.pdf. > > User buffers are a separate issue. MPI_Alltoall and MPI_Allgather acts on > O(nproc) user storage. MPI_Allgatherv, MPI_Alltoallv and MPI_Alltoallw > have O(nproc) input vectors. MPI experts often refer to the vector > collectives as unscalable, but of course this may not matter in practice > for many users. And in some of the cases where MPI_Alltoallv is used, one > can replace with a carefully written loop over Send-Recv calls that does > not require the user to allocate O(nproc) vectors specifically for MPI. > > There's a paper by Argonne+IBM that addresses this topic in more detail: > http://www.mcs.anl.gov/~thakur/papers/mpi-million.pdf > > Jeff > > > On Wed, Aug 24, 2016 at 10:28 AM, Dan Ibanez <dan.a.ibanez(a)gmail.com> > wrote: > >> Hello, >> >> This may be a silly question, but the reason >> I'm asking is to obtain a fairly definitive answer. >> Basically, does MPICH have data structures >> which are of size: >> 1) O(N) >> 2) O(N^2) >> Where N is the size of MPI_COMM_WORLD ? >> My initial guess would be no, because there >> exist machines (Mira) for which it is not >> possible to store N^2 bytes, and even N bytes >> becomes an issue. >> I understand there are MPI functions (MPI_alltoall) one can >> call that by definition will require at least O(N) memory, >> but supposing one does not use these, would the internal >> MPICH systems still have this memory complexity ? >> >> Thank you for looking at this anyway >> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] O(N^p) data
by Jeff Hammond 24 Aug '16

24 Aug '16

It depends on where you look in MPICH. I analyzed memory consumption of MPI on Blue Gene/Q, which was based on MPICH (and is OSS, so you can read all of it). There was O(nproc) memory usage at every node, but I recall it the prefactor was ~82 bytes, which is pretty lean (~25 MB per rank at 3M ranks). I don't know if the O(nproc) data was in MPICH itself or the underlying layer (PAMI), or both, but it doesn't really matter from a user perspective. Some _networks_ might make it hard not to have O(nproc) eager buffers on every rank, and there are other "features" of network HW/SW that may require O(nproc) data. Obviously, since this sort of thing is not scalable, networks that historically had such properties have evolved to support more scalable designs. Some of the low-level issues are addressed in https://www.open-mpi.org/papers/ipdps-2006/ipdps-2006-openmpi-ib-scalabilit… . User buffers are a separate issue. MPI_Alltoall and MPI_Allgather acts on O(nproc) user storage. MPI_Allgatherv, MPI_Alltoallv and MPI_Alltoallw have O(nproc) input vectors. MPI experts often refer to the vector collectives as unscalable, but of course this may not matter in practice for many users. And in some of the cases where MPI_Alltoallv is used, one can replace with a carefully written loop over Send-Recv calls that does not require the user to allocate O(nproc) vectors specifically for MPI. There's a paper by Argonne+IBM that addresses this topic in more detail: http://www.mcs.anl.gov/~thakur/papers/mpi-million.pdf Jeff On Wed, Aug 24, 2016 at 10:28 AM, Dan Ibanez <dan.a.ibanez(a)gmail.com> wrote: > Hello, > > This may be a silly question, but the reason > I'm asking is to obtain a fairly definitive answer. > Basically, does MPICH have data structures > which are of size: > 1) O(N) > 2) O(N^2) > Where N is the size of MPI_COMM_WORLD ? > My initial guess would be no, because there > exist machines (Mira) for which it is not > possible to store N^2 bytes, and even N bytes > becomes an issue. > I understand there are MPI functions (MPI_alltoall) one can > call that by definition will require at least O(N) memory, > but supposing one does not use these, would the internal > MPICH systems still have this memory complexity ? > > Thank you for looking at this anyway > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

O(N^p) data
by Dan Ibanez 24 Aug '16

24 Aug '16

Hello, This may be a silly question, but the reason I'm asking is to obtain a fairly definitive answer. Basically, does MPICH have data structures which are of size: 1) O(N) 2) O(N^2) Where N is the size of MPI_COMM_WORLD ? My initial guess would be no, because there exist machines (Mira) for which it is not possible to store N^2 bytes, and even N bytes becomes an issue. I understand there are MPI functions (MPI_alltoall) one can call that by definition will require at least O(N) memory, but supposing one does not use these, would the internal MPICH systems still have this memory complexity ? Thank you for looking at this anyway

1 0

Re: [mpich-devel] broken link to user guide
by Kenneth Raffenetti 01 Aug '16

01 Aug '16

The user guide link should be functional again. Ken On 07/30/2016 05:38 PM, Balaji, Pavan wrote: > > Sorry, something seems to have gone wrong after our power outage two days ago. We'll get it fixed. > > -- Pavan > >> On Jul 30, 2016, at 9:28 AM, Dan Ibanez <dan.a.ibanez(a)gmail.com> wrote: >> >> Hello, >> >> On this page: >> >> https://www.mpich.org/documentation/guides/ >> >> The link called "MPICH User's Guide" points to >> this location, which returns 404: >> >> https://www.mpich.org/static/downloads/3.2/mpich-3.2-userguide.pdf >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0