June 2018 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPI_Recv, blocking call concept
by Ali MoradiAlamdarloo 14 Jun '18

14 Jun '18

On Thu, Jun 14, 2018 at 9:18 AM, Jed Brown <jed(a)jedbrown.org> wrote: > > You've already lost if a significant fraction of the core-hours on your > machine are spent waiting. Optimizing CPU throttling when you know > you'll be waiting a long time is just picking up crumbs. > You are right. I wish I had asked my questions five month before. Let's finish the conversation. Thanks all.

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Jed Brown 13 Jun '18

13 Jun '18

Ali MoradiAlamdarloo <timndus(a)gmail.com> writes: > Thanks for your comment. > On Fri, Jun 8, 2018 at 9:08 PM, Jed Brown <jed(a)jedbrown.org> wrote: > >> Your assumption is that exascale machines will be homogeneous > > Of course Exascale machines will be heterogeneous in a global view. probably > two application running on the system has heterogeneous computing resource > comparing to each other, but I think each application must has their own > homogeneous computing resources, otherwise maybe this heterogeneity itself > makes load imbalances. I meant coprocessors or distinct latency- and throughput-optimized cores. A CPU spinning is a trivial energy cost when connected to several idle GPUs. > Yes we can do some static hardware reconfiguration and find the best > configuration(probably heterogeneous) for a specific application but I > don't think this is going to be feasible in Exascale systems. > > filled with dirt-cheap hardware that requires lots of power, and operated by >> low-wage workers > > This is not my assumption. Yes, the hardware guys are building low power > high efficient computing cores, but currently what is going to be run on > top of it is a software that waste power at slacks. > 1- there is a time frame that software waste the power. > 2- power is the most constrained resource in Exascale. It's money. It costs $30M to buy the hardware that draws 1 MW, so you'd need to run for 30 years to equal the acquisition cost (a MW-year costs on the order of $1M). Energy is a small fraction of the operating cost for current HPC facilities -- hardware and staffing are much more expensive. Note that there exist 300 MW commercial data centers. > 3- there will be a millions(maybe billions) of cores acting like this. You have terrible load balance if most cores are waiting for a few, thus an expensive mostly-idle machine. Load balancing attempts the opposite -- no stragglers, but it doesn't matter if a few complete very early. > So this must be handled, no matter how much our computing cores are power > efficient. > > and also that applications will be highly latency >> tolerant while also having such atrocious load imbalance that reducing >> power while waiting on communication will significantly reduce total >> operating costs? >> > Yes, probably there will be some plentiful latency insensitive Exascale > applications. > We would like the developers to write code where every processors is doing > pretty much the same thing, it takes the same amount of time, they all > communicate at exactly the same time and they do it again, in the real > world this never happens. until there are slacks we have the chance to go > forward in efficiency. You've already lost if a significant fraction of the core-hours on your machine are spent waiting. Optimizing CPU throttling when you know you'll be waiting a long time is just picking up crumbs.

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Ali MoradiAlamdarloo 13 Jun '18

13 Jun '18

Thanks for your comment. On Fri, Jun 8, 2018 at 9:08 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Your assumption is that exascale machines will be homogeneous Of course Exascale machines will be heterogeneous in a global view. probably two application running on the system has heterogeneous computing resource comparing to each other, but I think each application must has their own homogeneous computing resources, otherwise maybe this heterogeneity itself makes load imbalances. Yes we can do some static hardware reconfiguration and find the best configuration(probably heterogeneous) for a specific application but I don't think this is going to be feasible in Exascale systems. filled with dirt-cheap hardware that requires lots of power, and operated by > low-wage workers This is not my assumption. Yes, the hardware guys are building low power high efficient computing cores, but currently what is going to be run on top of it is a software that waste power at slacks. 1- there is a time frame that software waste the power. 2- power is the most constrained resource in Exascale. 3- there will be a millions(maybe billions) of cores acting like this. So this must be handled, no matter how much our computing cores are power efficient. and also that applications will be highly latency > tolerant while also having such atrocious load imbalance that reducing > power while waiting on communication will significantly reduce total > operating costs? > Yes, probably there will be some plentiful latency insensitive Exascale applications. We would like the developers to write code where every processors is doing pretty much the same thing, it takes the same amount of time, they all communicate at exactly the same time and they do it again, in the real world this never happens. until there are slacks we have the chance to go forward in efficiency.

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Jed Brown 08 Jun '18

08 Jun '18

Ali MoradiAlamdarloo <timndus(a)gmail.com> writes: > On Fri, Jun 8, 2018 at 2:30 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> It spins because that is optimal for latency and how the shared-memory >> protocols work. >> > > On Fri, Jun 8, 2018 at 4:22 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> As blocking poll is largely incompatible with low-latency and >> shared-memory protocols, I don't think there is any implementation that is >> going to do a good job at this, since it would not be very appealing to the >> majority of MPI users. >> > > > On Fri, Jun 8, 2018 at 4:48 AM, Jed Brown <jed(a)jedbrown.org> wrote: > >> On the flipside, MPI would be more attractive to some >> (latency-insensitive) users if the implementations were not so committed >> to spinning. >> > > I'm trying to have a bit contribution on preparing MPI for next generation > of computing systems (Exascale). It seems current MPI implementations cares > about latency and performance more than power. I believe future MPI > implementations must consider power as a high priority goal so i'm doing > something much similar to what @Jeff linked it as my MS.c thesis. Your assumption is that exascale machines will be homogeneous, filled with dirt-cheap hardware that requires lots of power, and operated by low-wage workers, and also that applications will be highly latency tolerant while also having such atrocious load imbalance that reducing power while waiting on communication will significantly reduce total operating costs? Dystopic.

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Ali MoradiAlamdarloo 08 Jun '18

08 Jun '18

On Fri, Jun 8, 2018 at 2:30 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > It spins because that is optimal for latency and how the shared-memory > protocols work. > On Fri, Jun 8, 2018 at 4:22 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > As blocking poll is largely incompatible with low-latency and > shared-memory protocols, I don't think there is any implementation that is > going to do a good job at this, since it would not be very appealing to the > majority of MPI users. > On Fri, Jun 8, 2018 at 4:48 AM, Jed Brown <jed(a)jedbrown.org> wrote: > On the flipside, MPI would be more attractive to some > (latency-insensitive) users if the implementations were not so committed > to spinning. > I'm trying to have a bit contribution on preparing MPI for next generation of computing systems (Exascale). It seems current MPI implementations cares about latency and performance more than power. I believe future MPI implementations must consider power as a high priority goal so i'm doing something much similar to what @Jeff linked it as my MS.c thesis. Thanks all, ______________________ Researcher School of Computer Engineering Iran University of Science and Technology http://webpages.iust.ac.ir/msharifi/dslab.html

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Lisandro Dalcin 08 Jun '18

08 Jun '18

On Fri, 8 Jun 2018 at 03:18, Jed Brown <jed(a)jedbrown.org> wrote: > > Jeff Hammond <jeff.science(a)gmail.com> writes: > > > The PETSc folks appear to be the biggest proponents of blocking poll > > (solely for purposes of running dozens of MPI processes on their > > laptops, it seems) and they seem to prefer ch3:sock. I defer to their > > experience as to whether a better implementation exists. > > Indeed, the test suite has several tests that use 9-16 ranks and take > less than 1 second even when oversubscribed using ch3:sock, but are > disturbingly slow with modern MPICH. The conundrum of developers not > being able to debug tests on their laptop and many CI solutions not > being able to execute those tests is reminiscent of the testing fiasco > that is multi-GPU development. Intel MPI seems to have some backoff support with I_MPI_SPIN_COUNT, I_MPI_SHM_SPIN_COUNT, and some others vars. I never tested how well these features work, though. I guess PETSc developers would just like to have a way to somehow alter behavior at runtime by exporting some environ vars in a CI script. This would benefit testing and latency-insensitive use cases. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Jed Brown 07 Jun '18

07 Jun '18

Jeff Hammond <jeff.science(a)gmail.com> writes: > As blocking poll is largely incompatible with low-latency and shared-memory > protocols, I don't think there is any implementation that is going to do a > good job at this, since it would not be very appealing to the majority of > MPI users. On the flipside, MPI would be more attractive to some (latency-insensitive) users if the implementations were not so committed to spinning. > The PETSc folks appear to be the biggest proponents of blocking poll > (solely for purposes of running dozens of MPI processes on their > laptops, it seems) and they seem to prefer ch3:sock. I defer to their > experience as to whether a better implementation exists. Indeed, the test suite has several tests that use 9-16 ranks and take less than 1 second even when oversubscribed using ch3:sock, but are disturbingly slow with modern MPICH. The conundrum of developers not being able to debug tests on their laptop and many CI solutions not being able to execute those tests is reminiscent of the testing fiasco that is multi-GPU development.

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Jeff Hammond 07 Jun '18

07 Jun '18

On Thu, Jun 7, 2018 at 4:39 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Though I acknowledge my solution is not very elegant, I had to find a > workaround for a async execution Python framework I wrote > (mpi4py.futures), and eventually decided to go with sleep() calls with > exponential backoff. I have no control and don't want users to be > forced to switch to ch3:sock, and this behavior is the common one in > other MPIs out there. > > @Ali, if you can read Python, you may find some inspiration here: > https://bitbucket.org/mpi4py/mpi4py/src/master/src/mpi4py/futures/_lib.py > This certainly adds latency, but you can somehow decide how much > (well, up to the accuracy of your kernel timer slack) and you use it > where really really needed. > I am not fluent in Python but you are just using nonblocking calls and testing them carefully, right? That's what I'd implement if I wanted to reduce spinning. A more exotic implementation could do something along the lines of Casper and offload progress to a different process, which would allow the calling process to block on a interprocess semaphore or something like that, but it might cause problems with threads in that process if the OS tries to put the whole process to sleep. > @Jeff, if you have a better implementation to suggest, please let us > know (maybe off-list, this is unrelated to MPICH development). > As blocking poll is largely incompatible with low-latency and shared-memory protocols, I don't think there is any implementation that is going to do a good job at this, since it would not be very appealing to the majority of MPI users. The PETSc folks appear to be the biggest proponents of blocking poll (solely for purposes of running dozens of MPI processes on their laptops, it seems) and they seem to prefer ch3:sock. I defer to their experience as to whether a better implementation exists. Jeff > > On Fri, 8 Jun 2018 at 01:00, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > It spins because that is optimal for latency and how the shared-memory > protocols work. If you want blocking semantics, use ch3:sock, which will > park the calling thread in the kernel. It is great for oversubscription > but terrible for performance in the common case of exact subscription or > undersubscription. > > > > You can't save much power unless you drop into lower P/C-states, but the > states that save you significant power will increase the latency a huge > amount. Dell did something a while back that turned down the frequency > during MPI calls (http://www.hpcadvisorycouncil.com/events/ > 2013/Spain-Workshop/pdf/5_Dell.pdf), which saved a bit of power. > > > > Jeff > > > > On Thu, Jun 7, 2018 at 4:27 AM, Ali MoradiAlamdarloo <timndus(a)gmail.com> > wrote: > >> > >> Dear all, > >> > >> The blocking call definition from my understanding is something like > this: > >> when a process(P0) do a blocking system call, scheduler block the > process and assign another process(P1) in order to efficiently use of CPU > core. Finally P0 response will be ready and scheduler can map it again on a > core. > >> > >> But this is not what happening In MPICH->MPI_Recv function. you call it > BLOCKING call, but the process that call this function actually doesn't > block, it just continue running on core WAITING for his response. > >> > >> Why you decide to do this? why we have a process waiting on a valuable > processing core and burning the power? > >> > >> _______________________________________________ > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/devel > >> > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Lisandro Dalcin 07 Jun '18

07 Jun '18

Though I acknowledge my solution is not very elegant, I had to find a workaround for a async execution Python framework I wrote (mpi4py.futures), and eventually decided to go with sleep() calls with exponential backoff. I have no control and don't want users to be forced to switch to ch3:sock, and this behavior is the common one in other MPIs out there. @Ali, if you can read Python, you may find some inspiration here: https://bitbucket.org/mpi4py/mpi4py/src/master/src/mpi4py/futures/_lib.py This certainly adds latency, but you can somehow decide how much (well, up to the accuracy of your kernel timer slack) and you use it where really really needed. @Jeff, if you have a better implementation to suggest, please let us know (maybe off-list, this is unrelated to MPICH development). On Fri, 8 Jun 2018 at 01:00, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > It spins because that is optimal for latency and how the shared-memory protocols work. If you want blocking semantics, use ch3:sock, which will park the calling thread in the kernel. It is great for oversubscription but terrible for performance in the common case of exact subscription or undersubscription. > > You can't save much power unless you drop into lower P/C-states, but the states that save you significant power will increase the latency a huge amount. Dell did something a while back that turned down the frequency during MPI calls (http://www.hpcadvisorycouncil.com/events/2013/Spain-Workshop/pdf/5_Dell.pdf) which saved a bit of power. > > Jeff > > On Thu, Jun 7, 2018 at 4:27 AM, Ali MoradiAlamdarloo <timndus(a)gmail.com> wrote: >> >> Dear all, >> >> The blocking call definition from my understanding is something like this: >> when a process(P0) do a blocking system call, scheduler block the process and assign another process(P1) in order to efficiently use of CPU core. Finally P0 response will be ready and scheduler can map it again on a core. >> >> But this is not what happening In MPICH->MPI_Recv function. you call it BLOCKING call, but the process that call this function actually doesn't block, it just continue running on core WAITING for his response. >> >> Why you decide to do this? why we have a process waiting on a valuable processing core and burning the power? >> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] MPI_Recv, blocking call concept
by Jeff Hammond 07 Jun '18

07 Jun '18

It spins because that is optimal for latency and how the shared-memory protocols work. If you want blocking semantics, use ch3:sock, which will park the calling thread in the kernel. It is great for oversubscription but terrible for performance in the common case of exact subscription or undersubscription. You can't save much power unless you drop into lower P/C-states, but the states that save you significant power will increase the latency a huge amount. Dell did something a while back that turned down the frequency during MPI calls ( http://www.hpcadvisorycouncil.com/events/2013/Spain-Workshop/pdf/5_Dell.pdf) which saved a bit of power. Jeff On Thu, Jun 7, 2018 at 4:27 AM, Ali MoradiAlamdarloo <timndus(a)gmail.com> wrote: > Dear all, > > The blocking call definition from my understanding is something like this: > when a process(P0) do a blocking system call, scheduler block the process > and assign another process(P1) in order to efficiently use of CPU core. > Finally P0 response will be ready and scheduler can map it again on a core. > > But this is not what happening In MPICH->MPI_Recv function. you call it > BLOCKING call, but the process that call this function actually doesn't > block, it just continue running on core WAITING for his response. > > Why you decide to do this? why we have a process waiting on a valuable > processing core and burning the power? > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0