- devel - mailman3.cels.anl.gov

mpich-3.1.1: MPI_Status_c2f
by Lisandro Dalcin 05 Jul '14

05 Jul '14

Was this change intentional? This way, code linked with "mpicc" is failing because of the missing symbol. IMHO, MPI_Status_c2f should be defined in libmpi.dylib. Not checked on Linux, but looking at the Homebrew formula, I cannot find anything obvious to indicate this is a brew and/or OS X issue: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/mpich2.rb $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpi.dylib | grep MPI_Status_c2f $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpifort.dylib | grep MPI_Status_c2f 0000000000008d60 T _MPI_Status_c2f -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Jeff Hammond 03 Jul '14

03 Jul '14

PAMI is open-source via https://repo.anl-external.org/repos/bgq-driver/. I believe ALCF has already reported this bug but you can contact support(a)alcf.anl.gov for an update. Best, Jeff On Thu, Jul 3, 2014 at 2:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > Hi, Sam, > I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling results > suggested the problem lies in a IBM PAMI library call, > PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access to the > PAMI source code and don't know why. I reported it to IBM and hope IBM will > fix it. > Alternatively, you can set an environment variable PAMID_COLLECTIVES=0 to > disables pami collectives. My tests showed it at least fixed the scalability > problem of Comm_split and Comm_dup. > Also through profiling, I found the qsort() called in MPICH code is > actually using the merge sort algorithm in Mira's libc library. > > > > --Junchao Zhang > > > On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: >> >> I've been conducting scaling experiments on the Mira (Blue Gene/Q) and K >> (Sparc) supercomputers. I've noticed that the time required for >> MPI_Comm_split and MPI_Comm_dup can grow quickly with scale (~P^2). As >> such, its performance eventually becomes a bottleneck. That is, although >> the benefit of using a subcommunicator is huge (multigrid solves are >> weak-scalable), the penalty of creating one (multigrid build time) is also >> huge. >> >> For example, when scaling from 1 to 46K nodes (= cubes of integers) on >> Mira, the time (in seconds) required to build a MG solver (including a >> subcommunicator) scales as >> 222335.output: Total time in MGBuild 0.056704 >> 222336.output: Total time in MGBuild 0.060834 >> 222348.output: Total time in MGBuild 0.064782 >> 222349.output: Total time in MGBuild 0.090229 >> 222350.output: Total time in MGBuild 0.075280 >> 222351.output: Total time in MGBuild 0.091852 >> 222352.output: Total time in MGBuild 0.137299 >> 222411.output: Total time in MGBuild 0.301552 >> 222413.output: Total time in MGBuild 0.606444 >> 222415.output: Total time in MGBuild 0.745272 >> 222417.output: Total time in MGBuild 0.779757 >> 222418.output: Total time in MGBuild 4.671838 >> 222419.output: Total time in MGBuild 15.123162 >> 222420.output: Total time in MGBuild 33.875626 >> 222421.output: Total time in MGBuild 49.494547 >> 222422.output: Total time in MGBuild 151.329026 >> >> If I disable the call to MPI_Comm_Split, my time scales as >> 224982.output: Total time in MGBuild 0.050143 >> 224983.output: Total time in MGBuild 0.052607 >> 224988.output: Total time in MGBuild 0.050697 >> 224989.output: Total time in MGBuild 0.078343 >> 224990.output: Total time in MGBuild 0.054634 >> 224991.output: Total time in MGBuild 0.052158 >> 224992.output: Total time in MGBuild 0.060286 >> 225008.output: Total time in MGBuild 0.062925 >> 225009.output: Total time in MGBuild 0.097357 >> 225010.output: Total time in MGBuild 0.061807 >> 225011.output: Total time in MGBuild 0.076617 >> 225012.output: Total time in MGBuild 0.099683 >> 225013.output: Total time in MGBuild 0.125580 >> 225014.output: Total time in MGBuild 0.190711 >> 225016.output: Total time in MGBuild 0.218329 >> 225017.output: Total time in MGBuild 0.282081 >> >> Although I didn't directly measure it, this suggests the time for >> MPI_Comm_Split is growing roughly quadratically with process concurrency. >> >> >> >> >> I see the same effect on the K machine (8...64K nodes) where the code uses >> comm_split/dup in conjunction: >> run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 seconds >> run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 seconds >> run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 seconds >> run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 seconds >> run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 seconds >> run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 seconds >> run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 seconds >> run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 seconds >> run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 seconds >> run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 seconds >> run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 seconds >> run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 seconds >> >> >> A glance at the implementation on Mira (I don't know if the implementation >> on K is stock) suggests it should be using qsort to sort based on keys. >> Unfortunately, qsort is not performance robust like heap/merge sort. If one >> were to be productive and call comm_split like... >> MPI_Comm_split(...,mycolor,myrank,...) >> then one runs the risk that the keys are presorted. This hits the worst >> case computational complexity for qsort... O(P^2). Demanding programmers >> avoid sending sorted keys seems unreasonable. >> >> >> I should note, I see a similar lack of scaling with MPI_Comm_dup on the K >> machine. Unfortunately, my BGQ data used an earlier version of the code >> that did not use comm_dup. As such, I can’t definitively say that it is a >> problem on that machine as well. >> >> Thus, I'm asking for scalable implementations of comm_split/dup using >> merge/heap sort whose worst case complexity is still PlogP to be prioritized >> in the next update. >> >> >> thanks >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Junchao Zhang 03 Jul '14

03 Jul '14

Hi, Sam, I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling results suggested the problem lies in a IBM PAMI library call, PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access to the PAMI source code and don't know why. I reported it to IBM and hope IBM will fix it. Alternatively, you can set an environment variable PAMID_COLLECTIVES=0 to disables pami collectives. My tests showed it at least fixed the scalability problem of Comm_split and Comm_dup. Also through profiling, I found the qsort() called in MPICH code is actually using the merge sort algorithm in Mira's libc library. --Junchao Zhang On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: > I've been conducting scaling experiments on the Mira (Blue Gene/Q) and K > (Sparc) supercomputers. I've noticed that the time required for > MPI_Comm_split and MPI_Comm_dup can grow quickly with scale (~P^2). As > such, its performance eventually becomes a bottleneck. That is, although > the benefit of using a subcommunicator is huge (multigrid solves are > weak-scalable), the penalty of creating one (multigrid build time) is also > huge. > > For example, when scaling from 1 to 46K nodes (= cubes of integers) on > Mira, the time (in seconds) required to build a MG solver (including a > subcommunicator) scales as > 222335.output: Total time in MGBuild 0.056704 > 222336.output: Total time in MGBuild 0.060834 > 222348.output: Total time in MGBuild 0.064782 > 222349.output: Total time in MGBuild 0.090229 > 222350.output: Total time in MGBuild 0.075280 > 222351.output: Total time in MGBuild 0.091852 > 222352.output: Total time in MGBuild 0.137299 > 222411.output: Total time in MGBuild 0.301552 > 222413.output: Total time in MGBuild 0.606444 > 222415.output: Total time in MGBuild 0.745272 > 222417.output: Total time in MGBuild 0.779757 > 222418.output: Total time in MGBuild 4.671838 > 222419.output: Total time in MGBuild 15.123162 > 222420.output: Total time in MGBuild 33.875626 > 222421.output: Total time in MGBuild 49.494547 > 222422.output: Total time in MGBuild 151.329026 > > If I disable the call to MPI_Comm_Split, my time scales as > 224982.output: Total time in MGBuild 0.050143 > 224983.output: Total time in MGBuild 0.052607 > 224988.output: Total time in MGBuild 0.050697 > 224989.output: Total time in MGBuild 0.078343 > 224990.output: Total time in MGBuild 0.054634 > 224991.output: Total time in MGBuild 0.052158 > 224992.output: Total time in MGBuild 0.060286 > 225008.output: Total time in MGBuild 0.062925 > 225009.output: Total time in MGBuild 0.097357 > 225010.output: Total time in MGBuild 0.061807 > 225011.output: Total time in MGBuild 0.076617 > 225012.output: Total time in MGBuild 0.099683 > 225013.output: Total time in MGBuild 0.125580 > 225014.output: Total time in MGBuild 0.190711 > 225016.output: Total time in MGBuild 0.218329 > 225017.output: Total time in MGBuild 0.282081 > > Although I didn't directly measure it, this suggests the time for > MPI_Comm_Split is growing roughly quadratically with process concurrency. > > > > > I see the same effect on the K machine (8...64K nodes) where the code uses > comm_split/dup in conjunction: > run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 seconds > run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 seconds > run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 seconds > run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 seconds > run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 seconds > run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 seconds > run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 seconds > run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 seconds > run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 seconds > run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 seconds > run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 seconds > run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 seconds > > > A glance at the implementation on Mira (I don't know if the implementation > on K is stock) suggests it should be using qsort to sort based on keys. > Unfortunately, qsort is not performance robust like heap/merge sort. If > one were to be productive and call comm_split like... > MPI_Comm_split(...,mycolor,myrank,...) > then one runs the risk that the keys are presorted. This hits the worst > case computational complexity for qsort... O(P^2). Demanding programmers > avoid sending sorted keys seems unreasonable. > > > I should note, I see a similar lack of scaling with MPI_Comm_dup on the K > machine. Unfortunately, my BGQ data used an earlier version of the code > that did not use comm_dup. As such, I can’t definitively say that it is a > problem on that machine as well. > > Thus, I'm asking for scalable implementations of comm_split/dup using > merge/heap sort whose worst case complexity is still PlogP to be > prioritized in the next update. > > > thanks > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Hal Finkel 30 Jun '14

30 Jun '14

----- Original Message ----- > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > To: "Hal Finkel" <hfinkel(a)anl.gov> > Cc: "Pavan Balaji" <balaji(a)anl.gov>, devel(a)mpich.org > Sent: Monday, June 30, 2014 3:59:50 PM > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Hi Hal, > > The functions of poking progress engine are MPID_Progress_poke(), > MPID_Progress_wait() and MPID_Progress_test(), but they are not > provided with a request pointer directly. > By saying "pass the single request pointer", I mean we pass it when > issuing operations (location I mentioned in the last email), and if > it is not completed, the request will be enqueued in a queue in the > lower layer. When we call MPID_Progress_ functions, the progress > engine will process on that queue and try to make more requests > completed. Thanks, now I understand what you meant. -Hal > > Xin > ________________________________________ > From: Hal Finkel [hfinkel(a)anl.gov] > Sent: Monday, June 30, 2014 3:45 PM > To: Zhao, Xin > Cc: Pavan Balaji; devel(a)mpich.org > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > ----- Original Message ----- > > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > > To: "Hal Finkel" <hfinkel(a)anl.gov> > > Cc: "Pavan Balaji" <balaji(a)anl.gov>, devel(a)mpich.org > > Sent: Monday, June 30, 2014 3:39:20 PM > > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > Hi Hal, > > > > I am not sure if I understand you correctly. If you mean the > > function > > that CH3 uses to pass the message to the lower layer, it indeed > > tests if it is completed or not upon return from nemesis. If > > completed, CH3 will free that operation immediately, if not, it > > will > > leave that operation in the list. You can refer to L1087~L1097 in > > src/mpid/ch3/src/ch3u_rma_sync.c in latest mpich/master about this. > > No, I mean where is the code in the CH3 layer that calls the nemesis > layer (pokes the progress engine?) providing it only with the single > request pointer? In short, where is this "We didn't pass the list to > the nemesis layer but only pass the single request pointer"? > > Thanks again, > Hal > > > > > Xin > > ________________________________________ > > From: Hal Finkel [hfinkel(a)anl.gov] > > Sent: Monday, June 30, 2014 3:07 PM > > To: Zhao, Xin > > Cc: Pavan Balaji; devel(a)mpich.org > > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > ----- Original Message ----- > > > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > > > To: "Hal Finkel" <hfinkel(a)anl.gov>, devel(a)mpich.org > > > Cc: "Pavan Balaji" <balaji(a)anl.gov> > > > Sent: Monday, June 30, 2014 2:14:15 PM > > > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > > > Hi Hal, > > > > > > Currently the RMA operation list is implemented in CH3 layer, > > > whereas > > > the progress engine is implemented in nemesis layer. We didn't > > > pass > > > the list to the nemesis layer but only pass the single request > > > pointer, so the progress engine now only modify on the request > > > pointer. > > > > Thanks for the explanation. FWIW, I don't understand why the > > calling > > code (in CH3) could not test for complete upon return from nemesis > > and handle it then. Which function handles this? > > > > -Hal > > > > > This is one of the problems we have with the current > > > separation of the CH3 and nemesis layers, we plan to merge these > > > layers in future and this should be fixed at that time. > > > > > > Thanks, > > > Xin > > > ________________________________________ > > > From: Hal Finkel [hfinkel(a)anl.gov] > > > Sent: Monday, June 30, 2014 1:28 PM > > > To: devel(a)mpich.org > > > Cc: Zhao, Xin; Pavan Balaji > > > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > > > Xin, et al., > > > > > > Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added > > > MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a > > > request > > > is completed by the progress engine, why can't the progress > > > engine > > > move it to the beginning of the list (or the end), and have > > > rma_list_gc only search from the beginning (or end) until it > > > finds > > > an incomplete request? > > > > > > Thanks in advance, > > > Hal > > > > > > -- > > > Hal Finkel > > > Assistant Computational Scientist > > > Leadership Computing Facility > > > Argonne National Laboratory > > > > > > > -- > > Hal Finkel > > Assistant Computational Scientist > > Leadership Computing Facility > > Argonne National Laboratory > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Zhao, Xin 30 Jun '14

30 Jun '14

Hi Hal, The functions of poking progress engine are MPID_Progress_poke(), MPID_Progress_wait() and MPID_Progress_test(), but they are not provided with a request pointer directly. By saying "pass the single request pointer", I mean we pass it when issuing operations (location I mentioned in the last email), and if it is not completed, the request will be enqueued in a queue in the lower layer. When we call MPID_Progress_ functions, the progress engine will process on that queue and try to make more requests completed. Xin ________________________________________ From: Hal Finkel [hfinkel(a)anl.gov] Sent: Monday, June 30, 2014 3:45 PM To: Zhao, Xin Cc: Pavan Balaji; devel(a)mpich.org Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED ----- Original Message ----- > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > To: "Hal Finkel" <hfinkel(a)anl.gov> > Cc: "Pavan Balaji" <balaji(a)anl.gov>, devel(a)mpich.org > Sent: Monday, June 30, 2014 3:39:20 PM > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Hi Hal, > > I am not sure if I understand you correctly. If you mean the function > that CH3 uses to pass the message to the lower layer, it indeed > tests if it is completed or not upon return from nemesis. If > completed, CH3 will free that operation immediately, if not, it will > leave that operation in the list. You can refer to L1087~L1097 in > src/mpid/ch3/src/ch3u_rma_sync.c in latest mpich/master about this. No, I mean where is the code in the CH3 layer that calls the nemesis layer (pokes the progress engine?) providing it only with the single request pointer? In short, where is this "We didn't pass the list to the nemesis layer but only pass the single request pointer"? Thanks again, Hal > > Xin > ________________________________________ > From: Hal Finkel [hfinkel(a)anl.gov] > Sent: Monday, June 30, 2014 3:07 PM > To: Zhao, Xin > Cc: Pavan Balaji; devel(a)mpich.org > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > ----- Original Message ----- > > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > > To: "Hal Finkel" <hfinkel(a)anl.gov>, devel(a)mpich.org > > Cc: "Pavan Balaji" <balaji(a)anl.gov> > > Sent: Monday, June 30, 2014 2:14:15 PM > > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > Hi Hal, > > > > Currently the RMA operation list is implemented in CH3 layer, > > whereas > > the progress engine is implemented in nemesis layer. We didn't pass > > the list to the nemesis layer but only pass the single request > > pointer, so the progress engine now only modify on the request > > pointer. > > Thanks for the explanation. FWIW, I don't understand why the calling > code (in CH3) could not test for complete upon return from nemesis > and handle it then. Which function handles this? > > -Hal > > > This is one of the problems we have with the current > > separation of the CH3 and nemesis layers, we plan to merge these > > layers in future and this should be fixed at that time. > > > > Thanks, > > Xin > > ________________________________________ > > From: Hal Finkel [hfinkel(a)anl.gov] > > Sent: Monday, June 30, 2014 1:28 PM > > To: devel(a)mpich.org > > Cc: Zhao, Xin; Pavan Balaji > > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > Xin, et al., > > > > Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added > > MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request > > is completed by the progress engine, why can't the progress engine > > move it to the beginning of the list (or the end), and have > > rma_list_gc only search from the beginning (or end) until it finds > > an incomplete request? > > > > Thanks in advance, > > Hal > > > > -- > > Hal Finkel > > Assistant Computational Scientist > > Leadership Computing Facility > > Argonne National Laboratory > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Hal Finkel 30 Jun '14

30 Jun '14

----- Original Message ----- > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > To: "Hal Finkel" <hfinkel(a)anl.gov> > Cc: "Pavan Balaji" <balaji(a)anl.gov>, devel(a)mpich.org > Sent: Monday, June 30, 2014 3:39:20 PM > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Hi Hal, > > I am not sure if I understand you correctly. If you mean the function > that CH3 uses to pass the message to the lower layer, it indeed > tests if it is completed or not upon return from nemesis. If > completed, CH3 will free that operation immediately, if not, it will > leave that operation in the list. You can refer to L1087~L1097 in > src/mpid/ch3/src/ch3u_rma_sync.c in latest mpich/master about this. No, I mean where is the code in the CH3 layer that calls the nemesis layer (pokes the progress engine?) providing it only with the single request pointer? In short, where is this "We didn't pass the list to the nemesis layer but only pass the single request pointer"? Thanks again, Hal > > Xin > ________________________________________ > From: Hal Finkel [hfinkel(a)anl.gov] > Sent: Monday, June 30, 2014 3:07 PM > To: Zhao, Xin > Cc: Pavan Balaji; devel(a)mpich.org > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > ----- Original Message ----- > > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > > To: "Hal Finkel" <hfinkel(a)anl.gov>, devel(a)mpich.org > > Cc: "Pavan Balaji" <balaji(a)anl.gov> > > Sent: Monday, June 30, 2014 2:14:15 PM > > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > Hi Hal, > > > > Currently the RMA operation list is implemented in CH3 layer, > > whereas > > the progress engine is implemented in nemesis layer. We didn't pass > > the list to the nemesis layer but only pass the single request > > pointer, so the progress engine now only modify on the request > > pointer. > > Thanks for the explanation. FWIW, I don't understand why the calling > code (in CH3) could not test for complete upon return from nemesis > and handle it then. Which function handles this? > > -Hal > > > This is one of the problems we have with the current > > separation of the CH3 and nemesis layers, we plan to merge these > > layers in future and this should be fixed at that time. > > > > Thanks, > > Xin > > ________________________________________ > > From: Hal Finkel [hfinkel(a)anl.gov] > > Sent: Monday, June 30, 2014 1:28 PM > > To: devel(a)mpich.org > > Cc: Zhao, Xin; Pavan Balaji > > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > > > Xin, et al., > > > > Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added > > MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request > > is completed by the progress engine, why can't the progress engine > > move it to the beginning of the list (or the end), and have > > rma_list_gc only search from the beginning (or end) until it finds > > an incomplete request? > > > > Thanks in advance, > > Hal > > > > -- > > Hal Finkel > > Assistant Computational Scientist > > Leadership Computing Facility > > Argonne National Laboratory > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Zhao, Xin 30 Jun '14

30 Jun '14

Hi Hal, I am not sure if I understand you correctly. If you mean the function that CH3 uses to pass the message to the lower layer, it indeed tests if it is completed or not upon return from nemesis. If completed, CH3 will free that operation immediately, if not, it will leave that operation in the list. You can refer to L1087~L1097 in src/mpid/ch3/src/ch3u_rma_sync.c in latest mpich/master about this. Xin ________________________________________ From: Hal Finkel [hfinkel(a)anl.gov] Sent: Monday, June 30, 2014 3:07 PM To: Zhao, Xin Cc: Pavan Balaji; devel(a)mpich.org Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED ----- Original Message ----- > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > To: "Hal Finkel" <hfinkel(a)anl.gov>, devel(a)mpich.org > Cc: "Pavan Balaji" <balaji(a)anl.gov> > Sent: Monday, June 30, 2014 2:14:15 PM > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Hi Hal, > > Currently the RMA operation list is implemented in CH3 layer, whereas > the progress engine is implemented in nemesis layer. We didn't pass > the list to the nemesis layer but only pass the single request > pointer, so the progress engine now only modify on the request > pointer. Thanks for the explanation. FWIW, I don't understand why the calling code (in CH3) could not test for complete upon return from nemesis and handle it then. Which function handles this? -Hal > This is one of the problems we have with the current > separation of the CH3 and nemesis layers, we plan to merge these > layers in future and this should be fixed at that time. > > Thanks, > Xin > ________________________________________ > From: Hal Finkel [hfinkel(a)anl.gov] > Sent: Monday, June 30, 2014 1:28 PM > To: devel(a)mpich.org > Cc: Zhao, Xin; Pavan Balaji > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Xin, et al., > > Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added > MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request > is completed by the progress engine, why can't the progress engine > move it to the beginning of the list (or the end), and have > rma_list_gc only search from the beginning (or end) until it finds > an incomplete request? > > Thanks in advance, > Hal > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Hal Finkel 30 Jun '14

30 Jun '14

----- Original Message ----- > From: "Xin Zhao" <xinzhao3(a)illinois.edu> > To: "Hal Finkel" <hfinkel(a)anl.gov>, devel(a)mpich.org > Cc: "Pavan Balaji" <balaji(a)anl.gov> > Sent: Monday, June 30, 2014 2:14:15 PM > Subject: RE: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Hi Hal, > > Currently the RMA operation list is implemented in CH3 layer, whereas > the progress engine is implemented in nemesis layer. We didn't pass > the list to the nemesis layer but only pass the single request > pointer, so the progress engine now only modify on the request > pointer. Thanks for the explanation. FWIW, I don't understand why the calling code (in CH3) could not test for complete upon return from nemesis and handle it then. Which function handles this? -Hal > This is one of the problems we have with the current > separation of the CH3 and nemesis layers, we plan to merge these > layers in future and this should be fixed at that time. > > Thanks, > Xin > ________________________________________ > From: Hal Finkel [hfinkel(a)anl.gov] > Sent: Monday, June 30, 2014 1:28 PM > To: devel(a)mpich.org > Cc: Zhao, Xin; Pavan Balaji > Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED > > Xin, et al., > > Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added > MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request > is completed by the progress engine, why can't the progress engine > move it to the beginning of the list (or the end), and have > rma_list_gc only search from the beginning (or end) until it finds > an incomplete request? > > Thanks in advance, > Hal > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Zhao, Xin 30 Jun '14

30 Jun '14

Hi Hal, Currently the RMA operation list is implemented in CH3 layer, whereas the progress engine is implemented in nemesis layer. We didn't pass the list to the nemesis layer but only pass the single request pointer, so the progress engine now only modify on the request pointer. This is one of the problems we have with the current separation of the CH3 and nemesis layers, we plan to merge these layers in future and this should be fixed at that time. Thanks, Xin ________________________________________ From: Hal Finkel [hfinkel(a)anl.gov] Sent: Monday, June 30, 2014 1:28 PM To: devel(a)mpich.org Cc: Zhao, Xin; Pavan Balaji Subject: Re: MPIR_CVAR_CH3_RMA_GC_NUM_TESTED Xin, et al., Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request is completed by the progress engine, why can't the progress engine move it to the beginning of the list (or the end), and have rma_list_gc only search from the beginning (or end) until it finds an incomplete request? Thanks in advance, Hal -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [mpich-devel] MPIR_CVAR_CH3_RMA_GC_NUM_TESTED
by Hal Finkel 30 Jun '14

30 Jun '14

Xin, et al., Regarding 283319f5d9d1cdbcf016dc0ff8ab284114c0be7e (which added MPIR_CVAR_CH3_RMA_GC_NUM_TESTED and associated code): If a request is completed by the progress engine, why can't the progress engine move it to the beginning of the list (or the end), and have rma_list_gc only search from the beginning (or end) until it finds an incomplete request? Thanks in advance, Hal -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0