January 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Poor performance of Waitany / Waitsome
by Jeff Hammond 15 Jan '14

15 Jan '14

Given that you're using shared memory on a bloated OS (anything driving a GUI Window Manager), software overhead is going to be significant. You can only do so much about this. You might want to compile MPICH yourself using all the optimization flags. For example, I decided that "--enable-static --enable-fast=O3,nochkmsg,notiming,ndebug,nompit --disable-weak-symbols --enable-threads=single" were configure options that someone in search of speed might use. I have not done any systematic testing yet so some MPICH developer might tell me I'm a clueless buffoon for bothering to (de)activate some of these options. If you were to assume that I was going to rerun your test with different builds of MPICH on my Mac laptop as soon as I get some coffee, you would be correct. Hence, apathy on your part has no impact on the experiments regarding MPICH build variants and speed :-) Jeff On Wed, Jan 15, 2014 at 3:10 PM, John Grime <jgrime(a)uchicago.edu> wrote: > Cheers for the help, Jeff! > > I just tried to mimic Waitall() using a variety of the “MPI_Test…” routines > (code attached), and the results are not what I would expect: > > Although Waitsome() seems to give consistently the worst performance ( > Waitall < Waitany < Waitsome ) , Testsome() *appears* to always be faster > than Testany(), and for larger numbers of requests the performance order > seems to actually reverse. > > > Now, I may have done something spectacularly dumb here (it would be the 5th > such example from today alone), but on the assumption I have not: is this > result expected given the underlying implementation? > > J. > > > ./time_routines.sh 4 50 > > nprocs = 4, ntokens = 16, ncycles = 50 > Method : Time Relative > MPI_Waitall : 1.526000e-03 1.000x > MPI_Waitany : 1.435000e-03 0.940x > MPI_Waitsome : 3.381000e-03 2.216x > MPI_Testall : 3.101000e-03 2.032x > MPI_Testany : 8.080000e-03 5.295x > MPI_Testsome : 3.037000e-03 1.990x > PMPI_Waitall : 1.603000e-03 1.050x > PMPI_Waitany : 1.404000e-03 0.920x > PMPI_Waitsome : 4.666000e-03 3.058x > > > > nprocs = 4, ntokens = 64, ncycles = 50 > Method : Time Relative > MPI_Waitall : 3.173000e-03 1.000x > MPI_Waitany : 5.362000e-03 1.690x > MPI_Waitsome : 1.809100e-02 5.702x > MPI_Testall : 1.364200e-02 4.299x > MPI_Testany : 2.309300e-02 7.278x > MPI_Testsome : 1.469800e-02 4.632x > PMPI_Waitall : 2.063000e-03 0.650x > PMPI_Waitany : 9.420000e-03 2.969x > PMPI_Waitsome : 1.890300e-02 5.957x > > > > nprocs = 4, ntokens = 128, ncycles = 50 > Method : Time Relative > MPI_Waitall : 4.730000e-03 1.000x > MPI_Waitany : 2.691000e-02 5.689x > MPI_Waitsome : 4.519000e-02 9.554x > MPI_Testall : 4.696900e-02 9.930x > MPI_Testany : 7.285200e-02 15.402x > MPI_Testsome : 3.773400e-02 7.978x > PMPI_Waitall : 5.158000e-03 1.090x > PMPI_Waitany : 2.223200e-02 4.700x > PMPI_Waitsome : 4.205000e-02 8.890x > > > > nprocs = 4, ntokens = 512, ncycles = 50 > Method : Time Relative > MPI_Waitall : 1.365900e-02 1.000x > MPI_Waitany : 3.261610e-01 23.879x > MPI_Waitsome : 3.944020e-01 28.875x > MPI_Testall : 5.408010e-01 39.593x > MPI_Testany : 4.865990e-01 35.625x > MPI_Testsome : 3.067470e-01 22.458x > PMPI_Waitall : 1.976100e-02 1.447x > PMPI_Waitany : 3.011500e-01 22.048x > PMPI_Waitsome : 3.791930e-01 27.761x > > > > nprocs = 4, ntokens = 1024, ncycles = 50 > Method : Time Relative > MPI_Waitall : 4.087800e-02 1.000x > MPI_Waitany : 1.245209e+00 30.462x > MPI_Waitsome : 1.704020e+00 41.686x > MPI_Testall : 1.940940e+00 47.481x > MPI_Testany : 1.618215e+00 39.586x > MPI_Testsome : 1.133568e+00 27.731x > PMPI_Waitall : 3.970200e-02 0.971x > PMPI_Waitany : 1.344188e+00 32.883x > PMPI_Waitsome : 1.685816e+00 41.240x > > > nprocs = 4, ntokens = 2048, ncycles = 50 > Method : Time Relative > MPI_Waitall : 1.173840e-01 1.000x > MPI_Waitany : 4.600552e+00 39.192x > MPI_Waitsome : 6.840568e+00 58.275x > MPI_Testall : 6.762144e+00 57.607x > MPI_Testany : 5.170525e+00 44.048x > MPI_Testsome : 4.260335e+00 36.294x > PMPI_Waitall : 1.291590e-01 1.100x > PMPI_Waitany : 5.161881e+00 43.974x > PMPI_Waitsome : 7.388439e+00 62.942x > > > > On Jan 15, 2014, at 2:53 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> On Wed, Jan 15, 2014 at 2:23 PM, John Grime <jgrime(a)uchicago.edu> wrote: >>> Hi Jeff, >>> >>>> If Waitall wasn't faster than Waitsome or Waitany, then it wouldn't >>>> exist since obviously one can implement the former in terms of the >>>> latter >>> >>> >>> I see no reason it wouldn’t exist in such a case, given that it’s an >>> elegant/convenient way to wait for all requests to complete vs. Waitsome / >>> Waitany. It makes sense to me that it would be in the API in any case, much >>> as I appreciate the value of the RISC-y approach you imply. >>> >>>> it shouldn't be surprising that they aren't as efficient. >>> >>> I would’t expect them to have identical performance - but nor would I >>> have expected a performance difference of ~50x for the same number of >>> outstanding requests, even given that a naive loop over the request array >>> will be O(N). That loop should be pretty cheap after all, even given that >>> you can’t use cache well due to the potential for background state changes >>> in the request object data or whatever (I’m not sure how it’s actually >>> implemented, which is why I’m asking about this issue on the mailing list). >>> >>>> The appropriate question to ask is whether Waitany is implemented >>>> optimally or not. >>> >>> >>> Well, yes. I kinda hoped that question was heavily implied by my original >>> email! >>> >>> >>>> If you find that emulating Waitany >>>> using Testall following by a loop, then that's useful information. >>> >>> I accidentally the whole thing, Jeff! ;) >>> >>> But that’s a good idea, thanks - I’ll give it a try and report back! >> >> Testall is the wrong semantic here. I thought it would test them all >> individually but it doesn't. I implemented it anyways and it is the >> worst of all. I attached your test with my modifications. Because I >> am an evil bastard, I made a ton of whitespace changes in addition to >> the nontrivial ones. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> <nb_ring.c>_______________________________________________ > >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Poor performance of Waitany / Waitsome
by John Grime 15 Jan '14

15 Jan '14

Cheers for the help, Jeff! I just tried to mimic Waitall() using a variety of the “MPI_Test…” routines (code attached), and the results are not what I would expect: Although Waitsome() seems to give consistently the worst performance ( Waitall < Waitany < Waitsome ) , Testsome() *appears* to always be faster than Testany(), and for larger numbers of requests the performance order seems to actually reverse. Now, I may have done something spectacularly dumb here (it would be the 5th such example from today alone), but on the assumption I have not: is this result expected given the underlying implementation? J. ./time_routines.sh 4 50 nprocs = 4, ntokens = 16, ncycles = 50 Method : Time Relative MPI_Waitall : 1.526000e-03 1.000x MPI_Waitany : 1.435000e-03 0.940x MPI_Waitsome : 3.381000e-03 2.216x MPI_Testall : 3.101000e-03 2.032x MPI_Testany : 8.080000e-03 5.295x MPI_Testsome : 3.037000e-03 1.990x PMPI_Waitall : 1.603000e-03 1.050x PMPI_Waitany : 1.404000e-03 0.920x PMPI_Waitsome : 4.666000e-03 3.058x nprocs = 4, ntokens = 64, ncycles = 50 Method : Time Relative MPI_Waitall : 3.173000e-03 1.000x MPI_Waitany : 5.362000e-03 1.690x MPI_Waitsome : 1.809100e-02 5.702x MPI_Testall : 1.364200e-02 4.299x MPI_Testany : 2.309300e-02 7.278x MPI_Testsome : 1.469800e-02 4.632x PMPI_Waitall : 2.063000e-03 0.650x PMPI_Waitany : 9.420000e-03 2.969x PMPI_Waitsome : 1.890300e-02 5.957x nprocs = 4, ntokens = 128, ncycles = 50 Method : Time Relative MPI_Waitall : 4.730000e-03 1.000x MPI_Waitany : 2.691000e-02 5.689x MPI_Waitsome : 4.519000e-02 9.554x MPI_Testall : 4.696900e-02 9.930x MPI_Testany : 7.285200e-02 15.402x MPI_Testsome : 3.773400e-02 7.978x PMPI_Waitall : 5.158000e-03 1.090x PMPI_Waitany : 2.223200e-02 4.700x PMPI_Waitsome : 4.205000e-02 8.890x nprocs = 4, ntokens = 512, ncycles = 50 Method : Time Relative MPI_Waitall : 1.365900e-02 1.000x MPI_Waitany : 3.261610e-01 23.879x MPI_Waitsome : 3.944020e-01 28.875x MPI_Testall : 5.408010e-01 39.593x MPI_Testany : 4.865990e-01 35.625x MPI_Testsome : 3.067470e-01 22.458x PMPI_Waitall : 1.976100e-02 1.447x PMPI_Waitany : 3.011500e-01 22.048x PMPI_Waitsome : 3.791930e-01 27.761x nprocs = 4, ntokens = 1024, ncycles = 50 Method : Time Relative MPI_Waitall : 4.087800e-02 1.000x MPI_Waitany : 1.245209e+00 30.462x MPI_Waitsome : 1.704020e+00 41.686x MPI_Testall : 1.940940e+00 47.481x MPI_Testany : 1.618215e+00 39.586x MPI_Testsome : 1.133568e+00 27.731x PMPI_Waitall : 3.970200e-02 0.971x PMPI_Waitany : 1.344188e+00 32.883x PMPI_Waitsome : 1.685816e+00 41.240x nprocs = 4, ntokens = 2048, ncycles = 50 Method : Time Relative MPI_Waitall : 1.173840e-01 1.000x MPI_Waitany : 4.600552e+00 39.192x MPI_Waitsome : 6.840568e+00 58.275x MPI_Testall : 6.762144e+00 57.607x MPI_Testany : 5.170525e+00 44.048x MPI_Testsome : 4.260335e+00 36.294x PMPI_Waitall : 1.291590e-01 1.100x PMPI_Waitany : 5.161881e+00 43.974x PMPI_Waitsome : 7.388439e+00 62.942x On Jan 15, 2014, at 2:53 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > On Wed, Jan 15, 2014 at 2:23 PM, John Grime <jgrime(a)uchicago.edu> wrote: >> Hi Jeff, >> >>> If Waitall wasn't faster than Waitsome or Waitany, then it wouldn't >>> exist since obviously one can implement the former in terms of the >>> latter >> >> >> I see no reason it wouldn’t exist in such a case, given that it’s an elegant/convenient way to wait for all requests to complete vs. Waitsome / Waitany. It makes sense to me that it would be in the API in any case, much as I appreciate the value of the RISC-y approach you imply. >> >>> it shouldn't be surprising that they aren't as efficient. >> >> I would’t expect them to have identical performance - but nor would I have expected a performance difference of ~50x for the same number of outstanding requests, even given that a naive loop over the request array will be O(N). That loop should be pretty cheap after all, even given that you can’t use cache well due to the potential for background state changes in the request object data or whatever (I’m not sure how it’s actually implemented, which is why I’m asking about this issue on the mailing list). >> >>> The appropriate question to ask is whether Waitany is implemented >>> optimally or not. >> >> >> Well, yes. I kinda hoped that question was heavily implied by my original email! >> >> >>> If you find that emulating Waitany >>> using Testall following by a loop, then that's useful information. >> >> I accidentally the whole thing, Jeff! ;) >> >> But that’s a good idea, thanks - I’ll give it a try and report back! > > Testall is the wrong semantic here. I thought it would test them all > individually but it doesn't. I implemented it anyways and it is the > worst of all. I attached your test with my modifications. Because I > am an evil bastard, I made a ton of whitespace changes in addition to > the nontrivial ones. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > <nb_ring.c>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Poor performance of Waitany / Waitsome
by Jeff Hammond 15 Jan '14

15 Jan '14

On Wed, Jan 15, 2014 at 2:23 PM, John Grime <jgrime(a)uchicago.edu> wrote: > Hi Jeff, > >> If Waitall wasn't faster than Waitsome or Waitany, then it wouldn't >> exist since obviously one can implement the former in terms of the >> latter > > > I see no reason it wouldn’t exist in such a case, given that it’s an elegant/convenient way to wait for all requests to complete vs. Waitsome / Waitany. It makes sense to me that it would be in the API in any case, much as I appreciate the value of the RISC-y approach you imply. > >> it shouldn't be surprising that they aren't as efficient. > > I would’t expect them to have identical performance - but nor would I have expected a performance difference of ~50x for the same number of outstanding requests, even given that a naive loop over the request array will be O(N). That loop should be pretty cheap after all, even given that you can’t use cache well due to the potential for background state changes in the request object data or whatever (I’m not sure how it’s actually implemented, which is why I’m asking about this issue on the mailing list). > >> The appropriate question to ask is whether Waitany is implemented >> optimally or not. > > > Well, yes. I kinda hoped that question was heavily implied by my original email! > > >> If you find that emulating Waitany >> using Testall following by a loop, then that's useful information. > > I accidentally the whole thing, Jeff! ;) > > But that’s a good idea, thanks - I’ll give it a try and report back! Testall is the wrong semantic here. I thought it would test them all individually but it doesn't. I implemented it anyways and it is the worst of all. I attached your test with my modifications. Because I am an evil bastard, I made a ton of whitespace changes in addition to the nontrivial ones. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Poor performance of Waitany / Waitsome
by John Grime 15 Jan '14

15 Jan '14

Hi Jeff, > If Waitall wasn't faster than Waitsome or Waitany, then it wouldn't > exist since obviously one can implement the former in terms of the > latter I see no reason it wouldn’t exist in such a case, given that it’s an elegant/convenient way to wait for all requests to complete vs. Waitsome / Waitany. It makes sense to me that it would be in the API in any case, much as I appreciate the value of the RISC-y approach you imply. > it shouldn't be surprising that they aren't as efficient. I would’t expect them to have identical performance - but nor would I have expected a performance difference of ~50x for the same number of outstanding requests, even given that a naive loop over the request array will be O(N). That loop should be pretty cheap after all, even given that you can’t use cache well due to the potential for background state changes in the request object data or whatever (I’m not sure how it’s actually implemented, which is why I’m asking about this issue on the mailing list). > The appropriate question to ask is whether Waitany is implemented > optimally or not. Well, yes. I kinda hoped that question was heavily implied by my original email! > If you find that emulating Waitany > using Testall following by a loop, then that's useful information. I accidentally the whole thing, Jeff! ;) But that’s a good idea, thanks - I’ll give it a try and report back! J. On Jan 15, 2014, at 1:52 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > If Waitall wasn't faster than Wait some or Waitany, then it wouldn't > exist, since obviously one can implement the former in terms of the > latter. The point of Waitall is to be an optimization for N calls to > Wait. Waitsome and Waitany are just weaker optimizations that could > be cast as Testall followed by output array inspection and it > shouldn't be surprising that they aren't as efficient. > > The appropriate question to ask is whether Waitany is implemented > optimally or not. Comparing Waitall to its emulation using Waitany > _does not_ answer this question. If you find that emulating Waitany > using Testall following by a loop, then that's useful information. > > Jeff > > On Wed, Jan 15, 2014 at 12:17 PM, John Grime <jgrime(a)uchicago.edu> wrote: >> Hi all, >> >> I noticed unexpectedly poor performance of the MPI_Waitany() routine (Mac >> OSX 10.9.1, MPICH v3.0.4 via Macports). >> >> I noticed that “wbland” had added relevant information to the “trac” system: >> >> http://trac.mpich.org/projects/mpich/ticket/1988 >> >> … so I downloaded his example code, modified it a little and wrote a wrapper >> script to examine the different routines (attached to this email, apologies >> in advance for any dumb contents). >> >> Results: >> >> ./time_routines.sh 4 50 >> >> nprocs = 4, ntokens = 16, ncycles = 50 >> Method : Time Relative >> MPI_Waitall : 1.358000e-03 1.000x >> MPI_Waitany : 1.491000e-03 1.098x >> MPI_Waitsome : 3.243000e-03 2.388x >> PMPI_Waitall : 9.860000e-04 0.726x >> PMPI_Waitany : 1.421000e-03 1.046x >> PMPI_Waitsome : 4.432000e-03 3.264x >> >> >> nprocs = 4, ntokens = 64, ncycles = 50 >> Method : Time Relative >> MPI_Waitall : 2.075000e-03 1.000x >> MPI_Waitany : 5.746000e-03 2.769x >> MPI_Waitsome : 1.314400e-02 6.334x >> PMPI_Waitall : 3.142000e-03 1.514x >> PMPI_Waitany : 5.450000e-03 2.627x >> PMPI_Waitsome : 1.891500e-02 9.116x >> >> >> nprocs = 4, ntokens = 128, ncycles = 50 >> Method : Time Relative >> MPI_Waitall : 5.159000e-03 1.000x >> MPI_Waitany : 1.615100e-02 3.131x >> MPI_Waitsome : 5.004100e-02 9.700x >> PMPI_Waitall : 3.480000e-03 0.675x >> PMPI_Waitany : 2.564000e-02 4.970x >> PMPI_Waitsome : 3.799700e-02 7.365x >> >> >> nprocs = 4, ntokens = 512, ncycles = 50 >> Method : Time Relative >> MPI_Waitall : 1.949800e-02 1.000x >> MPI_Waitany : 2.431020e-01 12.468x >> MPI_Waitsome : 3.643640e-01 18.687x >> PMPI_Waitall : 1.869800e-02 0.959x >> PMPI_Waitany : 2.491870e-01 12.780x >> PMPI_Waitsome : 3.500600e-01 17.954x >> >> >> nprocs = 4, ntokens = 1024, ncycles = 50 >> Method : Time Relative >> MPI_Waitall : 2.749100e-02 1.000x >> MPI_Waitany : 1.223122e+00 44.492x >> MPI_Waitsome : 1.554282e+00 56.538x >> PMPI_Waitall : 3.329800e-02 1.211x >> PMPI_Waitany : 1.232125e+00 44.819x >> PMPI_Waitsome : 1.531198e+00 55.698x >> >> … and so it seems the performance delta between the different approaches ( >> Waitall / Waitany / Waitsome ) increases as a function of the buffer size. >> >> This is a bit of a problem for me, as I make heavy use of Waitany() to >> overlap communication with calculations. Is there any way to avoid this >> behavior? >> >> Cheers, >> >> >> J. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with debugging using gdb in mpich-3.0.4
by Florin Isaila 15 Jan '14

15 Jan '14

Bingo! bash works fine. Thanks Florin On Wed, Jan 15, 2014 at 12:21 PM, Balaji, Pavan <balaji(a)mcs.anl.gov> wrote: > > Are you using tcsh? I remember seeing something like this with tcsh, which we were never quite able to figure out why it happened. > > — Pavan > > On Jan 15, 2014, at 10:36 AM, Florin Isaila <florin.isaila(a)gmail.com> wrote: > >> Hi, >> >> I have a problem when trying to use gdb in mpich-3.0.4 >> >> MPICH is configured with: >> ./configure --prefix=/homes/fisaila/software/mpich --enable-g=all >> --enable-fast=O0 --enable-shared >> >> When running a the cpi from examples directory with: >> mpiexec -np 1 xterm -e gdb cpi >> >> I am getting the xterm window correctly and: >> >> (gdb) r >> Starting program: /nfs2/fisaila/software/mpich-3.0.4/examples/cpi >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >> [cli_0]: write_line error; fd=6 buf=:cmd=init pmi_version=1 pmi_subversion=1 >> : >> system msg for write_line failure : Bad file descriptor >> [cli_0]: Unable to write to PMI_fd >> [cli_0]: write_line error; fd=6 buf=:cmd=get_appnum >> : >> system msg for write_line failure : Bad file descriptor >> Fatal error in MPI_Init: Other MPI error, error stack: >> MPIR_Init_thread(433): >> MPID_Init(139).......: channel initialization failed >> MPID_Init(421).......: PMI_Get_appnum returned -1 >> [Inferior 1 (process 14215) exited with code 01] >> >> The gdb version is: >> gdb -v >> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 >> >> >> Any suggestion? >> >> Thanks >> Florin >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Poor performance of Waitany / Waitsome
by Jeff Hammond 15 Jan '14

15 Jan '14

If Waitall wasn't faster than Waitsome or Waitany, then it wouldn't exist, since obviously one can implement the former in terms of the latter. The point of Waitall is to be an optimization for N calls to Wait. Waitsome and Waitany are just weaker optimizations that could be cast as Testall followed by output array inspection and it shouldn't be surprising that they aren't as efficient. The appropriate question to ask is whether Waitany is implemented optimally or not. Comparing Waitall to its emulation using Waitany _does not_ answer this question. If you find that emulating Waitany using Testall following by a loop, then that's useful information. Jeff On Wed, Jan 15, 2014 at 12:17 PM, John Grime <jgrime(a)uchicago.edu> wrote: > Hi all, > > I noticed unexpectedly poor performance of the MPI_Waitany() routine (Mac > OSX 10.9.1, MPICH v3.0.4 via Macports). > > I noticed that “wbland” had added relevant information to the “trac” system: > > http://trac.mpich.org/projects/mpich/ticket/1988 > > … so I downloaded his example code, modified it a little and wrote a wrapper > script to examine the different routines (attached to this email, apologies > in advance for any dumb contents). > > Results: > > ./time_routines.sh 4 50 > > nprocs = 4, ntokens = 16, ncycles = 50 > Method : Time Relative > MPI_Waitall : 1.358000e-03 1.000x > MPI_Waitany : 1.491000e-03 1.098x > MPI_Waitsome : 3.243000e-03 2.388x > PMPI_Waitall : 9.860000e-04 0.726x > PMPI_Waitany : 1.421000e-03 1.046x > PMPI_Waitsome : 4.432000e-03 3.264x > > > nprocs = 4, ntokens = 64, ncycles = 50 > Method : Time Relative > MPI_Waitall : 2.075000e-03 1.000x > MPI_Waitany : 5.746000e-03 2.769x > MPI_Waitsome : 1.314400e-02 6.334x > PMPI_Waitall : 3.142000e-03 1.514x > PMPI_Waitany : 5.450000e-03 2.627x > PMPI_Waitsome : 1.891500e-02 9.116x > > > nprocs = 4, ntokens = 128, ncycles = 50 > Method : Time Relative > MPI_Waitall : 5.159000e-03 1.000x > MPI_Waitany : 1.615100e-02 3.131x > MPI_Waitsome : 5.004100e-02 9.700x > PMPI_Waitall : 3.480000e-03 0.675x > PMPI_Waitany : 2.564000e-02 4.970x > PMPI_Waitsome : 3.799700e-02 7.365x > > > nprocs = 4, ntokens = 512, ncycles = 50 > Method : Time Relative > MPI_Waitall : 1.949800e-02 1.000x > MPI_Waitany : 2.431020e-01 12.468x > MPI_Waitsome : 3.643640e-01 18.687x > PMPI_Waitall : 1.869800e-02 0.959x > PMPI_Waitany : 2.491870e-01 12.780x > PMPI_Waitsome : 3.500600e-01 17.954x > > > nprocs = 4, ntokens = 1024, ncycles = 50 > Method : Time Relative > MPI_Waitall : 2.749100e-02 1.000x > MPI_Waitany : 1.223122e+00 44.492x > MPI_Waitsome : 1.554282e+00 56.538x > PMPI_Waitall : 3.329800e-02 1.211x > PMPI_Waitany : 1.232125e+00 44.819x > PMPI_Waitsome : 1.531198e+00 55.698x > > … and so it seems the performance delta between the different approaches ( > Waitall / Waitany / Waitsome ) increases as a function of the buffer size. > > This is a bit of a problem for me, as I make heavy use of Waitany() to > overlap communication with calculations. Is there any way to avoid this > behavior? > > Cheers, > > > J. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Problem with debugging using gdb in mpich-3.0.4
by Balaji, Pavan 15 Jan '14

15 Jan '14

Are you using tcsh? I remember seeing something like this with tcsh, which we were never quite able to figure out why it happened. — Pavan On Jan 15, 2014, at 10:36 AM, Florin Isaila <florin.isaila(a)gmail.com> wrote: > Hi, > > I have a problem when trying to use gdb in mpich-3.0.4 > > MPICH is configured with: > ./configure --prefix=/homes/fisaila/software/mpich --enable-g=all > --enable-fast=O0 --enable-shared > > When running a the cpi from examples directory with: > mpiexec -np 1 xterm -e gdb cpi > > I am getting the xterm window correctly and: > > (gdb) r > Starting program: /nfs2/fisaila/software/mpich-3.0.4/examples/cpi > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > [cli_0]: write_line error; fd=6 buf=:cmd=init pmi_version=1 pmi_subversion=1 > : > system msg for write_line failure : Bad file descriptor > [cli_0]: Unable to write to PMI_fd > [cli_0]: write_line error; fd=6 buf=:cmd=get_appnum > : > system msg for write_line failure : Bad file descriptor > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(433): > MPID_Init(139).......: channel initialization failed > MPID_Init(421).......: PMI_Get_appnum returned -1 > [Inferior 1 (process 14215) exited with code 01] > > The gdb version is: > gdb -v > GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 > > > Any suggestion? > > Thanks > Florin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Poor performance of Waitany / Waitsome
by John Grime 15 Jan '14

15 Jan '14

Hi all, I noticed unexpectedly poor performance of the MPI_Waitany() routine (Mac OSX 10.9.1, MPICH v3.0.4 via Macports). I noticed that “wbland” had added relevant information to the “trac” system: http://trac.mpich.org/projects/mpich/ticket/1988 … so I downloaded his example code, modified it a little and wrote a wrapper script to examine the different routines (attached to this email, apologies in advance for any dumb contents). Results: ./time_routines.sh 4 50 nprocs = 4, ntokens = 16, ncycles = 50 Method : Time Relative MPI_Waitall : 1.358000e-03 1.000x MPI_Waitany : 1.491000e-03 1.098x MPI_Waitsome : 3.243000e-03 2.388x PMPI_Waitall : 9.860000e-04 0.726x PMPI_Waitany : 1.421000e-03 1.046x PMPI_Waitsome : 4.432000e-03 3.264x nprocs = 4, ntokens = 64, ncycles = 50 Method : Time Relative MPI_Waitall : 2.075000e-03 1.000x MPI_Waitany : 5.746000e-03 2.769x MPI_Waitsome : 1.314400e-02 6.334x PMPI_Waitall : 3.142000e-03 1.514x PMPI_Waitany : 5.450000e-03 2.627x PMPI_Waitsome : 1.891500e-02 9.116x nprocs = 4, ntokens = 128, ncycles = 50 Method : Time Relative MPI_Waitall : 5.159000e-03 1.000x MPI_Waitany : 1.615100e-02 3.131x MPI_Waitsome : 5.004100e-02 9.700x PMPI_Waitall : 3.480000e-03 0.675x PMPI_Waitany : 2.564000e-02 4.970x PMPI_Waitsome : 3.799700e-02 7.365x nprocs = 4, ntokens = 512, ncycles = 50 Method : Time Relative MPI_Waitall : 1.949800e-02 1.000x MPI_Waitany : 2.431020e-01 12.468x MPI_Waitsome : 3.643640e-01 18.687x PMPI_Waitall : 1.869800e-02 0.959x PMPI_Waitany : 2.491870e-01 12.780x PMPI_Waitsome : 3.500600e-01 17.954x nprocs = 4, ntokens = 1024, ncycles = 50 Method : Time Relative MPI_Waitall : 2.749100e-02 1.000x MPI_Waitany : 1.223122e+00 44.492x MPI_Waitsome : 1.554282e+00 56.538x PMPI_Waitall : 3.329800e-02 1.211x PMPI_Waitany : 1.232125e+00 44.819x PMPI_Waitsome : 1.531198e+00 55.698x … and so it seems the performance delta between the different approaches ( Waitall / Waitany / Waitsome ) increases as a function of the buffer size. This is a bit of a problem for me, as I make heavy use of Waitany() to overlap communication with calculations. Is there any way to avoid this behavior? Cheers, J.

1 0

Re: [mpich-discuss] Adding libraries to the mpicc wrapper
by Matthieu Dorier 15 Jan '14

15 Jan '14

Thanks Rob! Matthieu Dorier PhD student at ENS Rennes http://people.irisa.fr/Matthieu.Dorier ----- Mail original ----- > De: "Rob Latham" <robl(a)mcs.anl.gov> > À: discuss(a)mpich.org > Envoyé: Mercredi 15 Janvier 2014 17:15:49 > Objet: Re: [mpich-discuss] Adding libraries to the mpicc wrapper > > > > On 01/15/2014 08:55 AM, Matthieu Dorier wrote: > > Hello, > > > > I have written an ADIO backend that depends on a library. So far to > > make > > the use of the library transparent when compiling a program, I > > added the > > library at line 226 of the mpicc wrapper (mpilibs=" -lmpich > > -lmylibrary"). I'd like to know if there is a "cleaner" way of > > adding > > it: mpicc being generated from a template, there should be a way to > > simply set a variable somewhere for that, shouldn't it? How is it > > done > > for other ADIO backends? > > Hey Mattieu! > > for old pvfs and lustre, add to LDFLAGS and LIBS when configuring > mpich > (and in turn, romio) > > for pvfs2, ROMIO's configure looked for and used pvfs2-config to add > the > appropriate libraries and headers. ROMIO would then add those to > 'src/mpi/romio/localdefs' so the needed libraries would propagate up > to > the mpicc wrapper. > > ==rob > > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Problem with debugging using gdb in mpich-3.0.4
by Florin Isaila 15 Jan '14

15 Jan '14

Hi, I have a problem when trying to use gdb in mpich-3.0.4 MPICH is configured with: ./configure --prefix=/homes/fisaila/software/mpich --enable-g=all --enable-fast=O0 --enable-shared When running a the cpi from examples directory with: mpiexec -np 1 xterm -e gdb cpi I am getting the xterm window correctly and: (gdb) r Starting program: /nfs2/fisaila/software/mpich-3.0.4/examples/cpi [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [cli_0]: write_line error; fd=6 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=6 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(433): MPID_Init(139).......: channel initialization failed MPID_Init(421).......: PMI_Get_appnum returned -1 [Inferior 1 (process 14215) exited with code 01] The gdb version is: gdb -v GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 Any suggestion? Thanks Florin

1 0