- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI Benchmark Suite
by Pavan Balaji 09 Jul '13

09 Jul '13

Nice collection. Thanks for putting it together. -- Pavan On 07/09/2013 02:53 PM, Jiri Simsa wrote: > I have compiled the pointers from Pavan, Rajeev, and Jeff into an > overview of MPI workloads (attached). > > Any comments or corrections are welcome. > > --Jiri > > > On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > See also the Mantevo mini-apps. http://www.mantevo.org/ > > > On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: > > > > > On 07/05/2013 10:38 AM, Jiri Simsa wrote: > >> Could anyone point me in the direction of programs that are > >> representative of MPI programs? I am looking for something like > PARSEC > >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am > >> interested in macro-bechnmarks not micro-benchmarks. Thank you. > > > > You could try the Sequoia suite. There's also the NAS parallel > benchmarks and Graph500 benchmarks. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Maciej.Golebiewski＠csiro.au 09 Jul '13

09 Jul '13

Hi Jeff, > Does your code actually scale to 240 MPI ranks per card? What is > the performance relative to peak or a fully-populated MPI-only run > on e.g SNB? I don't expect it to scale up to that many ranks, mentioning 240 ranks/card in my previous email was just to demonstrate that I was able to start that many ranks with Intel MPI. Performance peaks out at 60 ranks on MIC and is still about 4.5 times slower than 12 ranks on a dual slot system with 6 core X5670 Xeons (on which it scales up to 36 ranks over 3 nodes, the grid is only about 15K points which is probably what prevents further scaling). While MIC cores are slower that Xeon, 4.5 times slower with 5 times as many ranks was quite dissapointing. My first hunch was that perhaps Intel MPI does not handle well many immediate send/recv requests at a time (according to vampir traces node leader is always ready for workers to talk to them, but the workers take long to complete either receive of input or send of results), so I wanted to compare the perfrmance using another MPI implementation. After reading your comment about ring interconenct now I consider the possibility that instead I'm hitting the limit of the interconnect. > You're going to have to convert to MPI+Threads eventually so you That's one of these things that are obvious from a technical point of view but are not going to happen any time soon due to various non-technical impediments. > If you don't like OpenMP, you could consider Pthreads or TBB I've done my fair share of Pthreads programming in the past, and it's an evil thing to recommend it over OpenMP for a scientific application actively developed by people who are not expert programmers. Very, very evil. ;) > OpenMP, it's because you have an F77 code with common blocks that > cannot be made thread-safe for all the beer in Australia. Close enough, but not that bad. F90 with data in multiple modules, the main loop time loop could be nicely task-parallelized, the main effort would be to refactor the code so that no subroutine is accessing data through modules, only through the arguments. And then to forever keep policing the scientists to stick to this rule. :) Again, the technical side is not the problem, it's the external circumstances. Anyway, thanks for your comments. Replying to them helped me tidy my thoughts and get a better idea how to present my results and conclusions to the user. Cheers, Maciej

1 0

Re: [mpich-discuss] MPI Benchmark Suite
by Jiri Simsa 09 Jul '13

09 Jul '13

The overview is also available here: http://www.cs.cmu.edu/~jsimsa/mpi-macro.pdf On Tue, Jul 9, 2013 at 3:53 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > I have compiled the pointers from Pavan, Rajeev, and Jeff into an overview > of MPI workloads (attached). > > Any comments or corrections are welcome. > > --Jiri > > > On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >> See also the Mantevo mini-apps. http://www.mantevo.org/ >> >> >> On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: >> >> > >> > On 07/05/2013 10:38 AM, Jiri Simsa wrote: >> >> Could anyone point me in the direction of programs that are >> >> representative of MPI programs? I am looking for something like PARSEC >> >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am >> >> interested in macro-bechnmarks not micro-benchmarks. Thank you. >> > >> > You could try the Sequoia suite. There's also the NAS parallel >> benchmarks and Graph500 benchmarks. >> > >> > -- Pavan >> > >> > -- >> > Pavan Balaji >> > http://www.mcs.anl.gov/~balaji >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> >

1 0

Re: [mpich-discuss] MPI Benchmark Suite
by Jiri Simsa 09 Jul '13

09 Jul '13

I have compiled the pointers from Pavan, Rajeev, and Jeff into an overview of MPI workloads (attached). Any comments or corrections are welcome. --Jiri On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > See also the Mantevo mini-apps. http://www.mantevo.org/ > > > On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: > > > > > On 07/05/2013 10:38 AM, Jiri Simsa wrote: > >> Could anyone point me in the direction of programs that are > >> representative of MPI programs? I am looking for something like PARSEC > >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am > >> interested in macro-bechnmarks not micro-benchmarks. Thank you. > > > > You could try the Sequoia suite. There's also the NAS parallel > benchmarks and Graph500 benchmarks. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Jed Brown 09 Jul '13

09 Jul '13

Jeff Hammond <jeff.science(a)gmail.com> writes: > You're going to have to convert to MPI+Threads eventually so you might > as well start now. It is completely unreasonable to expect that > MPI-only is reasonable usage on a 60-core chip with a ring > interconnect. This is something that endpoints is supposed to address. Note that OpenMP doesn't have any decent way to do reductions scoped across subsets of threads, so for many applications, it's not even a reasonable level of expressivity. > I am so convinced of the impossibility that MPI_Allreduce scales to >>100 MPI procs in shared memory that I will learn Ada as punishment if > I'm wrong. Does 30 microsecond MPI_Allreduce at 512 cores qualify as "impossible to scale"? http://www.nas.nasa.gov/assets/pdf/techreports/2010/nas-10-001.pdf

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Sufeng Niu 09 Jul '13

09 Jul '13

I see. Thanks a lot! and that make sense. I tried fflush(stdout), it seems that still not work properly. but I understand that printf is not a proper way to check the timing. Thank you! Sufeng On Tue, Jul 9, 2013 at 12:24 PM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > ...and depending on your filesystem (assuming you pipe stdout to a > file), even that isn't sufficient. > > printf is a terrible way to evaluate temporal relationships and you > shouldn't try to reason about parallel programs with it unless you've > taken great care to ensure a global total ordering of output streams. > > Jeff > > On Tue, Jul 9, 2013 at 12:05 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > printf buffers output and doesn't write it out immediately (or in this > case, > > send it to mpiexec immediately). Try doing "fflush(stdout);" immediately > > after printf. > > > > -- Pavan > > > > > > On 07/09/2013 12:01 PM, Sufeng Niu wrote: > >> > >> Hello, > >> > >> Sorry to post this long, stupid and simple question. > >> I found that some time MPI_Barrier cannot stop all the process. I try to > >> write a simple test program to create data_struct shown below: > >> > >> #include <stdlib.h> > >> #include <stdio.h> > >> #include "mpi.h" > >> > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> int d; > >> } foo; > >> > >> int main(int argc, char *argv[]) > >> { > >> > >> int rank, size; > >> int i; > >> > >> foo x; > >> > >> MPI_Init(&argc, &argv); > >> MPI_Comm_size(MPI_COMM_WORLD, &size); > >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); > >> > >> char processor_name[MPI_MAX_PROCESSOR_NAME]; > >> int name_len; > >> MPI_Get_processor_name(processor_name, &name_len); > >> printf("-- processor %s, rank %d out of %d processors\n", > >> processor_name, rank, size); > >> > >> *MPI_Barrier(MPI_COMM_WORLD);* > >> > >> > >> int count=4; > >> > >> MPI_Datatype testtype; > >> MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; > >> int len[4] = {1, 1, 1, 1}; > >> MPI_Aint disp[4]; > >> long int base; > >> > >> MPI_Address(&x, disp); > >> MPI_Address(&(x.a), disp+1); > >> MPI_Address(&(x.b), disp+2); > >> MPI_Address(&(x.c), disp+3); > >> base = disp[0]; > >> for(i=0; i<4; i++) disp[i] -= base; > >> > >> MPI_Type_struct(count, len, disp, types, &testtype); > >> MPI_Type_commit(&testtype); > >> > >> if(rank == 0){ > >> x.a = 2; > >> x.b = 0; > >> x.c = 10; > >> x.d = 3; > >> } > >> > >> printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, > >> x.b, x.c, x.d); > >> *MPI_Barrier(MPI_COMM_WORLD);* > >> > >> MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); > >> > >> printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, > >> x.b, x.c, x.d); > >> > >> MPI_Finalize(); > >> > >> return 0; > >> } > >> > >> the output should be looks like: > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 1(before): x value is 1197535864, -1, 4994901, 0 > >> rank 2(before): x value is 1591464488, -1, 4994901, 0 > >> rank 3(before): x value is 1851622184, -1, 4994901, 0 > >> rank 0(after): x value is 2, 0, 10, 3 > >> rank 3(after): x value is 2, 0, 10, 3 > >> rank 1(after): x value is 2, 0, 10, 3 > >> rank 2(after): x value is 2, 0, 10, 3 > >> > >> but some time is shows as: > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 0(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> > >> out of 4 processors > >> rank 1(before): x value is -464731256, -1, 4994901, 0 > >> rank 1(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 2(before): x value is 1863042488, -1, 4994901, 0 > >> rank 2(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> > >> out of 4 processors > >> rank 3(before): x value is 1721065144, -1, 4994901, 0 > >> rank 3(after): x value is 2, 0, 10, 3 > >> > >> or > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> > >> out of 4 processors > >> rank 1(before): x value is -1883169624, -1, 4994901, 0 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 2(before): x value is -451256152, -1, 4994901, 0 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> > >> out of 4 processors > >> rank 3(before): x value is 1715067240, -1, 4994901, 0 > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 0(after): x value is 2, 0, 10, 3 > >> rank 1(after): x value is 2, 0, 10, 3 > >> rank 2(after): x value is 2, 0, 10, 3 > >> rank 3(after): x value is 2, 0, 10, 3 > >> > >> it is all randomly, I am not sure where is the problem. > >> > >> The second issue is I use MPI_Datatype to create a MPI struct for > >> broadcast. However, as the program shown above, if I change the struct: > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> *int d;* > >> > >> } foo; > >> as > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> *double d*; > >> > >> } foo; > >> > >> I found the result is: > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3.250000 > >> rank 3(before): x value is 0, 0, 547474368, 0.000000 > >> rank 1(before): x value is 0, 0, 547474368, 0.000000 > >> rank 2(before): x value is 0, 0, 547474368, 0.000000 > >> rank 0(after): x value is 2, 0, 10, 3.250000 > >> *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > >> > >> rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > >> rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 > >> > >> * > >> Do you have any clues on why this happened? Thanks a lot! > >> > >> -- > >> Best Regards, > >> Sufeng Niu > >> ECASP lab, ECE department, Illinois Institute of Technology > >> Tel: 312-731-7219 > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > -- Best Regards, Sufeng Niu ECASP lab, ECE department, Illinois Institute of Technology Tel: 312-731-7219

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Jeff Hammond 09 Jul '13

09 Jul '13

...and depending on your filesystem (assuming you pipe stdout to a file), even that isn't sufficient. printf is a terrible way to evaluate temporal relationships and you shouldn't try to reason about parallel programs with it unless you've taken great care to ensure a global total ordering of output streams. Jeff On Tue, Jul 9, 2013 at 12:05 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > printf buffers output and doesn't write it out immediately (or in this case, > send it to mpiexec immediately). Try doing "fflush(stdout);" immediately > after printf. > > -- Pavan > > > On 07/09/2013 12:01 PM, Sufeng Niu wrote: >> >> Hello, >> >> Sorry to post this long, stupid and simple question. >> I found that some time MPI_Barrier cannot stop all the process. I try to >> write a simple test program to create data_struct shown below: >> >> #include <stdlib.h> >> #include <stdio.h> >> #include "mpi.h" >> >> typedef struct >> { >> int a; >> char b; >> int c; >> int d; >> } foo; >> >> int main(int argc, char *argv[]) >> { >> >> int rank, size; >> int i; >> >> foo x; >> >> MPI_Init(&argc, &argv); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> char processor_name[MPI_MAX_PROCESSOR_NAME]; >> int name_len; >> MPI_Get_processor_name(processor_name, &name_len); >> printf("-- processor %s, rank %d out of %d processors\n", >> processor_name, rank, size); >> >> *MPI_Barrier(MPI_COMM_WORLD);* >> >> >> int count=4; >> >> MPI_Datatype testtype; >> MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; >> int len[4] = {1, 1, 1, 1}; >> MPI_Aint disp[4]; >> long int base; >> >> MPI_Address(&x, disp); >> MPI_Address(&(x.a), disp+1); >> MPI_Address(&(x.b), disp+2); >> MPI_Address(&(x.c), disp+3); >> base = disp[0]; >> for(i=0; i<4; i++) disp[i] -= base; >> >> MPI_Type_struct(count, len, disp, types, &testtype); >> MPI_Type_commit(&testtype); >> >> if(rank == 0){ >> x.a = 2; >> x.b = 0; >> x.c = 10; >> x.d = 3; >> } >> >> printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, >> x.b, x.c, x.d); >> *MPI_Barrier(MPI_COMM_WORLD);* >> >> MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); >> >> printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, >> x.b, x.c, x.d); >> >> MPI_Finalize(); >> >> return 0; >> } >> >> the output should be looks like: >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3 >> rank 1(before): x value is 1197535864, -1, 4994901, 0 >> rank 2(before): x value is 1591464488, -1, 4994901, 0 >> rank 3(before): x value is 1851622184, -1, 4994901, 0 >> rank 0(after): x value is 2, 0, 10, 3 >> rank 3(after): x value is 2, 0, 10, 3 >> rank 1(after): x value is 2, 0, 10, 3 >> rank 2(after): x value is 2, 0, 10, 3 >> >> but some time is shows as: >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3 >> rank 0(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> >> out of 4 processors >> rank 1(before): x value is -464731256, -1, 4994901, 0 >> rank 1(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 2(before): x value is 1863042488, -1, 4994901, 0 >> rank 2(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> >> out of 4 processors >> rank 3(before): x value is 1721065144, -1, 4994901, 0 >> rank 3(after): x value is 2, 0, 10, 3 >> >> or >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> >> out of 4 processors >> rank 1(before): x value is -1883169624, -1, 4994901, 0 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 2(before): x value is -451256152, -1, 4994901, 0 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> >> out of 4 processors >> rank 3(before): x value is 1715067240, -1, 4994901, 0 >> rank 0(before): x value is 2, 0, 10, 3 >> rank 0(after): x value is 2, 0, 10, 3 >> rank 1(after): x value is 2, 0, 10, 3 >> rank 2(after): x value is 2, 0, 10, 3 >> rank 3(after): x value is 2, 0, 10, 3 >> >> it is all randomly, I am not sure where is the problem. >> >> The second issue is I use MPI_Datatype to create a MPI struct for >> broadcast. However, as the program shown above, if I change the struct: >> typedef struct >> { >> int a; >> char b; >> int c; >> *int d;* >> >> } foo; >> as >> typedef struct >> { >> int a; >> char b; >> int c; >> *double d*; >> >> } foo; >> >> I found the result is: >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3.250000 >> rank 3(before): x value is 0, 0, 547474368, 0.000000 >> rank 1(before): x value is 0, 0, 547474368, 0.000000 >> rank 2(before): x value is 0, 0, 547474368, 0.000000 >> rank 0(after): x value is 2, 0, 10, 3.250000 >> *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 >> >> rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 >> rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 >> >> * >> Do you have any clues on why this happened? Thanks a lot! >> >> -- >> Best Regards, >> Sufeng Niu >> ECASP lab, ECE department, Illinois Institute of Technology >> Tel: 312-731-7219 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Pavan Balaji 09 Jul '13

09 Jul '13

printf buffers output and doesn't write it out immediately (or in this case, send it to mpiexec immediately). Try doing "fflush(stdout);" immediately after printf. -- Pavan On 07/09/2013 12:01 PM, Sufeng Niu wrote: > Hello, > > Sorry to post this long, stupid and simple question. > I found that some time MPI_Barrier cannot stop all the process. I try to > write a simple test program to create data_struct shown below: > > #include <stdlib.h> > #include <stdio.h> > #include "mpi.h" > > typedef struct > { > int a; > char b; > int c; > int d; > } foo; > > int main(int argc, char *argv[]) > { > > int rank, size; > int i; > > foo x; > > MPI_Init(&argc, &argv); > MPI_Comm_size(MPI_COMM_WORLD, &size); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > char processor_name[MPI_MAX_PROCESSOR_NAME]; > int name_len; > MPI_Get_processor_name(processor_name, &name_len); > printf("-- processor %s, rank %d out of %d processors\n", > processor_name, rank, size); > > *MPI_Barrier(MPI_COMM_WORLD);* > > int count=4; > > MPI_Datatype testtype; > MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; > int len[4] = {1, 1, 1, 1}; > MPI_Aint disp[4]; > long int base; > > MPI_Address(&x, disp); > MPI_Address(&(x.a), disp+1); > MPI_Address(&(x.b), disp+2); > MPI_Address(&(x.c), disp+3); > base = disp[0]; > for(i=0; i<4; i++) disp[i] -= base; > > MPI_Type_struct(count, len, disp, types, &testtype); > MPI_Type_commit(&testtype); > > if(rank == 0){ > x.a = 2; > x.b = 0; > x.c = 10; > x.d = 3; > } > > printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, > x.b, x.c, x.d); > *MPI_Barrier(MPI_COMM_WORLD);* > MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); > > printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, > x.b, x.c, x.d); > > MPI_Finalize(); > > return 0; > } > > the output should be looks like: > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3 > rank 1(before): x value is 1197535864, -1, 4994901, 0 > rank 2(before): x value is 1591464488, -1, 4994901, 0 > rank 3(before): x value is 1851622184, -1, 4994901, 0 > rank 0(after): x value is 2, 0, 10, 3 > rank 3(after): x value is 2, 0, 10, 3 > rank 1(after): x value is 2, 0, 10, 3 > rank 2(after): x value is 2, 0, 10, 3 > > but some time is shows as: > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3 > rank 0(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > rank 1(before): x value is -464731256, -1, 4994901, 0 > rank 1(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 2(before): x value is 1863042488, -1, 4994901, 0 > rank 2(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > rank 3(before): x value is 1721065144, -1, 4994901, 0 > rank 3(after): x value is 2, 0, 10, 3 > > or > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > rank 1(before): x value is -1883169624, -1, 4994901, 0 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 2(before): x value is -451256152, -1, 4994901, 0 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > rank 3(before): x value is 1715067240, -1, 4994901, 0 > rank 0(before): x value is 2, 0, 10, 3 > rank 0(after): x value is 2, 0, 10, 3 > rank 1(after): x value is 2, 0, 10, 3 > rank 2(after): x value is 2, 0, 10, 3 > rank 3(after): x value is 2, 0, 10, 3 > > it is all randomly, I am not sure where is the problem. > > The second issue is I use MPI_Datatype to create a MPI struct for > broadcast. However, as the program shown above, if I change the struct: > typedef struct > { > int a; > char b; > int c; > *int d;* > } foo; > as > typedef struct > { > int a; > char b; > int c; > *double d*; > } foo; > > I found the result is: > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3.250000 > rank 3(before): x value is 0, 0, 547474368, 0.000000 > rank 1(before): x value is 0, 0, 547474368, 0.000000 > rank 2(before): x value is 0, 0, 547474368, 0.000000 > rank 0(after): x value is 2, 0, 10, 3.250000 > *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 > * > Do you have any clues on why this happened? Thanks a lot! > > -- > Best Regards, > Sufeng Niu > ECASP lab, ECE department, Illinois Institute of Technology > Tel: 312-731-7219 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

MPI_Barrier cannot stop all process randomly
by Sufeng Niu 09 Jul '13

09 Jul '13

Hello, Sorry to post this long, stupid and simple question. I found that some time MPI_Barrier cannot stop all the process. I try to write a simple test program to create data_struct shown below: #include <stdlib.h> #include <stdio.h> #include "mpi.h" typedef struct { int a; char b; int c; int d; } foo; int main(int argc, char *argv[]) { int rank, size; int i; foo x; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); char processor_name[MPI_MAX_PROCESSOR_NAME]; int name_len; MPI_Get_processor_name(processor_name, &name_len); printf("-- processor %s, rank %d out of %d processors\n", processor_name, rank, size); * MPI_Barrier(MPI_COMM_WORLD);* int count=4; MPI_Datatype testtype; MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; int len[4] = {1, 1, 1, 1}; MPI_Aint disp[4]; long int base; MPI_Address(&x, disp); MPI_Address(&(x.a), disp+1); MPI_Address(&(x.b), disp+2); MPI_Address(&(x.c), disp+3); base = disp[0]; for(i=0; i<4; i++) disp[i] -= base; MPI_Type_struct(count, len, disp, types, &testtype); MPI_Type_commit(&testtype); if(rank == 0){ x.a = 2; x.b = 0; x.c = 10; x.d = 3; } printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, x.b, x.c, x.d); *MPI_Barrier(MPI_COMM_WORLD);* MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, x.b, x.c, x.d); MPI_Finalize(); return 0; } the output should be looks like: -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 0(before): x value is 2, 0, 10, 3 rank 1(before): x value is 1197535864, -1, 4994901, 0 rank 2(before): x value is 1591464488, -1, 4994901, 0 rank 3(before): x value is 1851622184, -1, 4994901, 0 rank 0(after): x value is 2, 0, 10, 3 rank 3(after): x value is 2, 0, 10, 3 rank 1(after): x value is 2, 0, 10, 3 rank 2(after): x value is 2, 0, 10, 3 but some time is shows as: -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors rank 0(before): x value is 2, 0, 10, 3 rank 0(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors rank 1(before): x value is -464731256, -1, 4994901, 0 rank 1(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 2(before): x value is 1863042488, -1, 4994901, 0 rank 2(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors rank 3(before): x value is 1721065144, -1, 4994901, 0 rank 3(after): x value is 2, 0, 10, 3 or -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors rank 1(before): x value is -1883169624, -1, 4994901, 0 -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 2(before): x value is -451256152, -1, 4994901, 0 -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors rank 3(before): x value is 1715067240, -1, 4994901, 0 rank 0(before): x value is 2, 0, 10, 3 rank 0(after): x value is 2, 0, 10, 3 rank 1(after): x value is 2, 0, 10, 3 rank 2(after): x value is 2, 0, 10, 3 rank 3(after): x value is 2, 0, 10, 3 it is all randomly, I am not sure where is the problem. The second issue is I use MPI_Datatype to create a MPI struct for broadcast. However, as the program shown above, if I change the struct: typedef struct { int a; char b; int c; *int d;* } foo; as typedef struct { int a; char b; int c; *double d*; } foo; I found the result is: -- processor ephesus.ece.iit.edu, rank 1 out of 4 processors -- processor ephesus.ece.iit.edu, rank 2 out of 4 processors -- processor ephesus.ece.iit.edu, rank 3 out of 4 processors -- processor ephesus.ece.iit.edu, rank 0 out of 4 processors rank 0(before): x value is 2, 0, 10, 3.250000 rank 3(before): x value is 0, 0, 547474368, 0.000000 rank 1(before): x value is 0, 0, 547474368, 0.000000 rank 2(before): x value is 0, 0, 547474368, 0.000000 rank 0(after): x value is 2, 0, 10, 3.250000 *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 rank 1(after): x value is 2, 0, 10, 0.000000** <- should be 3.25 * Do you have any clues on why this happened? Thanks a lot! -- Best Regards, Sufeng Niu ECASP lab, ECE department, Illinois Institute of Technology Tel: 312-731-7219

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Jeff Hammond 09 Jul '13

09 Jul '13

> Unfortunately the MPI application I'm trying to run would require too much effort to convert to hybrid OpenMP/MPI to be viable option at this point. Does your code actually scale to 240 MPI ranks per card? What is the performance relative to peak or a fully-populated MPI-only run on e.g SNB? You're going to have to convert to MPI+Threads eventually so you might as well start now. It is completely unreasonable to expect that MPI-only is reasonable usage on a 60-core chip with a ring interconnect. I am so convinced of the impossibility that MPI_Allreduce scales to >100 MPI procs in shared memory that I will learn Ada as punishment if I'm wrong. If you don't like OpenMP, you could consider Pthreads or TBB instead, but something tells me that if you're unwilling to use OpenMP, it's because you have an F77 code with common blocks that cannot be made thread-safe for all the beer in Australia. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0