July 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI Benchmark Suite
by Jiri Simsa 09 Jul '13

09 Jul '13

I have compiled the pointers from Pavan, Rajeev, and Jeff into an overview of MPI workloads (attached). Any comments or corrections are welcome. --Jiri On Sat, Jul 6, 2013 at 1:38 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > See also the Mantevo mini-apps. http://www.mantevo.org/ > > > On Jul 5, 2013, at 11:22 AM, Pavan Balaji wrote: > > > > > On 07/05/2013 10:38 AM, Jiri Simsa wrote: > >> Could anyone point me in the direction of programs that are > >> representative of MPI programs? I am looking for something like PARSEC > >> (http://parsec.cs.princeton.edu/) for MPI. In other words, I am > >> interested in macro-bechnmarks not micro-benchmarks. Thank you. > > > > You could try the Sequoia suite. There's also the NAS parallel > benchmarks and Graph500 benchmarks. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Jed Brown 09 Jul '13

09 Jul '13

Jeff Hammond <jeff.science(a)gmail.com> writes: > You're going to have to convert to MPI+Threads eventually so you might > as well start now. It is completely unreasonable to expect that > MPI-only is reasonable usage on a 60-core chip with a ring > interconnect. This is something that endpoints is supposed to address. Note that OpenMP doesn't have any decent way to do reductions scoped across subsets of threads, so for many applications, it's not even a reasonable level of expressivity. > I am so convinced of the impossibility that MPI_Allreduce scales to >>100 MPI procs in shared memory that I will learn Ada as punishment if > I'm wrong. Does 30 microsecond MPI_Allreduce at 512 cores qualify as "impossible to scale"? http://www.nas.nasa.gov/assets/pdf/techreports/2010/nas-10-001.pdf

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Sufeng Niu 09 Jul '13

09 Jul '13

I see. Thanks a lot! and that make sense. I tried fflush(stdout), it seems that still not work properly. but I understand that printf is not a proper way to check the timing. Thank you! Sufeng On Tue, Jul 9, 2013 at 12:24 PM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > ...and depending on your filesystem (assuming you pipe stdout to a > file), even that isn't sufficient. > > printf is a terrible way to evaluate temporal relationships and you > shouldn't try to reason about parallel programs with it unless you've > taken great care to ensure a global total ordering of output streams. > > Jeff > > On Tue, Jul 9, 2013 at 12:05 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > printf buffers output and doesn't write it out immediately (or in this > case, > > send it to mpiexec immediately). Try doing "fflush(stdout);" immediately > > after printf. > > > > -- Pavan > > > > > > On 07/09/2013 12:01 PM, Sufeng Niu wrote: > >> > >> Hello, > >> > >> Sorry to post this long, stupid and simple question. > >> I found that some time MPI_Barrier cannot stop all the process. I try to > >> write a simple test program to create data_struct shown below: > >> > >> #include <stdlib.h> > >> #include <stdio.h> > >> #include "mpi.h" > >> > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> int d; > >> } foo; > >> > >> int main(int argc, char *argv[]) > >> { > >> > >> int rank, size; > >> int i; > >> > >> foo x; > >> > >> MPI_Init(&argc, &argv); > >> MPI_Comm_size(MPI_COMM_WORLD, &size); > >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); > >> > >> char processor_name[MPI_MAX_PROCESSOR_NAME]; > >> int name_len; > >> MPI_Get_processor_name(processor_name, &name_len); > >> printf("-- processor %s, rank %d out of %d processors\n", > >> processor_name, rank, size); > >> > >> *MPI_Barrier(MPI_COMM_WORLD);* > >> > >> > >> int count=4; > >> > >> MPI_Datatype testtype; > >> MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; > >> int len[4] = {1, 1, 1, 1}; > >> MPI_Aint disp[4]; > >> long int base; > >> > >> MPI_Address(&x, disp); > >> MPI_Address(&(x.a), disp+1); > >> MPI_Address(&(x.b), disp+2); > >> MPI_Address(&(x.c), disp+3); > >> base = disp[0]; > >> for(i=0; i<4; i++) disp[i] -= base; > >> > >> MPI_Type_struct(count, len, disp, types, &testtype); > >> MPI_Type_commit(&testtype); > >> > >> if(rank == 0){ > >> x.a = 2; > >> x.b = 0; > >> x.c = 10; > >> x.d = 3; > >> } > >> > >> printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, > >> x.b, x.c, x.d); > >> *MPI_Barrier(MPI_COMM_WORLD);* > >> > >> MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); > >> > >> printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, > >> x.b, x.c, x.d); > >> > >> MPI_Finalize(); > >> > >> return 0; > >> } > >> > >> the output should be looks like: > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 1(before): x value is 1197535864, -1, 4994901, 0 > >> rank 2(before): x value is 1591464488, -1, 4994901, 0 > >> rank 3(before): x value is 1851622184, -1, 4994901, 0 > >> rank 0(after): x value is 2, 0, 10, 3 > >> rank 3(after): x value is 2, 0, 10, 3 > >> rank 1(after): x value is 2, 0, 10, 3 > >> rank 2(after): x value is 2, 0, 10, 3 > >> > >> but some time is shows as: > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 0(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> > >> out of 4 processors > >> rank 1(before): x value is -464731256, -1, 4994901, 0 > >> rank 1(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 2(before): x value is 1863042488, -1, 4994901, 0 > >> rank 2(after): x value is 2, 0, 10, 3 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> > >> out of 4 processors > >> rank 3(before): x value is 1721065144, -1, 4994901, 0 > >> rank 3(after): x value is 2, 0, 10, 3 > >> > >> or > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > >> out of 4 processors > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > >> > >> out of 4 processors > >> rank 1(before): x value is -1883169624, -1, 4994901, 0 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > >> > >> out of 4 processors > >> rank 2(before): x value is -451256152, -1, 4994901, 0 > >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > >> > >> out of 4 processors > >> rank 3(before): x value is 1715067240, -1, 4994901, 0 > >> rank 0(before): x value is 2, 0, 10, 3 > >> rank 0(after): x value is 2, 0, 10, 3 > >> rank 1(after): x value is 2, 0, 10, 3 > >> rank 2(after): x value is 2, 0, 10, 3 > >> rank 3(after): x value is 2, 0, 10, 3 > >> > >> it is all randomly, I am not sure where is the problem. > >> > >> The second issue is I use MPI_Datatype to create a MPI struct for > >> broadcast. However, as the program shown above, if I change the struct: > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> *int d;* > >> > >> } foo; > >> as > >> typedef struct > >> { > >> int a; > >> char b; > >> int c; > >> *double d*; > >> > >> } foo; > >> > >> I found the result is: > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 > >> out of 4 processors > >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 > >> > >> out of 4 processors > >> rank 0(before): x value is 2, 0, 10, 3.250000 > >> rank 3(before): x value is 0, 0, 547474368, 0.000000 > >> rank 1(before): x value is 0, 0, 547474368, 0.000000 > >> rank 2(before): x value is 0, 0, 547474368, 0.000000 > >> rank 0(after): x value is 2, 0, 10, 3.250000 > >> *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > >> > >> rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > >> rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 > >> > >> * > >> Do you have any clues on why this happened? Thanks a lot! > >> > >> -- > >> Best Regards, > >> Sufeng Niu > >> ECASP lab, ECE department, Illinois Institute of Technology > >> Tel: 312-731-7219 > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > -- Best Regards, Sufeng Niu ECASP lab, ECE department, Illinois Institute of Technology Tel: 312-731-7219

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Jeff Hammond 09 Jul '13

09 Jul '13

...and depending on your filesystem (assuming you pipe stdout to a file), even that isn't sufficient. printf is a terrible way to evaluate temporal relationships and you shouldn't try to reason about parallel programs with it unless you've taken great care to ensure a global total ordering of output streams. Jeff On Tue, Jul 9, 2013 at 12:05 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > printf buffers output and doesn't write it out immediately (or in this case, > send it to mpiexec immediately). Try doing "fflush(stdout);" immediately > after printf. > > -- Pavan > > > On 07/09/2013 12:01 PM, Sufeng Niu wrote: >> >> Hello, >> >> Sorry to post this long, stupid and simple question. >> I found that some time MPI_Barrier cannot stop all the process. I try to >> write a simple test program to create data_struct shown below: >> >> #include <stdlib.h> >> #include <stdio.h> >> #include "mpi.h" >> >> typedef struct >> { >> int a; >> char b; >> int c; >> int d; >> } foo; >> >> int main(int argc, char *argv[]) >> { >> >> int rank, size; >> int i; >> >> foo x; >> >> MPI_Init(&argc, &argv); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> char processor_name[MPI_MAX_PROCESSOR_NAME]; >> int name_len; >> MPI_Get_processor_name(processor_name, &name_len); >> printf("-- processor %s, rank %d out of %d processors\n", >> processor_name, rank, size); >> >> *MPI_Barrier(MPI_COMM_WORLD);* >> >> >> int count=4; >> >> MPI_Datatype testtype; >> MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; >> int len[4] = {1, 1, 1, 1}; >> MPI_Aint disp[4]; >> long int base; >> >> MPI_Address(&x, disp); >> MPI_Address(&(x.a), disp+1); >> MPI_Address(&(x.b), disp+2); >> MPI_Address(&(x.c), disp+3); >> base = disp[0]; >> for(i=0; i<4; i++) disp[i] -= base; >> >> MPI_Type_struct(count, len, disp, types, &testtype); >> MPI_Type_commit(&testtype); >> >> if(rank == 0){ >> x.a = 2; >> x.b = 0; >> x.c = 10; >> x.d = 3; >> } >> >> printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, >> x.b, x.c, x.d); >> *MPI_Barrier(MPI_COMM_WORLD);* >> >> MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); >> >> printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, >> x.b, x.c, x.d); >> >> MPI_Finalize(); >> >> return 0; >> } >> >> the output should be looks like: >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3 >> rank 1(before): x value is 1197535864, -1, 4994901, 0 >> rank 2(before): x value is 1591464488, -1, 4994901, 0 >> rank 3(before): x value is 1851622184, -1, 4994901, 0 >> rank 0(after): x value is 2, 0, 10, 3 >> rank 3(after): x value is 2, 0, 10, 3 >> rank 1(after): x value is 2, 0, 10, 3 >> rank 2(after): x value is 2, 0, 10, 3 >> >> but some time is shows as: >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3 >> rank 0(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> >> out of 4 processors >> rank 1(before): x value is -464731256, -1, 4994901, 0 >> rank 1(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 2(before): x value is 1863042488, -1, 4994901, 0 >> rank 2(after): x value is 2, 0, 10, 3 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> >> out of 4 processors >> rank 3(before): x value is 1721065144, -1, 4994901, 0 >> rank 3(after): x value is 2, 0, 10, 3 >> >> or >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 >> out of 4 processors >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 >> >> out of 4 processors >> rank 1(before): x value is -1883169624, -1, 4994901, 0 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 >> >> out of 4 processors >> rank 2(before): x value is -451256152, -1, 4994901, 0 >> -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 >> >> out of 4 processors >> rank 3(before): x value is 1715067240, -1, 4994901, 0 >> rank 0(before): x value is 2, 0, 10, 3 >> rank 0(after): x value is 2, 0, 10, 3 >> rank 1(after): x value is 2, 0, 10, 3 >> rank 2(after): x value is 2, 0, 10, 3 >> rank 3(after): x value is 2, 0, 10, 3 >> >> it is all randomly, I am not sure where is the problem. >> >> The second issue is I use MPI_Datatype to create a MPI struct for >> broadcast. However, as the program shown above, if I change the struct: >> typedef struct >> { >> int a; >> char b; >> int c; >> *int d;* >> >> } foo; >> as >> typedef struct >> { >> int a; >> char b; >> int c; >> *double d*; >> >> } foo; >> >> I found the result is: >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 >> out of 4 processors >> -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 >> >> out of 4 processors >> rank 0(before): x value is 2, 0, 10, 3.250000 >> rank 3(before): x value is 0, 0, 547474368, 0.000000 >> rank 1(before): x value is 0, 0, 547474368, 0.000000 >> rank 2(before): x value is 0, 0, 547474368, 0.000000 >> rank 0(after): x value is 2, 0, 10, 3.250000 >> *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 >> >> rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 >> rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 >> >> * >> Do you have any clues on why this happened? Thanks a lot! >> >> -- >> Best Regards, >> Sufeng Niu >> ECASP lab, ECE department, Illinois Institute of Technology >> Tel: 312-731-7219 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] MPI_Barrier cannot stop all process randomly
by Pavan Balaji 09 Jul '13

09 Jul '13

printf buffers output and doesn't write it out immediately (or in this case, send it to mpiexec immediately). Try doing "fflush(stdout);" immediately after printf. -- Pavan On 07/09/2013 12:01 PM, Sufeng Niu wrote: > Hello, > > Sorry to post this long, stupid and simple question. > I found that some time MPI_Barrier cannot stop all the process. I try to > write a simple test program to create data_struct shown below: > > #include <stdlib.h> > #include <stdio.h> > #include "mpi.h" > > typedef struct > { > int a; > char b; > int c; > int d; > } foo; > > int main(int argc, char *argv[]) > { > > int rank, size; > int i; > > foo x; > > MPI_Init(&argc, &argv); > MPI_Comm_size(MPI_COMM_WORLD, &size); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > char processor_name[MPI_MAX_PROCESSOR_NAME]; > int name_len; > MPI_Get_processor_name(processor_name, &name_len); > printf("-- processor %s, rank %d out of %d processors\n", > processor_name, rank, size); > > *MPI_Barrier(MPI_COMM_WORLD);* > > int count=4; > > MPI_Datatype testtype; > MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; > int len[4] = {1, 1, 1, 1}; > MPI_Aint disp[4]; > long int base; > > MPI_Address(&x, disp); > MPI_Address(&(x.a), disp+1); > MPI_Address(&(x.b), disp+2); > MPI_Address(&(x.c), disp+3); > base = disp[0]; > for(i=0; i<4; i++) disp[i] -= base; > > MPI_Type_struct(count, len, disp, types, &testtype); > MPI_Type_commit(&testtype); > > if(rank == 0){ > x.a = 2; > x.b = 0; > x.c = 10; > x.d = 3; > } > > printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, > x.b, x.c, x.d); > *MPI_Barrier(MPI_COMM_WORLD);* > MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); > > printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, > x.b, x.c, x.d); > > MPI_Finalize(); > > return 0; > } > > the output should be looks like: > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3 > rank 1(before): x value is 1197535864, -1, 4994901, 0 > rank 2(before): x value is 1591464488, -1, 4994901, 0 > rank 3(before): x value is 1851622184, -1, 4994901, 0 > rank 0(after): x value is 2, 0, 10, 3 > rank 3(after): x value is 2, 0, 10, 3 > rank 1(after): x value is 2, 0, 10, 3 > rank 2(after): x value is 2, 0, 10, 3 > > but some time is shows as: > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3 > rank 0(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > rank 1(before): x value is -464731256, -1, 4994901, 0 > rank 1(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 2(before): x value is 1863042488, -1, 4994901, 0 > rank 2(after): x value is 2, 0, 10, 3 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > rank 3(before): x value is 1721065144, -1, 4994901, 0 > rank 3(after): x value is 2, 0, 10, 3 > > or > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 0 > out of 4 processors > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 1 > out of 4 processors > rank 1(before): x value is -1883169624, -1, 4994901, 0 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 2 > out of 4 processors > rank 2(before): x value is -451256152, -1, 4994901, 0 > -- processor iocfccd3.aps.anl.gov <http://iocfccd3.aps.anl.gov>, rank 3 > out of 4 processors > rank 3(before): x value is 1715067240, -1, 4994901, 0 > rank 0(before): x value is 2, 0, 10, 3 > rank 0(after): x value is 2, 0, 10, 3 > rank 1(after): x value is 2, 0, 10, 3 > rank 2(after): x value is 2, 0, 10, 3 > rank 3(after): x value is 2, 0, 10, 3 > > it is all randomly, I am not sure where is the problem. > > The second issue is I use MPI_Datatype to create a MPI struct for > broadcast. However, as the program shown above, if I change the struct: > typedef struct > { > int a; > char b; > int c; > *int d;* > } foo; > as > typedef struct > { > int a; > char b; > int c; > *double d*; > } foo; > > I found the result is: > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 1 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 2 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 3 > out of 4 processors > -- processor ephesus.ece.iit.edu <http://ephesus.ece.iit.edu>, rank 0 > out of 4 processors > rank 0(before): x value is 2, 0, 10, 3.250000 > rank 3(before): x value is 0, 0, 547474368, 0.000000 > rank 1(before): x value is 0, 0, 547474368, 0.000000 > rank 2(before): x value is 0, 0, 547474368, 0.000000 > rank 0(after): x value is 2, 0, 10, 3.250000 > *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 > rank 1(after): x value is 2, 0, 10, 0.000000**<- should be 3.25 > * > Do you have any clues on why this happened? Thanks a lot! > > -- > Best Regards, > Sufeng Niu > ECASP lab, ECE department, Illinois Institute of Technology > Tel: 312-731-7219 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

MPI_Barrier cannot stop all process randomly
by Sufeng Niu 09 Jul '13

09 Jul '13

Hello, Sorry to post this long, stupid and simple question. I found that some time MPI_Barrier cannot stop all the process. I try to write a simple test program to create data_struct shown below: #include <stdlib.h> #include <stdio.h> #include "mpi.h" typedef struct { int a; char b; int c; int d; } foo; int main(int argc, char *argv[]) { int rank, size; int i; foo x; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); char processor_name[MPI_MAX_PROCESSOR_NAME]; int name_len; MPI_Get_processor_name(processor_name, &name_len); printf("-- processor %s, rank %d out of %d processors\n", processor_name, rank, size); * MPI_Barrier(MPI_COMM_WORLD);* int count=4; MPI_Datatype testtype; MPI_Datatype types[4] = {MPI_INT, MPI_CHAR, MPI_INT, MPI_DOUBLE}; int len[4] = {1, 1, 1, 1}; MPI_Aint disp[4]; long int base; MPI_Address(&x, disp); MPI_Address(&(x.a), disp+1); MPI_Address(&(x.b), disp+2); MPI_Address(&(x.c), disp+3); base = disp[0]; for(i=0; i<4; i++) disp[i] -= base; MPI_Type_struct(count, len, disp, types, &testtype); MPI_Type_commit(&testtype); if(rank == 0){ x.a = 2; x.b = 0; x.c = 10; x.d = 3; } printf("rank %d(before): x value is %d, %d, %d, %d\n", rank, x.a, x.b, x.c, x.d); *MPI_Barrier(MPI_COMM_WORLD);* MPI_Bcast(&x, 1, testtype, 0, MPI_COMM_WORLD); printf("rank %d(after): x value is %d, %d, %d, %d\n", rank, x.a, x.b, x.c, x.d); MPI_Finalize(); return 0; } the output should be looks like: -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 0(before): x value is 2, 0, 10, 3 rank 1(before): x value is 1197535864, -1, 4994901, 0 rank 2(before): x value is 1591464488, -1, 4994901, 0 rank 3(before): x value is 1851622184, -1, 4994901, 0 rank 0(after): x value is 2, 0, 10, 3 rank 3(after): x value is 2, 0, 10, 3 rank 1(after): x value is 2, 0, 10, 3 rank 2(after): x value is 2, 0, 10, 3 but some time is shows as: -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors rank 0(before): x value is 2, 0, 10, 3 rank 0(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors rank 1(before): x value is -464731256, -1, 4994901, 0 rank 1(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 2(before): x value is 1863042488, -1, 4994901, 0 rank 2(after): x value is 2, 0, 10, 3 -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors rank 3(before): x value is 1721065144, -1, 4994901, 0 rank 3(after): x value is 2, 0, 10, 3 or -- processor iocfccd3.aps.anl.gov, rank 0 out of 4 processors -- processor iocfccd3.aps.anl.gov, rank 1 out of 4 processors rank 1(before): x value is -1883169624, -1, 4994901, 0 -- processor iocfccd3.aps.anl.gov, rank 2 out of 4 processors rank 2(before): x value is -451256152, -1, 4994901, 0 -- processor iocfccd3.aps.anl.gov, rank 3 out of 4 processors rank 3(before): x value is 1715067240, -1, 4994901, 0 rank 0(before): x value is 2, 0, 10, 3 rank 0(after): x value is 2, 0, 10, 3 rank 1(after): x value is 2, 0, 10, 3 rank 2(after): x value is 2, 0, 10, 3 rank 3(after): x value is 2, 0, 10, 3 it is all randomly, I am not sure where is the problem. The second issue is I use MPI_Datatype to create a MPI struct for broadcast. However, as the program shown above, if I change the struct: typedef struct { int a; char b; int c; *int d;* } foo; as typedef struct { int a; char b; int c; *double d*; } foo; I found the result is: -- processor ephesus.ece.iit.edu, rank 1 out of 4 processors -- processor ephesus.ece.iit.edu, rank 2 out of 4 processors -- processor ephesus.ece.iit.edu, rank 3 out of 4 processors -- processor ephesus.ece.iit.edu, rank 0 out of 4 processors rank 0(before): x value is 2, 0, 10, 3.250000 rank 3(before): x value is 0, 0, 547474368, 0.000000 rank 1(before): x value is 0, 0, 547474368, 0.000000 rank 2(before): x value is 0, 0, 547474368, 0.000000 rank 0(after): x value is 2, 0, 10, 3.250000 *rank 2(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 rank 3(after): x value is 2, 0, 10, 0.000000 <- should be 3.25 rank 1(after): x value is 2, 0, 10, 0.000000** <- should be 3.25 * Do you have any clues on why this happened? Thanks a lot! -- Best Regards, Sufeng Niu ECASP lab, ECE department, Illinois Institute of Technology Tel: 312-731-7219

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Jeff Hammond 09 Jul '13

09 Jul '13

> Unfortunately the MPI application I'm trying to run would require too much effort to convert to hybrid OpenMP/MPI to be viable option at this point. Does your code actually scale to 240 MPI ranks per card? What is the performance relative to peak or a fully-populated MPI-only run on e.g SNB? You're going to have to convert to MPI+Threads eventually so you might as well start now. It is completely unreasonable to expect that MPI-only is reasonable usage on a 60-core chip with a ring interconnect. I am so convinced of the impossibility that MPI_Allreduce scales to >100 MPI procs in shared memory that I will learn Ada as punishment if I'm wrong. If you don't like OpenMP, you could consider Pthreads or TBB instead, but something tells me that if you're unwilling to use OpenMP, it's because you have an F77 code with common blocks that cannot be made thread-safe for all the beer in Australia. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Support for MIC in MPICH 3.0.4
by Pavan Balaji 09 Jul '13

09 Jul '13

Hi Maciej, I checked with Intel and they mentioned that they are using a different netmod called IBSCIF on their big machines, not SCIF directly. -- Pavan On 07/08/2013 06:21 PM, Maciej.Golebiewski(a)csiro.au wrote: > Hi Pavan, > > Thanks for your reply, although it is not what I hoped to hear. > > Unfortunately the MPI application I'm trying to run would require too much effort to convert to hybrid OpenMP/MPI to be viable option at this point. > > I wonder how IntelMPI is working around this limit (I've run it with up to 2 (host) + 2 x 240 (MIC) ranks). > > Anyway, thanks again and I guess I will have to wait for the updates to MPSS/SCIF. > > Cheers, > > Maciej > > >> -----Original Message----- >> From: Pavan Balaji [mailto:[email protected]] >> Sent: Tuesday, July 09, 2013 12:32 AM >> To: discuss(a)mpich.org >> Cc: Golebiewski, Maciej (CSIRO IM&T, Docklands) >> Subject: Re: [mpich-discuss] Support for MIC in MPICH 3.0.4 >> >> Hi Maciej, >> >> Intel is aware of this problem that occurs when a large number of >> MIC processes are used. They are looking into it. Right now, the >> only workaround we can offer is to use fewer MIC processes and use >> threads on the MIC instead. >> >> -- Pavan >> >> On 07/08/2013 01:34 AM, Maciej.Golebiewski(a)csiro.au wrote: >>> Hi, >>> >>> I have managed to build MPICH for MIC device and for host system >> with support for SCIF and using Intel compilers. I am able to start >> MPI processes on MIC device using mpirun from host or mpiexec on >> the device itself. >>> >>> I can also run application with some ranks on the host and some >> on the device. >>> >>> I can't however start more than 4 ranks on any device, if I try >> to run my application across more than 1 node (be it host or MIC >> cards): >>> >>> env I_MPI_MIC=enable mpirun -n 1 -host mike.it.csiro.au ./mpptest >> -bcast : -n 30 -host mic0 /tmp/mpptest.scif -bcast >>> 0: 1: 00000011: 00000170: readv err 0 Fatal error in >> MPI_Allreduce: >>> Other MPI error, error stack: >>> MPI_Allreduce(861)...............: >> MPI_Allreduce(sbuf=0x7fff62945a08, >>> rbuf=0x7c5de0, count=1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD) >> failed >>> MPIR_Allreduce_impl(719).........: >>> MPIR_Allreduce_intra(201)........: >>> allreduce_intra_or_coll_fn(110)..: >>> MPIR_Allreduce_intra(362)........: >>> MPIC_Sendrecv(213)...............: >>> MPIC_Wait(569)...................: >>> MPIDI_CH3I_Progress(367).........: >>> MPID_nem_mpich_blocking_recv(894): >>> state_commrdy_handler(175).......: >>> state_commrdy_handler(138).......: >>> MPID_nem_scif_recv_handler(115)..: Communication error with rank >> 1 >>> [...bunch of more error messages follows...] >>> >>> This problem does not occur of all ranks run on the same node: >>> >>> env I_MPI_MIC=enable mpirun -n 30 -host mic0 /tmp/mpptest.scif - >> bcast >>> set default set font variable set curve window y 0.15 0.90 set >> order d >>> d d x y d title left 'time (us)', bottom 'Size (bytes)', >>> top 'Comm Perf for MPI (pkmic-mic0)', >>> 'type = blocking' >>> >>> #p0 p1 dist len ave time (us) rate >>> 0 29 29 0 11.448860 0.00 >>> 0 29 29 32 12.009144 2.665e+6 >>> [...more expected output follows...] >>> >>> Last November there was a short thread on this list (archive: >> http://lists.mpich.org/pipermail/discuss/2012-November/000109.html) >> about seemingly similiar problem, but seemed to fizzle out before >> any solution was offered. >>> >>> I wonder whether there was any progress on it, or perhaps someone >> else has encountered this problem, too and managed to find a >> workaround/solution or at least has an explanation as to what is >> happening. >>> >>> In case someone would like to try and reproduce it, my configure >> invocation for MPICH on MIC was: >>> >>> ./configure CC="icc" CXX=icpc F77="ifort -warn noalignments" >> FC="ifort >>> -warn noalignments" \ CXXFLAGS=-mmic CFLAGS=-mmic FFLAGS=-mmic >>> FCFLAGS=-mmic \ MPICHLIB_CFLAGS="-g" MPICHLIB_CXXFLAGS="-g" >>> MPICHLIB_FFLAGS="-g" MPICHLIB_FCFLAGS="-g" \ MPICHLIB_LDFLAGS="- >> g" >>> LIBS=-lscif LDFLAGS=-L/opt/intel/mic/filesystem/base/lib64 \ >>> --with-device=ch3:nemesis:scif,tcp --disable-romio --disable-mpe >>> --enable-smpcoll \ --enable-fast=defopt >>> --prefix=/apps/mpich-scif/3.0.4/mic --enable-error- >> checking=runtime \ >>> --enable-timing=runtime --host=i686-linux-gnu \ >>> --with-cross=maint/fcrosscompile/cross_values.txt.mike >>> >>> The settings in cross_values.txt.mike were: >>> >>> # The Fortran related cross compilation values. >>> CROSS_F77_SIZEOF_INTEGER="4" >>> CROSS_F77_SIZEOF_REAL="4" >>> CROSS_F77_SIZEOF_DOUBLE_PRECISION="8" >>> CROSS_F77_TRUE_VALUE="-1" >>> CROSS_F77_FALSE_VALUE="0" >>> CROSS_F90_ADDRESS_KIND="8" >>> CROSS_F90_OFFSET_KIND="8" >>> CROSS_F90_INTEGER_KIND="4" >>> CROSS_F90_REAL_MODEL=" 6 , 37" >>> CROSS_F90_DOUBLE_MODEL=" 15 , 307" >>> CROSS_F90_INTEGER_MODEL=" 9" >>> CROSS_F90_ALL_INTEGER_MODELS=" 2 , 1, 4 , 2, 9 , 4, 18 , 8," >>> CROSS_F90_INTEGER_MODEL_MAP=" { 2 , 1 , 1 }, { 4 , 2 , 2 }, { >> 9 , 4 , 4 }, { 18 , 8 , 8 }," >>> >>> And finally, the configure invocation for MPICH on the host was: >>> >>> ./configure CC="icc" CXX=icpc F77="ifort -warn noalignments" >> FC="ifort >>> -warn noalignments" \ MPICHLIB_CFLAGS="-g" MPICHLIB_CXXFLAGS="-g" >>> MPICHLIB_FFLAGS="-g" MPICHLIB_FCFLAGS="-g" \ MPICHLIB_LDFLAGS="- >> g" >>> LIBS=-lscif LDFLAGS=-L/usr/lib64 --with- >> device=ch3:nemesis:scif,tcp \ >>> --disable-romio --disable-mpe --enable-smpcoll --enable- >> fast=defopt \ >>> --prefix=/apps/mpich-scif/3.0.4/intel64 >>> --enable-error-checking=runtime \ --enable-timing=runtime >>> >>> Thanks, >>> >>> Maciej >>> >>> Maciej Golebiewski >>> Applications Support, Advanced Scientific Computing CSIRO IM&T >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] an FC-Serial symbol
by Jeff Hammond 09 Jul '13

09 Jul '13

Read a tutorial on gmake on the Internet. This is not an MPICH issue. You can also ask on Stackoverflow or any other programmer Q&A site. Just not here. Jeff Sent from my iPhone On Jul 9, 2013, at 1:54 AM, Sara Yzdn <sarayzdn(a)gmail.com> wrote: > > Hi, > > I have built a code by mpich, but I want to define an FC_SERIAL symbol on top of the makefile. How can I do it correctly? > > Regards, > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segmentation fault (signal 11) and Exit code 139
by Jeff Hammond 09 Jul '13

09 Jul '13

Not an MPICH problem. Contact WRF support. Jeff Sent from my iPhone On Jul 9, 2013, at 2:09 AM, Zheng Li <seraphzl(a)gmail.com> wrote: Dear All, I tried to run the default January 2000 Case (the East Coast Winter Storm of January 24-25, 2000) in WRF v3.5, and failed when running wrf.exe. The error is as follows: [sl@sl0 em_real]$ mpirun -np 4 ./wrf.exe starting wrf task 0 of 4 starting wrf task 1 of 4 starting wrf task 2 of 4 starting wrf task 3 of 4 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Does someone have idea of resolving this issue? Thanks in advance. Lee <seraphzl(a)gmail.com> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0