May 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] bug in MPI_File_write_all?
by Rajeev Thakur 20 May '14

20 May '14

Are you writing to an NFS file system or local disk? Rajeev On May 20, 2014, at 7:09 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > Hi, I have tested with the last version I have found (mpich-master-v3.1-248-g6732f002). It is strange, the first time I have tried everything have been ok, then I have tried to increment the number of iterations (repeat<40) and then the test failed again. I have run several times the test code and now it seems a little bit more robust (I do not know if this makes sense, but seems that need more iterations to fail. I do not check statistically, this could be only an impression), but fails. > > Oriol > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Rajeev Thakur > Sent: 19 May 2014 17:05 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] bug in MPI_File_write_all? > > The program works on my Mac laptop with the latest MPICH source. Can you try with the latest nightly snapshot of MPICH from here: > http://www.mpich.org/static/downloads/nightly/master/mpich/ > > Rajeev > > On May 19, 2014, at 4:49 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > >> Yes, a attach below a test program. The error can be reproduced running it with 9 MPI threads. >> >> #include <mpi.h> >> >> #include <iostream> >> #include <sstream> >> #include <string> >> #include <vector> >> >> /** >> * get the number of local rows and columns */ int getNumroc(int >> globalSize, int myProc, int nProcs, int blockSize) { int myDist = >> myProc % nProcs; int nBlocks = globalSize / blockSize; int numroc = >> nBlocks / nProcs; numroc *= blockSize; int extraBlocks = nBlocks % >> nProcs; if(myDist < extraBlocks) { >> numroc += blockSize; >> } >> else if(myDist == extraBlocks) >> { >> numroc += globalSize % blockSize; >> } >> return numroc; >> } >> >> int main(int argc, char **argv) >> { >> //MPI vars >> bool mpiRoot; >> int mpiRank; >> int mpiNumTasks; >> char hostName[MPI_MAX_PROCESSOR_NAME]; int lenHostName; int >> myProcRow; int myProcCol; >> >> // Initiate MPI >> int tmp; >> tmp = MPI_Init(&argc, &argv); >> if (tmp != MPI_SUCCESS) { >> std::cerr << "Error: MPI can not be started. Terminating." << std::endl; >> MPI_Abort(MPI_COMM_WORLD, 1); >> } >> MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank); >> MPI_Comm_size(MPI_COMM_WORLD, &mpiNumTasks); >> MPI_Get_processor_name(hostName, &lenHostName); >> >> if(mpiNumTasks != 9) >> { >> std::cerr << "Error: This is a test program designed for running with 9 threads." << std::endl; >> MPI_Abort(MPI_COMM_WORLD, 1); >> } >> >> //Id of the process in a 2d grid >> myProcRow = mpiRank / 3; >> myProcCol = mpiRank % 3; >> >> double *m; ///<A pointer to the distributed matrix >> double *mRead; ///<A pointer to the distributed matrix >> >> int nGlobRows = 5; ///<Number of rows of the global matrix >> int nGlobCols = 4; ///<Number of columns of the global matrix >> int nRows; ///<Number of rows of the local matrix >> int nCols; ///<Number of columns of the local matrix >> int nBlockRows = 1; ///<Number of rows of the distributed matrix blocks >> int nBlockCols = 3; ///<Number of columns of the distributed matrix blocks >> >> nRows = getNumroc(nGlobRows, myProcRow, 3, nBlockRows); nCols = >> getNumroc(nGlobCols, myProcCol, 3, nBlockCols); >> >> m = new double[nRows*nCols]; >> mRead = new double[nRows*nCols]; >> for(int i = 0; i < nRows; i++) >> { >> for(int j = 0; j < nCols; j++) >> { >> m[i*nCols + j] = 1; >> } >> } >> >> for(int repeat = 0; repeat < 10; repeat++) { >> int dims[] = {nGlobRows, nGlobCols}; >> int dargs[] = {nBlockRows, nBlockCols}; >> int distribs[] = {MPI_DISTRIBUTE_CYCLIC, MPI_DISTRIBUTE_CYCLIC}; >> int dim[] = {3, 3}; >> char nat[] = "native"; >> int rc; >> MPI_Datatype dcarray; >> MPI_File cFile; >> MPI_Status status; >> >> MPI_Type_create_darray(mpiNumTasks, mpiRank, 2, dims, distribs, dargs, dim, MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); >> MPI_Type_commit(&dcarray); >> >> std::stringstream ss; >> ss << "test_" << repeat << ".bin"; >> std::string fname = ss.str(); //"test.bin"; >> std::vector<char> fn(fname.begin(), fname.end()); >> fn.push_back('\0'); >> MPI_File_delete (&fn[0], MPI_INFO_NULL); >> >> //Write file >> rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); >> if(rc){ >> std::cerr << "Error: Failed to open file." << std::endl; >> MPI_Abort(MPI_COMM_WORLD, 1); >> } >> else >> { >> MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >> MPI_File_write_all(cFile, m, nRows*nCols, MPI_DOUBLE, &status); >> } >> MPI_Barrier(MPI_COMM_WORLD); >> MPI_File_close(&cFile); >> >> >> //Initialize matrix before reading to 0; >> for(int i = 0; i < nRows; i++) >> { >> for(int j = 0; j < nCols; j++) >> { >> mRead[i*nCols + j] = 0; >> } >> } >> //Read file >> rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_RDONLY, MPI_INFO_NULL, &cFile); >> if(rc){ >> std::cerr << "Error: Failed to open file." << std::endl; >> MPI_Abort(MPI_COMM_WORLD, 1); >> } >> else >> { >> MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >> MPI_File_read_all(cFile, mRead, nRows*nCols, MPI_DOUBLE, &status); >> } >> MPI_Barrier(MPI_COMM_WORLD); >> MPI_File_close(&cFile); >> MPI_Type_free(&dcarray); >> >> //Check data >> for(int i = 0; i < nRows; i++) >> { >> for(int j = 0; j < nCols; j++) >> { >> if(mRead[i*nCols + j] != 1) >> { >> std::cerr << "Error in data. " << repeat << " iteration." << std::endl; >> MPI_Abort(MPI_COMM_WORLD, 1); >> } >> } >> } >> } >> >> delete [] m; >> delete [] mRead; >> >> return 0; >> } >> >> Oriol >> >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Rajeev Thakur >> Sent: 16 May 2014 20:22 >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] bug in MPI_File_write_all? >> >> Can you send us a small test program that can be compiled and run? >> >> Rajeev >> >> >> On May 16, 2014, at 11:42 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: >> >>> I also tried with the 3.0.4 (for which I have a distribution package) version with same results. If you think the result could change with 3.1, I will try with it. >>> >>> Oriol >>> >>> >>> -- >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >>> >>> -----Original Message----- >>> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >>> Behalf Of Rajeev Thakur >>> Sent: 16 May 2014 17:14 >>> To: discuss(a)mpich.org >>> Subject: Re: [mpich-discuss] bug in MPI_File_write_all? >>> >>> 1.4.1 is a very old version of MPICH. Can you try with the latest release, 3.1? >>> >>> Rajeev >>> >>> On May 16, 2014, at 8:50 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: >>> >>>> Hello, >>>> >>>> I am writing a function for storing a block cyclic distributed array using MPI_Type_create_darray, MPI_File_set_view, and MPI_File_write_all functions. After some testing I found that for some particular cases, sometimes, some of the values of the array have been not stored properly on the file. The error seems random, but always affects the same number in the same position (where instead of a number, there is a nan, it seems that the position is not written). I tried with other implementations (e.g. openmpi) and seems everything works well (I am using MPICH2 Version 1.4.1). >>>> >>>> For instance, I saw the error for a matrix with 5 rows and 4 columns. Block size for rows = 1 and for columns = 3. There are 9 MPI processes distributed in a 3x3 array. My function is (where nBlockRows/nBlockCols define the size of the blocks, nGlobRows/nGlobCols define the global size of the matrix, nProcRows/nProcCols define the dimensions of the process grid, and fname is the name of the file.): >>>> >>>> void Matrix::writeMatrixMPI(std::string fname) { int dims[] = >>>> {this->nGlobRows, this->nGlobCols}; int dargs[] = >>>> {this->nBlockRows, >>>> this->nBlockCols}; int distribs[] = {MPI_DISTRIBUTE_CYCLIC, >>>> MPI_DISTRIBUTE_CYCLIC}; int dim[] = {communicator->nProcRows, >>>> communicator->nProcCols}; char nat[] = "native"; int rc; >>>> MPI_Datatype dcarray; MPI_File cFile; MPI_Status status; >>>> >>>> MPI_Type_create_darray(communicator->mpiNumTasks, >>>> communicator->mpiRank, 2, dims, distribs, dargs, dim, >>>> MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); >>>> MPI_Type_commit(&dcarray); >>>> >>>> std::vector<char> fn(fname.begin(), fname.end()); >>>> fn.push_back('\0'); rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], >>>> MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, >>>> &cFile); if(rc){ ss << "Error: Failed to open file: " << rc; >>>> misc.error(ss.str(), 0); } else { MPI_File_set_view(cFile, 0, >>>> MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >>>> MPI_File_write_all(cFile, this->m, this->nRows*this->nCols, MPI_DOUBLE, &status); >>>> } >>>> MPI_File_close(&cFile); >>>> MPI_Type_free(&dcarray); >>>> } >>>> >>>> If I call the function a repeated number of times without deleting the file between calls, the file seems to be always correct. If I delete the file between calls or create a file with a different name each time, then I see the error in some of the files. The error is a nan always in the same position. For me it is very strange, but as I said, with other implementations, I do not have the problem. >>>> >>>> -- >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] optimization of MPI_Alltoall(..)
by William Gropp 20 May '14

20 May '14

Yes, there are. There are more things that you can do to take into account the interconnect topology, and there are things that can be done to take better advantage of the SMP nodes. The algorithm in MPICH is rather generic and tries not to overstress the network, since contention and queue search times can have a significant impact on performance of alltoall. With the appropriate algorithm, you should be able to sustain near the peak interconnect bandwidth (assuming also that the memory system on the node is fast enough to keep up with the network). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On May 19, 2014, at 9:19 PM, Jan T. Balewski wrote: > Hi, > I want to use MPI to transpose a big NxM matrix. > Currently the size is: N=20k, M=40k, type=ushort, total size =16GB (it will be > larger in the future) > > To evaluate the code I report the processing speed in MB/sec, it is dominated > by the cost of MPI_Alltoall(...) between 32 MPI processes. > By changing the order in which MPI jobs are assigned to the cores on the blades > I was able to increase the overall speed by ~15% , from 349 MB/sec to 398 > MB/sec. > > My question is are there more tricks I can play to accelerate MPI_Alltoall(...) > further ? > (For now using the existing hardware) > Thanks for any suggestions > Jan > > P.S. Below are gory details: > > The transposition is done in 2 stages on 32 MPI jobs running on 4 8-core Dell > 1950 blades. > The 4 blades are connected via eth0 to the same card in the Cisco 1 GBit switch. > This is the order of operations: > - big matrix is divided on 32x32 non-square blocks > - each of 32 blocks is transposed individually on CPU in each MPI job > - the 32x32 blocks are exchanged between MPI jobs and transposed using > MPI_Alltoall(...) command > - the time is measured using MPI_Wtime() > FYI, the whole code is visible at: > https://bitbucket.org/balewski/kruk-mpi/src/8d3c4b7deb566f2768f132135b12e58… > > > ***** mode 1: uses 8 consecutive cores per blade, next fill 2nd blade, etc > $ mpiexec -f machinefile -n 32 ./kruk2 > where machinefile is: > oswrk139.lns.mit.edu:8 > oswrk140.lns.mit.edu:8 > oswrk145.lns.mit.edu:8 > oswrk150.lns.mit.edu:8 > > Summary: Ncpu=32 avrT/sec=40.83, totMB=16257.0 avrSpeed=398.2(MB/sec) > > > ***** mode 2: uses 1st core from all blades, next use 2nd core from all, etc: > $ mpiexec -f machinefile -n 32 ./kruk2 > where machinefile is: > oswrk139.lns.mit.edu:1 > oswrk140.lns.mit.edu:1 > oswrk145.lns.mit.edu:1 > oswrk150.lns.mit.edu:1 > > Summary: Ncpu=32 avrT/sec=46.61, totMB=16257.0 avrSpeed=348.8(MB/sec) > (3) > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] bug in MPI_File_write_all?
by CANELA-XANDRI Oriol 20 May '14

20 May '14

Hi, I have tested with the last version I have found (mpich-master-v3.1-248-g6732f002). It is strange, the first time I have tried everything have been ok, then I have tried to increment the number of iterations (repeat<40) and then the test failed again. I have run several times the test code and now it seems a little bit more robust (I do not know if this makes sense, but seems that need more iterations to fail. I do not check statistically, this could be only an impression), but fails. Oriol -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Rajeev Thakur Sent: 19 May 2014 17:05 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] bug in MPI_File_write_all? The program works on my Mac laptop with the latest MPICH source. Can you try with the latest nightly snapshot of MPICH from here: http://www.mpich.org/static/downloads/nightly/master/mpich/ Rajeev On May 19, 2014, at 4:49 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > Yes, a attach below a test program. The error can be reproduced running it with 9 MPI threads. > > #include <mpi.h> > > #include <iostream> > #include <sstream> > #include <string> > #include <vector> > > /** > * get the number of local rows and columns */ int getNumroc(int > globalSize, int myProc, int nProcs, int blockSize) { int myDist = > myProc % nProcs; int nBlocks = globalSize / blockSize; int numroc = > nBlocks / nProcs; numroc *= blockSize; int extraBlocks = nBlocks % > nProcs; if(myDist < extraBlocks) { > numroc += blockSize; > } > else if(myDist == extraBlocks) > { > numroc += globalSize % blockSize; > } > return numroc; > } > > int main(int argc, char **argv) > { > //MPI vars > bool mpiRoot; > int mpiRank; > int mpiNumTasks; > char hostName[MPI_MAX_PROCESSOR_NAME]; int lenHostName; int > myProcRow; int myProcCol; > > // Initiate MPI > int tmp; > tmp = MPI_Init(&argc, &argv); > if (tmp != MPI_SUCCESS) { > std::cerr << "Error: MPI can not be started. Terminating." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank); > MPI_Comm_size(MPI_COMM_WORLD, &mpiNumTasks); > MPI_Get_processor_name(hostName, &lenHostName); > > if(mpiNumTasks != 9) > { > std::cerr << "Error: This is a test program designed for running with 9 threads." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > > //Id of the process in a 2d grid > myProcRow = mpiRank / 3; > myProcCol = mpiRank % 3; > > double *m; ///<A pointer to the distributed matrix > double *mRead; ///<A pointer to the distributed matrix > > int nGlobRows = 5; ///<Number of rows of the global matrix > int nGlobCols = 4; ///<Number of columns of the global matrix > int nRows; ///<Number of rows of the local matrix > int nCols; ///<Number of columns of the local matrix > int nBlockRows = 1; ///<Number of rows of the distributed matrix blocks > int nBlockCols = 3; ///<Number of columns of the distributed matrix blocks > > nRows = getNumroc(nGlobRows, myProcRow, 3, nBlockRows); nCols = > getNumroc(nGlobCols, myProcCol, 3, nBlockCols); > > m = new double[nRows*nCols]; > mRead = new double[nRows*nCols]; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > m[i*nCols + j] = 1; > } > } > > for(int repeat = 0; repeat < 10; repeat++) { > int dims[] = {nGlobRows, nGlobCols}; > int dargs[] = {nBlockRows, nBlockCols}; > int distribs[] = {MPI_DISTRIBUTE_CYCLIC, MPI_DISTRIBUTE_CYCLIC}; > int dim[] = {3, 3}; > char nat[] = "native"; > int rc; > MPI_Datatype dcarray; > MPI_File cFile; > MPI_Status status; > > MPI_Type_create_darray(mpiNumTasks, mpiRank, 2, dims, distribs, dargs, dim, MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); > MPI_Type_commit(&dcarray); > > std::stringstream ss; > ss << "test_" << repeat << ".bin"; > std::string fname = ss.str(); //"test.bin"; > std::vector<char> fn(fname.begin(), fname.end()); > fn.push_back('\0'); > MPI_File_delete (&fn[0], MPI_INFO_NULL); > > //Write file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_write_all(cFile, m, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > > > //Initialize matrix before reading to 0; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > mRead[i*nCols + j] = 0; > } > } > //Read file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_RDONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_read_all(cFile, mRead, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > MPI_Type_free(&dcarray); > > //Check data > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > if(mRead[i*nCols + j] != 1) > { > std::cerr << "Error in data. " << repeat << " iteration." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > } > } > } > > delete [] m; > delete [] mRead; > > return 0; > } > > Oriol > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Rajeev Thakur > Sent: 16 May 2014 20:22 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] bug in MPI_File_write_all? > > Can you send us a small test program that can be compiled and run? > > Rajeev > > > On May 16, 2014, at 11:42 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > >> I also tried with the 3.0.4 (for which I have a distribution package) version with same results. If you think the result could change with 3.1, I will try with it. >> >> Oriol >> >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Rajeev Thakur >> Sent: 16 May 2014 17:14 >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] bug in MPI_File_write_all? >> >> 1.4.1 is a very old version of MPICH. Can you try with the latest release, 3.1? >> >> Rajeev >> >> On May 16, 2014, at 8:50 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: >> >>> Hello, >>> >>> I am writing a function for storing a block cyclic distributed array using MPI_Type_create_darray, MPI_File_set_view, and MPI_File_write_all functions. After some testing I found that for some particular cases, sometimes, some of the values of the array have been not stored properly on the file. The error seems random, but always affects the same number in the same position (where instead of a number, there is a nan, it seems that the position is not written). I tried with other implementations (e.g. openmpi) and seems everything works well (I am using MPICH2 Version 1.4.1). >>> >>> For instance, I saw the error for a matrix with 5 rows and 4 columns. Block size for rows = 1 and for columns = 3. There are 9 MPI processes distributed in a 3x3 array. My function is (where nBlockRows/nBlockCols define the size of the blocks, nGlobRows/nGlobCols define the global size of the matrix, nProcRows/nProcCols define the dimensions of the process grid, and fname is the name of the file.): >>> >>> void Matrix::writeMatrixMPI(std::string fname) { int dims[] = >>> {this->nGlobRows, this->nGlobCols}; int dargs[] = >>> {this->nBlockRows, >>> this->nBlockCols}; int distribs[] = {MPI_DISTRIBUTE_CYCLIC, >>> MPI_DISTRIBUTE_CYCLIC}; int dim[] = {communicator->nProcRows, >>> communicator->nProcCols}; char nat[] = "native"; int rc; >>> MPI_Datatype dcarray; MPI_File cFile; MPI_Status status; >>> >>> MPI_Type_create_darray(communicator->mpiNumTasks, >>> communicator->mpiRank, 2, dims, distribs, dargs, dim, >>> MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); >>> MPI_Type_commit(&dcarray); >>> >>> std::vector<char> fn(fname.begin(), fname.end()); >>> fn.push_back('\0'); rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], >>> MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, >>> &cFile); if(rc){ ss << "Error: Failed to open file: " << rc; >>> misc.error(ss.str(), 0); } else { MPI_File_set_view(cFile, 0, >>> MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >>> MPI_File_write_all(cFile, this->m, this->nRows*this->nCols, MPI_DOUBLE, &status); >>> } >>> MPI_File_close(&cFile); >>> MPI_Type_free(&dcarray); >>> } >>> >>> If I call the function a repeated number of times without deleting the file between calls, the file seems to be always correct. If I delete the file between calls or create a file with a different name each time, then I see the error in some of the files. The error is a nan always in the same position. For me it is very strange, but as I said, with other implementations, I do not have the problem. >>> >>> -- >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] problem on configuring MPI with slurm
by Balaji, Pavan 20 May '14

20 May '14

Please keep the mailing list cc’ed. Run configure without those options: ./configure --prefix=/zhuxm/mpich-3.1 --enable-fast=all,03 CC=icc CXX=icpc F77=ifort FC=ifort — Pavan On May 20, 2014, at 1:02 AM, 朱学明 <zhuxm(a)nmefc.gov.cn> wrote: > Pavan, > > Thank you for your quickly response. Can you tell me that should I get rid of --with-slurm option only, or any other friend options, such as --with-pmi=slurm --with-pm=no ? > > Can you show me the exact options what I need? > > > -- > Xueming Zhu 朱学明 > Key Laboratory of Research on Marine Hazards Forecasting (LoMF), SOA > National Marine Environmental Forecasting Center (NMEFC) > No.8 Dahuisi Road, Haidian District, Beijing, 100081 > People's Republic of China > Tel: +86-10-82481923 > > > 在 2014-05-20 14:24:44 ， Balaji, Pavan <balaji(a)anl.gov> 写道： > > We should just get rid of the --with-slurm option in MPICH. It’s pretty useless for two reasons: > > 1. You don’t need to configure mpich with slurm at all, unless you are planning to use srun/salloc directly. You can just build mpich normally and run with mpiexec. mpiexec will internally detect that you are using a slurm environment and use srun internally. > > 2. If you really want to use srun/salloc, you can just build your application with mpicc -L/path/to/slurm foo.c -lpmi and it’ll pick up slurm’s PMI library. Then you can use srun/salloc. > > Xueming, > > My suggestion is, forget the slurm option. Just build mpich normally and run it with mpiexec. It’ll automatically use slurm internally. > > — Pavan > > On May 19, 2014, at 9:28 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > > > configure expects a directory named slurm under the location you specify. Try using '--with-slurm=/usr/local' in your configure line instead. > > > > Ken > > > > On 05/19/2014 07:06 PM, 朱学明 wrote: > >> Hello, > >> > >> I am trying to install MPICH-3.1 on our linux server with Slurm > >> 1.2.25. I configured it with the command as follow, > >> > >> > >> ./configure --prefix=/zhuxm/mpich-3.1 CC=icc CXX=icpc F77=ifort > >> FC=ifort -enable-fast=all,O3 -enable-shared --with-rdma=gen2 > >> --with-pmi=slurm --with-pm=no --with-slurm=/usr/local/slurm > >> --with-ib-include=/usr/local/ib_hpc/include > >> --with-ib-libpath=/usr/local/ib_hpc/lib64 --with-ch3-rank-bits=32 > >> --without-mpe --without-hwloc --disable-rdma-cm -disable-fuse 2>&1 |tee > >> config-min.log > >> > >> > >> Then I have got the error messages as follow, > >> > >> > >> Checking slurm/pmi.h usability ... no > >> > >> Checking slurm/pmi.h presence... no > >> > >> Checking for slurm/pmi.h... no > >> > >> configure: error: could not find slurm/pmi.h. Configure aborted > >> > >> > >> Can you give mu any suggestion how to figure out the issue? Thanks. > >> > >> > >> -- > >> *Xueming Zhu 朱学明* > >> Key Laboratory of Research on Marine Hazards Forecasting (*LoMF*), SOA > >> National Marine Environmental Forecasting Center (*NMEFC*) > >> No.8 Dahuisi Road, Haidian District, Beijing, 100081 > >> People's Republic of China > >> Tel: +86-10-82481923 > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] problem on configuring MPI with slurm
by Balaji, Pavan 20 May '14

20 May '14

We should just get rid of the --with-slurm option in MPICH. It’s pretty useless for two reasons: 1. You don’t need to configure mpich with slurm at all, unless you are planning to use srun/salloc directly. You can just build mpich normally and run with mpiexec. mpiexec will internally detect that you are using a slurm environment and use srun internally. 2. If you really want to use srun/salloc, you can just build your application with mpicc -L/path/to/slurm foo.c -lpmi and it’ll pick up slurm’s PMI library. Then you can use srun/salloc. Xueming, My suggestion is, forget the slurm option. Just build mpich normally and run it with mpiexec. It’ll automatically use slurm internally. — Pavan On May 19, 2014, at 9:28 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > configure expects a directory named slurm under the location you specify. Try using '--with-slurm=/usr/local' in your configure line instead. > > Ken > > On 05/19/2014 07:06 PM, 朱学明 wrote: >> Hello, >> >> I am trying to install MPICH-3.1 on our linux server with Slurm >> 1.2.25. I configured it with the command as follow, >> >> >> ./configure --prefix=/zhuxm/mpich-3.1 CC=icc CXX=icpc F77=ifort >> FC=ifort -enable-fast=all,O3 -enable-shared --with-rdma=gen2 >> --with-pmi=slurm --with-pm=no --with-slurm=/usr/local/slurm >> --with-ib-include=/usr/local/ib_hpc/include >> --with-ib-libpath=/usr/local/ib_hpc/lib64 --with-ch3-rank-bits=32 >> --without-mpe --without-hwloc --disable-rdma-cm -disable-fuse 2>&1 |tee >> config-min.log >> >> >> Then I have got the error messages as follow, >> >> >> Checking slurm/pmi.h usability ... no >> >> Checking slurm/pmi.h presence... no >> >> Checking for slurm/pmi.h... no >> >> configure: error: could not find slurm/pmi.h. Configure aborted >> >> >> Can you give mu any suggestion how to figure out the issue? Thanks. >> >> >> -- >> *Xueming Zhu 朱学明* >> Key Laboratory of Research on Marine Hazards Forecasting (*LoMF*), SOA >> National Marine Environmental Forecasting Center (*NMEFC*) >> No.8 Dahuisi Road, Haidian District, Beijing, 100081 >> People's Republic of China >> Tel: +86-10-82481923 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

configure MPICH with Absoft Fortran
by Yan Bai 19 May '14

19 May '14

Dear all I haven been trying to install MPICH-3.1 for a month, but always failed. I use Absoft Fortran and configured the MPICH with the following command ./configure --prefix=/opt/mpich2-install F77=$ABSOFT/bin/af90 \ FC=$ABSOFT/bin/af90 --with-pm=mpd I got the following error message: configure: error: The selected Fortran 90 compiler /Applications/Absoft10.2/bin/af90 does not work with the selected Fortran 77 compiler /Applications/Absoft10.2/bin/af90. Use the environment variables FC and F77 respectively to select compatible Fortran compilers. The check here tests to see if a main program compiled with the Fortran 90 compiler can link with a subroutine compiled with the Fortran 77 compiler. I also tried the configure line in the Readme file of MPICH-3.1 and the following one, ./configure --prefix=/opt/mpich2-install F77=$ABSOFT/bin/af77 \ FC=$ABSOFT/bin/af90 --with-pm=mp All gave me the similar error message. Attached please find the config.log file. Any help/suggestions will be greatly appreciated! Best Yan

1 0

Re: [mpich-discuss] problem on configuring MPI with slurm
by Kenneth Raffenetti 19 May '14

19 May '14

configure expects a directory named slurm under the location you specify. Try using '--with-slurm=/usr/local' in your configure line instead. Ken On 05/19/2014 07:06 PM, 朱学明 wrote: > Hello, > > I am trying to install MPICH-3.1 on our linux server with Slurm > 1.2.25. I configured it with the command as follow, > > > ./configure --prefix=/zhuxm/mpich-3.1 CC=icc CXX=icpc F77=ifort > FC=ifort -enable-fast=all,O3 -enable-shared --with-rdma=gen2 > --with-pmi=slurm --with-pm=no --with-slurm=/usr/local/slurm > --with-ib-include=/usr/local/ib_hpc/include > --with-ib-libpath=/usr/local/ib_hpc/lib64 --with-ch3-rank-bits=32 > --without-mpe --without-hwloc --disable-rdma-cm -disable-fuse 2>&1 |tee > config-min.log > > > Then I have got the error messages as follow, > > > Checking slurm/pmi.h usability ... no > > Checking slurm/pmi.h presence... no > > Checking for slurm/pmi.h... no > > configure: error: could not find slurm/pmi.h. Configure aborted > > > Can you give mu any suggestion how to figure out the issue? Thanks. > > > -- > *Xueming Zhu 朱学明* > Key Laboratory of Research on Marine Hazards Forecasting (*LoMF*), SOA > National Marine Environmental Forecasting Center (*NMEFC*) > No.8 Dahuisi Road, Haidian District, Beijing, 100081 > People's Republic of China > Tel: +86-10-82481923 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

optimization of MPI_Alltoall(..)
by Jan T. Balewski 19 May '14

19 May '14

Hi, I want to use MPI to transpose a big NxM matrix. Currently the size is: N=20k, M=40k, type=ushort, total size =16GB (it will be larger in the future) To evaluate the code I report the processing speed in MB/sec, it is dominated by the cost of MPI_Alltoall(...) between 32 MPI processes. By changing the order in which MPI jobs are assigned to the cores on the blades I was able to increase the overall speed by ~15% , from 349 MB/sec to 398 MB/sec. My question is are there more tricks I can play to accelerate MPI_Alltoall(...) further ? (For now using the existing hardware) Thanks for any suggestions Jan P.S. Below are gory details: The transposition is done in 2 stages on 32 MPI jobs running on 4 8-core Dell 1950 blades. The 4 blades are connected via eth0 to the same card in the Cisco 1 GBit switch. This is the order of operations: - big matrix is divided on 32x32 non-square blocks - each of 32 blocks is transposed individually on CPU in each MPI job - the 32x32 blocks are exchanged between MPI jobs and transposed using MPI_Alltoall(...) command - the time is measured using MPI_Wtime() FYI, the whole code is visible at: https://bitbucket.org/balewski/kruk-mpi/src/8d3c4b7deb566f2768f132135b12e58… ***** mode 1: uses 8 consecutive cores per blade, next fill 2nd blade, etc $ mpiexec -f machinefile -n 32 ./kruk2 where machinefile is: oswrk139.lns.mit.edu:8 oswrk140.lns.mit.edu:8 oswrk145.lns.mit.edu:8 oswrk150.lns.mit.edu:8 Summary: Ncpu=32 avrT/sec=40.83, totMB=16257.0 avrSpeed=398.2(MB/sec) ***** mode 2: uses 1st core from all blades, next use 2nd core from all, etc: $ mpiexec -f machinefile -n 32 ./kruk2 where machinefile is: oswrk139.lns.mit.edu:1 oswrk140.lns.mit.edu:1 oswrk145.lns.mit.edu:1 oswrk150.lns.mit.edu:1 Summary: Ncpu=32 avrT/sec=46.61, totMB=16257.0 avrSpeed=348.8(MB/sec) (3)

1 0

Re: [mpich-discuss] problem on configuring MPI with slurm
by Lu, Huiwei 19 May '14

19 May '14

Looks like the slurm path is not correct. MPICH will automatically detect slurm and use slurm capabilities. Could you try again with just “--with-pmi=slurm --with-pm=no”, and no “--with-slurm=/usr/local/slurm”? — Huiwei Lu Postdoc Appointee Mathematics and Computer Science Division, Argonne National Laboratory http://www.mcs.anl.gov/~huiweilu/ On May 19, 2014, at 7:06 PM, 朱学明 <zhuxm(a)nmefc.gov.cn> wrote: > Hello, > > I am trying to install MPICH-3.1 on our linux server with Slurm 1.2.25. I configured it with the command as follow, > > > ./configure --prefix=/zhuxm/mpich-3.1 CC=icc CXX=icpc F77=ifort FC=ifort -enable-fast=all,O3 -enable-shared --with-rdma=gen2 --with-pmi=slurm --with-pm=no --with-slurm=/usr/local/slurm --with-ib-include=/usr/local/ib_hpc/include --with-ib-libpath=/usr/local/ib_hpc/lib64 --with-ch3-rank-bits=32 --without-mpe --without-hwloc --disable-rdma-cm -disable-fuse 2>&1 |tee config-min.log > > > Then I have got the error messages as follow, > > > Checking slurm/pmi.h usability ... no > > Checking slurm/pmi.h presence... no > > Checking for slurm/pmi.h... no > > configure: error: could not find slurm/pmi.h. Configure aborted > > > Can you give mu any suggestion how to figure out the issue? Thanks. > > > -- > Xueming Zhu 朱学明 > Key Laboratory of Research on Marine Hazards Forecasting (LoMF), SOA > National Marine Environmental Forecasting Center (NMEFC) > No.8 Dahuisi Road, Haidian District, Beijing, 100081 > People's Republic of China > Tel: +86-10-82481923 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] problem on configuring MPI with slurm
by Seo, Sangmin 19 May '14

19 May '14

Hi Xueming, Could you try to specify Slurm’s include path with '--with-slurm-include’? — Sangmin On May 19, 2014, at 7:06 PM, 朱学明 <zhuxm(a)nmefc.gov.cn<mailto:[email protected]>> wrote: Hello, I am trying to install MPICH-3.1 on our linux server with Slurm 1.2.25. I configured it with the command as follow, ./configure --prefix=/zhuxm/mpich-3.1 CC=icc CXX=icpc F77=ifort FC=ifort -enable-fast=all,O3 -enable-shared --with-rdma=gen2 --with-pmi=slurm --with-pm=no --with-slurm=/usr/local/slurm --with-ib-include=/usr/local/ib_hpc/include --with-ib-libpath=/usr/local/ib_hpc/lib64 --with-ch3-rank-bits=32 --without-mpe --without-hwloc --disable-rdma-cm -disable-fuse 2>&1 |tee config-min.log Then I have got the error messages as follow, Checking slurm/pmi.h usability ... no Checking slurm/pmi.h presence... no Checking for slurm/pmi.h... no configure: error: could not find slurm/pmi.h. Configure aborted Can you give mu any suggestion how to figure out the issue? Thanks. -- Xueming Zhu 朱学明 Key Laboratory of Research on Marine Hazards Forecasting (LoMF), SOA National Marine Environmental Forecasting Center (NMEFC) No.8 Dahuisi Road, Haidian District, Beijing, 100081 People's Republic of China Tel: +86-10-82481923 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0