May 2014 - discuss - mailman3.cels.anl.gov

problem on configuring MPI with slurm
by 朱学明 19 May '14

19 May '14

1 0

Re: [mpich-discuss] -configfile detail
by Balaji, Pavan 19 May '14

19 May '14

There was a limit which we increased in the past. Are you using the latest version of mpich (or Hydra)? — Pavan On May 19, 2014, at 12:38 PM, Raeth . Peter <PRaeth(a)drc.com> wrote: > > Is there a limit to the number of process lines on -configfile? In our case, we are getting a segmentation fault with large numbers but the run goes fine for small numbers. For example: Our scenario has ten nodes. Each node has 16 cores. > > The job starts with: mpiexec -configfile mpich.txt > > For a small number of process lines, mpich.txt contains: > > -v -launcher ssh -launcher-exec /app/local/pbs_ssh -print-all-exitcodes -prepend-rank -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 0 0 0 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 1 1 1 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 2 2 2 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 3 3 3 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 1 1 1 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 2 2 2 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 3 3 3 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 > > > For a large number of process lines, mpich.txt contains more -host lines, 160 total. For each group of 16 processes, a different node is specified so that only 16 processes go on each of 10 nodes. The memory occupied by simpleTest is extremely small. It is this large run that fails with a seqmentation fault prior to any processes actually running. > > The reason for wanting to run this way is that different inputs are needed for each process. Plus, we want to be able to support users who want to physically co-locate processes that spend a lot of time communicating with each other. > > Thank you for whatever insights you can offer. > > > Best, > > Peter. > > ________________________________ > This electronic message transmission and any attachments that accompany it contain information from DRC® (Dynamics Research Corporation) or its subsidiaries, or the intended recipient, which is privileged, proprietary, business confidential, or otherwise protected from disclosure and is the exclusive property of DRC and/or the intended recipient. The information in this email is solely intended for the use of the individual or entity that is the intended recipient. If you are not the intended recipient, any use, dissemination, distribution, retention, or copying of this communication, attachments, or substance is prohibited. If you have received this electronic transmission in error, please immediately reply to the author via email that you received the message by mistake and also promptly and permanently delete this message and all copies of this email and any attachments. We thank you for your assistance and apologize for any inconvenience. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

-configfile detail
by Raeth . Peter 19 May '14

19 May '14

Is there a limit to the number of process lines on -configfile? In our case, we are getting a segmentation fault with large numbers but the run goes fine for small numbers. For example: Our scenario has ten nodes. Each node has 16 cores. The job starts with: mpiexec -configfile mpich.txt For a small number of process lines, mpich.txt contains: -v -launcher ssh -launcher-exec /app/local/pbs_ssh -print-all-exitcodes -prepend-rank -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 0 0 0 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 1 1 1 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 2 2 2 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 3 3 3 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 1 1 1 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 2 2 2 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 3 3 3 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 -host wputilc-0012 -np 1 ./simpleTest/distrib/simpleTest 4 4 4 22 16 33 For a large number of process lines, mpich.txt contains more -host lines, 160 total. For each group of 16 processes, a different node is specified so that only 16 processes go on each of 10 nodes. The memory occupied by simpleTest is extremely small. It is this large run that fails with a seqmentation fault prior to any processes actually running. The reason for wanting to run this way is that different inputs are needed for each process. Plus, we want to be able to support users who want to physically co-locate processes that spend a lot of time communicating with each other. Thank you for whatever insights you can offer. Best, Peter. ________________________________ This electronic message transmission and any attachments that accompany it contain information from DRC® (Dynamics Research Corporation) or its subsidiaries, or the intended recipient, which is privileged, proprietary, business confidential, or otherwise protected from disclosure and is the exclusive property of DRC and/or the intended recipient. The information in this email is solely intended for the use of the individual or entity that is the intended recipient. If you are not the intended recipient, any use, dissemination, distribution, retention, or copying of this communication, attachments, or substance is prohibited. If you have received this electronic transmission in error, please immediately reply to the author via email that you received the message by mistake and also promptly and permanently delete this message and all copies of this email and any attachments. We thank you for your assistance and apologize for any inconvenience.

1 0

Re: [mpich-discuss] bug in MPI_File_write_all?
by Rob Latham 19 May '14

19 May '14

On 05/19/2014 04:49 AM, CANELA-XANDRI Oriol wrote: > Yes, a attach below a test program. The error can be reproduced running it with 9 MPI threads. > Thank you for the test case. I probably fixed this last week, which is why Rajeev no longer sees this on his laptop. It is likely a bug in the way ROMIO flattens darray datatypes. Open-mpi from last week dies with a "signal 7" with this test case. ==rob > #include <mpi.h> > > #include <iostream> > #include <sstream> > #include <string> > #include <vector> > > /** > * get the number of local rows and columns > */ > int getNumroc(int globalSize, int myProc, int nProcs, int blockSize) > { > int myDist = myProc % nProcs; > int nBlocks = globalSize / blockSize; > int numroc = nBlocks / nProcs; > numroc *= blockSize; > int extraBlocks = nBlocks % nProcs; > if(myDist < extraBlocks) > { > numroc += blockSize; > } > else if(myDist == extraBlocks) > { > numroc += globalSize % blockSize; > } > return numroc; > } > > int main(int argc, char **argv) > { > //MPI vars > bool mpiRoot; > int mpiRank; > int mpiNumTasks; > char hostName[MPI_MAX_PROCESSOR_NAME]; > int lenHostName; > int myProcRow; > int myProcCol; > > // Initiate MPI > int tmp; > tmp = MPI_Init(&argc, &argv); > if (tmp != MPI_SUCCESS) { > std::cerr << "Error: MPI can not be started. Terminating." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank); > MPI_Comm_size(MPI_COMM_WORLD, &mpiNumTasks); > MPI_Get_processor_name(hostName, &lenHostName); > > if(mpiNumTasks != 9) > { > std::cerr << "Error: This is a test program designed for running with 9 threads." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > > //Id of the process in a 2d grid > myProcRow = mpiRank / 3; > myProcCol = mpiRank % 3; > > double *m; ///<A pointer to the distributed matrix > double *mRead; ///<A pointer to the distributed matrix > > int nGlobRows = 5; ///<Number of rows of the global matrix > int nGlobCols = 4; ///<Number of columns of the global matrix > int nRows; ///<Number of rows of the local matrix > int nCols; ///<Number of columns of the local matrix > int nBlockRows = 1; ///<Number of rows of the distributed matrix blocks > int nBlockCols = 3; ///<Number of columns of the distributed matrix blocks > > nRows = getNumroc(nGlobRows, myProcRow, 3, nBlockRows); > nCols = getNumroc(nGlobCols, myProcCol, 3, nBlockCols); > > m = new double[nRows*nCols]; > mRead = new double[nRows*nCols]; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > m[i*nCols + j] = 1; > } > } > > for(int repeat = 0; repeat < 10; repeat++) > { > int dims[] = {nGlobRows, nGlobCols}; > int dargs[] = {nBlockRows, nBlockCols}; > int distribs[] = {MPI_DISTRIBUTE_CYCLIC, MPI_DISTRIBUTE_CYCLIC}; > int dim[] = {3, 3}; > char nat[] = "native"; > int rc; > MPI_Datatype dcarray; > MPI_File cFile; > MPI_Status status; > > MPI_Type_create_darray(mpiNumTasks, mpiRank, 2, dims, distribs, dargs, dim, MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); > MPI_Type_commit(&dcarray); > > std::stringstream ss; > ss << "test_" << repeat << ".bin"; > std::string fname = ss.str(); //"test.bin"; > std::vector<char> fn(fname.begin(), fname.end()); > fn.push_back('\0'); > MPI_File_delete (&fn[0], MPI_INFO_NULL); > > //Write file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_write_all(cFile, m, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > > > //Initialize matrix before reading to 0; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > mRead[i*nCols + j] = 0; > } > } > //Read file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_RDONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_read_all(cFile, mRead, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > MPI_Type_free(&dcarray); > > //Check data > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > if(mRead[i*nCols + j] != 1) > { > std::cerr << "Error in data. " << repeat << " iteration." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > } > } > } > > delete [] m; > delete [] mRead; > > return 0; > } > > Oriol > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] bug in MPI_File_write_all?
by Rajeev Thakur 19 May '14

19 May '14

The program works on my Mac laptop with the latest MPICH source. Can you try with the latest nightly snapshot of MPICH from here: http://www.mpich.org/static/downloads/nightly/master/mpich/ Rajeev On May 19, 2014, at 4:49 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > Yes, a attach below a test program. The error can be reproduced running it with 9 MPI threads. > > #include <mpi.h> > > #include <iostream> > #include <sstream> > #include <string> > #include <vector> > > /** > * get the number of local rows and columns > */ > int getNumroc(int globalSize, int myProc, int nProcs, int blockSize) > { > int myDist = myProc % nProcs; > int nBlocks = globalSize / blockSize; > int numroc = nBlocks / nProcs; > numroc *= blockSize; > int extraBlocks = nBlocks % nProcs; > if(myDist < extraBlocks) > { > numroc += blockSize; > } > else if(myDist == extraBlocks) > { > numroc += globalSize % blockSize; > } > return numroc; > } > > int main(int argc, char **argv) > { > //MPI vars > bool mpiRoot; > int mpiRank; > int mpiNumTasks; > char hostName[MPI_MAX_PROCESSOR_NAME]; > int lenHostName; > int myProcRow; > int myProcCol; > > // Initiate MPI > int tmp; > tmp = MPI_Init(&argc, &argv); > if (tmp != MPI_SUCCESS) { > std::cerr << "Error: MPI can not be started. Terminating." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank); > MPI_Comm_size(MPI_COMM_WORLD, &mpiNumTasks); > MPI_Get_processor_name(hostName, &lenHostName); > > if(mpiNumTasks != 9) > { > std::cerr << "Error: This is a test program designed for running with 9 threads." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > > //Id of the process in a 2d grid > myProcRow = mpiRank / 3; > myProcCol = mpiRank % 3; > > double *m; ///<A pointer to the distributed matrix > double *mRead; ///<A pointer to the distributed matrix > > int nGlobRows = 5; ///<Number of rows of the global matrix > int nGlobCols = 4; ///<Number of columns of the global matrix > int nRows; ///<Number of rows of the local matrix > int nCols; ///<Number of columns of the local matrix > int nBlockRows = 1; ///<Number of rows of the distributed matrix blocks > int nBlockCols = 3; ///<Number of columns of the distributed matrix blocks > > nRows = getNumroc(nGlobRows, myProcRow, 3, nBlockRows); > nCols = getNumroc(nGlobCols, myProcCol, 3, nBlockCols); > > m = new double[nRows*nCols]; > mRead = new double[nRows*nCols]; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > m[i*nCols + j] = 1; > } > } > > for(int repeat = 0; repeat < 10; repeat++) > { > int dims[] = {nGlobRows, nGlobCols}; > int dargs[] = {nBlockRows, nBlockCols}; > int distribs[] = {MPI_DISTRIBUTE_CYCLIC, MPI_DISTRIBUTE_CYCLIC}; > int dim[] = {3, 3}; > char nat[] = "native"; > int rc; > MPI_Datatype dcarray; > MPI_File cFile; > MPI_Status status; > > MPI_Type_create_darray(mpiNumTasks, mpiRank, 2, dims, distribs, dargs, dim, MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); > MPI_Type_commit(&dcarray); > > std::stringstream ss; > ss << "test_" << repeat << ".bin"; > std::string fname = ss.str(); //"test.bin"; > std::vector<char> fn(fname.begin(), fname.end()); > fn.push_back('\0'); > MPI_File_delete (&fn[0], MPI_INFO_NULL); > > //Write file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_write_all(cFile, m, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > > > //Initialize matrix before reading to 0; > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > mRead[i*nCols + j] = 0; > } > } > //Read file > rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_RDONLY, MPI_INFO_NULL, &cFile); > if(rc){ > std::cerr << "Error: Failed to open file." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > else > { > MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); > MPI_File_read_all(cFile, mRead, nRows*nCols, MPI_DOUBLE, &status); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_File_close(&cFile); > MPI_Type_free(&dcarray); > > //Check data > for(int i = 0; i < nRows; i++) > { > for(int j = 0; j < nCols; j++) > { > if(mRead[i*nCols + j] != 1) > { > std::cerr << "Error in data. " << repeat << " iteration." << std::endl; > MPI_Abort(MPI_COMM_WORLD, 1); > } > } > } > } > > delete [] m; > delete [] mRead; > > return 0; > } > > Oriol > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Rajeev Thakur > Sent: 16 May 2014 20:22 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] bug in MPI_File_write_all? > > Can you send us a small test program that can be compiled and run? > > Rajeev > > > On May 16, 2014, at 11:42 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > >> I also tried with the 3.0.4 (for which I have a distribution package) version with same results. If you think the result could change with 3.1, I will try with it. >> >> Oriol >> >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> -----Original Message----- >> From: discuss-bounces(a)mpich.org [mailto:[email protected]] On >> Behalf Of Rajeev Thakur >> Sent: 16 May 2014 17:14 >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] bug in MPI_File_write_all? >> >> 1.4.1 is a very old version of MPICH. Can you try with the latest release, 3.1? >> >> Rajeev >> >> On May 16, 2014, at 8:50 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: >> >>> Hello, >>> >>> I am writing a function for storing a block cyclic distributed array using MPI_Type_create_darray, MPI_File_set_view, and MPI_File_write_all functions. After some testing I found that for some particular cases, sometimes, some of the values of the array have been not stored properly on the file. The error seems random, but always affects the same number in the same position (where instead of a number, there is a nan, it seems that the position is not written). I tried with other implementations (e.g. openmpi) and seems everything works well (I am using MPICH2 Version 1.4.1). >>> >>> For instance, I saw the error for a matrix with 5 rows and 4 columns. Block size for rows = 1 and for columns = 3. There are 9 MPI processes distributed in a 3x3 array. My function is (where nBlockRows/nBlockCols define the size of the blocks, nGlobRows/nGlobCols define the global size of the matrix, nProcRows/nProcCols define the dimensions of the process grid, and fname is the name of the file.): >>> >>> void Matrix::writeMatrixMPI(std::string fname) { int dims[] = >>> {this->nGlobRows, this->nGlobCols}; int dargs[] = {this->nBlockRows, >>> this->nBlockCols}; int distribs[] = {MPI_DISTRIBUTE_CYCLIC, >>> MPI_DISTRIBUTE_CYCLIC}; int dim[] = {communicator->nProcRows, >>> communicator->nProcCols}; char nat[] = "native"; int rc; >>> MPI_Datatype dcarray; MPI_File cFile; MPI_Status status; >>> >>> MPI_Type_create_darray(communicator->mpiNumTasks, >>> communicator->mpiRank, 2, dims, distribs, dargs, dim, >>> MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); MPI_Type_commit(&dcarray); >>> >>> std::vector<char> fn(fname.begin(), fname.end()); fn.push_back('\0'); >>> rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | >>> MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); if(rc){ >>> ss << "Error: Failed to open file: " << rc; >>> misc.error(ss.str(), 0); >>> } >>> else >>> { >>> MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >>> MPI_File_write_all(cFile, this->m, this->nRows*this->nCols, MPI_DOUBLE, &status); >>> } >>> MPI_File_close(&cFile); >>> MPI_Type_free(&dcarray); >>> } >>> >>> If I call the function a repeated number of times without deleting the file between calls, the file seems to be always correct. If I delete the file between calls or create a file with a different name each time, then I see the error in some of the files. The error is a nan always in the same position. For me it is very strange, but as I said, with other implementations, I do not have the problem. >>> >>> -- >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fault tolerance MPICH2 hydra vs MPICH3.0.4 hydra
by Balaji, Pavan 19 May '14

19 May '14

Hydra should be compatible between different versions of mpich (or mpich derivatives). However, there’s always a possibility that there was a bug in mpich-3.0.4’s hydra that was fixed in mpich-3.1. So we recommend using the latest version. — Pavan On May 19, 2014, at 1:15 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Hi Wesley. > Thank you very much for quick response. > I executed your's code. Master can't finish it's execution. It stalled on MPI_Wait on iteration 7. > > But if I use MPICH2 hydra, Master process will finish executing by reporting number of times on failure of slave process. > > Can you please advice if it's safe to make hybrid system of build with MPICH3.0.4, but using MPICH2 hydra? > Or may be any other solution. > Does MPICH 3.0.4 include all MPICH2 hydra functionality? > May be my configuration of MPICH 3.0.4 is wrong? > > Regards, > Anatoly. > > > On Sun, May 18, 2014 at 8:40 PM, Wesley Bland <wbland(a)anl.gov> wrote: > Hi Anatoly, > > I think the problem may be the way that you're aborting. MPICH catches the system abort call and kills the entire application when it's called. Instead, I suggest using MPI_Abort(MPI_COMM_WORLD, 1); That's what I use in my tests and it works fine. It also seemed to work for your code when I tried. I'll attach my modified version of your code. I switched it to C since I happened to have C++ support disabled on my local install, but that shouldn't change anything. > > Thanks, > Wesley > > > On Sun, May 18, 2014 at 5:18 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Dear MPICH2, > Can you please help me with understanding Fault Tolerance in MPICH3.0.4 > I have a simple MPI program: > Master calls MPI_Irecv + MPI_Wait in loop. > Single slave: calls MPI_Send x 5 times, then calls abort. > > When I execute program with MPICH2 hydra I get multiple times Master process prints about fail in slave. In MPICH3 hydra I get a single message about fail of slave and then Master process enters to endless wait for next Irecv completion. > In both cases I compiled program with MPICH3.0.4 > > In other words, with MPICH2 hydra each Irecv completes (even if slave died before execution of Irecv) but in MPICH3 hydra not. Causes MPI_Irecv endless wait. > > If I compile same program with MPICH2 and use MPICH2 hydra, I get the same result as compiling with MPICH3.0.4 and running with MPICH2 hydra. > > Execution command: > mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines1.txt -launcher=rsh -n 2 mpi_irecv_ft_simple > > > Both hydra's configured with: > $ ./configure --prefix=/space/local/mpich-3.0.4/ --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --no-recursion > > $ ./configure --prefix=/space/local/mpich2-1.5b2/ --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc > > Can you advice please? > > Regards, > Anatoly. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] bug in MPI_File_write_all?
by CANELA-XANDRI Oriol 19 May '14

19 May '14

Yes, a attach below a test program. The error can be reproduced running it with 9 MPI threads. #include <mpi.h> #include <iostream> #include <sstream> #include <string> #include <vector> /** * get the number of local rows and columns */ int getNumroc(int globalSize, int myProc, int nProcs, int blockSize) { int myDist = myProc % nProcs; int nBlocks = globalSize / blockSize; int numroc = nBlocks / nProcs; numroc *= blockSize; int extraBlocks = nBlocks % nProcs; if(myDist < extraBlocks) { numroc += blockSize; } else if(myDist == extraBlocks) { numroc += globalSize % blockSize; } return numroc; } int main(int argc, char **argv) { //MPI vars bool mpiRoot; int mpiRank; int mpiNumTasks; char hostName[MPI_MAX_PROCESSOR_NAME]; int lenHostName; int myProcRow; int myProcCol; // Initiate MPI int tmp; tmp = MPI_Init(&argc, &argv); if (tmp != MPI_SUCCESS) { std::cerr << "Error: MPI can not be started. Terminating." << std::endl; MPI_Abort(MPI_COMM_WORLD, 1); } MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank); MPI_Comm_size(MPI_COMM_WORLD, &mpiNumTasks); MPI_Get_processor_name(hostName, &lenHostName); if(mpiNumTasks != 9) { std::cerr << "Error: This is a test program designed for running with 9 threads." << std::endl; MPI_Abort(MPI_COMM_WORLD, 1); } //Id of the process in a 2d grid myProcRow = mpiRank / 3; myProcCol = mpiRank % 3; double *m; ///<A pointer to the distributed matrix double *mRead; ///<A pointer to the distributed matrix int nGlobRows = 5; ///<Number of rows of the global matrix int nGlobCols = 4; ///<Number of columns of the global matrix int nRows; ///<Number of rows of the local matrix int nCols; ///<Number of columns of the local matrix int nBlockRows = 1; ///<Number of rows of the distributed matrix blocks int nBlockCols = 3; ///<Number of columns of the distributed matrix blocks nRows = getNumroc(nGlobRows, myProcRow, 3, nBlockRows); nCols = getNumroc(nGlobCols, myProcCol, 3, nBlockCols); m = new double[nRows*nCols]; mRead = new double[nRows*nCols]; for(int i = 0; i < nRows; i++) { for(int j = 0; j < nCols; j++) { m[i*nCols + j] = 1; } } for(int repeat = 0; repeat < 10; repeat++) { int dims[] = {nGlobRows, nGlobCols}; int dargs[] = {nBlockRows, nBlockCols}; int distribs[] = {MPI_DISTRIBUTE_CYCLIC, MPI_DISTRIBUTE_CYCLIC}; int dim[] = {3, 3}; char nat[] = "native"; int rc; MPI_Datatype dcarray; MPI_File cFile; MPI_Status status; MPI_Type_create_darray(mpiNumTasks, mpiRank, 2, dims, distribs, dargs, dim, MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); MPI_Type_commit(&dcarray); std::stringstream ss; ss << "test_" << repeat << ".bin"; std::string fname = ss.str(); //"test.bin"; std::vector<char> fn(fname.begin(), fname.end()); fn.push_back('\0'); MPI_File_delete (&fn[0], MPI_INFO_NULL); //Write file rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); if(rc){ std::cerr << "Error: Failed to open file." << std::endl; MPI_Abort(MPI_COMM_WORLD, 1); } else { MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); MPI_File_write_all(cFile, m, nRows*nCols, MPI_DOUBLE, &status); } MPI_Barrier(MPI_COMM_WORLD); MPI_File_close(&cFile); //Initialize matrix before reading to 0; for(int i = 0; i < nRows; i++) { for(int j = 0; j < nCols; j++) { mRead[i*nCols + j] = 0; } } //Read file rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_RDONLY, MPI_INFO_NULL, &cFile); if(rc){ std::cerr << "Error: Failed to open file." << std::endl; MPI_Abort(MPI_COMM_WORLD, 1); } else { MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); MPI_File_read_all(cFile, mRead, nRows*nCols, MPI_DOUBLE, &status); } MPI_Barrier(MPI_COMM_WORLD); MPI_File_close(&cFile); MPI_Type_free(&dcarray); //Check data for(int i = 0; i < nRows; i++) { for(int j = 0; j < nCols; j++) { if(mRead[i*nCols + j] != 1) { std::cerr << "Error in data. " << repeat << " iteration." << std::endl; MPI_Abort(MPI_COMM_WORLD, 1); } } } } delete [] m; delete [] mRead; return 0; } Oriol -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Rajeev Thakur Sent: 16 May 2014 20:22 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] bug in MPI_File_write_all? Can you send us a small test program that can be compiled and run? Rajeev On May 16, 2014, at 11:42 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > I also tried with the 3.0.4 (for which I have a distribution package) version with same results. If you think the result could change with 3.1, I will try with it. > > Oriol > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > -----Original Message----- > From: discuss-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Rajeev Thakur > Sent: 16 May 2014 17:14 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] bug in MPI_File_write_all? > > 1.4.1 is a very old version of MPICH. Can you try with the latest release, 3.1? > > Rajeev > > On May 16, 2014, at 8:50 AM, CANELA-XANDRI Oriol <Oriol.CAnela-Xandri(a)roslin.ed.ac.uk> wrote: > >> Hello, >> >> I am writing a function for storing a block cyclic distributed array using MPI_Type_create_darray, MPI_File_set_view, and MPI_File_write_all functions. After some testing I found that for some particular cases, sometimes, some of the values of the array have been not stored properly on the file. The error seems random, but always affects the same number in the same position (where instead of a number, there is a nan, it seems that the position is not written). I tried with other implementations (e.g. openmpi) and seems everything works well (I am using MPICH2 Version 1.4.1). >> >> For instance, I saw the error for a matrix with 5 rows and 4 columns. Block size for rows = 1 and for columns = 3. There are 9 MPI processes distributed in a 3x3 array. My function is (where nBlockRows/nBlockCols define the size of the blocks, nGlobRows/nGlobCols define the global size of the matrix, nProcRows/nProcCols define the dimensions of the process grid, and fname is the name of the file.): >> >> void Matrix::writeMatrixMPI(std::string fname) { int dims[] = >> {this->nGlobRows, this->nGlobCols}; int dargs[] = {this->nBlockRows, >> this->nBlockCols}; int distribs[] = {MPI_DISTRIBUTE_CYCLIC, >> MPI_DISTRIBUTE_CYCLIC}; int dim[] = {communicator->nProcRows, >> communicator->nProcCols}; char nat[] = "native"; int rc; >> MPI_Datatype dcarray; MPI_File cFile; MPI_Status status; >> >> MPI_Type_create_darray(communicator->mpiNumTasks, >> communicator->mpiRank, 2, dims, distribs, dargs, dim, >> MPI_ORDER_FORTRAN, MPI_DOUBLE, &dcarray); MPI_Type_commit(&dcarray); >> >> std::vector<char> fn(fname.begin(), fname.end()); fn.push_back('\0'); >> rc = MPI_File_open(MPI_COMM_WORLD, &fn[0], MPI_MODE_EXCL | >> MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &cFile); if(rc){ >> ss << "Error: Failed to open file: " << rc; >> misc.error(ss.str(), 0); >> } >> else >> { >> MPI_File_set_view(cFile, 0, MPI_DOUBLE, dcarray, nat, MPI_INFO_NULL); >> MPI_File_write_all(cFile, this->m, this->nRows*this->nCols, MPI_DOUBLE, &status); >> } >> MPI_File_close(&cFile); >> MPI_Type_free(&dcarray); >> } >> >> If I call the function a repeated number of times without deleting the file between calls, the file seems to be always correct. If I delete the file between calls or create a file with a different name each time, then I see the error in some of the files. The error is a nan always in the same position. For me it is very strange, but as I said, with other implementations, I do not have the problem. >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fault tolerance MPICH2 hydra vs MPICH3.0.4 hydra
by Anatoly G 19 May '14

19 May '14

Hi Wesley. Thank you very much for quick response. I executed your's code. Master can't finish it's execution. It stalled on MPI_Wait on iteration 7. But if I use MPICH2 hydra, Master process will finish executing by reporting number of times on failure of slave process. Can you please advice if it's safe to make hybrid system of build with MPICH3.0.4, but using MPICH2 hydra? Or may be any other solution. Does MPICH 3.0.4 include all MPICH2 hydra functionality? May be my configuration of MPICH 3.0.4 is wrong? Regards, Anatoly. On Sun, May 18, 2014 at 8:40 PM, Wesley Bland <wbland(a)anl.gov> wrote: > Hi Anatoly, > > I think the problem may be the way that you're aborting. MPICH catches the > system abort call and kills the entire application when it's called. > Instead, I suggest using MPI_Abort(MPI_COMM_WORLD, 1); That's what I use in > my tests and it works fine. It also seemed to work for your code when I > tried. I'll attach my modified version of your code. I switched it to C > since I happened to have C++ support disabled on my local install, but that > shouldn't change anything. > > Thanks, > Wesley > > > On Sun, May 18, 2014 at 5:18 AM, Anatoly G <anatolyrishon(a)gmail.com>wrote: > >> Dear MPICH2, >> Can you please help me with understanding Fault Tolerance in MPICH3.0.4 >> I have a simple MPI program: >> Master calls MPI_Irecv + MPI_Wait in loop. >> Single slave: calls MPI_Send x 5 times, then calls abort. >> >> When I execute program with MPICH2 hydra I get multiple times Master >> process prints about fail in slave. In MPICH3 hydra I get a single message >> about fail of slave and then Master process enters to endless wait for next >> Irecv completion. >> In both cases I compiled program with MPICH3.0.4 >> >> In other words, with MPICH2 hydra each Irecv completes (even if slave >> died before execution of Irecv) but in MPICH3 hydra not. Causes MPI_Irecvendless wait. >> >> If I compile same program with MPICH2 and use MPICH2 hydra, I get the >> same result as compiling with MPICH3.0.4 and running with MPICH2 hydra. >> >> Execution command: >> mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines1.txt-launcher= >> rsh -n 2 mpi_irecv_ft_simple >> >> >> Both hydra's configured with: >> $ ./configure --prefix=/space/local/mpich-3.0.4/ >> --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=- >> fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview--enable-static --disable-f77 --disable- >> fc --no-recursion >> >> $ ./configure --prefix=/space/local/mpich2-1.5b2/ >> --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=- >> fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview--enable-static --disable-f77 --disable- >> fc >> >> Can you advice please? >> >> Regards, >> Anatoly. >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fault tolerance MPICH2 hydra vs MPICH3.0.4 hydra
by Wesley Bland 18 May '14

18 May '14

Hi Anatoly, I think the problem may be the way that you're aborting. MPICH catches the system abort call and kills the entire application when it's called. Instead, I suggest using MPI_Abort(MPI_COMM_WORLD, 1); That's what I use in my tests and it works fine. It also seemed to work for your code when I tried. I'll attach my modified version of your code. I switched it to C since I happened to have C++ support disabled on my local install, but that shouldn't change anything. Thanks, Wesley On Sun, May 18, 2014 at 5:18 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Dear MPICH2, > Can you please help me with understanding Fault Tolerance in MPICH3.0.4 > I have a simple MPI program: > Master calls MPI_Irecv + MPI_Wait in loop. > Single slave: calls MPI_Send x 5 times, then calls abort. > > When I execute program with MPICH2 hydra I get multiple times Master > process prints about fail in slave. In MPICH3 hydra I get a single message > about fail of slave and then Master process enters to endless wait for next > Irecv completion. > In both cases I compiled program with MPICH3.0.4 > > In other words, with MPICH2 hydra each Irecv completes (even if slave > died before execution of Irecv) but in MPICH3 hydra not. Causes MPI_Irecvendless wait. > > If I compile same program with MPICH2 and use MPICH2 hydra, I get the > same result as compiling with MPICH3.0.4 and running with MPICH2 hydra. > > Execution command: > mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines1.txt-launcher= > rsh -n 2 mpi_irecv_ft_simple > > > Both hydra's configured with: > $ ./configure --prefix=/space/local/mpich-3.0.4/ > --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC > FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static > --disable-f77 --disable-fc --no-recursion > > $ ./configure --prefix=/space/local/mpich2-1.5b2/ > --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC > FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static > --disable-f77 --disable-fc > > Can you advice please? > > Regards, > Anatoly. >

1 0

Fault tolerance MPICH2 hydra vs MPICH3.0.4 hydra
by Anatoly G 18 May '14

18 May '14

Dear MPICH2, Can you please help me with understanding Fault Tolerance in MPICH3.0.4 I have a simple MPI program: Master calls MPI_Irecv + MPI_Wait in loop. Single slave: calls MPI_Send x 5 times, then calls abort. When I execute program with MPICH2 hydra I get multiple times Master process prints about fail in slave. In MPICH3 hydra I get a single message about fail of slave and then Master process enters to endless wait for next Irecv completion. In both cases I compiled program with MPICH3.0.4 In other words, with MPICH2 hydra each Irecv completes (even if slave died before execution of Irecv) but in MPICH3 hydra not. Causes MPI_Irecvendless wait. If I compile same program with MPICH2 and use MPICH2 hydra, I get the same result as compiling with MPICH3.0.4 and running with MPICH2 hydra. Execution command: mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines1.txt-launcher= rsh -n 2 mpi_irecv_ft_simple Both hydra's configured with: $ ./configure --prefix=/space/local/mpich-3.0.4/ --enable-error-checking= runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic--enable-threads= runtime --enable-totalview --enable-static --disable-f77 --disable-fc--no-recursion $ ./configure --prefix=/space/local/mpich2-1.5b2/ --enable-error-checking= runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic--enable-threads= runtime --enable-totalview --enable-static --disable-f77 --disable-fc Can you advice please? Regards, Anatoly.

1 0