- devel - mailman3.cels.anl.gov

Re: [mpich-devel] datatype accessors with uncommited datatypes
by Jeff Hammond 27 Dec '12

27 Dec '12

On Thu, Dec 27, 2012 at 1:01 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 12/27/2012 12:56 PM US Central Time, Jeff Hammond wrote: >> I don't disagree with your interpretation, but I am always hesitant to >> assert my understanding of the text in the absence of input from those >> who were present when it was written. >> >> I'll file an MPI Forum ticket and see what happens. > > This does look like something that's ill-defined in the standard. A > ticket to address in MPI-3.1 is a good idea. > > Lisandro: till this is fixed in the standard, the workaround will be to > commit all datatypes. See https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/356. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] datatype accessors with uncommited datatypes
by Pavan Balaji 27 Dec '12

27 Dec '12

On 12/27/2012 12:56 PM US Central Time, Jeff Hammond wrote: > I don't disagree with your interpretation, but I am always hesitant to > assert my understanding of the text in the absence of input from those > who were present when it was written. > > I'll file an MPI Forum ticket and see what happens. This does look like something that's ill-defined in the standard. A ticket to address in MPI-3.1 is a good idea. Lisandro: till this is fixed in the standard, the workaround will be to commit all datatypes. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-devel] datatype accessors with uncommited datatypes
by Jeff Hammond 27 Dec '12

27 Dec '12

I don't disagree with your interpretation, but I am always hesitant to assert my understanding of the text in the absence of input from those who were present when it was written. I'll file an MPI Forum ticket and see what happens. Jeff On Thu, Dec 27, 2012 at 12:48 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > On 27 December 2012 15:36, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> This is what the MPI standard says: >> >> 4.1.9 Commit and Free >> A datatype object has to be committed before it can be used in a >> communication. As an argument in datatype constructors, uncommitted >> and also committed datatypes can be used. There is no need to commit >> basic datatypes. They are “pre-committed.” >> >> There is some ambiguity since the functions to which you refer lie >> outside of the union of "communication" and "datatype constructors". >> However, since MPI_TYPE_SIZE(_X) determine "the total size of the data >> in a message that would be created with this datatype", I wonder if >> there is an implementation wherein the "compilation" that can >> potentially occur during MPI_TYPE_COMMIT is required in order to >> satisfy MPI_TYPE_SIZE(_X). >> > > I would argue that a "datatype constructor" is a much more complex > operation that a "datatype accessor", so if construction should work > with uncommited, I would expect accessors to work, too. Moreover, the > current implementation of MPI_Type_size() seems to agree with my POV. > >> Is it necessary for mpi4py to have the behavior of the older version >> of MPICH to function properly or can you work around this for now? >> > > It is not a problem of mpi4py, actually, buy user's code out there > that could potentially break (BTW, this issue also applies to C and > Fortran codes). > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] datatype accessors with uncommited datatypes
by Lisandro Dalcin 27 Dec '12

27 Dec '12

On 27 December 2012 15:36, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > This is what the MPI standard says: > > 4.1.9 Commit and Free > A datatype object has to be committed before it can be used in a > communication. As an argument in datatype constructors, uncommitted > and also committed datatypes can be used. There is no need to commit > basic datatypes. They are “pre-committed.” > > There is some ambiguity since the functions to which you refer lie > outside of the union of "communication" and "datatype constructors". > However, since MPI_TYPE_SIZE(_X) determine "the total size of the data > in a message that would be created with this datatype", I wonder if > there is an implementation wherein the "compilation" that can > potentially occur during MPI_TYPE_COMMIT is required in order to > satisfy MPI_TYPE_SIZE(_X). > I would argue that a "datatype constructor" is a much more complex operation that a "datatype accessor", so if construction should work with uncommited, I would expect accessors to work, too. Moreover, the current implementation of MPI_Type_size() seems to agree with my POV. > Is it necessary for mpi4py to have the behavior of the older version > of MPICH to function properly or can you work around this for now? > It is not a problem of mpi4py, actually, buy user's code out there that could potentially break (BTW, this issue also applies to C and Fortran codes). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] datatype accessors with uncommited datatypes
by Jeff Hammond 27 Dec '12

27 Dec '12

This is what the MPI standard says: 4.1.9 Commit and Free A datatype object has to be committed before it can be used in a communication. As an argument in datatype constructors, uncommitted and also committed datatypes can be used. There is no need to commit basic datatypes. They are “pre-committed.” There is some ambiguity since the functions to which you refer lie outside of the union of "communication" and "datatype constructors". However, since MPI_TYPE_SIZE(_X) determine "the total size of the data in a message that would be created with this datatype", I wonder if there is an implementation wherein the "compilation" that can potentially occur during MPI_TYPE_COMMIT is required in order to satisfy MPI_TYPE_SIZE(_X). Is it necessary for mpi4py to have the behavior of the older version of MPICH to function properly or can you work around this for now? Best, Jeff On Thu, Dec 27, 2012 at 10:50 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > My understanding of the MPI standard is that datatype committing is > only required for communication, but not for calls like > MPI_Type_size_x(). I'm testing mpi4py with mpich-3.0.1, and getting > the following errors (IIRC, 3.0rc1 did not have this issue) > > mpi4py.MPI.Exception: Invalid datatype, error stack: > PMPI_Type_size_x(125): MPI_Type_size_x(datatype=dtype=USER<vector>, > size=0x7fffeb4ef658) > PMPI_Type_size_x(98).: Datatype has not been committed > > > mpi4py.MPI.Exception: Invalid datatype, error stack: > PMPI_Type_get_extent_x(128): > MPI_Type_get_extent_x(datatype=dtype=USER<struct>, lb=0x7fff97046bb0, > extent=0x7fff97046bb8) > PMPI_Type_get_extent_x(102): Datatype has not been committed > > Looking more closely to type_size_x.c and type_size.c in > src/mpi/datatype/, I see MPID_Datatype_committed_ptr() in the former, > but not in the later. IMHO, these checks should be removed from > type_size_x.c, type_get_extent_x.c, and type_get_true_extent_x.c > (please not I'm not listing get_elements_x.c, this is a different > beast) > > Thanks, > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

datatype accessors with uncommited datatypes
by Lisandro Dalcin 27 Dec '12

27 Dec '12

My understanding of the MPI standard is that datatype committing is only required for communication, but not for calls like MPI_Type_size_x(). I'm testing mpi4py with mpich-3.0.1, and getting the following errors (IIRC, 3.0rc1 did not have this issue) mpi4py.MPI.Exception: Invalid datatype, error stack: PMPI_Type_size_x(125): MPI_Type_size_x(datatype=dtype=USER<vector>, size=0x7fffeb4ef658) PMPI_Type_size_x(98).: Datatype has not been committed mpi4py.MPI.Exception: Invalid datatype, error stack: PMPI_Type_get_extent_x(128): MPI_Type_get_extent_x(datatype=dtype=USER<struct>, lb=0x7fff97046bb0, extent=0x7fff97046bb8) PMPI_Type_get_extent_x(102): Datatype has not been committed Looking more closely to type_size_x.c and type_size.c in src/mpi/datatype/, I see MPID_Datatype_committed_ptr() in the former, but not in the later. IMHO, these checks should be removed from type_size_x.c, type_get_extent_x.c, and type_get_true_extent_x.c (please not I'm not listing get_elements_x.c, this is a different beast) Thanks, -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] LLONG_MAX undeclared in status_set_elements_x.c ?
by William Gropp 21 Dec '12

21 Dec '12

Thanks. This test erroneously uses long long and LLONG_MAX without ensuring that they are available (and for which the configure for the tests already checks). I'll fix it. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Dec 21, 2012, at 2:25 AM, P.-H. Lin wrote: > Hello Howard, > I have encountered the same problem with mpich-3.0.1 today. (And > that's why I found this thread :P) > After raising the optimization level, it didn't complain about that > undeclared problem anymore, and the installation goes well. > This is my configure option: > ./configure --prefix /opt/app/MPICH --enable-fast=all,O3 > > BTW, the problem still exist for "indexed-misc.c" under the test directory > > Regards

1 0

LLONG_MAX undeclared in status_set_elements_x.c ?
by P.-H. Lin 21 Dec '12

21 Dec '12

Hello Howard, I have encountered the same problem with mpich-3.0.1 today. (And that's why I found this thread :P) After raising the optimization level, it didn't complain about that undeclared problem anymore, and the installation goes well. This is my configure option: ./configure --prefix /opt/app/MPICH --enable-fast=all,O3 BTW, the problem still exist for "indexed-misc.c" under the test directory Regards

1 0

Re: [mpich-devel] MPICH2 hang
by Jim Dinan 17 Dec '12

17 Dec '12

Hi Bob, I committed a fix to the MPICH trunk in r10771 and several tests cases in r10772. I tried out the test cases that you sent, and with this fix, MPICH detects context ID exhaustion rather than hanging. Best, ~Jim. On 12/15/12 11:39 PM, Jim Dinan wrote: > Hi Bob, > > I think I have a fix for detecting this error, and I should be able to > send it along next week. > > AFAIK, MPICH2 has never detected the particular context ID exhaustion > scenario from the test case you sent. Is it possible that the size of > the context ID space was increased on the BG/P until the error went > away? If that was the case (would be in src/include/mpiimpl.h where > MPIR_MAX_CONTEXT_MASK is defined), it may make sense to do the same for > BG/Q, in case there are apps that need more communicators than the > default limit. > > Best, > ~Jim. > > On 12/15/12 9:23 AM, Bob Cernohous wrote: >> I didn't try it myself on BG/P. All I really know is someone at ANL >> (possibly Nick?) reported (about ftmain.f90) : >> >> >> This programs create as many communicators as there are MPI tasks. It is >> a bad program provided by a user. On BG/P, this type of program threw a >> proper warning. It should do the same on BG/Q. >> >> Business impact ( BusImpact ) >> It is mostly a nuisance to those who don't understand the inherent >> limitations in MPICH2. >> >> >> I don't have access to PMR's but it was *41473,122,000 . *The CPS issue >> that was opened to me from that PMR had no other details. >> >> Bob Cernohous: (T/L 553) 507-253-6093 >> >> BobC(a)us.ibm.com >> IBM Rochester, Building 030-2(C335), Department 61L >> 3605 Hwy 52 North, Rochester, MN 55901-7829 >> >> > Chaos reigns within. >> > Reflect, repent, and reboot. >> > Order shall return. >> >> >> >> >> From: Jim Dinan <dinan(a)mcs.anl.gov> >> To: devel(a)mpich.org, >> Date: 12/14/2012 11:52 PM >> Subject: Re: [mpich-devel] MPICH2 hang >> Sent by: devel-bounces(a)mpich.org >> ------------------------------------------------------------------------ >> >> >> >> Hi Bob, >> >> The ftmain2.f90 test fails on MPICH2 1.2.1p1, which was released on >> 2-22-2010, well before any of the MPI-3 changes. Could you provide some >> more information on when this test was reporting a failure instead of >> hanging? >> >> It looks like this test case generates a context ID exhaustion pattern >> where context IDs are available at all processes, but the processes have >> no free context IDs in common. Because there is no common context ID >> available, allocation can't succeed and it loops indefinitely. This is >> a resource exhaustion pattern that AFAIK, MPICH has not detected in the >> past. >> >> I attached for reference a C translation of this test that is a little >> easier to grok and also fails on MPICH, going back to MPICH2 1.2.1p1. >> >> ~Jim. >> >> On 12/14/12 11:27 AM, Jim Dinan wrote: >> > Hi Bob, >> > >> > Thanks for the detailed bug report and test cases. I confirmed >> that the >> > failure you are seeing on the MPICH trunk. This is likely related to >> > changes we made to support MPI-3 MPI_Comm_create_group(). I created a >> > ticket to track this: >> > >> > https://trac.mpich.org/projects/mpich/ticket/1768 >> > >> > ~Jim. >> > >> > On 12/12/12 5:38 PM, Bob Cernohous wrote: >> >> >> >> I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. >> >> >> >> It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on >> BG/Q. >> >> >> >> It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. >> >> >> >> On older mpich 1.? (BG/P) it failed with 'too many >> communicators' and >> >> didn't hang, which is what they expected. >> >> >> >> It seems like it's stuck in the while (*context_id == 0) loop >> >> repeatedly calling allreduce and never settling on a context id in >> >> commutil.c. I didn't do a lot of debug but seems like it's in >> >> vanilla mpich code, not something we modified. >> >> >> >> ftmain.f90 fails if you run it on >2k ranks (creates one comm per >> >> rank). This was the original customer testcase. >> >> >> >> ftmain2.f90 fails by looping so you can run on fewer ranks. >> >> >> >> >> >> >> >> >> >> I just noticed that with --np 1, I get the 'too many >> communicators' from >> >> ftmain2. But --np 2 and up hangs. >> >> >> >> stdout[0]: check_newcomm do-start 0 , repeat 2045 , >> total >> >> 2046 >> >> stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal >> error >> >> in PMPI_Comm_create: Other MPI error, error stack: >> >> stderr[0]: PMPI_Comm_create(609).........: >> >> MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, >> new_comm=0x1dbfffb520) >> >> failed >> >> stderr[0]: PMPI_Comm_create(590).........: >> >> stderr[0]: MPIR_Comm_create_intra(250)...: >> >> stderr[0]: MPIR_Get_contextid(521).......: >> >> stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators >> [attachment "too_many_comms3.c" deleted by Bob Cernohous/Rochester/IBM]

1 0

Re: [mpich-devel] MPICH2 hang
by Jeff Hammond 16 Dec '12

16 Dec '12

I am copying Nick on this since he was the one who reported the PMR to IBM. My understanding of MPICH is that increasing the maximum communicator count is not easy. I recall reading an article by Cray on how they had to reduce the max to accomodate extra bits in order to run MPI at scale on Jaguar. I agree that this should not hang; rather, it should fail with an appropriate error message. On the other hand, I don't think that any effort should be put forth to accomodate users who want thousands of communicators. Jeff On Sat, Dec 15, 2012 at 11:39 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > Hi Bob, > > I think I have a fix for detecting this error, and I should be able to send > it along next week. > > AFAIK, MPICH2 has never detected the particular context ID exhaustion > scenario from the test case you sent. Is it possible that the size of the > context ID space was increased on the BG/P until the error went away? If > that was the case (would be in src/include/mpiimpl.h where > MPIR_MAX_CONTEXT_MASK is defined), it may make sense to do the same for > BG/Q, in case there are apps that need more communicators than the default > limit. > > Best, > ~Jim. > > > On 12/15/12 9:23 AM, Bob Cernohous wrote: >> >> I didn't try it myself on BG/P. All I really know is someone at ANL >> (possibly Nick?) reported (about ftmain.f90) : >> >> >> This programs create as many communicators as there are MPI tasks. It is >> a bad program provided by a user. On BG/P, this type of program threw a >> proper warning. It should do the same on BG/Q. >> >> Business impact ( BusImpact ) >> It is mostly a nuisance to those who don't understand the inherent >> limitations in MPICH2. >> >> >> I don't have access to PMR's but it was *41473,122,000 . *The CPS issue >> >> that was opened to me from that PMR had no other details. >> >> Bob Cernohous: (T/L 553) 507-253-6093 >> >> BobC(a)us.ibm.com >> IBM Rochester, Building 030-2(C335), Department 61L >> 3605 Hwy 52 North, Rochester, MN 55901-7829 >> >> > Chaos reigns within. >> > Reflect, repent, and reboot. >> > Order shall return. >> >> >> >> >> From: Jim Dinan <dinan(a)mcs.anl.gov> >> To: devel(a)mpich.org, >> Date: 12/14/2012 11:52 PM >> Subject: Re: [mpich-devel] MPICH2 hang >> Sent by: devel-bounces(a)mpich.org >> ------------------------------------------------------------------------ >> >> >> >> >> Hi Bob, >> >> The ftmain2.f90 test fails on MPICH2 1.2.1p1, which was released on >> 2-22-2010, well before any of the MPI-3 changes. Could you provide some >> more information on when this test was reporting a failure instead of >> hanging? >> >> It looks like this test case generates a context ID exhaustion pattern >> where context IDs are available at all processes, but the processes have >> no free context IDs in common. Because there is no common context ID >> available, allocation can't succeed and it loops indefinitely. This is >> a resource exhaustion pattern that AFAIK, MPICH has not detected in the >> past. >> >> I attached for reference a C translation of this test that is a little >> easier to grok and also fails on MPICH, going back to MPICH2 1.2.1p1. >> >> ~Jim. >> >> On 12/14/12 11:27 AM, Jim Dinan wrote: >> > Hi Bob, >> > >> > Thanks for the detailed bug report and test cases. I confirmed that >> the >> > failure you are seeing on the MPICH trunk. This is likely related to >> > changes we made to support MPI-3 MPI_Comm_create_group(). I created a >> > ticket to track this: >> > >> > https://trac.mpich.org/projects/mpich/ticket/1768 >> > >> > ~Jim. >> > >> > On 12/12/12 5:38 PM, Bob Cernohous wrote: >> >> >> >> I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. >> >> >> >> It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on >> BG/Q. >> >> >> >> It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. >> >> >> >> On older mpich 1.? (BG/P) it failed with 'too many communicators' >> and >> >> didn't hang, which is what they expected. >> >> >> >> It seems like it's stuck in the while (*context_id == 0) loop >> >> repeatedly calling allreduce and never settling on a context id in >> >> commutil.c. I didn't do a lot of debug but seems like it's in >> >> vanilla mpich code, not something we modified. >> >> >> >> ftmain.f90 fails if you run it on >2k ranks (creates one comm per >> >> rank). This was the original customer testcase. >> >> >> >> ftmain2.f90 fails by looping so you can run on fewer ranks. >> >> >> >> >> >> >> >> >> >> I just noticed that with --np 1, I get the 'too many communicators' >> from >> >> ftmain2. But --np 2 and up hangs. >> >> >> >> stdout[0]: check_newcomm do-start 0 , repeat 2045 , >> total >> >> 2046 >> >> stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error >> >> in PMPI_Comm_create: Other MPI error, error stack: >> >> stderr[0]: PMPI_Comm_create(609).........: >> >> MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, >> new_comm=0x1dbfffb520) >> >> failed >> >> stderr[0]: PMPI_Comm_create(590).........: >> >> stderr[0]: MPIR_Comm_create_intra(250)...: >> >> stderr[0]: MPIR_Get_contextid(521).......: >> >> stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators >> [attachment "too_many_comms3.c" deleted by Bob Cernohous/Rochester/IBM] -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0