- devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPICH2 hang
by Jim Dinan 15 Dec '12

15 Dec '12

Hi Bob, I think I have a fix for detecting this error, and I should be able to send it along next week. AFAIK, MPICH2 has never detected the particular context ID exhaustion scenario from the test case you sent. Is it possible that the size of the context ID space was increased on the BG/P until the error went away? If that was the case (would be in src/include/mpiimpl.h where MPIR_MAX_CONTEXT_MASK is defined), it may make sense to do the same for BG/Q, in case there are apps that need more communicators than the default limit. Best, ~Jim. On 12/15/12 9:23 AM, Bob Cernohous wrote: > I didn't try it myself on BG/P. All I really know is someone at ANL > (possibly Nick?) reported (about ftmain.f90) : > > > This programs create as many communicators as there are MPI tasks. It is > a bad program provided by a user. On BG/P, this type of program threw a > proper warning. It should do the same on BG/Q. > > Business impact ( BusImpact ) > It is mostly a nuisance to those who don't understand the inherent > limitations in MPICH2. > > > I don't have access to PMR's but it was *41473,122,000 . *The CPS issue > that was opened to me from that PMR had no other details. > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > > > From: Jim Dinan <dinan(a)mcs.anl.gov> > To: devel(a)mpich.org, > Date: 12/14/2012 11:52 PM > Subject: Re: [mpich-devel] MPICH2 hang > Sent by: devel-bounces(a)mpich.org > ------------------------------------------------------------------------ > > > > Hi Bob, > > The ftmain2.f90 test fails on MPICH2 1.2.1p1, which was released on > 2-22-2010, well before any of the MPI-3 changes. Could you provide some > more information on when this test was reporting a failure instead of > hanging? > > It looks like this test case generates a context ID exhaustion pattern > where context IDs are available at all processes, but the processes have > no free context IDs in common. Because there is no common context ID > available, allocation can't succeed and it loops indefinitely. This is > a resource exhaustion pattern that AFAIK, MPICH has not detected in the > past. > > I attached for reference a C translation of this test that is a little > easier to grok and also fails on MPICH, going back to MPICH2 1.2.1p1. > > ~Jim. > > On 12/14/12 11:27 AM, Jim Dinan wrote: > > Hi Bob, > > > > Thanks for the detailed bug report and test cases. I confirmed that the > > failure you are seeing on the MPICH trunk. This is likely related to > > changes we made to support MPI-3 MPI_Comm_create_group(). I created a > > ticket to track this: > > > > https://trac.mpich.org/projects/mpich/ticket/1768 > > > > ~Jim. > > > > On 12/12/12 5:38 PM, Bob Cernohous wrote: > >> > >> I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. > >> > >> It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on BG/Q. > >> > >> It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. > >> > >> On older mpich 1.? (BG/P) it failed with 'too many communicators' and > >> didn't hang, which is what they expected. > >> > >> It seems like it's stuck in the while (*context_id == 0) loop > >> repeatedly calling allreduce and never settling on a context id in > >> commutil.c. I didn't do a lot of debug but seems like it's in > >> vanilla mpich code, not something we modified. > >> > >> ftmain.f90 fails if you run it on >2k ranks (creates one comm per > >> rank). This was the original customer testcase. > >> > >> ftmain2.f90 fails by looping so you can run on fewer ranks. > >> > >> > >> > >> > >> I just noticed that with --np 1, I get the 'too many communicators' from > >> ftmain2. But --np 2 and up hangs. > >> > >> stdout[0]: check_newcomm do-start 0 , repeat 2045 , total > >> 2046 > >> stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error > >> in PMPI_Comm_create: Other MPI error, error stack: > >> stderr[0]: PMPI_Comm_create(609).........: > >> MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, new_comm=0x1dbfffb520) > >> failed > >> stderr[0]: PMPI_Comm_create(590).........: > >> stderr[0]: MPIR_Comm_create_intra(250)...: > >> stderr[0]: MPIR_Get_contextid(521).......: > >> stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators > [attachment "too_many_comms3.c" deleted by Bob Cernohous/Rochester/IBM]

1 0

Re: [mpich-devel] MPICH2 hang
by Bob Cernohous 15 Dec '12

15 Dec '12

I didn't try it myself on BG/P. All I really know is someone at ANL (possibly Nick?) reported (about ftmain.f90) : This programs create as many communicators as there are MPI tasks. It is a bad program provided by a user. On BG/P, this type of program threw a proper warning. It should do the same on BG/Q. Business impact ( BusImpact ) It is mostly a nuisance to those who don't understand the inherent limitations in MPICH2. I don't have access to PMR's but it was 41473,122,000 . The CPS issue that was opened to me from that PMR had no other details. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. From: Jim Dinan <dinan(a)mcs.anl.gov> To: devel(a)mpich.org, Date: 12/14/2012 11:52 PM Subject: Re: [mpich-devel] MPICH2 hang Sent by: devel-bounces(a)mpich.org Hi Bob, The ftmain2.f90 test fails on MPICH2 1.2.1p1, which was released on 2-22-2010, well before any of the MPI-3 changes. Could you provide some more information on when this test was reporting a failure instead of hanging? It looks like this test case generates a context ID exhaustion pattern where context IDs are available at all processes, but the processes have no free context IDs in common. Because there is no common context ID available, allocation can't succeed and it loops indefinitely. This is a resource exhaustion pattern that AFAIK, MPICH has not detected in the past. I attached for reference a C translation of this test that is a little easier to grok and also fails on MPICH, going back to MPICH2 1.2.1p1. ~Jim. On 12/14/12 11:27 AM, Jim Dinan wrote: > Hi Bob, > > Thanks for the detailed bug report and test cases. I confirmed that the > failure you are seeing on the MPICH trunk. This is likely related to > changes we made to support MPI-3 MPI_Comm_create_group(). I created a > ticket to track this: > > https://trac.mpich.org/projects/mpich/ticket/1768 > > ~Jim. > > On 12/12/12 5:38 PM, Bob Cernohous wrote: >> >> I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. >> >> It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on BG/Q. >> >> It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. >> >> On older mpich 1.? (BG/P) it failed with 'too many communicators' and >> didn't hang, which is what they expected. >> >> It seems like it's stuck in the while (*context_id == 0) loop >> repeatedly calling allreduce and never settling on a context id in >> commutil.c. I didn't do a lot of debug but seems like it's in >> vanilla mpich code, not something we modified. >> >> ftmain.f90 fails if you run it on >2k ranks (creates one comm per >> rank). This was the original customer testcase. >> >> ftmain2.f90 fails by looping so you can run on fewer ranks. >> >> >> >> >> I just noticed that with --np 1, I get the 'too many communicators' from >> ftmain2. But --np 2 and up hangs. >> >> stdout[0]: check_newcomm do-start 0 , repeat 2045 , total >> 2046 >> stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error >> in PMPI_Comm_create: Other MPI error, error stack: >> stderr[0]: PMPI_Comm_create(609).........: >> MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, new_comm=0x1dbfffb520) >> failed >> stderr[0]: PMPI_Comm_create(590).........: >> stderr[0]: MPIR_Comm_create_intra(250)...: >> stderr[0]: MPIR_Get_contextid(521).......: >> stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators [attachment "too_many_comms3.c" deleted by Bob Cernohous/Rochester/IBM]

1 0

Re: [mpich-devel] MPICH2 hang
by Jim Dinan 14 Dec '12

14 Dec '12

Hi Bob, The ftmain2.f90 test fails on MPICH2 1.2.1p1, which was released on 2-22-2010, well before any of the MPI-3 changes. Could you provide some more information on when this test was reporting a failure instead of hanging? It looks like this test case generates a context ID exhaustion pattern where context IDs are available at all processes, but the processes have no free context IDs in common. Because there is no common context ID available, allocation can't succeed and it loops indefinitely. This is a resource exhaustion pattern that AFAIK, MPICH has not detected in the past. I attached for reference a C translation of this test that is a little easier to grok and also fails on MPICH, going back to MPICH2 1.2.1p1. ~Jim. On 12/14/12 11:27 AM, Jim Dinan wrote: > Hi Bob, > > Thanks for the detailed bug report and test cases. I confirmed that the > failure you are seeing on the MPICH trunk. This is likely related to > changes we made to support MPI-3 MPI_Comm_create_group(). I created a > ticket to track this: > > https://trac.mpich.org/projects/mpich/ticket/1768 > > ~Jim. > > On 12/12/12 5:38 PM, Bob Cernohous wrote: >> >> I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. >> >> It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on BG/Q. >> >> It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. >> >> On older mpich 1.? (BG/P) it failed with 'too many communicators' and >> didn't hang, which is what they expected. >> >> It seems like it's stuck in the while (*context_id == 0) loop >> repeatedly calling allreduce and never settling on a context id in >> commutil.c. I didn't do a lot of debug but seems like it's in >> vanilla mpich code, not something we modified. >> >> ftmain.f90 fails if you run it on >2k ranks (creates one comm per >> rank). This was the original customer testcase. >> >> ftmain2.f90 fails by looping so you can run on fewer ranks. >> >> >> >> >> I just noticed that with --np 1, I get the 'too many communicators' from >> ftmain2. But --np 2 and up hangs. >> >> stdout[0]: check_newcomm do-start 0 , repeat 2045 , total >> 2046 >> stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error >> in PMPI_Comm_create: Other MPI error, error stack: >> stderr[0]: PMPI_Comm_create(609).........: >> MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, new_comm=0x1dbfffb520) >> failed >> stderr[0]: PMPI_Comm_create(590).........: >> stderr[0]: MPIR_Comm_create_intra(250)...: >> stderr[0]: MPIR_Get_contextid(521).......: >> stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators

1 0

Re: [mpich-devel] MPICH2 hang
by Jim Dinan 14 Dec '12

14 Dec '12

Hi Bob, Thanks for the detailed bug report and test cases. I confirmed that the failure you are seeing on the MPICH trunk. This is likely related to changes we made to support MPI-3 MPI_Comm_create_group(). I created a ticket to track this: https://trac.mpich.org/projects/mpich/ticket/1768 ~Jim. On 12/12/12 5:38 PM, Bob Cernohous wrote: > > I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. > > It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on BG/Q. > > It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. > > On older mpich 1.? (BG/P) it failed with 'too many communicators' and > didn't hang, which is what they expected. > > It seems like it's stuck in the while (*context_id == 0) loop > repeatedly calling allreduce and never settling on a context id in > commutil.c. I didn't do a lot of debug but seems like it's in > vanilla mpich code, not something we modified. > > ftmain.f90 fails if you run it on >2k ranks (creates one comm per > rank). This was the original customer testcase. > > ftmain2.f90 fails by looping so you can run on fewer ranks. > > > > > I just noticed that with --np 1, I get the 'too many communicators' from > ftmain2. But --np 2 and up hangs. > > stdout[0]: check_newcomm do-start 0 , repeat 2045 , total > 2046 > stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error > in PMPI_Comm_create: Other MPI error, error stack: > stderr[0]: PMPI_Comm_create(609).........: > MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, new_comm=0x1dbfffb520) > failed > stderr[0]: PMPI_Comm_create(590).........: > stderr[0]: MPIR_Comm_create_intra(250)...: > stderr[0]: MPIR_Get_contextid(521).......: > stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators

1 0

MPICH2 hang
by Bob Cernohous 12 Dec '12

12 Dec '12

I've had a hang reported on BG/Q after about 2K MPI_Comm_create's. It hangs on the latest 2 releases (mpich2 v1.5.x and v1.4.x) on BG/Q. It also hangs on linux: 64bit (MPI over PAMI) MPICH2 library. On older mpich 1.? (BG/P) it failed with 'too many communicators' and didn't hang, which is what they expected. It seems like it's stuck in the while (*context_id == 0) loop repeatedly calling allreduce and never settling on a context id in commutil.c. I didn't do a lot of debug but seems like it's in vanilla mpich code, not something we modified. ftmain.f90 fails if you run it on >2k ranks (creates one comm per rank). This was the original customer testcase. ftmain2.f90 fails by looping so you can run on fewer ranks. I just noticed that with --np 1, I get the 'too many communicators' from ftmain2. But --np 2 and up hangs. stdout[0]: check_newcomm do-start 0 , repeat 2045 , total 2046 stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error in PMPI_Comm_create: Other MPI error, error stack: stderr[0]: PMPI_Comm_create(609).........: MPI_Comm_create(MPI_COMM_WORLD, group=0xc80700f6, new_comm=0x1dbfffb520) failed stderr[0]: PMPI_Comm_create(590).........: stderr[0]: MPIR_Comm_create_intra(250)...: stderr[0]: MPIR_Get_contextid(521).......: stderr[0]: MPIR_Get_contextid_sparse(752): Too many communicators

1 0

Re: [mpich-devel] LLONG_MAX undeclared in status_set_elements_x.c ?
by Dave Goodell 29 Nov '12

29 Nov '12

Howard, what configure options are you using? We've seen a problem like this when "--enable-strict=c89" is passed, since the features.h header deliberately avoids defining LLONG_MAX when in C89 mode. I haven't had the time to come up with a fix for this yet. -Dave On Nov 29, 2012, at 3:08 PM CST, Howard Pritchard wrote: > HI Folks, > > I've done a fresh update of mpich2 trunk on my sles 11 machine, done a new autogen.sh > and config of mpich. But when I compile, I get this error in this fairly recently introduced > file: > > src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': > src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) > src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once > src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) > make[2]: *** [src/mpi/datatype/status_set_elements_x.lo] Error 1 > > It looks like mpiimpl.h isn't pulling in limits.h. > > Howard > > -- > Howard Pritchard > Software Engineering > Cray, Inc. >

1 0

LLONG_MAX undeclared in status_set_elements_x.c ?
by Howard Pritchard 29 Nov '12

29 Nov '12

HI Folks, I've done a fresh update of mpich2 trunk on my sles 11 machine, done a new autogen.sh and config of mpich. But when I compile, I get this error in this fairly recently introduced file: src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) make[2]: *** [src/mpi/datatype/status_set_elements_x.lo] Error 1 It looks like mpiimpl.h isn't pulling in limits.h. Howard -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [mpich-devel] \\Re: 404 — File not found. mpich2-1.5.tar.gz
by Pavan Balaji 23 Nov '12

23 Nov '12

On 11/23/2012 08:00 AM US Central Time, Donald Moore wrote: > I find found some more issues with downloads BOF slides . (low > priority) This was fixed a few hours ago. I'm able to download the files fine. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

\\Re: 404 — File not found. mpich2-1.5.tar.gz
by Donald Moore 23 Nov '12

23 Nov '12

Dave-- I find found some more issues with downloads BOF slides . (low priority) http://www.mpich.org/2012/11/16/mpich-bof-slides-from-sc12-available/ wget http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-Intro.pdf --2012-11-23 07:58:17-- http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-Intro.pdf Resolving www.mpich.org (www.mpich.org) 140.221.6.71 Connecting to www.mpich.org (www.mpich.org)|140.221.6.71|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-11-23 07:58:17 ERROR 403: Forbidden. wget http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-20_Years… --2012-11-23 07:58:41-- http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-20_Years… Resolving www.mpich.org (www.mpich.org) 140.221.6.71 Connecting to www.mpich.org (www.mpich.org)|140.221.6.71|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-11-23 07:58:41 ERROR 403: Forbidden. wget http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-30_and_B… --2012-11-23 07:59:00-- http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-30_and_B… Resolving www.mpich.org (www.mpich.org) 140.221.6.71 Connecting to www.mpich.org (www.mpich.org)|140.221.6.71|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-11-23 07:59:00 ERROR 403: Forbidden. wget http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-IBM.pdf --2012-11-23 07:59:16-- http://www.mpich.org/static/docs/slides/2012-sc-bof/MPICH_BOF_SC12-IBM.pdf Resolving www.mpich.org (www.mpich.org) 140.221.6.71 Connecting to www.mpich.org (www.mpich.org)|140.221.6.71|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-11-23 07:59:17 ERROR 403: Forbidden. ----- Original Message ----- From: "Dave Goodell" <goodell(a)mcs.anl.gov> To: devel(a)mpich.org Sent: Monday, November 19, 2012 2:07:26 PM Subject: Re: [mpich-devel] 404 — File not found. mpich2-1.5.tar.gz I've fixed this now. We are still getting used to our new content management system and continue to make occasional mistakes. We appreciate the bug reports. Thanks, -Dave On Nov 19, 2012, at 1:00 PM CST, Donald Moore wrote: > > Hi- > > I can't download mpich2-1.5.tar.gz since the recent update ( when mpich-3.0rc1 added_) > > -/Don > wget http://www.mpich.org/files/tarballs1.5/mpich2-1.5.tar.gz > --2012-11-19 12:54:14-- http://www.mpich.org/files/tarballs1.5/mpich2-1.5.tar.gz > Resolving www.mpich.org... 140.221.6.71 > Connecting to www.mpich.org|140.221.6.71|:80... connected. > HTTP request sent, awaiting response... 404 Not Found > 2012-11-19 12:54:14 ERROR 404: Not Found. > --- > wget http://www.mpich.org/files/tarballs1.5/hydra-1.5.tar.gz > --2012-11-19 12:59:05-- http://www.mpich.org/files/tarballs1.5/hydra-1.5.tar.gz > Resolving www.mpich.org... 140.221.6.71 > Connecting to www.mpich.org|140.221.6.71|:80... connected. > HTTP request sent, awaiting response... 404 Not Found > 2012-11-19 12:59:05 ERROR 404: Not Found. > > --- > wget http://www.mpich.org/files/tarballs/3.0rc1/mpich-3.0rc1.tar.gz > --2012-11-19 12:58:20-- http://www.mpich.org/files/tarballs/3.0rc1/mpich-3.0rc1.tar.gz > Resolving www.mpich.org... 140.221.6.71 > Connecting to www.mpich.org|140.221.6.71|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 11107912 (11M) [application/x-gzip] > Saving to: “mpich-3.0rc1.tar.gz” > > 60% [======================> ] 6,666,208 1.68M/s eta 3s

1 0

Re: [mpich-devel] MPICH 3.0rc1 on Windows
by Dave Goodell 21 Nov '12

21 Nov '12

On Nov 21, 2012, at 4:17 PM CST, Lisandro Dalcin wrote: > Is it possible to get a build of 3.0rc1 for Windows 32bit and 64bit? I > would like to test my updates to mpi4py on these platforms. Our Windows development situation has changed over the past year, and we do not have any estimate for when Windows support/builds may become available again. It certainly will not happen within the next few weeks. -Dave

1 0