- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Accumulate with MAXLOC does not error for incompatible types
by Jed Brown 14 Jan '13

14 Jan '13

Thanks, Dave. On Mon, Jan 14, 2013 at 6:39 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Fixed by http://git.mpich.org/mpich.git/commitdiff/ba2e444 > > It looks like I accidentally put the wrong URL into the commit message for > some reason. The correct mailing list URL should be: > http://lists.mpich.org/pipermail/discuss/2013-January/000227.html > > -Dave > > On Jan 1, 2013, at 1:09 PM CST, Jed Brown wrote: > > > With MPI_Reduce, we get a nice helpful error message: > > > > Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack: > > PMPI_Reduce(1217)...........: MPI_Reduce(sbuf=0x7fff3e39fa50, > rbuf=0x7fff3e39fa40, count=1, dtype=USER<contig>, MPI_MAXLOC, root=0, > MPI_COMM_WORLD) failed > > MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined > for this datatype > > > > > > With MPI_Accumulate, we just get wrong results. See attached code using > a user-defined contiguous results. > > > > I care about this because I'm stuck [1] dispatching into alternative > mechanisms for type combinations that are not directly supported by MPI > (e.g., struct {int u; long long k;} and __float128). I missed a spot and > had a confusing bug that I think MPICH should be catching. > > > > > > [1] As usual, the crud would evaporate if only MPI_Accumulate were > extensible over datatypes. > > <maxloc.c>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPI_Accumulate with MAXLOC does not error for incompatible types
by Dave Goodell 14 Jan '13

14 Jan '13

Fixed by http://git.mpich.org/mpich.git/commitdiff/ba2e444 It looks like I accidentally put the wrong URL into the commit message for some reason. The correct mailing list URL should be: http://lists.mpich.org/pipermail/discuss/2013-January/000227.html -Dave On Jan 1, 2013, at 1:09 PM CST, Jed Brown wrote: > With MPI_Reduce, we get a nice helpful error message: > > Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack: > PMPI_Reduce(1217)...........: MPI_Reduce(sbuf=0x7fff3e39fa50, rbuf=0x7fff3e39fa40, count=1, dtype=USER<contig>, MPI_MAXLOC, root=0, MPI_COMM_WORLD) failed > MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for this datatype > > > With MPI_Accumulate, we just get wrong results. See attached code using a user-defined contiguous results. > > I care about this because I'm stuck [1] dispatching into alternative mechanisms for type combinations that are not directly supported by MPI (e.g., struct {int u; long long k;} and __float128). I missed a spot and had a confusing bug that I think MPICH should be catching. > > > [1] As usual, the crud would evaporate if only MPI_Accumulate were extensible over datatypes. > <maxloc.c>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5
by Dave Goodell 14 Jan '13

14 Jan '13

Thanks for letting us know. This is a known issue related to our recent switch to git. We're working the kinks out of our nightly crons still. -Dave On Jan 14, 2013, at 12:02 PM CST, Yauheni Zelenko wrote: > Hi, Dave! > > Thank you for notice! > > Somehow latest nightly build listed on http://www.mpich.org/static/tarballs/nightly/trunk/ is from January 4. > > Eugene. > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] On Behalf Of Dave Goodell [goodell(a)mcs.anl.gov] > Sent: Friday, January 11, 2013 3:50 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5 > > FYI, the LLONG_MAX issue was resolved recently: http://git.mpich.org/mpich.git/commitdiff/465ca32 > > Expect to see this in an upcoming MPICH bug release, likely named v3.0.2. > > -Dave > > On Nov 14, 2012, at 4:46 PM CST, Dave Goodell wrote: > >> This is because you have selected a bare "--enable-strict" option to configure, which defaults to c89. I'll see if there's a more pleasant way around this, but in the meantime is it acceptable for you to pass "--enable-strict=c99" instead? >> >> The problem occurs because your compiler still responds to the "long long" type but disables the "LLONG_MAX" constant when asked for c89 mode. >> >> -Dave >> >> On Nov 14, 2012, at 4:14 PM CST, Yauheni Zelenko wrote: >> >>> Hi! >>> >>> I'm trying to compile MPICH 3.0 rc1 on Red Hat Enterprise Linux 5 and compilation failed because of undefined LLONG_MAX. >>> >>> LLONG_MAX definition in /usr/include/limits.h is depend on C99 condition. -std=c89 is passed to GCC. >>> >>> My configure options: >>> >>> ./configure --prefix=${InstallDir} \ >>> --with-pm=hydra --with-device=ch3:nemesis \ >>> --disable-chekpointing \ >>> --enable-strict --enable-fast=O3 \ >>> --disable-f77 --disable-fc \ >>> CC=/grid/common/pkgs/gcc/v4.4.5/bin/gcc CFLAGS="-m32 -fPIC" \ >>> CPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp \ >>> CXX=/grid/common/pkgs/gcc/v4.4.5/bin/g++ CXXFLAGS="-m32 -fPIC" \ >>> CXXCPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp >>> >>> GCC 4.4.5 is also produce next warnings: >>> >>> mpi-io/write.c: In function 'MPIOI_File_write': >>> mpi-io/write.c:126: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/write_all.c: In function 'MPIOI_File_write_all': >>> mpi-io/write_all.c:109: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/write_allb.c: In function 'MPIOI_File_write_all_begin': >>> mpi-io/write_allb.c:105: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/write_ord.c: In function 'PMPI_File_write_ordered': >>> mpi-io/write_ord.c:98: warning: assignment discards qualifiers from pointer target type >>> mpi-io/write_ord.c:100: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/write_ordb.c: In function 'PMPI_File_write_ordered_begin': >>> mpi-io/write_ordb.c:104: warning: assignment discards qualifiers from pointer target type >>> mpi-io/write_ordb.c:106: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/write_sh.c: In function 'PMPI_File_write_shared': >>> mpi-io/write_sh.c:100: warning: assignment discards qualifiers from pointer target type >>> mpi-io/write_sh.c:102: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >>> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >>> mpi-io/mpiu_external32.c: In function 'MPIU_write_external32_conversion_fn': >>> mpi-io/mpiu_external32.c:24: warning: implicit declaration of function 'MPIR_Datatype_iscontig' >>> mpi-io/mpiu_external32.c:24: warning: nested extern declaration of 'MPIR_Datatype_iscontig' >>> mpi-io/mpiu_external32.c: In function 'MPIU_datatype_full_size': >>> mpi-io/mpiu_external32.c:123: warning: unused variable 'extent' >>> ./pm/pmiserv/pmip_cb.c: In function 'parse_exec_params': >>> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >>> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >>> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >>> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >>> src/mpi/datatype/status_set_elements.c: In function 'PMPI_Status_set_elements': >>> src/mpi/datatype/status_set_elements.c:54: warning: unused variable 'size' >>> src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': >>> src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) >>> src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once >>> src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) >>> >>> Eugene. >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5
by Yauheni Zelenko 14 Jan '13

14 Jan '13

Hi, Dave! Thank you for notice! Somehow latest nightly build listed on http://www.mpich.org/static/tarballs/nightly/trunk/ is from January 4. Eugene. ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] On Behalf Of Dave Goodell [goodell(a)mcs.anl.gov] Sent: Friday, January 11, 2013 3:50 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5 FYI, the LLONG_MAX issue was resolved recently: http://git.mpich.org/mpich.git/commitdiff/465ca32 Expect to see this in an upcoming MPICH bug release, likely named v3.0.2. -Dave On Nov 14, 2012, at 4:46 PM CST, Dave Goodell wrote: > This is because you have selected a bare "--enable-strict" option to configure, which defaults to c89. I'll see if there's a more pleasant way around this, but in the meantime is it acceptable for you to pass "--enable-strict=c99" instead? > > The problem occurs because your compiler still responds to the "long long" type but disables the "LLONG_MAX" constant when asked for c89 mode. > > -Dave > > On Nov 14, 2012, at 4:14 PM CST, Yauheni Zelenko wrote: > >> Hi! >> >> I'm trying to compile MPICH 3.0 rc1 on Red Hat Enterprise Linux 5 and compilation failed because of undefined LLONG_MAX. >> >> LLONG_MAX definition in /usr/include/limits.h is depend on C99 condition. -std=c89 is passed to GCC. >> >> My configure options: >> >> ./configure --prefix=${InstallDir} \ >> --with-pm=hydra --with-device=ch3:nemesis \ >> --disable-chekpointing \ >> --enable-strict --enable-fast=O3 \ >> --disable-f77 --disable-fc \ >> CC=/grid/common/pkgs/gcc/v4.4.5/bin/gcc CFLAGS="-m32 -fPIC" \ >> CPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp \ >> CXX=/grid/common/pkgs/gcc/v4.4.5/bin/g++ CXXFLAGS="-m32 -fPIC" \ >> CXXCPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp >> >> GCC 4.4.5 is also produce next warnings: >> >> mpi-io/write.c: In function 'MPIOI_File_write': >> mpi-io/write.c:126: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_all.c: In function 'MPIOI_File_write_all': >> mpi-io/write_all.c:109: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_allb.c: In function 'MPIOI_File_write_all_begin': >> mpi-io/write_allb.c:105: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ord.c: In function 'PMPI_File_write_ordered': >> mpi-io/write_ord.c:98: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ord.c:100: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ordb.c: In function 'PMPI_File_write_ordered_begin': >> mpi-io/write_ordb.c:104: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ordb.c:106: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_sh.c: In function 'PMPI_File_write_shared': >> mpi-io/write_sh.c:100: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_sh.c:102: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/mpiu_external32.c: In function 'MPIU_write_external32_conversion_fn': >> mpi-io/mpiu_external32.c:24: warning: implicit declaration of function 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c:24: warning: nested extern declaration of 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c: In function 'MPIU_datatype_full_size': >> mpi-io/mpiu_external32.c:123: warning: unused variable 'extent' >> ./pm/pmiserv/pmip_cb.c: In function 'parse_exec_params': >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> src/mpi/datatype/status_set_elements.c: In function 'PMPI_Status_set_elements': >> src/mpi/datatype/status_set_elements.c:54: warning: unused variable 'size' >> src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': >> src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) >> src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once >> src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) >> >> Eugene. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 12 Jan '13

12 Jan '13

On 01/12/2013 02:51 PM US Central Time, Michael Colonno wrote: > I'm only using two hosts in this test. cxxcpi works successfully on > any one host (n <= 16) but fails on any two hosts (n > 16). So > whatever the issue is it's systematic to all the systems (which I > suppose makes sense). FWIW, you can pass -ppn 1 to mpiexec to ask it to pretend that each host has only one core. That way, you should be able to reproduce this with just n == 2. > All these systems have firewalls disabled so I can confirm that > firewall is not an issue. The only thing that's not completely > standard is they use a bond of several eth devices for communication > and some fancier load-balancing network equipment. Hmm.. Interesting. Is it possible that the hostname provided by slurm matches an eth device that does not have full connectivity? Can you try passing -disable-hostname-propagation to mpiexec to see if that helps? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Michael Colonno 12 Jan '13

12 Jan '13

Hi Pavan ~ I'm only using two hosts in this test. cxxcpi works successfully on any one host (n <= 16) but fails on any two hosts (n > 16). So whatever the issue is it's systematic to all the systems (which I suppose makes sense). All these systems have firewalls disabled so I can confirm that firewall is not an issue. The only thing that's not completely standard is they use a bond of several eth devices for communication and some fancier load-balancing network equipment. That should all be external to the OS however. I'm almost ready to test out the little VM test network; I will report what I find. Thanks, ~Mike C. -----Original Message----- From: Pavan Balaji [mailto:[email protected]] Sent: Saturday, January 12, 2013 10:34 AM To: Michael Colonno Cc: discuss(a)mpich.org Subject: Re: [mpich-discuss] Fatal error in PMPI_Reduce Hi Michael, On 01/12/2013 12:29 PM US Central Time, Michael Colonno wrote: > Thanks for the reply. I did verify connectivity prior to > trying the program. In this test case I have two systems, each with 16 > cores (cv-hpcn1 and cv-hpcn2). ssh between the two systems appears to > be working as far as I can tell: Even if ssh is working, there might still be problems. For example, many firewalls allow ssh (port 22), but not other ports. So even if ssh appears to be working, MPICH cannot. Similarly, if one of the nodes has a messed up /etc/hosts file pointing to a wrong IP address for another host, again mpich will fail. My recommendation would be to use selectively remove hosts from your host file to figure out the problematic host first. > cv-hpcn1$ /usr/local/apps/MPICH2/standalone/bin/mpirun -n 32 -f > /usr/local/apps/hosts /usr/local/apps/cxxcpi FWIW, in a SLURM environment, mpiexec/mpirun automatically detects the default "-f" parameter. You can override it with whatever you want, but if you want to use the default set of allocated hosts, -f is not required. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 12 Jan '13

12 Jan '13

Hi Michael, On 01/12/2013 12:29 PM US Central Time, Michael Colonno wrote: > Thanks for the reply. I did verify connectivity prior to > trying the program. In this test case I have two systems, each with 16 > cores (cv-hpcn1 and cv-hpcn2). ssh between the two systems appears to be > working as far as I can tell: Even if ssh is working, there might still be problems. For example, many firewalls allow ssh (port 22), but not other ports. So even if ssh appears to be working, MPICH cannot. Similarly, if one of the nodes has a messed up /etc/hosts file pointing to a wrong IP address for another host, again mpich will fail. My recommendation would be to use selectively remove hosts from your host file to figure out the problematic host first. > cv-hpcn1$ /usr/local/apps/MPICH2/standalone/bin/mpirun -n 32 -f > /usr/local/apps/hosts /usr/local/apps/cxxcpi FWIW, in a SLURM environment, mpiexec/mpirun automatically detects the default "-f" parameter. You can override it with whatever you want, but if you want to use the default set of allocated hosts, -f is not required. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Michael Colonno 12 Jan '13

12 Jan '13

Hi Pavan ~ Thanks for the reply. I did verify connectivity prior to trying the program. In this test case I have two systems, each with 16 cores (cv-hpcn1 and cv-hpcn2). ssh between the two systems appears to be working as far as I can tell: cv-hpcn1$ ssh cv-hpcn1 date Sat Jan 12 10:20:01 PST 2013 cv-hpcn1$ ssh cv-hpcn2 date Sat Jan 12 10:19:23 PST 2013 cv-hpcn1$ ssh cv-hpcn2 cv-hpcn2$ ssh cv-hpcn1 date Sat Jan 12 10:20:43 PST 2013 Here's the full test run of cxxcpi: cv-hpcn1$ /usr/local/apps/MPICH2/standalone/bin/mpirun -n 32 -f /usr/local/apps/hosts /usr/local/apps/cxxcpi Process 17 of 32 is on cv-hpcn2 Process 19 of 32 is on cv-hpcn2 Process 20 of 32 is on cv-hpcn2 Process 21 of 32 is on cv-hpcn2 Process 22 of 32 is on cv-hpcn2 Process 23 of 32 is on cv-hpcn2 Process 24 of 32 is on cv-hpcn2 Process 26 of 32 is on cv-hpcn2 Process 28 of 32 is on cv-hpcn2 Process 29 of 32 is on cv-hpcn2 Process 30 of 32 is on cv-hpcn2 Process 31 of 32 is on cv-hpcn2 Process 16 of 32 is on cv-hpcn2 Process 18 of 32 is on cv-hpcn2 Process 25 of 32 is on cv-hpcn2 Process 27 of 32 is on cv-hpcn2 Process 1 of 32 is on cv-hpcn1 Process 2 of 32 is on cv-hpcn1 Process 10 of 32 is on cv-hpcn1 Process 12 of 32 is on cv-hpcn1 Process 14 of 32 is on cv-hpcn1 Process 15 of 32 is on cv-hpcn1 Process 0 of 32 is on cv-hpcn1 Process 5 of 32 is on cv-hpcn1 Process 8 of 32 is on cv-hpcn1 Process 13 of 32 is on cv-hpcn1 Process 11 of 32 is on cv-hpcn1 Process 3 of 32 is on cv-hpcn1 Process 9 of 32 is on cv-hpcn1 Process 7 of 32 is on cv-hpcn1 Process 4 of 32 is on cv-hpcn1 Process 6 of 32 is on cv-hpcn1 Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7ffff0ab2320, rbuf=0x7ffff0ab2328, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective ============================================================================ ======= = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ============================================================================ ======= [proxy:0:1@cv-hpcn2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:1@cv-hpcn2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:1@cv-hpcn2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@cv-hpcn2] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@cv-hpcn2] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@cv-hpcn2] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@cv-hpcn2] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion Thanks for any advice. I've built many different versions of MPICH many times and this one has me scratching me head. I'm rebuilding everything identically on a pair of little VMs now to see if I can repeat the behavior outside of these systems. ~Mike C. -----Original Message----- From: Pavan Balaji [mailto:[email protected]] Sent: Friday, January 11, 2013 9:20 PM To: Michael Colonno Cc: discuss(a)mpich.org Subject: Re: [mpich-discuss] Fatal error in PMPI_Reduce On 01/11/2013 09:18 PM US Central Time, Michael Colonno wrote: > The first output below is from cxxcpi; I can also run cpi is it's helpful. Ah, sorry, I didn't realize you were in fact running one of the test programs distributed in MPICH. Based on the error, some of the nodes are not able to connect to each other based on the hostnames they are publishing. Did you try the solutions listed on the FAQ: <http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_ program_aborts_with_an_error_saying_it_cannot_communicate_with_other_process es> http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_p rogram_aborts_with_an_error_saying_it_cannot_communicate_with_other_processe s -- Pavan -- Pavan Balaji <http://www.mcs.anl.gov/~balaji> http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 11 Jan '13

11 Jan '13

On 01/11/2013 09:18 PM US Central Time, Michael Colonno wrote: > The first output below is from cxxcpi; I can also run cpi is it's helpful. Ah, sorry, I didn't realize you were in fact running one of the test programs distributed in MPICH. Based on the error, some of the nodes are not able to connect to each other based on the hostnames they are publishing. Did you try the solutions listed on the FAQ: http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_… -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Michael Colonno 11 Jan '13

11 Jan '13

The first output below is from cxxcpi; I can also run cpi is it's helpful. Thanks, Mike C. On Jan 11, 2013, at 6:27 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Ok, this doesn't seem to have anything to do with slurm. Can you try > running simple programs from the mpich install to make sure it's > correctly installed? For example, can you try examples/cpi in mpich? > > -- Pavan > > On 01/11/2013 05:47 PM US Central Time, Michael Colonno wrote: >> Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM >> and repeated the experiment with the same result: >> >> >> >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, >> rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, >> MPI_COMM_WORLD) failed >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(779)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(144).......: >> >> MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 >> >> MPIR_Reduce_intra(799)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(206).......: Failure during collective >> >> >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> = EXIT CODE: 1 >> >> = CLEANING UP REMAINING PROCESSES >> >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> =================================================================================== >> >> [proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb >> (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed >> >> [proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event >> (./tools/demux/demux_poll.c:77): callback returned error status >> >> [proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error >> waiting for event >> >> [mpiexec@n1] HYDT_bscu_wait_for_completion >> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes >> terminated badly; aborting >> >> [mpiexec@n1] HYDT_bsci_wait_for_completion >> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting >> for completion >> >> [mpiexec@n1] HYD_pmci_wait_for_completion >> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for >> completion >> >> [mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error >> waiting for completion >> >> >> >> This is now a vanilla install trying to run the cxxcpi >> example code. Any help is appreciated. >> >> >> >> Thanks, >> >> ~Mike C. >> >> >> >> *From:*Michael Colonno [mailto:[email protected]] >> *Sent:* Friday, January 11, 2013 1:32 PM >> *To:* 'discuss(a)mpich.org' >> *Subject:* Fatal error in PMPI_Reduce >> >> >> >> Hi All ~ >> >> >> >> I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on >> a CentOS 6.3 x64 system using SLURM as the process manager. My configure >> was simply: >> >> >> >> ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 >> >> >> >> No errors during build or install. When I compile and run the example >> program cxxcpi I get (truncated): >> >> >> >> $ srun -n32 /usr/local/apps/cxxcpi >> >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, >> rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, >> MPI_COMM_WORLD) failed >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(779)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(144).......: >> >> MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 >> >> MPIR_Reduce_intra(799)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(206).......: Failure during collective >> >> srun: error: task 0: Exited with exit code 1 >> >> >> >> This error is experienced with many of my MPI programs. A >> different application yields: >> >> >> >> PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, >> root=0, MPI_COMM_WORLD) failed >> >> MPIR_Bcast_impl(1369).: >> >> MPIR_Bcast_intra(1160): >> >> MPIR_SMP_Bcast(1077)..: Failure during collective >> >> >> >> Can anyone point me in the right direction? >> >> >> >> Thanks, >> >> ~Mike C. >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0