- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 11 Jan '13

11 Jan '13

Ok, this doesn't seem to have anything to do with slurm. Can you try running simple programs from the mpich install to make sure it's correctly installed? For example, can you try examples/cpi in mpich? -- Pavan On 01/11/2013 05:47 PM US Central Time, Michael Colonno wrote: > Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM > and repeated the experiment with the same result: > > > > Fatal error in PMPI_Reduce: A process has failed, error stack: > > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, > rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, > MPI_COMM_WORLD) failed > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(779)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(144).......: > > MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 > > MPIR_Reduce_intra(799)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(206).......: Failure during collective > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = EXIT CODE: 1 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > [proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb > (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed > > [proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > > [proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error > waiting for event > > [mpiexec@n1] HYDT_bscu_wait_for_completion > (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes > terminated badly; aborting > > [mpiexec@n1] HYDT_bsci_wait_for_completion > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting > for completion > > [mpiexec@n1] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for > completion > > [mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error > waiting for completion > > > > This is now a vanilla install trying to run the cxxcpi > example code. Any help is appreciated. > > > > Thanks, > > ~Mike C. > > > > *From:*Michael Colonno [mailto:[email protected]] > *Sent:* Friday, January 11, 2013 1:32 PM > *To:* 'discuss(a)mpich.org' > *Subject:* Fatal error in PMPI_Reduce > > > > Hi All ~ > > > > I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on > a CentOS 6.3 x64 system using SLURM as the process manager. My configure > was simply: > > > > ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 > > > > No errors during build or install. When I compile and run the example > program cxxcpi I get (truncated): > > > > $ srun -n32 /usr/local/apps/cxxcpi > > Fatal error in PMPI_Reduce: A process has failed, error stack: > > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, > rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, > MPI_COMM_WORLD) failed > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(779)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(144).......: > > MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 > > MPIR_Reduce_intra(799)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(206).......: Failure during collective > > srun: error: task 0: Exited with exit code 1 > > > > This error is experienced with many of my MPI programs. A > different application yields: > > > > PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, > root=0, MPI_COMM_WORLD) failed > > MPIR_Bcast_impl(1369).: > > MPIR_Bcast_intra(1160): > > MPIR_SMP_Bcast(1077)..: Failure during collective > > > > Can anyone point me in the right direction? > > > > Thanks, > > ~Mike C. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5
by Dave Goodell 11 Jan '13

11 Jan '13

FYI, the LLONG_MAX issue was resolved recently: http://git.mpich.org/mpich.git/commitdiff/465ca32 Expect to see this in an upcoming MPICH bug release, likely named v3.0.2. -Dave On Nov 14, 2012, at 4:46 PM CST, Dave Goodell wrote: > This is because you have selected a bare "--enable-strict" option to configure, which defaults to c89. I'll see if there's a more pleasant way around this, but in the meantime is it acceptable for you to pass "--enable-strict=c99" instead? > > The problem occurs because your compiler still responds to the "long long" type but disables the "LLONG_MAX" constant when asked for c89 mode. > > -Dave > > On Nov 14, 2012, at 4:14 PM CST, Yauheni Zelenko wrote: > >> Hi! >> >> I'm trying to compile MPICH 3.0 rc1 on Red Hat Enterprise Linux 5 and compilation failed because of undefined LLONG_MAX. >> >> LLONG_MAX definition in /usr/include/limits.h is depend on C99 condition. -std=c89 is passed to GCC. >> >> My configure options: >> >> ./configure --prefix=${InstallDir} \ >> --with-pm=hydra --with-device=ch3:nemesis \ >> --disable-chekpointing \ >> --enable-strict --enable-fast=O3 \ >> --disable-f77 --disable-fc \ >> CC=/grid/common/pkgs/gcc/v4.4.5/bin/gcc CFLAGS="-m32 -fPIC" \ >> CPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp \ >> CXX=/grid/common/pkgs/gcc/v4.4.5/bin/g++ CXXFLAGS="-m32 -fPIC" \ >> CXXCPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp >> >> GCC 4.4.5 is also produce next warnings: >> >> mpi-io/write.c: In function 'MPIOI_File_write': >> mpi-io/write.c:126: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_all.c: In function 'MPIOI_File_write_all': >> mpi-io/write_all.c:109: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_allb.c: In function 'MPIOI_File_write_all_begin': >> mpi-io/write_allb.c:105: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ord.c: In function 'PMPI_File_write_ordered': >> mpi-io/write_ord.c:98: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ord.c:100: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ordb.c: In function 'PMPI_File_write_ordered_begin': >> mpi-io/write_ordb.c:104: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ordb.c:106: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_sh.c: In function 'PMPI_File_write_shared': >> mpi-io/write_sh.c:100: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_sh.c:102: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/mpiu_external32.c: In function 'MPIU_write_external32_conversion_fn': >> mpi-io/mpiu_external32.c:24: warning: implicit declaration of function 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c:24: warning: nested extern declaration of 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c: In function 'MPIU_datatype_full_size': >> mpi-io/mpiu_external32.c:123: warning: unused variable 'extent' >> ./pm/pmiserv/pmip_cb.c: In function 'parse_exec_params': >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> src/mpi/datatype/status_set_elements.c: In function 'PMPI_Status_set_elements': >> src/mpi/datatype/status_set_elements.c:54: warning: unused variable 'size' >> src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': >> src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) >> src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once >> src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) >> >> Eugene. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Michael Colonno 11 Jan '13

11 Jan '13

Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM and repeated the experiment with the same result: Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective ============================================================================ ======= = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ============================================================================ ======= [proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@n1] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@n1] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@n1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion This is now a vanilla install trying to run the cxxcpi example code. Any help is appreciated. Thanks, ~Mike C. From: Michael Colonno [mailto:[email protected]] Sent: Friday, January 11, 2013 1:32 PM To: 'discuss(a)mpich.org' Subject: Fatal error in PMPI_Reduce Hi All ~ I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply: ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 No errors during build or install. When I compile and run the example program cxxcpi I get (truncated): $ srun -n32 /usr/local/apps/cxxcpi Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective srun: error: task 0: Exited with exit code 1 This error is experienced with many of my MPI programs. A different application yields: PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1369).: MPIR_Bcast_intra(1160): MPIR_SMP_Bcast(1077)..: Failure during collective Can anyone point me in the right direction? Thanks, ~Mike C.

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Michael Colonno 11 Jan '13

11 Jan '13

With the options I used to configure, mpiexec / mpirun are not built or installed (presumably due to SLURM interface). Thanks, ~Mike C. -----Original Message----- From: discuss-bounces(a)mpich.org [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Friday, January 11, 2013 2:20 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Fatal error in PMPI_Reduce FYI, the reason I suggested this is because mpiexec will automatically detect and use slurm internally. -- Pavan On 01/11/2013 04:19 PM US Central Time, Pavan Balaji wrote: > Michael, > > Did you try just using mpiexec? > > mpiexec -n 32 /usr/local/apps/cxxcpi > > -- Pavan > > On 01/11/2013 03:31 PM US Central Time, Michael Colonno wrote: >> Hi All ~ >> >> >> >> I've compiled MPICH2 3.0 with the Intel compiler (v. 13) >> on a CentOS 6.3 x64 system using SLURM as the process manager. My >> configure was simply: >> >> >> >> ./configure --with-pmi=slurm --with-pm=no >> --prefix=/usr/local/apps/MPICH2 >> >> >> >> No errors during build or install. When I compile and run the example >> program cxxcpi I get (truncated): >> >> >> >> $ srun -n32 /usr/local/apps/cxxcpi >> >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, >> rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, >> MPI_COMM_WORLD) failed >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(779)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(144).......: >> >> MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 >> >> MPIR_Reduce_intra(799)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(206).......: Failure during collective >> >> srun: error: task 0: Exited with exit code 1 >> >> >> >> This error is experienced with many of my MPI programs. A >> different application yields: >> >> >> >> PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, >> MPI_INT, root=0, MPI_COMM_WORLD) failed >> >> MPIR_Bcast_impl(1369).: >> >> MPIR_Bcast_intra(1160): >> >> MPIR_SMP_Bcast(1077)..: Failure during collective >> >> >> >> Can anyone point me in the right direction? >> >> >> >> Thanks, >> >> ~Mike C. >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 11 Jan '13

11 Jan '13

FYI, the reason I suggested this is because mpiexec will automatically detect and use slurm internally. -- Pavan On 01/11/2013 04:19 PM US Central Time, Pavan Balaji wrote: > Michael, > > Did you try just using mpiexec? > > mpiexec -n 32 /usr/local/apps/cxxcpi > > -- Pavan > > On 01/11/2013 03:31 PM US Central Time, Michael Colonno wrote: >> Hi All ~ >> >> >> >> I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on >> a CentOS 6.3 x64 system using SLURM as the process manager. My configure >> was simply: >> >> >> >> ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 >> >> >> >> No errors during build or install. When I compile and run the example >> program cxxcpi I get (truncated): >> >> >> >> $ srun -n32 /usr/local/apps/cxxcpi >> >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, >> rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, >> MPI_COMM_WORLD) failed >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(779)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(144).......: >> >> MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 >> >> MPIR_Reduce_intra(799)..........: >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(206).......: Failure during collective >> >> srun: error: task 0: Exited with exit code 1 >> >> >> >> This error is experienced with many of my MPI programs. A >> different application yields: >> >> >> >> PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, >> root=0, MPI_COMM_WORLD) failed >> >> MPIR_Bcast_impl(1369).: >> >> MPIR_Bcast_intra(1160): >> >> MPIR_SMP_Bcast(1077)..: Failure during collective >> >> >> >> Can anyone point me in the right direction? >> >> >> >> Thanks, >> >> ~Mike C. >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 11 Jan '13

11 Jan '13

Michael, Did you try just using mpiexec? mpiexec -n 32 /usr/local/apps/cxxcpi -- Pavan On 01/11/2013 03:31 PM US Central Time, Michael Colonno wrote: > Hi All ~ > > > > I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on > a CentOS 6.3 x64 system using SLURM as the process manager. My configure > was simply: > > > > ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 > > > > No errors during build or install. When I compile and run the example > program cxxcpi I get (truncated): > > > > $ srun -n32 /usr/local/apps/cxxcpi > > Fatal error in PMPI_Reduce: A process has failed, error stack: > > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, > rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, > MPI_COMM_WORLD) failed > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(779)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(144).......: > > MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 > > MPIR_Reduce_intra(799)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(206).......: Failure during collective > > srun: error: task 0: Exited with exit code 1 > > > > This error is experienced with many of my MPI programs. A > different application yields: > > > > PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, > root=0, MPI_COMM_WORLD) failed > > MPIR_Bcast_impl(1369).: > > MPIR_Bcast_intra(1160): > > MPIR_SMP_Bcast(1077)..: Failure during collective > > > > Can anyone point me in the right direction? > > > > Thanks, > > ~Mike C. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Fatal error in PMPI_Reduce
by Michael Colonno 11 Jan '13

11 Jan '13

Hi All ~ I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply: ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 No errors during build or install. When I compile and run the example program cxxcpi I get (truncated): $ srun -n32 /usr/local/apps/cxxcpi Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective srun: error: task 0: Exited with exit code 1 This error is experienced with many of my MPI programs. A different application yields: PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1369).: MPIR_Bcast_intra(1160): MPIR_SMP_Bcast(1077)..: Failure during collective Can anyone point me in the right direction? Thanks, ~Mike C.

1 0

Re: [mpich-discuss] start MPD daemons on 2 different subnets
by Dave Goodell 11 Jan '13

11 Jan '13

Use hydra instead of MPD: http://wiki.mpich.org/mpich/index.php/FAQ#Q:_I_don.27t_like_.3CWHATEVER.3E_… -Dave On Jan 11, 2013, at 12:55 PM CST, Na Zhang wrote: > Dear developers, > > We want to start MPD daemons on 2 different subnets: for example, > subnet #1 192.168.0.0/24 and subnet #2: 192.168.1.0/24 > > The two subnets are connected via switches and they can talk to each > other. Next, we'd start MPD daemons on two nodes: > > Node #1 (in subnet #1): (hostname:node_001) IP=192.168.0.1 > Node #2 (in subnet #2): (hostname:node_002) IP=192.168.1.1 > > We used the following commands: > > on Node #1: mpd --ifhn=192.168.0.1 --daemon > (daemon is successfully started on node_001) > > on Node #2: mpd -h node_001 -p <node1's_port_number> > --ifhn=192.168.1.1 --daemon > (daemon cannot be started on Node #2. there is no error message, we > use "mpdtrace" to check on Node #1, it shows no daemon started on Node > #2 ) > > Node #2 cannot join the ring that is generated by node #1. > > How should we do? > > Thank you in advance. > > Sincerely, > Na Zhang > > -- > Na Zhang, Ph.D. Candidate > Dept. of Applied Mathematics and Statistics > Stony Brook University > Phone: 631-838-3205 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

start MPD daemons on 2 different subnets
by Na Zhang 11 Jan '13

11 Jan '13

Dear developers, We want to start MPD daemons on 2 different subnets: for example, subnet #1 192.168.0.0/24 and subnet #2: 192.168.1.0/24 The two subnets are connected via switches and they can talk to each other. Next, we'd start MPD daemons on two nodes: Node #1 (in subnet #1): (hostname:node_001) IP=192.168.0.1 Node #2 (in subnet #2): (hostname:node_002) IP=192.168.1.1 We used the following commands: on Node #1: mpd --ifhn=192.168.0.1 --daemon (daemon is successfully started on node_001) on Node #2: mpd -h node_001 -p <node1's_port_number> --ifhn=192.168.1.1 --daemon (daemon cannot be started on Node #2. there is no error message, we use "mpdtrace" to check on Node #1, it shows no daemon started on Node #2 ) Node #2 cannot join the ring that is generated by node #1. How should we do? Thank you in advance. Sincerely, Na Zhang -- Na Zhang, Ph.D. Candidate Dept. of Applied Mathematics and Statistics Stony Brook University Phone: 631-838-3205

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Dmitri Gribenko 09 Jan '13

09 Jan '13

On Wed, Jan 9, 2013 at 10:00 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Jan 9, 2013 12:56 PM, "Dmitri Gribenko" <gribozavr(a)gmail.com> wrote: >> I don't think GCC isn't smart enough to do the same. Do you compile >> with optimization? > > Dmitri, as discussed in the other thread, it's smart enough, but only when > optimization is turned on. There's no reason to needlessly make debug builds > heavier than necessary. This is not a big deal either way. Oh, now I see -- in debug builds it still emits these. Thank you for fixing! Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0