January 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich startup without networking
by Balaji, Pavan 22 Jan '16

22 Jan '16

That's gross. I think nemesis has the same problem that I just fixed in Hydra. The correct solution is to move this check for localhost/local IP to MPL and use it in both places. I'm in the middle of a different code hack right now and can't context switch to this, unfortunately. But given that you have a workaround for now, I've created a ticket for it and will deal with this later. https://trac.mpich.org/projects/mpich/ticket/2329 -- Pavan > On Jan 22, 2016, at 9:58 AM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > > On 01/22/2016 08:52 AM, Balaji, Pavan wrote: >> >>> On Jan 22, 2016, at 9:40 AM, Orion Poplawski <orion(a)cora.nwra.com> wrote: >>> >>> On 01/21/2016 04:07 PM, Balaji, Pavan wrote: >>>> >>>> I don't have a way to easily test this, so I'm going to do >>>> remote-email-testing on it. Can you try out the attached patch? >>>> >>>> -- Pavan >>> >>> That appears to allow -host localhost to work, thanks! >> >> Great. Does it also allow you to run when "-host localhost" is not provided? >> >> -- Pavan > > No, I still get: > > > Testing MPI parallel I/O without netCDF... > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(474)..............: > MPID_Init(190).....................: channel initialization failed > MPIDI_CH3_Init(89).................: > MPID_nem_init(320).................: > MPID_nem_tcp_init(173).............: > MPID_nem_tcp_get_business_card(420): > MPID_nem_tcp_init(379).............: gethostbyname failed, > copr-builder-194568281.novalocal (errno 2) > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(474)..............: > MPID_Init(190).....................: channel initialization failed > MPIDI_CH3_Init(89).................: > MPID_nem_init(320).................: > MPID_nem_tcp_init(173).............: > MPID_nem_tcp_get_business_card(420): > MPID_nem_tcp_init(379).............: gethostbyname failed, > copr-builder-194568281.novalocal (errno 2) > FAIL run_par_test.sh (exit status: 15) > > > -- > Orion Poplawski > Technical Manager 303-415-9701 x222 > NWRA, Boulder/CoRA Office FAX: 303-415-9702 > 3380 Mitchell Lane orion(a)nwra.com > Boulder, CO 80301 http://www.nwra.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Orion Poplawski 22 Jan '16

22 Jan '16

On 01/22/2016 08:52 AM, Balaji, Pavan wrote: > >> On Jan 22, 2016, at 9:40 AM, Orion Poplawski <orion(a)cora.nwra.com> wrote: >> >> On 01/21/2016 04:07 PM, Balaji, Pavan wrote: >>> >>> I don't have a way to easily test this, so I'm going to do >>> remote-email-testing on it. Can you try out the attached patch? >>> >>> -- Pavan >> >> That appears to allow -host localhost to work, thanks! > > Great. Does it also allow you to run when "-host localhost" is not provided? > > -- Pavan No, I still get: Testing MPI parallel I/O without netCDF... Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(474)..............: MPID_Init(190).....................: channel initialization failed MPIDI_CH3_Init(89).................: MPID_nem_init(320).................: MPID_nem_tcp_init(173).............: MPID_nem_tcp_get_business_card(420): MPID_nem_tcp_init(379).............: gethostbyname failed, copr-builder-194568281.novalocal (errno 2) Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(474)..............: MPID_Init(190).....................: channel initialization failed MPIDI_CH3_Init(89).................: MPID_nem_init(320).................: MPID_nem_tcp_init(173).............: MPID_nem_tcp_get_business_card(420): MPID_nem_tcp_init(379).............: gethostbyname failed, copr-builder-194568281.novalocal (errno 2) FAIL run_par_test.sh (exit status: 15) -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Balaji, Pavan 22 Jan '16

22 Jan '16

> On Jan 22, 2016, at 9:40 AM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > > On 01/21/2016 04:07 PM, Balaji, Pavan wrote: >> >> I don't have a way to easily test this, so I'm going to do >> remote-email-testing on it. Can you try out the attached patch? >> >> -- Pavan > > That appears to allow -host localhost to work, thanks! Great. Does it also allow you to run when "-host localhost" is not provided? -- Pavan _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Orion Poplawski 22 Jan '16

22 Jan '16

On 01/21/2016 04:07 PM, Balaji, Pavan wrote: > > I don't have a way to easily test this, so I'm going to do > remote-email-testing on it. Can you try out the attached patch? > > -- Pavan That appears to allow -host localhost to work, thanks! - Oiron > >> On Jan 21, 2016, at 4:48 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> >> Hmm. We could do that. Alternatively, we could cover for the case that we are not able to resolve the IP address for the local hostname, i.e., "localhost", for systems where this is not added in /etc/hosts. Let me see if I can work up a patch. Give me a few minutes. >> >> -- Pavan >> >>> On Jan 21, 2016, at 3:20 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: >>> >>> More Fedora builder issues. I was able to get farther by using the nemesis >>> channel on all platforms. Now I'm running into: >>> >>> Testing MPI parallel I/O without netCDF... >>> Fatal error in MPI_Init: Other MPI error, error stack: >>> MPIR_Init_thread(474)..............: >>> MPID_Init(190).....................: channel initialization failed >>> MPIDI_CH3_Init(89).................: >>> MPID_nem_init(320).................: >>> MPID_nem_tcp_init(173).............: >>> MPID_nem_tcp_get_business_card(420): >>> MPID_nem_tcp_init(379).............: gethostbyname failed, >>> buildvm-12.phx2.fedoraproject.org (errno 2) >>> >>> Okay, so lets try -host localhost: >>> >>> Testing MPI parallel I/O without netCDF... >>> [mpiexec(a)buildhw-05.phx2.fedoraproject.org] HYDU_sock_is_local >>> (utils/sock/sock.c:532): gethostbyname error on >>> buildhw-05.phx2.fedoraproject.org: Host name lookup failure >>> [mpiexec(a)buildhw-05.phx2.fedoraproject.org] main (ui/mpich/mpiexec.c:291): >>> unable to check if localhost is local >>> >>> Perhaps we can encode that "localhost" is always local? >>> >>> >>> >>> -- >>> Orion Poplawski >>> Technical Manager 303-415-9701 x222 >>> NWRA, Boulder/CoRA Office FAX: 303-415-9702 >>> 3380 Mitchell Lane orion(a)nwra.com >>> Boulder, CO 80301 http://www.nwra.com >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Balaji, Pavan 21 Jan '16

21 Jan '16

I don't have a way to easily test this, so I'm going to do remote-email-testing on it. Can you try out the attached patch? -- Pavan > On Jan 21, 2016, at 4:48 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > Hmm. We could do that. Alternatively, we could cover for the case that we are not able to resolve the IP address for the local hostname, i.e., "localhost", for systems where this is not added in /etc/hosts. Let me see if I can work up a patch. Give me a few minutes. > > -- Pavan > >> On Jan 21, 2016, at 3:20 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: >> >> More Fedora builder issues. I was able to get farther by using the nemesis >> channel on all platforms. Now I'm running into: >> >> Testing MPI parallel I/O without netCDF... >> Fatal error in MPI_Init: Other MPI error, error stack: >> MPIR_Init_thread(474)..............: >> MPID_Init(190).....................: channel initialization failed >> MPIDI_CH3_Init(89).................: >> MPID_nem_init(320).................: >> MPID_nem_tcp_init(173).............: >> MPID_nem_tcp_get_business_card(420): >> MPID_nem_tcp_init(379).............: gethostbyname failed, >> buildvm-12.phx2.fedoraproject.org (errno 2) >> >> Okay, so lets try -host localhost: >> >> Testing MPI parallel I/O without netCDF... >> [mpiexec(a)buildhw-05.phx2.fedoraproject.org] HYDU_sock_is_local >> (utils/sock/sock.c:532): gethostbyname error on >> buildhw-05.phx2.fedoraproject.org: Host name lookup failure >> [mpiexec(a)buildhw-05.phx2.fedoraproject.org] main (ui/mpich/mpiexec.c:291): >> unable to check if localhost is local >> >> Perhaps we can encode that "localhost" is always local? >> >> >> >> -- >> Orion Poplawski >> Technical Manager 303-415-9701 x222 >> NWRA, Boulder/CoRA Office FAX: 303-415-9702 >> 3380 Mitchell Lane orion(a)nwra.com >> Boulder, CO 80301 http://www.nwra.com >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Balaji, Pavan 21 Jan '16

21 Jan '16

Hmm. We could do that. Alternatively, we could cover for the case that we are not able to resolve the IP address for the local hostname, i.e., "localhost", for systems where this is not added in /etc/hosts. Let me see if I can work up a patch. Give me a few minutes. -- Pavan > On Jan 21, 2016, at 3:20 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > > More Fedora builder issues. I was able to get farther by using the nemesis > channel on all platforms. Now I'm running into: > > Testing MPI parallel I/O without netCDF... > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(474)..............: > MPID_Init(190).....................: channel initialization failed > MPIDI_CH3_Init(89).................: > MPID_nem_init(320).................: > MPID_nem_tcp_init(173).............: > MPID_nem_tcp_get_business_card(420): > MPID_nem_tcp_init(379).............: gethostbyname failed, > buildvm-12.phx2.fedoraproject.org (errno 2) > > Okay, so lets try -host localhost: > > Testing MPI parallel I/O without netCDF... > [mpiexec(a)buildhw-05.phx2.fedoraproject.org] HYDU_sock_is_local > (utils/sock/sock.c:532): gethostbyname error on > buildhw-05.phx2.fedoraproject.org: Host name lookup failure > [mpiexec(a)buildhw-05.phx2.fedoraproject.org] main (ui/mpich/mpiexec.c:291): > unable to check if localhost is local > > Perhaps we can encode that "localhost" is always local? > > > > -- > Orion Poplawski > Technical Manager 303-415-9701 x222 > NWRA, Boulder/CoRA Office FAX: 303-415-9702 > 3380 Mitchell Lane orion(a)nwra.com > Boulder, CO 80301 http://www.nwra.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich startup without networking
by Orion Poplawski 21 Jan '16

21 Jan '16

More Fedora builder issues. I was able to get farther by using the nemesis channel on all platforms. Now I'm running into: Testing MPI parallel I/O without netCDF... Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(474)..............: MPID_Init(190).....................: channel initialization failed MPIDI_CH3_Init(89).................: MPID_nem_init(320).................: MPID_nem_tcp_init(173).............: MPID_nem_tcp_get_business_card(420): MPID_nem_tcp_init(379).............: gethostbyname failed, buildvm-12.phx2.fedoraproject.org (errno 2) Okay, so lets try -host localhost: Testing MPI parallel I/O without netCDF... [mpiexec(a)buildhw-05.phx2.fedoraproject.org] HYDU_sock_is_local (utils/sock/sock.c:532): gethostbyname error on buildhw-05.phx2.fedoraproject.org: Host name lookup failure [mpiexec(a)buildhw-05.phx2.fedoraproject.org] main (ui/mpich/mpiexec.c:291): unable to check if localhost is local Perhaps we can encode that "localhost" is always local? -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hanging on startup
by Balaji, Pavan 20 Jan '16

20 Jan '16

No, you can use nemesis on all platforms. -- Pavan > On Jan 20, 2016, at 8:25 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > > On 01/20/2016 05:05 PM, Orion Poplawski wrote: >> I'm got a strange situation - I'm trying to build hdf5 1.8.16 for Fedora. The >> Fedora builders do not have network access (/etc/resolv.conf set to >> "nameserver 127.0.0.1" and no local nameserver) for security purposes. The >> hdf5 t_mpi parallel test hangs on launch with mpich 3.2, but only on the arm >> builders. >> >> The processes seem to deadlock in the MPI code, each blocking in poll(), >> presumably waiting for communication from the other process. > > > Apparently the Fedora package builds using --with-device=ch3:nemesis on x86 and --with-device=ch3:sock on all other platforms, so the arm builds are using a different channel. Is that still a necessary and/or recommended configuration? > > > -- > Orion Poplawski > Technical Manager 303-415-9701 x222 > NWRA/CoRA Division FAX: 303-415-9702 > 3380 Mitchell Lane orion(a)cora.nwra.com > Boulder, CO 80301 http://www.cora.nwra.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hanging on startup
by Orion Poplawski 20 Jan '16

20 Jan '16

On 01/20/2016 05:05 PM, Orion Poplawski wrote: > I'm got a strange situation - I'm trying to build hdf5 1.8.16 for Fedora. The > Fedora builders do not have network access (/etc/resolv.conf set to > "nameserver 127.0.0.1" and no local nameserver) for security purposes. The > hdf5 t_mpi parallel test hangs on launch with mpich 3.2, but only on the arm > builders. > > The processes seem to deadlock in the MPI code, each blocking in poll(), > presumably waiting for communication from the other process. Apparently the Fedora package builds using --with-device=ch3:nemesis on x86 and --with-device=ch3:sock on all other platforms, so the arm builds are using a different channel. Is that still a necessary and/or recommended configuration? -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich hanging on startup
by Orion Poplawski 20 Jan '16

20 Jan '16

I'm got a strange situation - I'm trying to build hdf5 1.8.16 for Fedora. The Fedora builders do not have network access (/etc/resolv.conf set to "nameserver 127.0.0.1" and no local nameserver) for security purposes. The hdf5 t_mpi parallel test hangs on launch with mpich 3.2, but only on the arm builders. The processes seem to deadlock in the MPI code, each blocking in poll(), presumably waiting for communication from the other process. here are gdb stack traces of a two process t_mpi job: (gdb) bt #0 0xb69176ec in poll () from /lib/libc.so.6 #1 0xb6b4b178 in poll (__timeout=-1, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 MPIDU_Sock_wait (sock_set=0x7f57b3e0, millisecond_timeout=millisecond_timeout@entry=-1, eventp=0xb6fc6000, eventp@entry=0xbeffe054) at src/mpid/common/sock/poll/sock_wait.i:123 #3 0xb6b30960 in MPIDI_CH3i_Progress_wait (progress_state=0xbeffe0a0) at src/mpid/ch3/channels/sock/src/ch3_progress.c:221 #4 MPIDI_CH3I_Progress (blocking=blocking@entry=1, state=state@entry=0xbeffe0a0) at src/mpid/ch3/channels/sock/src/ch3_progress.c:962 #5 0xb6abfeac in MPIC_Wait (request_ptr=0xb6c0a180 <MPID_Request_direct>, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/helper_fns.c:225 #6 0xb6ac01cc in MPIC_Recv (buf=buf@entry=0x7f5a0820, count=count@entry=65, datatype=datatype@entry=1275069445, source=<optimized out>, tag=tag@entry=11, comm_ptr=0xb6bf0908 <MPID_Comm_direct>, comm_ptr@entry=0x7f586250, status=0xbeffe190, status@entry=0xb6ffec88 <__stack_chk_guard>, errflag=0xbeffe330, errflag@entry=0x7f586c80) at src/mpi/coll/helper_fns.c:355 #7 0xb6a0326c in MPIR_Reduce_binomial (errflag=0x7f586c80, comm_ptr=0x7f586250, root=-1231016932, op=-1090526356, datatype=1275069445, count=65, recvbuf=0xb6bf0908 <MPID_Comm_direct>, sendbuf=<optimized out>) at src/mpi/coll/reduce.c:181 #8 MPIR_Reduce_intra (sendbuf=sendbuf@entry=0xffffffff, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=datatype@entry=1275069445, op=1476395014, op@entry=-1228891236, root=root@entry=0, comm_ptr=0xb6bf0908 <MPID_Comm_direct>, comm_ptr@entry=0x58000006, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/reduce.c:874 #9 0xb6a02c1c in MPIR_Reduce_impl (sendbuf=0xffffffff, sendbuf@entry=0xb6bf1288 <MPID_Comm_builtin>, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=datatype@entry=1275069445, op=1476395014, op@entry=-1228991864, root=root@entry=0, comm_ptr=comm_ptr@entry=0xb6bf0908 <MPID_Comm_direct>, errflag=0xbeffe330, errflag@entry=0xb6ffec88 <__stack_chk_guard>) at src/mpi/coll/reduce.c:1068 #10 0xb69f7b64 in MPIR_Allreduce_intra (sendbuf=0xb6bf1288 <MPID_Comm_builtin>, sendbuf@entry=0xffffffff, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=datatype@entry=1275069445, op=op@entry=1476395014, comm_ptr=0xb6bf1288 <MPID_Comm_builtin>, comm_ptr@entry=0xb6acdd88 <MPIR_Get_contextid_sparse_group+824>, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/allreduce.c:234 #11 0xb69f9414 in MPIR_Allreduce_impl (sendbuf=sendbuf@entry=0xffffffff, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=1275069445, op=1476395014, comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, errflag=0xbeffe330, errflag@entry=0xbeffe4c8) at src/mpi/coll/allreduce.c:763 #12 0xb6acdd88 in MPIR_Get_contextid_sparse_group ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, group_ptr=0xb6998228 <__libc_malloc_initialized>, group_ptr@entry=0x0, tag=-1090526416, context_id=0xfffff000, context_id@entry=0xbeffe4c0, ignore_id=ignore_id@entry=0) at src/mpi/comm/contextid.c:496 #13 0xb6ace5a4 in MPIR_Get_contextid_sparse ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, context_id=context_id@entry=0xbeffe4c0, ignore_id=ignore_id@entry=0) at src/mpi/comm/contextid.c:298 #14 0xb6acc14c in MPIR_Comm_copy ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, size=2, outcomm_ptr=outcomm_ptr@entry=0xbeffe538) at src/mpi/comm/commutil.c:736 #15 0xb6a50a48 in MPIR_Comm_dup_impl ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, newcomm_ptr=0xbeffe538, newcomm_ptr@entry=0xbeffe530) at src/mpi/comm/comm_dup.c:56 #16 0xb6a50cf0 in PMPI_Comm_dup (comm=1140850688, newcomm=0xbeffe574, newcomm@entry=0xbeffe56c) at src/mpi/comm/comm_dup.c:161 #17 0xb6daa0dc in H5FD_mpi_comm_info_dup (comm=<optimized out>, info=469762048, comm_new=comm_new@entry=0x7f5a0810, info_new=info_new@entry=0x7f5a0814) at ../../src/H5FDmpi.c:271 #18 0xb6daaa80 in H5FD_mpio_fapl_copy (_old_fa=0xbeffe64c) at ../../src/H5FDmpio.c:789 #19 0xb6d9e0ec in H5FD_pl_copy (copied_pl=0xbeffe5d0, old_pl=0xbeffe64c, pl_size=<optimized out>, copy_func=<optimized out>) at ../../src/H5FD.c:625 #20 H5FD_fapl_copy (copied_fapl=0xbeffe5d0, old_fapl=0xbeffe64c, driver_id=<optimized out>) at ../../src/H5FD.c:800 #21 H5FD_fapl_open (plist=plist@entry=0x7f5a06c8, driver_id=-1224955460, driver_id@entry=134217729, driver_info=driver_info@entry=0xbeffe64c) at ../../src/H5FD.c:753 #22 0xb6e6aea4 in H5P_set_driver (plist=plist@entry=0x7f5a06c8, new_driver_id=134217729, new_driver_info=0xbeffe64c, new_driver_info@entry=0xbeffe644) at ../../src/H5Pfapl.c:633 #23 0xb6dacec4 in H5Pset_fapl_mpio (fapl_id=<optimized out>, comm=1140850688, info=469762048) at ../../src/H5FDmpio.c:353 #24 0x7f5570e8 in parse_options (argv=<optimized out>, argc=<optimized out>) at ../../testpar/t_mpi.c:1041 #25 main (argc=0, argv=0x0) at ../../testpar/t_mpi.c:1106 (gdb) up 2 #2 MPIDU_Sock_wait (sock_set=0x7f57b3e0, millisecond_timeout=millisecond_timeout@entry=-1, eventp=0xb6fc6000, eventp@entry=0xbeffe054) at src/mpid/common/sock/poll/sock_wait.i:123 123 n_fds = poll(sock_set->pollfds, sock_set->poll_array_elems, (gdb) print millisecond_timeout $1 = -1 (gdb) print *sock_set $2 = {id = 0, starting_elem = 0, poll_array_sz = 32, poll_array_elems = 1, pollfds = 0x7f57b588, pollinfos = 0x7f57b690, eventq_head = 0x0, eventq_tail = 0x0, pollfds_active = 0x0, pollfds_updated = 0, wakeup_posted = 0, intr_sock = 0x0, intr_fds = {-1, -1}} (gdb) print *sock_set->pollfds $3 = {fd = 10, events = 1, revents = 0} #0 0xb69176ec in poll () from /lib/libc.so.6 #1 0xb6b4b178 in poll (__timeout=-1, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 MPIDU_Sock_wait (sock_set=0x7f57b3e0, millisecond_timeout=millisecond_timeout@entry=-1, eventp=0x7f5e5700, eventp@entry=0xbeffe074) at src/mpid/common/sock/poll/sock_wait.i:123 #3 0xb6b30960 in MPIDI_CH3i_Progress_wait (progress_state=0xbeffe0c0) at src/mpid/ch3/channels/sock/src/ch3_progress.c:221 #4 MPIDI_CH3I_Progress (blocking=blocking@entry=1, state=state@entry=0xbeffe0c0) at src/mpid/ch3/channels/sock/src/ch3_progress.c:962 #5 0xb6abfeac in MPIC_Wait (request_ptr=0xb6c0a180 <MPID_Request_direct>, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/helper_fns.c:225 #6 0xb6ac000c in MPIC_Send (buf=buf@entry=0xb6bf0908 <MPID_Comm_direct>, count=count@entry=65, datatype=datatype@entry=1275069445, dest=<optimized out>, tag=tag@entry=11, comm_ptr=0xb6bf0908 <MPID_Comm_direct>, comm_ptr@entry=0x7f586250, errflag=0xbeffe330, errflag@entry=0x7f586c80) at src/mpi/coll/helper_fns.c:302 #7 0xb6a03438 in MPIR_Reduce_binomial (errflag=0x7f586c80, comm_ptr=0x7f586250, root=-1231016932, op=-1090526356, datatype=1275069445, count=65, recvbuf=0xb6bf0908 <MPID_Comm_direct>, sendbuf=<optimized out>) at src/mpi/coll/reduce.c:210 #8 MPIR_Reduce_intra (sendbuf=sendbuf@entry=0xbeffe36c, recvbuf=recvbuf@entry=0x0, count=count@entry=65, datatype=datatype@entry=1275069445, op=1476395014, op@entry=-1228891236, root=root@entry=0, comm_ptr=0xb6bf0908 <MPID_Comm_direct>, comm_ptr@entry=0x58000006, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/reduce.c:874 #9 0xb6a02c1c in MPIR_Reduce_impl (sendbuf=sendbuf@entry=0xbeffe36c, recvbuf=recvbuf@entry=0x0, count=count@entry=65, datatype=datatype@entry=1275069445, op=1476395014, op@entry=-1228991864, root=root@entry=0, comm_ptr=comm_ptr@entry=0xb6bf0908 <MPID_Comm_direct>, errflag=0xbeffe330, errflag@entry=0xb6ffec88 <__stack_chk_guard>) at src/mpi/coll/reduce.c:1068 #10 0xb69f8b94 in MPIR_Allreduce_intra ( sendbuf=0xb6bf1288 <MPID_Comm_builtin>, sendbuf@entry=0xffffffff, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=datatype@entry=1275069445, op=op@entry=1476395014, comm_ptr=0xb6bf1288 <MPID_Comm_builtin>, comm_ptr@entry=0xb6acdd88 <MPIR_Get_contextid_sparse_group+824>, errflag=errflag@entry=0xbeffe330) at src/mpi/coll/allreduce.c:226 #11 0xb69f9414 in MPIR_Allreduce_impl (sendbuf=sendbuf@entry=0xffffffff, recvbuf=recvbuf@entry=0xbeffe36c, count=count@entry=65, datatype=1275069445, op=1476395014, comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, errflag=0xbeffe330, errflag@entry=0xbeffe4c8) at src/mpi/coll/allreduce.c:763 #12 0xb6acdd88 in MPIR_Get_contextid_sparse_group ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, group_ptr=0xb6998228 <__libc_malloc_initialized>, group_ptr@entry=0x0, tag=-1090526416, context_id=0xfffff000, context_id@entry=0xbeffe4c0, ignore_id=ignore_id@entry=0) at src/mpi/comm/contextid.c:496 #13 0xb6ace5a4 in MPIR_Get_contextid_sparse ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, context_id=context_id@entry=0xbeffe4c0, ignore_id=ignore_id@entry=0) at src/mpi/comm/contextid.c:298 #14 0xb6acc14c in MPIR_Comm_copy ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, size=2, outcomm_ptr=outcomm_ptr@entry=0xbeffe538) at src/mpi/comm/commutil.c:736 #15 0xb6a50a48 in MPIR_Comm_dup_impl ( comm_ptr=comm_ptr@entry=0xb6bf1288 <MPID_Comm_builtin>, newcomm_ptr=0xbeffe538, newcomm_ptr@entry=0xbeffe530) at src/mpi/comm/comm_dup.c:56 #16 0xb6a50cf0 in PMPI_Comm_dup (comm=1140850688, newcomm=0xbeffe574, newcomm@entry=0xbeffe56c) at src/mpi/comm/comm_dup.c:161 #17 0xb6daa0dc in H5FD_mpi_comm_info_dup (comm=<optimized out>, info=469762048, comm_new=comm_new@entry=0x7f5a0810, info_new=info_new@entry=0x7f5a0814) at ../../src/H5FDmpi.c:271 #18 0xb6daaa80 in H5FD_mpio_fapl_copy (_old_fa=0xbeffe64c) at ../../src/H5FDmpio.c:789 #19 0xb6d9e0ec in H5FD_pl_copy (copied_pl=0xbeffe5d0, old_pl=0xbeffe64c, pl_size=<optimized out>, copy_func=<optimized out>) at ../../src/H5FD.c:625 #20 H5FD_fapl_copy (copied_fapl=0xbeffe5d0, old_fapl=0xbeffe64c, driver_id=<optimized out>) at ../../src/H5FD.c:800 #21 H5FD_fapl_open (plist=plist@entry=0x7f5a06c8, driver_id=-1224955460, driver_id@entry=134217729, driver_info=driver_info@entry=0xbeffe64c) at ../../src/H5FD.c:753 #22 0xb6e6aea4 in H5P_set_driver (plist=plist@entry=0x7f5a06c8, new_driver_id=134217729, new_driver_info=0xbeffe64c, new_driver_info@entry=0xbeffe644) at ../../src/H5Pfapl.c:633 #23 0xb6dacec4 in H5Pset_fapl_mpio (fapl_id=<optimized out>, comm=1140850688, info=469762048) at ../../src/H5FDmpio.c:353 #24 0x7f5570e8 in parse_options (argv=<optimized out>, argc=<optimized out>) at ../../testpar/t_mpi.c:1041 #25 main (argc=0, argv=0x0) at ../../testpar/t_mpi.c:1106 (gdb) up #2 MPIDU_Sock_wait (sock_set=0x7f57b3e0, millisecond_timeout=millisecond_timeout@entry=-1, eventp=0x7f5e5700, eventp@entry=0xbeffe074) at src/mpid/common/sock/poll/sock_wait.i:123 123 n_fds = poll(sock_set->pollfds, sock_set->poll_array_elems, (gdb) list 118 just use the same code as above. Otherwise, use 119 multithreaded code (and we don't then need the 120 MPIU_THREAD_CHECK_BEGIN/END macros) */ 121 if (!MPIR_ThreadInfo.isThreaded) { 122 MPIDI_FUNC_ENTER(MPID_STATE_POLL); 123 n_fds = poll(sock_set->pollfds, sock_set->poll_array_elems, 124 millisecond_timeout); 125 MPIDI_FUNC_EXIT(MPID_STATE_POLL); 126 } 127 else (gdb) print *sock_set $2 = {id = 0, starting_elem = 0, poll_array_sz = 32, poll_array_elems = 1, pollfds = 0x7f57b588, pollinfos = 0x7f57b690, eventq_head = 0x0, eventq_tail = 0x0, pollfds_active = 0x0, pollfds_updated = 0, wakeup_posted = 0, intr_sock = 0x0, intr_fds = {-1, -1}} (gdb) print millisecond_timeout $3 = -1 (gdb) print *sock_set->pollfds $4 = {fd = 6, events = 1, revents = 0} Is there anything else that might be useful in tracking this down? -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0