October 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error: failed to allocate memory for an unexpected message
by XingFENG 02 Oct '14

02 Oct '14

Hi Wesley Bland, Thanks for your reply. My codes is relatively big( around 2000 lines). I will try to make and post one small example later. On Thu, Oct 2, 2014 at 8:50 PM, Wesley Bland <wbland(a)anl.gov> wrote: > Can you provide a minimal example code that reproduced the problem? > > > > On Oct 2, 2014, at 2:13 AM, XingFENG <xingfeng(a)cse.unsw.edu.au> wrote: > > Hi Wesley Bland, > > Thanks for your reply. > > I have modified my codes. For each process, it first receives then sends > message from/to others. However, same error still appears. > > I also noted that the code works fine for single node machine. It crushed > with this error on multi-node cluster. > > > On Sun, Sep 28, 2014 at 10:44 PM, Wesley Bland <wbland(a)anl.gov> wrote: > >> The problem in this situation usually is that you're not posting enough >> receives and too many of your messages are getting buffered by the MPI >> library. Make sure you match up your sends and receives and whenever >> possible you post your receives early. >> >> Wesley >> >> >> >> > On Sep 28, 2014, at 7:13 AM, XingFENG <xingfeng(a)cse.unsw.edu.au> wrote: >> > >> > Hi all, >> > >> > I am running a MPI program on two machines. I got errors as follows: >> > >> > >> > ==================================================================== >> > Fatal error in MPI_Test: Other MPI error, error stack: >> > MPI_Test(153)......................: MPI_Test(request=0xa0a088, >> flag=0x7fff470e86fc, status=0x7fff470e86e0) failed >> > MPIDI_CH3I_Progress(150)...........: >> > MPID_nem_mpich2_test_recv(800).....: >> > MPID_nem_tcp_connpoll(1720)........: >> > state_commrdy_handler(1556)........: >> > MPID_nem_tcp_recv_handler(1459)....: >> > MPID_nem_handle_pkt(493)...........: >> > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an >> unexpected message. 261892 unexpected messages queued. >> > Fatal error in MPI_Test: Other MPI error, error stack: >> > MPI_Test(153)......................: MPI_Test(request=0xadb128, >> flag=0x7fff33cba448, status=0x7fff33cba430) failed >> > MPIDI_CH3I_Progress(150)...........: >> > MPID_nem_mpich2_test_recv(800).....: >> > MPID_nem_tcp_connpoll(1720)........: >> > state_commrdy_handler(1556)........: >> > MPID_nem_tcp_recv_handler(1459)....: >> > MPID_nem_handle_pkt(493)...........: >> > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an >> unexpected message. 261890 unexpected messages queued. >> > rank 1 in job 11 slave_36134 caused collective abort of all ranks >> > exit status of rank 1: killed by signal 9 >> > >> > ==================================================================== >> > >> > >> > I have never seen such errors before. What is the cause of this error? >> Is it an out of memory error? ( There is 20% remaining memory on machines ) >> > >> > Any help would be greatly appreciated. Thanks in advance! >> > >> > >> > -- >> > Best Regards. >> > --- >> > Xing FENG >> > PhD Candidate >> > Database Research Group >> > >> > School of Computer Science and Engineering >> > University of New South Wales >> > NSW 2052, Sydney >> > >> > Phone: (+61) 413 857 288 >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > -- > Best Regards. > --- > Xing FENG > PhD Candidate > Database Research Group > > School of Computer Science and Engineering > University of New South Wales > NSW 2052, Sydney > > Phone: (+61) 413 857 288 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Best Regards. --- Xing FENG PhD Candidate Database Research Group School of Computer Science and Engineering University of New South Wales NSW 2052, Sydney Phone: (+61) 413 857 288 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error: failed to allocate memory for an unexpected message
by Wesley Bland 02 Oct '14

02 Oct '14

Can you provide a minimal example code that reproduced the problem? > On Oct 2, 2014, at 2:13 AM, XingFENG <xingfeng(a)cse.unsw.edu.au> wrote: > > Hi Wesley Bland, > > Thanks for your reply. > > I have modified my codes. For each process, it first receives then sends message from/to others. However, same error still appears. > > I also noted that the code works fine for single node machine. It crushed with this error on multi-node cluster. > > >> On Sun, Sep 28, 2014 at 10:44 PM, Wesley Bland <wbland(a)anl.gov> wrote: >> The problem in this situation usually is that you're not posting enough receives and too many of your messages are getting buffered by the MPI library. Make sure you match up your sends and receives and whenever possible you post your receives early. >> >> Wesley >> >> >> >> > On Sep 28, 2014, at 7:13 AM, XingFENG <xingfeng(a)cse.unsw.edu.au> wrote: >> > >> > Hi all, >> > >> > I am running a MPI program on two machines. I got errors as follows: >> > >> > >> > ==================================================================== >> > Fatal error in MPI_Test: Other MPI error, error stack: >> > MPI_Test(153)......................: MPI_Test(request=0xa0a088, flag=0x7fff470e86fc, status=0x7fff470e86e0) failed >> > MPIDI_CH3I_Progress(150)...........: >> > MPID_nem_mpich2_test_recv(800).....: >> > MPID_nem_tcp_connpoll(1720)........: >> > state_commrdy_handler(1556)........: >> > MPID_nem_tcp_recv_handler(1459)....: >> > MPID_nem_handle_pkt(493)...........: >> > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an unexpected message. 261892 unexpected messages queued. >> > Fatal error in MPI_Test: Other MPI error, error stack: >> > MPI_Test(153)......................: MPI_Test(request=0xadb128, flag=0x7fff33cba448, status=0x7fff33cba430) failed >> > MPIDI_CH3I_Progress(150)...........: >> > MPID_nem_mpich2_test_recv(800).....: >> > MPID_nem_tcp_connpoll(1720)........: >> > state_commrdy_handler(1556)........: >> > MPID_nem_tcp_recv_handler(1459)....: >> > MPID_nem_handle_pkt(493)...........: >> > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an unexpected message. 261890 unexpected messages queued. >> > rank 1 in job 11 slave_36134 caused collective abort of all ranks >> > exit status of rank 1: killed by signal 9 >> > >> > ==================================================================== >> > >> > >> > I have never seen such errors before. What is the cause of this error? Is it an out of memory error? ( There is 20% remaining memory on machines ) >> > >> > Any help would be greatly appreciated. Thanks in advance! >> > >> > >> > -- >> > Best Regards. >> > --- >> > Xing FENG >> > PhD Candidate >> > Database Research Group >> > >> > School of Computer Science and Engineering >> > University of New South Wales >> > NSW 2052, Sydney >> > >> > Phone: (+61) 413 857 288 >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Best Regards. > --- > Xing FENG > PhD Candidate > Database Research Group > > School of Computer Science and Engineering > University of New South Wales > NSW 2052, Sydney > > Phone: (+61) 413 857 288 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error: failed to allocate memory for an unexpected message
by XingFENG 02 Oct '14

02 Oct '14

Hi Wesley Bland, Thanks for your reply. I have modified my codes. For each process, it first receives then sends message from/to others. However, same error still appears. I also noted that the code works fine for single node machine. It crushed with this error on multi-node cluster. On Sun, Sep 28, 2014 at 10:44 PM, Wesley Bland <wbland(a)anl.gov> wrote: > The problem in this situation usually is that you're not posting enough > receives and too many of your messages are getting buffered by the MPI > library. Make sure you match up your sends and receives and whenever > possible you post your receives early. > > Wesley > > > > > On Sep 28, 2014, at 7:13 AM, XingFENG <xingfeng(a)cse.unsw.edu.au> wrote: > > > > Hi all, > > > > I am running a MPI program on two machines. I got errors as follows: > > > > > > ==================================================================== > > Fatal error in MPI_Test: Other MPI error, error stack: > > MPI_Test(153)......................: MPI_Test(request=0xa0a088, > flag=0x7fff470e86fc, status=0x7fff470e86e0) failed > > MPIDI_CH3I_Progress(150)...........: > > MPID_nem_mpich2_test_recv(800).....: > > MPID_nem_tcp_connpoll(1720)........: > > state_commrdy_handler(1556)........: > > MPID_nem_tcp_recv_handler(1459)....: > > MPID_nem_handle_pkt(493)...........: > > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an > unexpected message. 261892 unexpected messages queued. > > Fatal error in MPI_Test: Other MPI error, error stack: > > MPI_Test(153)......................: MPI_Test(request=0xadb128, > flag=0x7fff33cba448, status=0x7fff33cba430) failed > > MPIDI_CH3I_Progress(150)...........: > > MPID_nem_mpich2_test_recv(800).....: > > MPID_nem_tcp_connpoll(1720)........: > > state_commrdy_handler(1556)........: > > MPID_nem_tcp_recv_handler(1459)....: > > MPID_nem_handle_pkt(493)...........: > > MPIDI_CH3_PktHandler_EagerSend(589): Failed to allocate memory for an > unexpected message. 261890 unexpected messages queued. > > rank 1 in job 11 slave_36134 caused collective abort of all ranks > > exit status of rank 1: killed by signal 9 > > > > ==================================================================== > > > > > > I have never seen such errors before. What is the cause of this error? > Is it an out of memory error? ( There is 20% remaining memory on machines ) > > > > Any help would be greatly appreciated. Thanks in advance! > > > > > > -- > > Best Regards. > > --- > > Xing FENG > > PhD Candidate > > Database Research Group > > > > School of Computer Science and Engineering > > University of New South Wales > > NSW 2052, Sydney > > > > Phone: (+61) 413 857 288 > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Best Regards. --- Xing FENG PhD Candidate Database Research Group School of Computer Science and Engineering University of New South Wales NSW 2052, Sydney Phone: (+61) 413 857 288 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Installing MPICH in Windows using Cygwin
by "Antonio J. Peña" 01 Oct '14

01 Oct '14

Hi Mike, Sorry, we are not providing support for Windows since some time ago already: http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_can… Please use Microsoft's (http://msdn.microsoft.com/en-us/library/bb524831%28v=vs.85%29.aspx) or Intel's MPICH derivatives. Best, Antonio On 10/01/2014 05:42 PM, mhgardner wrote: > Hi, > > I'm trying to install MPICH in Windows using Cygwin, but when I try to > build it says no targets are specified. I'm following the install > instructions from the following page: > http://www.mpich.org/static/downloads/3.1.2/mpich-3.1.2-installguide.pdf > The c.txt and m.txt files are attached. > Not sure where things are going wrong, so any guidance would great. > > Thanks, > > Mike > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Installing MPICH in Windows using Cygwin
by mhgardner 01 Oct '14

01 Oct '14

Hi, I'm trying to install MPICH in Windows using Cygwin, but when I try to build it says no targets are specified. I'm following the install instructions from the following page: http://www.mpich.org/static/downloads/3.1.2/mpich-3.1.2-installguide.pdf The c.txt and m.txt files are attached. Not sure where things are going wrong, so any guidance would great. Thanks, Mike _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [mvapich-discuss] Lustre 2.4 bug is back in MV2-2.1a
by Jonathan Perkins 01 Oct '14

01 Oct '14

On Wed, Oct 01, 2014 at 11:25:33AM -0500, Rob Latham wrote: > > > On 10/01/2014 10:33 AM, Jonathan Perkins wrote: > >Hello MPICH community. Recently we had a report of a regression on the > >MVAPICH discuss list and I've found that the regression appears in > >MPICH-3.1.2 itself. > > > >We plan on just removing the _XOPEN_SOURCE lines but thought that you'd > >like to know about this so that it could be fixed in MPICH as well. > >Thanks! > > so, on solaris we needed to put back _XOPEN_SOURCE lines with > --enable-strict in order to bring in pread/pwrite declarations, I think. > but i had forgotten about the caddr_t time compile issue. > > thanks for the report. Ah, thanks for that info. I figured that there was a chance that the proper fix might be a little bit more involved. We'll keep our eye on how you guys deal with this. Thanks! -- Jonathan Perkins _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [mvapich-discuss] Lustre 2.4 bug is back in MV2-2.1a
by Rob Latham 01 Oct '14

01 Oct '14

On 10/01/2014 10:33 AM, Jonathan Perkins wrote: > Hello MPICH community. Recently we had a report of a regression on the > MVAPICH discuss list and I've found that the regression appears in > MPICH-3.1.2 itself. > > We plan on just removing the _XOPEN_SOURCE lines but thought that you'd > like to know about this so that it could be fixed in MPICH as well. > Thanks! so, on solaris we needed to put back _XOPEN_SOURCE lines with --enable-strict in order to bring in pread/pwrite declarations, I think. but i had forgotten about the caddr_t time compile issue. thanks for the report. ==rob > > On Tue, Sep 30, 2014 at 07:18:45PM -0400, Jonathan Perkins wrote: >> Thanks for the report Adam. We'll review the fix and ticket and see >> what we do to prevent this from appearing in future releases. >> >> On Tue, Sep 30, 2014 at 04:07:24PM -0700, Adam T. Moody wrote: >>> Hello MVAPICH team, >>> While building MV2-2.1a on our systems, I'm getting the following error: >>> >>> In file included from /usr/include/linux/lustre_user.h:46, >>> from /usr/include/lustre/lustre_user.h:54, >>> from adio/ad_lustre/ad_lustre.h:30, >>> from adio/ad_lustre/ad_lustre_rwcontig.c:20: >>> /usr/include/sys/quota.h:221: error: expected declaration specifiers or >>> '...' before 'caddr_t' >>> make[3]: *** [adio/ad_lustre/ad_lustre_rwcontig.lo] Error 1 >>> >>> This happens while trying to build >>> mvapich2-2.1a/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c. >>> >>> This is due to an incompatibility with Lustre 2.4. I first saw this with >>> MV2-1.9, it was fixed in MV2-2.0, but now it's back with MV2-2.1a. >>> >>> The problem lines are here, which can be found starting at line 5: >>> >>> #ifdef _STDC_C99 >>> #define _XOPEN_SOURCE 600 >>> #else >>> #define _XOPEN_SOURCE 500 >>> #endif >>> >>> The solution before was to not define _XOPEN_SOURCE in this file. If I >>> comment out these definitions, I've verified that MV2-2.1a builds. >>> >>> Here is a related MPICH ticket reported by Yan Li and Jerome Vienne: >>> >>> http://trac.mpich.org/projects/mpich/ticket/1973 >>> -Adam >>> _______________________________________________ >>> mvapich-discuss mailing list >>> mvapich-discuss(a)cse.ohio-state.edu >>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss >> >> -- >> Jonathan Perkins > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [mvapich-discuss] Lustre 2.4 bug is back in MV2-2.1a
by Jonathan Perkins 01 Oct '14

01 Oct '14

Hello MPICH community. Recently we had a report of a regression on the MVAPICH discuss list and I've found that the regression appears in MPICH-3.1.2 itself. We plan on just removing the _XOPEN_SOURCE lines but thought that you'd like to know about this so that it could be fixed in MPICH as well. Thanks! On Tue, Sep 30, 2014 at 07:18:45PM -0400, Jonathan Perkins wrote: > Thanks for the report Adam. We'll review the fix and ticket and see > what we do to prevent this from appearing in future releases. > > On Tue, Sep 30, 2014 at 04:07:24PM -0700, Adam T. Moody wrote: > > Hello MVAPICH team, > > While building MV2-2.1a on our systems, I'm getting the following error: > > > > In file included from /usr/include/linux/lustre_user.h:46, > > from /usr/include/lustre/lustre_user.h:54, > > from adio/ad_lustre/ad_lustre.h:30, > > from adio/ad_lustre/ad_lustre_rwcontig.c:20: > > /usr/include/sys/quota.h:221: error: expected declaration specifiers or > > '...' before 'caddr_t' > > make[3]: *** [adio/ad_lustre/ad_lustre_rwcontig.lo] Error 1 > > > > This happens while trying to build > > mvapich2-2.1a/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c. > > > > This is due to an incompatibility with Lustre 2.4. I first saw this with > > MV2-1.9, it was fixed in MV2-2.0, but now it's back with MV2-2.1a. > > > > The problem lines are here, which can be found starting at line 5: > > > > #ifdef _STDC_C99 > > #define _XOPEN_SOURCE 600 > > #else > > #define _XOPEN_SOURCE 500 > > #endif > > > > The solution before was to not define _XOPEN_SOURCE in this file. If I > > comment out these definitions, I've verified that MV2-2.1a builds. > > > > Here is a related MPICH ticket reported by Yan Li and Jerome Vienne: > > > > http://trac.mpich.org/projects/mpich/ticket/1973 > > -Adam > > _______________________________________________ > > mvapich-discuss mailing list > > mvapich-discuss(a)cse.ohio-state.edu > > http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss > > -- > Jonathan Perkins -- Jonathan Perkins _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0