October 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Pavan Balaji 23 Oct '13

23 Oct '13

Luiz, You don’t need to know how many messages you will receive. You only need to make sure that whenever a message comes in, it knows which buffer it should go into. One way to fix that is to post N receives with ANY_SOURCE and ANY_TAG: MPI_Irecv(.., MPI_ANY_SOURCE, MPI_ANY_TAG, ..); Then test for the first one to see if something came in. Whenever one message comes in, process it and repost it as another Irecv. In this case, you should not have any unexpected messages (unless you have multiple communicators, which is a different story). —- Pavan On Oct 23, 2013, at 4:22 PM, Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br> wrote: > Hi Antonio, > > Thanks for your quick reply. > > I confess I have to study a bit more about this, but I think I understood your suggestion. > After a little research, I understood that when I use MPI_IRecv I am performing a non-blocking operation to just inform that I have a message to receive and then after MPI_Waitany returns I can access data to write it on the file. In this kind of implementation, I inform that I have a bunch of messages to receive from worker processes and, because of that, I would have to have a buffer in the writer process to receive messages asynchronously. The higher efficiency is archived because, while the writer process is performing the IO operation (writing to hard disk), MPI can transfer the received data in the meanwhile to my pre-allocated buffer in my application. Is all this right? > > But, what if I don't know the number of messages I will receive? > That's my case, actually... I don't even know which worker process will send me data and I also don't know how many times each one of them will send me messages. I use a master-slaves scheme to distribute tasks, so the number of calculations done in each worker process (and thus, data send to writer process) depends on the speed each worker and how many times they ask for tasks. It is easy to deal with messages coming from a unknown sender, but I don't know how to deal with an unknown number of messages (i.e., how many MPI_IRecv I have to call?). Any idea? > > As I said before, I considered having more than one writer process, but I can't see how this would solve the problem, that seems to be related to IO disk speed. In other words, why it would be worth having more than one writer process if, at the end, I have only one hard disk to perform IO operations? > > Thanks again. > > Regards, Luiz > > > On 23 October 2013 17:42, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: > > Hi Luiz, > > Your error trace indicates that the receiver went out of memory due to a too large amount (261,895) of eager unexpected messages received, i.e., small messages received without a matching receive operation. Whenever this happens, the receiver allocates a temporary buffer to hold the received message. This exhausted the available memory in the computer where the receiver was executing. > > To avoid this, try to pre-post receives before messages arrive. Indeed, this is far more efficient. Maybe you could do an MPI_IRecv per worker in your writer process, and process them after an MPI_Waitany. You may also consider having multiple writer processes if your use case permits and the volume of received messages is too high to be processed by a single writer. > > Antonio > > > On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior wrote: > Hi, > > I am getting the following error when running my parallel application: > > MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed > MPIDI_CH3I_Progress(402)...........: > MPID_nem_mpich2_blocking_recv(905).: > MPID_nem_tcp_connpoll(1838)........: > state_commrdy_handler(1676)........: > MPID_nem_tcp_recv_handler(1564)....: > MPID_nem_handle_pkt(636)...........: > MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued. > Fatal error in MPI_Send: Other MPI error, error stack: > MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed > MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer > > I went to MPICH's FAQ (http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_…) > It says that most likely the receiver process can't cope to process the high number of messages it is receiving. > > In my application, the worker processes perform a very large number of small computations and, after some computation is complete, they sent the data to a special "writer" process that is responsible to write the output to disk. > This scheme use to work in a very reasonable fashion, until we faced some new data with larger parameters that caused the problem above. > > Even though we can redesign the application, for example, by creating a pool of writer process we still have only one hard disk, so the bottleneck would not be solved. So, this doesn't seem to be a good approach. > > As far as I understood, MPICH saves the content of every MPI_SEND in a internal buffer (I don't know where the buffer in located, sender or receiver?) to allow asynchronous sender's computation while the messages are being received. > The problem is that buffer has been exhausted due some resource limitation. > > It is very interesting to have a buffer but if the buffer in the writer process is close to its limit the workers processes should stop and wait until it frees some space to restart sending new data to be written to disk. > > Is it possible to check this buffer in MPICH? Or is it possible to check the number of messages to be received? > Can anyone suggest a better (easy to implement) solution? > > Thanks in advance. > > Regards, > Luiz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Compilation error of mpich-3.0.4 with Lustre 2.4.0
by Yan Li 23 Oct '13

23 Oct '13

Hello, I'm trying to compile mpich-3.0.4 on CentOS 6.4 x86-64 running Lustre 2.4.0, which was installed by using RPMs from http://zfsonlinux.org/lustre.html Configure command used: ./configure --prefix=/share/ --with-file-system=ufs+lustre There's a compilation error: ~~~~ [snip] ~~~~ CC adio/ad_lustre/ad_lustre_open.lo CC adio/ad_lustre/ad_lustre_rwcontig.lo In file included from /usr/include/linux/lustre_user.h:46, from /usr/include/lustre/lustre_user.h:54, from adio/ad_lustre/ad_lustre.h:30, from adio/ad_lustre/ad_lustre_rwcontig.c:14: /usr/include/sys/quota.h:221: error: expected declaration specifiers or '...' before 'caddr_t' ~~~~ [snip] ~~~~ As you can see that the file ad_lustre_open.c compiles OK. The difference between it and ad_lustre_rwcontig.c is that the latter includes this line: #define _XOPEN_SOURCE 600 before including the lustre header file. It seems to me that defining _XOPEN_SOURCE to 600 on Linux implies that caddr_t is defined somewhere else, suppressing the definition of caddr_t in <sys/types.h>, thus leads to the compilation error of <sys/quota.h>. Removing the definition of _XOPEN_SOURCE from ad_lustre_rwcontig.c seems fixed this problem, but I want to know why the definition was put there in the first place and whether removing it can cause any potential problem. Thanks for your help in advance. Yan Li Graduate Student Researcher, Storage Systems Research Center, Computer Science, University of California, Santa Cruz

1 0

Re: [mpich-discuss] mpiexec crash
by Cherukumilli, Vasu 23 Oct '13

23 Oct '13

Hi, Was wondering if you guys had any insight in to this assert failure. We are seeing this intermittently. The mpiexec command we are executing is similar to the one pasted below: mpiexec -n 1 <exec1> : -n 1 <exec2> : -n 1 <exec3> : -n 1 <exec4> : -n 1 <exec5> : -n 1 <exec6> : -n 1 <exec7> Crash that we are seeing: [proxy:0:0@gretel] HYD_pmcd_pmip_control_cmd_cb (</PATH/TO>/src/pm/hydra/pm/pmiserv/pmip_cb.c:934): assert (!closed) failed [proxy:0:0@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [proxy:0:0@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmip.c:210): demux engine error waiting for event [mpiexec@gretel] control_cb (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:201): assert (!closed) failed [mpiexec@gretel] HYDT_dmxu_poll_wait_for_event (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [mpiexec@gretel] HYD_pmci_wait_for_completion (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec@gretel] main (</PATH/TO>/src/mpich2-1.5/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion No such file or directory. (errno = ENOENT) Command exited with non-zero status 20 Thanks, Vasu

1 0

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Luiz Carlos da Costa Junior 23 Oct '13

23 Oct '13

Hi Antonio, Thanks for your quick reply. I confess I have to study a bit more about this, but I think I understood your suggestion. After a little research, I understood that when I use MPI_IRecv I am performing a non-blocking operation to just inform that I have a message to receive and then after MPI_Waitany returns I can access data to write it on the file. In this kind of implementation, I inform that I have a bunch of messages to receive from worker processes and, because of that, I would have to have a buffer in the writer process to receive messages asynchronously. The higher efficiency is archived because, while the writer process is performing the IO operation (writing to hard disk), MPI can transfer the received data in the meanwhile to my pre-allocated buffer in my application. Is all this right? But, what if I don't know the number of messages I will receive? That's my case, actually... I don't even know which worker process will send me data and I also don't know how many times each one of them will send me messages. I use a master-slaves scheme to distribute tasks, so the number of calculations done in each worker process (and thus, data send to writer process) depends on the speed each worker and how many times they ask for tasks. It is easy to deal with messages coming from a unknown sender, but I don't know how to deal with an unknown number of messages (i.e., how many MPI_IRecv I have to call?). Any idea? As I said before, I considered having more than one writer process, but I can't see how this would solve the problem, that seems to be related to IO disk speed. In other words, why it would be worth having more than one writer process if, at the end, I have only one hard disk to perform IO operations? Thanks again. Regards, Luiz On 23 October 2013 17:42, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: > ** > > > > Hi Luiz, > > > > Your error trace indicates that the receiver went out of memory due to a > too large amount (261,895) of eager unexpected messages received, i.e., > small messages received without a matching receive operation. Whenever this > happens, the receiver allocates a temporary buffer to hold the received > message. This exhausted the available memory in the computer where the > receiver was executing. > > > > To avoid this, try to pre-post receives before messages arrive. Indeed, > this is far more efficient. Maybe you could do an MPI_IRecv per worker in > your writer process, and process them after an MPI_Waitany. You may also > consider having multiple writer processes if your use case permits and the > volume of received messages is too high to be processed by a single writer. > > > > Antonio > > > > > > On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior > wrote: > > Hi, > > > I am getting the following error when running my parallel application: > > > MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, > MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed > > MPIDI_CH3I_Progress(402)...........: > > MPID_nem_mpich2_blocking_recv(905).: > > MPID_nem_tcp_connpoll(1838)........: > > state_commrdy_handler(1676)........: > > MPID_nem_tcp_recv_handler(1564)....: > > MPID_nem_handle_pkt(636)...........: > > MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an > unexpected message. 261895 unexpected messages queued. > > Fatal error in MPI_Send: Other MPI error, error stack: > > MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, > MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed > > MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection > reset by peer > > > I went to MPICH's FAQ ( > http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_…) > > > It says that most likely the receiver process can't cope to process the > high number of messages it is receiving. > > > In my application, the worker processes perform a very large number of > small computations and, after some computation is complete, they sent the > data to a special "writer" process that is responsible to write the output > to disk. > > This scheme use to work in a very reasonable fashion, until we faced some > new data with larger parameters that caused the problem above. > > > Even though we can redesign the application, for example, by creating a > pool of writer process we still have only one hard disk, so the bottleneck > would not be solved. So, this doesn't seem to be a good approach. > > > As far as I understood, MPICH saves the content of every MPI_SEND in a > internal buffer (I don't know where the buffer in located, sender or > receiver?) to allow asynchronous sender's computation while the messages > are being received. > > The problem is that buffer has been exhausted due some resource limitation. > > > It is very interesting to have a buffer but if the buffer in the writer > process is close to its limit the workers processes should stop and wait > until it frees some space to restart sending new data to be written to > disk. > > > Is it possible to check this buffer in MPICH? Or is it possible to check > the number of messages to be received? > > Can anyone suggest a better (easy to implement) solution? > > > Thanks in advance. > > > Regards, > > Luiz > > >

1 0

Re: [mpich-discuss] running parallel job issue
by Alexandra Betouni 23 Oct '13

23 Oct '13

Ok Antonio I know what you're talking about‎. I will try this and I hope to solve it! Thanks anyway to all of you :) Sent from my BlackBerry 10 smartphone. From: discuss-request(a)mpich.org Sent: Wednesday, 23 October 2013 23:22 To: discuss(a)mpich.org Reply To: discuss(a)mpich.org Subject: discuss Digest, Vol 12, Issue 15 Send discuss mailing list submissions to discuss(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to discuss-request(a)mpich.org You can reach the person managing the list at discuss-owner(a)mpich.org When replying, please edit your Subject line so it is more specific than "Re: Contents of discuss digest..." Today's Topics: 1. Re: running parralel job issue(alexandra) (Antonio J. Pe?a) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Oct 2013 15:21:57 -0500 From: Antonio J. Pe?a <apenya(a)mcs.anl.gov> To: discuss(a)mpich.org Subject: Re: [mpich-discuss] running parralel job issue(alexandra) Message-ID: <2206013.AaZ6q0B8ti(a)localhost.localdomain> Content-Type: text/plain; charset="utf-8" The address you got is nothing but a loopback address. I mean, that's not the real IP address of your network interfaces, and is only used for self communications through the sockets interface. In Linux, you can determine the IP address of your network interface with something like: /sbin/ifconfig eth0 (you may need to replace eth0 with the identifier of your network interface, but in most cases this should work). You should assign different IP addresses to all your computers in the same network in order to be able to perform any communication among them. You most likely will want to assign private addresses, such as 192.168.0.1, 192.168.0.2, etc. You can easily find how to do this by googling a little bit. Antonio On Wednesday, October 23, 2013 11:10:48 PM Alexandra Betouni wrote: ?Well yes, the job runs on host2 locally, but parallel execution does the same thing like on host1. Someone here said that if all the computers have the same ip direction it won't work.. Well, every node has ?127.0.1.1 As ip, and all of them had same host name till I changed the two of them. Hydra is the default launcher. I also forgot to mencione that ping host2 and same, ping host1 works fine... Sent from my BlackBerry 10 smartphone. *From: *?discuss-request(a)mpich.org *Sent: *?Wednesday, 23 October 2013 22:42 *To: *discuss(a)mpich.org *Reply To: *discuss(a)mpich.org *Subject: *discuss Digest, Vol 12, Issue 13 Send discuss mailing list submissions to discuss(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss[1] _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ http://lists.mpich.org/pipermail/discuss/attachments/20131023/fdf9d820/at tachment-0001.html[2]> ------------------------------ Message: 3Date: Wed, 23 Oct 2013 17:27:27 -0200From: Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br>To: MPICH Discuss <mpich- discuss(a)mcs.anl.gov>Subject: [mpich-discuss] Failed to allocate memory for an unexpected messageMessage-ID: <CAOv4ofRY4ajVZecZcDN3d3tdENV=XBMd=5i1TjX3310ZnEFUdg(a)mail.gmai l.com>Content-Type: text/plain; charset="iso-8859-1" Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060,MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failedMPIDI_CH3I_Progress(402)...........:MPID_nem_mpich2_blocking_recv(90 5).:MPID_nem_tcp_connpoll(1838)........:state_commrdy_handler(1676)........ :MPID_nem_tcp_recv_handler(1564)....:MPID_nem_handle_pkt(636)...........:M PIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for anunexpected message. 261895 unexpected messages queued.Fatal error in MPI_Send: Other MPI error, error stack:MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060,MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failedMPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connectionreset by peer I went to MPICH's FAQ ( http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why _am_I_getting_so_many_unexpected_messages.3F[3] http://lists.mpich.org/pipermail/discuss/attachments/20131023/3a02fa51/a ttachment-0001.html[4]> ------------------------------ Message: 4Date: Wed, 23 Oct 2013 14:42:15 -0500From: Antonio J. Pe?a <apenya(a)mcs.anl.gov>To: discuss(a)mpich.orgCc: MPICH Discuss <mpich- discuss(a)mcs.anl.gov>Subject: Re: [mpich-discuss] Failed to allocate memory for an unexpected messageMessage-ID: <1965559.SsluspJNke(a)localhost.localdomain>Content-Type: text/plain; charset="iso-8859-1" Hi Luiz, Your error trace indicates that the receiver went out of memory due to a too large amount (261,895) of eager unexpected messages received, i.e., small messages received without a matching receive operation. Whenever this happens, the receiver allocates a temporary buffer to hold the received message. This exhausted the available memory in the computer where the receiver was executing. To avoid this, try to pre-post receives before messages arrive. Indeed, this is far more efficient. Maybe you could do an MPI_IRecv per worker in your writer process, and process them after an MPI_Waitany. You may also consider having multiple writer processes if your use case permits and the volume of received messages is too high to be processed by a single writer. Antonio On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior wrote: Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed MPIDI_CH3I_Progress(402)...........: MPID_nem_mpich2_blocking_recv(905).: MPID_nem_tcp_connpoll(1838)........: state_commrdy_handler(1676)........: MPID_nem_tcp_recv_handler(1564)....: MPID_nem_handle_pkt(636)...........: MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued. Fatal error in MPI_Send: Other MPI error, error stack:MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer I went to MPICH's FAQ (http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Wh y_am_I_getting_so_many_unexpected_messages.3F[1]). It says that most likely the receiver process can't cope to process the high number of messages it is receiving.

1 0

Re: [mpich-discuss] running parralel job issue(alexandra)
by Antonio J. Peña 23 Oct '13

23 Oct '13

The address you got is nothing but a loopback address. I mean, that's not the real IP address of your network interfaces, and is only used for self communications through the sockets interface. In Linux, you can determine the IP address of your network interface with something like: /sbin/ifconfig eth0 (you may need to replace eth0 with the identifier of your network interface, but in most cases this should work). You should assign different IP addresses to all your computers in the same network in order to be able to perform any communication among them. You most likely will want to assign private addresses, such as 192.168.0.1, 192.168.0.2, etc. You can easily find how to do this by googling a little bit. Antonio On Wednesday, October 23, 2013 11:10:48 PM Alexandra Betouni wrote: ‎Well yes, the job runs on host2 locally, but parallel execution does the same thing like on host1. Someone here said that if all the computers have the same ip direction it won't work.. Well, every node has ‎127.0.1.1 As ip, and all of them had same host name till I changed the two of them. Hydra is the default launcher. I also forgot to mencione that ping host2 and same, ping host1 works fine... Sent from my BlackBerry 10 smartphone. *From: *‎discuss-request(a)mpich.org *Sent: *‎Wednesday, 23 October 2013 22:42 *To: *discuss(a)mpich.org *Reply To: *discuss(a)mpich.org *Subject: *discuss Digest, Vol 12, Issue 13 Send discuss mailing list submissions to discuss(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss[1] _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ _https://lists.mpich.org/mailman/listinfo/discuss_ http://lists.mpich.org/pipermail/discuss/attachments/20131023/fdf9d820/at tachment-0001.html[2]> ------------------------------ Message: 3Date: Wed, 23 Oct 2013 17:27:27 -0200From: Luiz Carlos da Costa Junior <lcjunior(a)ufrj.br>To: MPICH Discuss <mpich- discuss(a)mcs.anl.gov>Subject: [mpich-discuss] Failed to allocate memory for an unexpected messageMessage-ID: <CAOv4ofRY4ajVZecZcDN3d3tdENV=XBMd=5i1TjX3310ZnEFUdg(a)mail.gmai l.com>Content-Type: text/plain; charset="iso-8859-1" Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060,MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failedMPIDI_CH3I_Progress(402)...........:MPID_nem_mpich2_blocking_recv(90 5).:MPID_nem_tcp_connpoll(1838)........:state_commrdy_handler(1676)........ :MPID_nem_tcp_recv_handler(1564)....:MPID_nem_handle_pkt(636)...........:M PIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for anunexpected message. 261895 unexpected messages queued.Fatal error in MPI_Send: Other MPI error, error stack:MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060,MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failedMPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connectionreset by peer I went to MPICH's FAQ ( http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why _am_I_getting_so_many_unexpected_messages.3F[3] http://lists.mpich.org/pipermail/discuss/attachments/20131023/3a02fa51/a ttachment-0001.html[4]> ------------------------------ Message: 4Date: Wed, 23 Oct 2013 14:42:15 -0500From: Antonio J. Pe?a <apenya(a)mcs.anl.gov>To: discuss(a)mpich.orgCc: MPICH Discuss <mpich- discuss(a)mcs.anl.gov>Subject: Re: [mpich-discuss] Failed to allocate memory for an unexpected messageMessage-ID: <1965559.SsluspJNke(a)localhost.localdomain>Content-Type: text/plain; charset="iso-8859-1" Hi Luiz, Your error trace indicates that the receiver went out of memory due to a too large amount (261,895) of eager unexpected messages received, i.e., small messages received without a matching receive operation. Whenever this happens, the receiver allocates a temporary buffer to hold the received message. This exhausted the available memory in the computer where the receiver was executing. To avoid this, try to pre-post receives before messages arrive. Indeed, this is far more efficient. Maybe you could do an MPI_IRecv per worker in your writer process, and process them after an MPI_Waitany. You may also consider having multiple writer processes if your use case permits and the volume of received messages is too high to be processed by a single writer. Antonio On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior wrote: Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed MPIDI_CH3I_Progress(402)...........: MPID_nem_mpich2_blocking_recv(905).: MPID_nem_tcp_connpoll(1838)........: state_commrdy_handler(1676)........: MPID_nem_tcp_recv_handler(1564)....: MPID_nem_handle_pkt(636)...........: MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued. Fatal error in MPI_Send: Other MPI error, error stack:MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer I went to MPICH's FAQ (http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Wh y_am_I_getting_so_many_unexpected_messages.3F[1]). It says that most likely the receiver process can't cope to process the high number of messages it is receiving.

1 0

Re: [mpich-discuss] running parralel job issue(alexandra)
by Alexandra Betouni 23 Oct '13

23 Oct '13

‎Well yes, the job runs on host2 locally, but parallel execution does the same thing like on host1. Someone here said that if all the computers have the same ip direction it won't work.. Well, every node has ‎127.0.1.1 As ip, and all of them had same host name till I changed the two of them. Hydra is the default launcher. I also forgot to mencione that ping host2 and same, ping host1 works fine... Sent from my BlackBerry 10 smartphone. From: ‎discuss-request(a)mpich.org Sent: ‎Wednesday, 23 October 2013 22:42 To: discuss(a)mpich.org Reply To: discuss(a)mpich.org Subject: discuss Digest, Vol 12, Issue 13 Send discuss mailing list submissions to discuss(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to discuss-request(a)mpich.org You can reach the person managing the list at discuss-owner(a)mpich.org When replying, please edit your Subject line so it is more specific than "Re: Contents of discuss digest..." Today's Topics: 1. Re: running parallel job issue! (Reuti) 2. Re: running parallel job issue! (Ricardo Rom?n Brenes) 3. Failed to allocate memory for an unexpected message (Luiz Carlos da Costa Junior) 4. Re: Failed to allocate memory for an unexpected message (Antonio J. Pe?a) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Oct 2013 17:00:53 +0200 From: Reuti <reuti(a)staff.uni-marburg.de> To: discuss(a)mpich.org Subject: Re: [mpich-discuss] running parallel job issue! Message-ID: <071A0111-ECEF-437E-9D3C-7412FDCC84D5(a)staff.uni-marburg.de> Content-Type: text/plain; charset=iso-8859-1 Am 23.10.2013 um 16:51 schrieb Ricardo Rom?n Brenes: > Hi. I'm not sure how it's done on that version of mpich, but you need to be sure that the mpd daemon is running on both hosts (mpdboot or maybe mpiexec starts it ). No. The `mpdboot` is gone for some time - nowadays Hydra is used (since MPICH2 v1.3). -- Reuti > Does cpi runs in hosts 2 locally? > > On Oct 23, 2013 8:47 AM, "Alexandra Betouni" <alexandra_99(a)windowslive.com> wrote: > > > > > Hey there, I am trying to set up a parallel invironment with 14 machines, running Linux XUbuntu, connected via ethernet. > They all have same IP's and same hostnames. Well I started installing mpich-3.0.4 on a single machine, I run the cpi example on localhost by giving mpiexec -host localhost -n 4 ./examples/cpi and everything worked fine! > So I continued changing the hostnames of 2 pc's for a start, and setting up the ssh in these two, also I installed the mpich-3.0.4 on the other machine too. > By giving the ssh <othermachine> date commant , I get the date of the other host without giving a password, so I think I passed that step too. > Next step was to check if the mpich-3.0.4 runs parallel, so I created a machine file (I made a text file giving the hostnames of the two computers , host1 and host2), and save it in my mpich-3.0.4 build directory. Though when I am trying to parallel run the cpi code by giving mpiexec -n 4 -f machinefile ./examples/cpi on my working directory, I get NO errors but neither parallel job... > All processes still running on host1 which is my work station. > What am I doing wrong? > Thanks > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss ------------------------------ Message: 2 Date: Wed, 23 Oct 2013 09:04:29 -0600 From: Ricardo Rom?n Brenes <roman.ricardo(a)gmail.com> To: discuss(a)mpich.org Subject: Re: [mpich-discuss] running parallel job issue! Message-ID: <CAG-vK_yyAQ_CNFhYBxqoMniE01HVMA2YxJH=wC79m9KkBDVWFg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Then the mpiexec should handle it... sorry, I use an outdated version that is in centos repos On Oct 23, 2013 9:01 AM, "Reuti" <reuti(a)staff.uni-marburg.de> wrote: > Am 23.10.2013 um 16:51 schrieb Ricardo Rom?n Brenes: > > > Hi. I'm not sure how it's done on that version of mpich, but you need to > be sure that the mpd daemon is running on both hosts (mpdboot or maybe > mpiexec starts it ). > > No. The `mpdboot` is gone for some time - nowadays Hydra is used (since > MPICH2 v1.3). > > -- Reuti > > > > Does cpi runs in hosts 2 locally? > > > > On Oct 23, 2013 8:47 AM, "Alexandra Betouni" < > alexandra_99(a)windowslive.com> wrote: > > > > > > > > > > Hey there, I am trying to set up a parallel invironment with 14 > machines, running Linux XUbuntu, connected via ethernet. > > They all have same IP's and same hostnames. Well I started installing > mpich-3.0.4 on a single machine, I run the cpi example on localhost by > giving mpiexec -host localhost -n 4 ./examples/cpi and everything worked > fine! > > So I continued changing the hostnames of 2 pc's for a start, and setting > up the ssh in these two, also I installed the mpich-3.0.4 on the other > machine too. > > By giving the ssh <othermachine> date commant , I get the date of the > other host without giving a password, so I think I passed that step too. > > Next step was to check if the mpich-3.0.4 runs parallel, so I created a > machine file (I made a text file giving the hostnames of the two computers > , host1 and host2), and save it in my mpich-3.0.4 build directory. Though > when I am trying to parallel run the cpi code by giving mpiexec -n 4 -f > machinefile ./examples/cpi on my working directory, I get NO errors but > neither parallel job... > > All processes still running on host1 which is my work station. > > What am I doing wrong? > > Thanks > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Failed to allocate memory for an unexpected message
by Antonio J. Peña 23 Oct '13

23 Oct '13

Hi Luiz, Your error trace indicates that the receiver went out of memory due to a too large amount (261,895) of eager unexpected messages received, i.e., small messages received without a matching receive operation. Whenever this happens, the receiver allocates a temporary buffer to hold the received message. This exhausted the available memory in the computer where the receiver was executing. To avoid this, try to pre-post receives before messages arrive. Indeed, this is far more efficient. Maybe you could do an MPI_IRecv per worker in your writer process, and process them after an MPI_Waitany. You may also consider having multiple writer processes if your use case permits and the volume of received messages is too high to be processed by a single writer. Antonio On Wednesday, October 23, 2013 05:27:27 PM Luiz Carlos da Costa Junior wrote: Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed MPIDI_CH3I_Progress(402)...........: MPID_nem_mpich2_blocking_recv(905).: MPID_nem_tcp_connpoll(1838)........: state_commrdy_handler(1676)........: MPID_nem_tcp_recv_handler(1564)....: MPID_nem_handle_pkt(636)...........: MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued. Fatal error in MPI_Send: Other MPI error, error stack: MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer I went to MPICH's FAQ (http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Wh y_am_I_getting_so_many_unexpected_messages.3F[1]). It says that most likely the receiver process can't cope to process the high number of messages it is receiving. In my application, the worker processes perform a very large number of small computations and, after some computation is complete, they sent the data to a special "writer" process that is responsible to write the output to disk. This scheme use to work in a very reasonable fashion, until we faced some new data with larger parameters that caused the problem above. Even though we can redesign the application, for example, by creating a pool of writer process we still have only one hard disk, so the bottleneck would not be solved. So, this doesn't seem to be a good approach. As far as I understood, MPICH saves the content of every MPI_SEND in a internal buffer (I don't know where the buffer in located, sender or receiver?) to allow asynchronous sender's computation while the messages are being received. The problem is that buffer has been exhausted due some resource limitation. It is very interesting to have a buffer but if the buffer in the writer process is close to its limit the workers processes should stop and wait until it frees some space to restart sending new data to be written to disk. Is it possible to check this buffer in MPICH? Or is it possible to check the number of messages to be received? Can anyone suggest a better (easy to implement) solution? Thanks in advance. Regards, Luiz -------- [1] http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why _am_I_getting_so_many_unexpected_messages.3F

1 0

Failed to allocate memory for an unexpected message
by Luiz Carlos da Costa Junior 23 Oct '13

23 Oct '13

Hi, I am getting the following error when running my parallel application: MPI_Recv(186)......................: MPI_Recv(buf=0x125bd840, count=2060, MPI_CHARACTER, src=24, tag=94, comm=0x84000002, status=0x125fcff0) failed MPIDI_CH3I_Progress(402)...........: MPID_nem_mpich2_blocking_recv(905).: MPID_nem_tcp_connpoll(1838)........: state_commrdy_handler(1676)........: MPID_nem_tcp_recv_handler(1564)....: MPID_nem_handle_pkt(636)...........: MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued. Fatal error in MPI_Send: Other MPI error, error stack: MPI_Send(173)..............: MPI_Send(buf=0x765d2e60, count=2060, MPI_CHARACTER, dest=0, tag=94, comm=0x84000004) failed MPID_nem_tcp_connpoll(1826): Communication error with rank 1: Connection reset by peer I went to MPICH's FAQ ( http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_Why_am_… ). It says that most likely the receiver process can't cope to process the high number of messages it is receiving. In my application, the worker processes perform a very large number of small computations and, after some computation is complete, they sent the data to a special "writer" process that is responsible to write the output to disk. This scheme use to work in a very reasonable fashion, until we faced some new data with larger parameters that caused the problem above. Even though we can redesign the application, for example, by creating a pool of writer process we still have only one hard disk, so the bottleneck would not be solved. So, this doesn't seem to be a good approach. As far as I understood, MPICH saves the content of every MPI_SEND in a internal buffer (I don't know where the buffer in located, sender or receiver?) to allow asynchronous sender's computation while the messages are being received. The problem is that buffer has been exhausted due some resource limitation. It is very interesting to have a buffer but if the buffer in the writer process is close to its limit the workers processes should stop and wait until it frees some space to restart sending new data to be written to disk. Is it possible to check this buffer in MPICH? Or is it possible to check the number of messages to be received? Can anyone suggest a better (easy to implement) solution? Thanks in advance. Regards, Luiz

1 0

Re: [mpich-discuss] running parallel job issue!
by Ricardo Román Brenes 23 Oct '13

23 Oct '13

Then the mpiexec should handle it... sorry, I use an outdated version that is in centos repos On Oct 23, 2013 9:01 AM, "Reuti" <reuti(a)staff.uni-marburg.de> wrote: > Am 23.10.2013 um 16:51 schrieb Ricardo Román Brenes: > > > Hi. I'm not sure how it's done on that version of mpich, but you need to > be sure that the mpd daemon is running on both hosts (mpdboot or maybe > mpiexec starts it ). > > No. The `mpdboot` is gone for some time - nowadays Hydra is used (since > MPICH2 v1.3). > > -- Reuti > > > > Does cpi runs in hosts 2 locally? > > > > On Oct 23, 2013 8:47 AM, "Alexandra Betouni" < > alexandra_99(a)windowslive.com> wrote: > > > > > > > > > > Hey there, I am trying to set up a parallel invironment with 14 > machines, running Linux XUbuntu, connected via ethernet. > > They all have same IP's and same hostnames. Well I started installing > mpich-3.0.4 on a single machine, I run the cpi example on localhost by > giving mpiexec -host localhost -n 4 ./examples/cpi and everything worked > fine! > > So I continued changing the hostnames of 2 pc's for a start, and setting > up the ssh in these two, also I installed the mpich-3.0.4 on the other > machine too. > > By giving the ssh <othermachine> date commant , I get the date of the > other host without giving a password, so I think I passed that step too. > > Next step was to check if the mpich-3.0.4 runs parallel, so I created a > machine file (I made a text file giving the hostnames of the two computers > , host1 and host2), and save it in my mpich-3.0.4 build directory. Though > when I am trying to parallel run the cpi code by giving mpiexec -n 4 -f > machinefile ./examples/cpi on my working directory, I get NO errors but > neither parallel job... > > All processes still running on host1 which is my work station. > > What am I doing wrong? > > Thanks > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0