May 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] question on correct use of mpi_isend/mpi_irecv
by Rajeev Thakur 27 May '14

27 May '14

And you are allowed to post a receive of a larger size than the send. So if you know that the send is not going to be larger than 1000 bytes but you don't know the exact size, you can post a receive for 1000 bytes (and allocate that much memory), and whatever was the actual send size (say 800 bytes) will be received. Rajeev On May 27, 2014, at 11:23 AM, Wesley Bland <wbland(a)anl.gov> wrote: > The way you’re doing things is theoretically fine, but unnecessary. In MPI, it’s fine to not know the size of the message before you receive it (as long as you know the type). Instead of sending around “pre-messages” with all of the meta-data, you can use MPI_Probe (http://www.mpich.org/static/docs/latest/www3/MPI_Probe.html) which will give you information about the next message that you’ll receive (there are other variants that you might prefer to use like MPI_Iprobe, MPI_Mprobe, etc.). > > Have a look at the documentation at the link above, but generally, you’ll probe MPI to find out information about the next message such as the source and the size. Then you can create your buffers appropriately and avoid lots of extra buffer creation / destruction calls. > > Thanks, > Wesley > On Tuesday, May 27, 2014 at 11:15 AM, Grindeanu, Iulian R. wrote: > >> Hello, >> We are trying to track down a memory leak, and we are not sure if we are using correctly MPI_ISend and MPI_Irecv >> >> We would like to use non-blocking communication, between pairs of processes. >> >> (it happens during a computation, at every time step; every processor has a list of processors it needs to communicate with; what we do not know, is the size of the messages, in advance) >> >> Let's assume A needs to send to B >> Because we do not know the size in advance, we send first a fixed size message from A to B; this first message has info about how big the total message needs to be; A knows it has to send more. When B receives the first part, it sends an ack (small size, int 4) to A, acknowledging, and also resizing the local buffer, to fill more data; then A sends the rest. >> >> All these sends / receives use Isend and Irecv, and we try to match the messages, using proper tags. We use different tags for fixed size, ack, and rest of the message. >> >> Before sending the code, I would like to know is what we try is feasible and doable? >> >> We notice that the memory use on processor A keeps increasing, when we do this in a loop, on the order of about 1000-2000 bytes per iteration. >> >> I think we are matching correctly isends and ireceives, and that our buffers are not leaking (we may be wrong) >> >> So again, an iteration is like this: >> A sends a fixed size message to B; B sends back an ack, and when A receives it, sends the rest, because B has now the proper buffer to receive. >> >> Next iteration, the size of the message is again unknown, so we do again the dance. >> >> Do you have an example like that we can use? Should we use other types of sends/receives? >> >> Thanks, >> Iulian >> >> Attachments: >> - ATT00001.txt > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question on correct use of mpi_isend/mpi_irecv
by Wesley Bland 27 May '14

27 May '14

The way you’re doing things is theoretically fine, but unnecessary. In MPI, it’s fine to not know the size of the message before you receive it (as long as you know the type). Instead of sending around “pre-messages” with all of the meta-data, you can use MPI_Probe (http://www.mpich.org/static/docs/latest/www3/MPI_Probe.html) which will give you information about the next message that you’ll receive (there are other variants that you might prefer to use like MPI_Iprobe, MPI_Mprobe, etc.). Have a look at the documentation at the link above, but generally, you’ll probe MPI to find out information about the next message such as the source and the size. Then you can create your buffers appropriately and avoid lots of extra buffer creation / destruction calls. Thanks, Wesley On Tuesday, May 27, 2014 at 11:15 AM, Grindeanu, Iulian R. wrote: > Hello, > We are trying to track down a memory leak, and we are not sure if we are using correctly MPI_ISend and MPI_Irecv > > We would like to use non-blocking communication, between pairs of processes. > > (it happens during a computation, at every time step; every processor has a list of processors it needs to communicate with; what we do not know, is the size of the messages, in advance) > > Let's assume A needs to send to B > Because we do not know the size in advance, we send first a fixed size message from A to B; this first message has info about how big the total message needs to be; A knows it has to send more. When B receives the first part, it sends an ack (small size, int 4) to A, acknowledging, and also resizing the local buffer, to fill more data; then A sends the rest. > > All these sends / receives use Isend and Irecv, and we try to match the messages, using proper tags. We use different tags for fixed size, ack, and rest of the message. > > Before sending the code, I would like to know is what we try is feasible and doable? > > We notice that the memory use on processor A keeps increasing, when we do this in a loop, on the order of about 1000-2000 bytes per iteration. > > I think we are matching correctly isends and ireceives, and that our buffers are not leaking (we may be wrong) > > So again, an iteration is like this: > A sends a fixed size message to B; B sends back an ack, and when A receives it, sends the rest, because B has now the proper buffer to receive. > > Next iteration, the size of the message is again unknown, so we do again the dance. > > Do you have an example like that we can use? Should we use other types of sends/receives? > > Thanks, > Iulian > > Attachments: > - ATT00001.txt >

1 0

question on correct use of mpi_isend/mpi_irecv
by Grindeanu, Iulian R. 27 May '14

27 May '14

Hello, We are trying to track down a memory leak, and we are not sure if we are using correctly MPI_ISend and MPI_Irecv We would like to use non-blocking communication, between pairs of processes. (it happens during a computation, at every time step; every processor has a list of processors it needs to communicate with; what we do not know, is the size of the messages, in advance) Let's assume A needs to send to B Because we do not know the size in advance, we send first a fixed size message from A to B; this first message has info about how big the total message needs to be; A knows it has to send more. When B receives the first part, it sends an ack (small size, int 4) to A, acknowledging, and also resizing the local buffer, to fill more data; then A sends the rest. All these sends / receives use Isend and Irecv, and we try to match the messages, using proper tags. We use different tags for fixed size, ack, and rest of the message. Before sending the code, I would like to know is what we try is feasible and doable? We notice that the memory use on processor A keeps increasing, when we do this in a loop, on the order of about 1000-2000 bytes per iteration. I think we are matching correctly isends and ireceives, and that our buffers are not leaking (we may be wrong) So again, an iteration is like this: A sends a fixed size message to B; B sends back an ack, and when A receives it, sends the rest, because B has now the proper buffer to receive. Next iteration, the size of the message is again unknown, so we do again the dance. Do you have an example like that we can use? Should we use other types of sends/receives? Thanks, Iulian

1 0

Re: [mpich-discuss] Error: "MPIR_Get_contextid_sparse_group(1193): Too many communicators (0/2048)"
by Ted Sariyski 27 May '14

27 May '14

Pavan and Rajeev, Thanks for all your kind help. Best regards, --Ted On 05/23/2014 01:29 PM, Rajeev Thakur wrote: > I don't fully understand your use case, but: > > The window memory passed to MPI_Win_create on each process must be a contiguous memory. You can internally manage it in whatever way. > > If you create a datatype on a process, the datatype is local to that process. Other processes can't use that datatype. > > Also the MPI standard says the following about the target datatype passed to put/get/accumulate (MPI-3, pg 420, line 33): > "The target datatype must contain only relative displacements, not absolute addresses." > > Rajeev > > > On May 23, 2014, at 9:44 AM, Ted Sariyski <tsariysk(a)craft-tech.com> wrote: > >> I need a little bit more help. My question relates to the structure of the array (referred below as A), for which a window object will be created. >> >> Let say that particles are of type particle_t: >> >> type particle_t >> integer :: host, id,... >> real,pointer :: data(:)=>null() >> end type particle_t >> >> I built two corresponding mpi types, one with data allocated, mpi_particles_t, and mpi_particles_null_t, when p%data=>null(). >> >> What I want to do is the following: >> ï¿½ All processes allocates an array of type particles_t for all particles, but data is allocated only if a particle is assigned to the domain: >> allocate(A(tot_num_particles)) >> do n=1,tot_num_particles >> if ( particle(n)%host == myid) allocate(particle(n)%data(data_size)) >> enddo >> >> 2. All processes create their mpi data type mpi_A_t: >> >> do n=1,tot_num_particles >> >> if ( particle(n)%host == myid) then >> blockcounts(n) = 1 ; oldtypes(n) = mpi_particles_t >> else >> blockcounts(n) = 1 ; oldtypes(n) = mpi_particles_null_t >> endif >> call MPI_GET_ADDRESS(A(n), offsets(n), ierr) >> enddo >> >> call mpi_type_create_struct(tot_num_particles,blockcounts,offsets,oldtypes,mpi_A_myid_t,ierr) >> call mpi_type_commit(mpi_A_myid_t,ierr) >> >> I gave a try but it fails. Before I proceed I want to make sure that there is nothing fundamentally wrong with this approach. Besides, even if it is correct, I am not sure that this is the 'best' solution. I will highly appreciate your comments. >> >> Thanks in advance, >> --Ted >> >> >> >> >> >> On 05/22/2014 03:31 PM, Ted Sariyski wrote: >>> I see. Thanks a lot. >>> --Ted >>> >>> >>> On 05/22/2014 03:15 PM, Rajeev Thakur wrote: >>>>> What do you mean with: "Why canï¿½t all processes open one large window?" I guess I miss something. >>>> If each process has an array of objects that belong to it (called array A say) , then with a single call to MPI_Win_create you can create a window object that has everyone's A arrays in it. >>>> >>>> Rajeev >>>> >>>> >>>> >>>> On May 22, 2014, at 9:46 AM, Ted Sariyski <tsariysk(a)craft-tech.com> wrote: >>>> >>>>> It is about MPI_win. Here is the problem as it relates to MPI (it is a Boltzmann type equation). >>>>> >>>>> There are N particles interacting with each other. Interaction is directional, so that a particle interacts only with those particles, which are within a narrow cone. The first step is to segregate the initial set of N particles into subsets of particles (I called it 'objects'), which interact with each other. Here is what I do: >>>>> ï¿½ Assign each object to a process. >>>>> ï¿½ The process which owns a object: >>>>> ï¿½ Makes a guess for the maximum number of particles expected in this object. >>>>> ï¿½ Allocates memory for it. >>>>> ï¿½ Opens a shared window. >>>>> ï¿½ All processes >>>>> ï¿½ Each particle identify which object it belongs to, and PUTs its data there. >>>>> ï¿½ After assembly is done, objects are passed to a solver. >>>>> ï¿½ Repeat >>>>> What do you mean with: "Why canï¿½t all processes open one large window?" I guess I miss something. >>>>> Thanks, >>>>> --Ted >>>>> >>>>> >>>>> On 05/21/2014 11:03 PM, Balaji, Pavan wrote: >>>>>> On May 21, 2014, at 6:02 PM, Ted Sariyski <tsariysk(a)craft-tech.com> >>>>>> wrote: >>>>>> >>>>>>> Memory limitations. With one large window all processes have to allocate memory for the objects they own as well as for objects assigned to other process. >>>>>>> >>>>>> Are we talking about the same thing here? Iï¿½m referring to an MPI_Win. What objects do processes need to keep track of? >>>>>> >>>>>> ï¿½ Pavan >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list >>>>>> discuss(a)mpich.org >>>>>> >>>>>> To manage subscription options or unsubscribe: >>>>>> >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] communication device selection
by Balaji, Pavan 26 May '14

26 May '14

On May 26, 2014, at 4:28 PM, Yida Wang <yidawa(a)gmail.com> wrote: > Never mind. It looks like that I can pick communication methods by setting MPIR_CVAR_NEMESIS_NETMOD. Yup, that’s it.

1 0

Re: [mpich-discuss] communication device selection
by Yida Wang 26 May '14

26 May '14

Never mind. It looks like that I can pick communication methods by setting MPIR_CVAR_NEMESIS_NETMOD. Please let me know if I miss anything. Thanks! On Mon, May 26, 2014 at 4:43 PM, Yida Wang <yidawa(a)gmail.com> wrote: > Hi Pavan, > > Thanks for letting me know. One follow-up (maybe silly) question, how can > I pick tcp at runtime instead of scif? > > Yida > > > On Mon, May 26, 2014 at 3:06 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> Hi Yida, >> >> Using SCIF + TCP (+ shared memory, which is implicit) together within the >> same job is not supported right now. When you configure with scif,tcp, you >> can pick either one at runtime, but not both. >> >> We are working on some improvements to this for the next major release >> (3.2). Please stay tuned. >> >> Regards, >> >> — Pavan >> >> On May 26, 2014, at 10:50 AM, Yida Wang <yidawa(a)gmail.com> wrote: >> >> > Hi Guys, >> > >> > During configuration, I put in an optional argument as >> "--with-device=ch3:nemesis:scif,tcp" to hopefully enable scif communication >> through PCIe, and use tcp for remote communication. >> > >> > However, it looks like that tcp channel is disabled by this setting. >> That is, when I wanted to communicate between remote Xeon Phis, it returned >> error saying that "channel initialization failed" and "scif_connect >> failed", where I assume that tcp channel should be used instead of scif. >> > >> > Moreover, when communicating between two remote hosts, it would get >> stuck probably because it still tried to use scif channel. >> > >> > Did I configure the communication device incorrectly? >> > >> > Thanks, >> > YW >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > >

1 0

Re: [mpich-discuss] communication device selection
by Yida Wang 26 May '14

26 May '14

Hi Pavan, Thanks for letting me know. One follow-up (maybe silly) question, how can I pick tcp at runtime instead of scif? Yida On Mon, May 26, 2014 at 3:06 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Hi Yida, > > Using SCIF + TCP (+ shared memory, which is implicit) together within the > same job is not supported right now. When you configure with scif,tcp, you > can pick either one at runtime, but not both. > > We are working on some improvements to this for the next major release > (3.2). Please stay tuned. > > Regards, > > — Pavan > > On May 26, 2014, at 10:50 AM, Yida Wang <yidawa(a)gmail.com> wrote: > > > Hi Guys, > > > > During configuration, I put in an optional argument as > "--with-device=ch3:nemesis:scif,tcp" to hopefully enable scif communication > through PCIe, and use tcp for remote communication. > > > > However, it looks like that tcp channel is disabled by this setting. > That is, when I wanted to communicate between remote Xeon Phis, it returned > error saying that "channel initialization failed" and "scif_connect > failed", where I assume that tcp channel should be used instead of scif. > > > > Moreover, when communicating between two remote hosts, it would get > stuck probably because it still tried to use scif channel. > > > > Did I configure the communication device incorrectly? > > > > Thanks, > > YW > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] communication device selection
by Balaji, Pavan 26 May '14

26 May '14

Hi Yida, Using SCIF + TCP (+ shared memory, which is implicit) together within the same job is not supported right now. When you configure with scif,tcp, you can pick either one at runtime, but not both. We are working on some improvements to this for the next major release (3.2). Please stay tuned. Regards, — Pavan On May 26, 2014, at 10:50 AM, Yida Wang <yidawa(a)gmail.com> wrote: > Hi Guys, > > During configuration, I put in an optional argument as "--with-device=ch3:nemesis:scif,tcp" to hopefully enable scif communication through PCIe, and use tcp for remote communication. > > However, it looks like that tcp channel is disabled by this setting. That is, when I wanted to communicate between remote Xeon Phis, it returned error saying that "channel initialization failed" and "scif_connect failed", where I assume that tcp channel should be used instead of scif. > > Moreover, when communicating between two remote hosts, it would get stuck probably because it still tried to use scif channel. > > Did I configure the communication device incorrectly? > > Thanks, > YW > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

communication device selection
by Yida Wang 26 May '14

26 May '14

Hi Guys, During configuration, I put in an optional argument as "--with-device=ch3:nemesis:scif,tcp" to hopefully enable scif communication through PCIe, and use tcp for remote communication. However, it looks like that tcp channel is disabled by this setting. That is, when I wanted to communicate between remote Xeon Phis, it returned error saying that "channel initialization failed" and "scif_connect failed", where I assume that tcp channel should be used instead of scif. Moreover, when communicating between two remote hosts, it would get stuck probably because it still tried to use scif channel. Did I configure the communication device incorrectly? Thanks, YW

1 0

Re: [mpich-discuss] process failing...
by Ron Palmer 25 May '14

25 May '14

Antonio and Pavan, thanks for your comments and suggestions, I have passed them on to the group that writes this program. I believe he had assumed that all processes execute at the same speed. I will write a comment once this is under control. Thanks, Ron On 25/05/2014 10:50, Balaji, Pavan wrote: > Actually, the below example will not cause any unexpected messages since you are doing an ANY_SOURCE receive. To get unexpected messages you need to be making progress on the receive side, but the incoming messages should be messages you donï¿½t have a receive buffer for. Hereï¿½s a modified example: > > Host A: > for(int i=0; i<n; i++) { > MPI_Isend(..to C..); > MPI_Send(..to B..); > } > > Host B: > for(int i=0; i<n; i++) { > MPI_Recv(..from A..); > MPI_Isend(..to C..); > } > > Host C: > for(int i=0; i<n; i++) { > MPI_Recv(..from B..); > } > for(int i=0; i<n; i++) { > MPI_Recv(..from A..); > } > > ï¿½ Pavan > > On May 24, 2014, at 7:19 PM, Antonio J. Peï¿½a <apenya(a)mcs.anl.gov> wrote: > >> Hi Ron, >> >> Depending on how the algorithm is structured, I may well be that the faster computers are generating messages faster than the slow computer is able to process them. As the problem size increases, the amount of data in unprocessed messages may get too high for that computer. In a very simple example (but note that there are many other possible situations): >> >> Hosts A and B: >> for(int i=0; i<n; i++) { >> MPI_Send(buf, count, MPI_INT, host_c, 0, MPI_COMM_WORLD); >> buf += count; >> } >> >> Host C: >> for(int i=0; i<2*n; i++) { >> MPI_Recv(buf, count, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status); >> process_data(buf); >> } >> >> In that case the senders keep sending data to host_c faster than it's calling MPI_Recv, and get received by MPICH as unexpected messages (MPICH doesn't know where to place that data, and uses temporary internal buffers). Note that this situation is not necessarily caused by different processing speeds, since the culprit may well be the used algorithm itself. A very simple solution to avoid the prior potential problem could be: >> >> Host A and B: >> MPI_Barrier(MPI_COMM_WORLD); >> for(int i=0; i<n; i++) { >> MPI_Send(buf, count, MPI_INT, host_c, 0, MPI_COMM_WORLD); >> buf += n; >> } >> >> Host C: >> for(int i=0; i<2*n; i++) { >> MPI_Irecv(buf, count, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &requests[i]); >> buf += count; >> } >> MPI_Barrier(MPI_COMM_WORLD); >> for(int i=0; i<2*n; i++) { >> MPI_Waitany(2*n, requests, &idx, &status); >> process_data(buf+idx*count); >> } >> >> Now the data is placed on its final destination, since it's guaranteed that the MPI implementation knows its destination before receiving it. That solution would assume that there is enough memory for preallocating all receiving buffers, which may not be true and may require, for example, implementing a high-level protocol (such as based on credits) to synchronize the sends and receives to make sure that there is room on the receiver for them. >> >> I hope this helps. >> >> Best, >> Antonio >> >> >> On 05/24/2014 02:00 AM, Ron Palmer wrote: >>> Antonio, Rajeev and others, >>> thanks for your replies and comments on possible causes for the error messages and failure, I have passed them on to the programmers of the underlying application. I must admit I do not understand what unexpected messages are (I am but a mere user), could you perhaps give examples of typical causes of them? Eg, the cluster it runs on consists of 3 dual xeon computers with varying cpu clock rating - could these error messages be due to getting out of synch, expecting results but not getting them from the slower computer? I have re-started the process but excluded the slowest computer (2.27GHz, the other two are running at 2.87 and 3.2) as I was running out of ideas. >>> >>> For your information, this runs well on smaller problems (few computations). >>> >>> Thanks, >>> Ron >>> >>> On 24/05/2014 3:10 AM, Rajeev Thakur wrote: >>>> Yes. The message below says some process has received 261,895 messages for which no matching receives have been posted yet. >>>> >>>> <Mail Attachment> >>>> >>>> Rajeev >>>> >>>> >>>>> It looks like at least one of your processes is receiving too many unexpected messages, leading to get out of memory. Unexpected messages are those not matching a posted receive on the receiver side. You may check with the application developers to make them review the algorithm or look for any possible bug. >>>>> >>>>> Antonio >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list >>>> discuss(a)mpich.org >>>> >>>> To manage subscription options or unsubscribe: >>>> >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> _______________________________________________ >>> discuss mailing list >>> discuss(a)mpich.org >>> >>> To manage subscription options or unsubscribe: >>> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Antonio J. Peï¿½a >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- *Ron Palmer*MSc MBA. Principal Geophysicist ron.palmer(a)pgcgroup.com.au <mailto:[email protected]> 0413 579 099 07 3103 4963

1 0