December 2013 - discuss - mailman3.cels.anl.gov

I have a question about using MPICH2 1.4.1p1 for Windows
by Ярослав Иванов 05 Dec '13

05 Dec '13

Hello! My name is Jaroslaw, I am student of CSD (Tomsk State University). I am using MPICH2 1.4.1p1 for Windows. It is for my exam work. I have trouble, but I don't know, how to question about my problem on http://lists.mpich.org I read: *Using discuss*To post a message to all the list members, send email to discuss(a)mpich.org. You can subscribe to the list, or change your existing subscription, in the sections below. So, do I just send you my question and it display in http://lists.mpich.org ? Or no? So, message about problem: "Hello! I have a question about using MPICH2 1.4.1p1 for Windows. http://www.mpich.org/static/tarballs/1.4.1p1/mpich2-1.4.1p1-win-ia32.msi I recompilated examples (Visual Studio 2010) about calculating pi. So. If I exec program on pure C - it works fine on network of 2 machines. But, If I exec program on C++ - it don't work on 2 machines (but work on 1 machine). When I try to exec C++ version on 2 machines, console return me next issue: http://pastebin.com/JimcBW2N Thank you for your help and I sorry, if I badly use English - it is no my native language."

1 0

Re: [mpich-discuss] MPI memory allocation.
by Anatoly G 05 Dec '13

05 Dec '13

I think I need MPICH2, but not MPICH. Does MPICH 3.0.4 includes all features of MPICH2? Anatoly. On Thu, Dec 5, 2013 at 12:00 PM, Reuti <reuti(a)staff.uni-marburg.de> wrote: > Hi, > > Am 05.12.2013 um 09:58 schrieb Anatoly G: > > > I"m using MPICH2 1.5. > > My system contains master and 16 slaves. > > System uses number of communicators. > > The single communicator used for below scenario: > > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits > using MPI_Wait. > > Master starts with MPI_Irecv to each slave > > Then in endless loop: > > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. > > > > Another communicator used for broadcast communication (commands between > master + slaves), > > but it's not used in parallel with previous communicator, > > only before or after data transfer. > > > > The system executed on two computers linked by 1Gbit/s Ethernet. > > Master executed on first computer, all slaves on other one. > > Network traffic is ~800Mbit/s. > > > > After 1-2 minutes of execution, master process starts to increase it's > memory allocation and network traffic becomes low. > > This memory allocation & network traffic slow down continues until fail > of MPI, > > without core file save. > > This might depend on the ulimit: core file size > > > > My program doesn't allocate memory. Can you please explain this > behaviour. > > How can I cause MPI to stop sending slaves if Master can't serve such > traffic, instead of memory allocation and fail? > > Can you please try with the latest MPICH 3.0.4 to check whether it behaves > in the same way. > > -- Reuti > > > > Thank you, > > Anatoly. > > > > P.S. > > On my stand alone test, I simulate similar behaviour, but with single > thread on each process (master & hosts). > > When I start stand alone test, master stops slaves until it completes > accumulated data processing and MPI doesn't increase memory allocation. > > When Master is free slaves continue to send data. > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPI memory allocation.
by Reuti 05 Dec '13

05 Dec '13

Hi, Am 05.12.2013 um 09:58 schrieb Anatoly G: > I"m using MPICH2 1.5. > My system contains master and 16 slaves. > System uses number of communicators. > The single communicator used for below scenario: > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits using MPI_Wait. > Master starts with MPI_Irecv to each slave > Then in endless loop: > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. > > Another communicator used for broadcast communication (commands between master + slaves), > but it's not used in parallel with previous communicator, > only before or after data transfer. > > The system executed on two computers linked by 1Gbit/s Ethernet. > Master executed on first computer, all slaves on other one. > Network traffic is ~800Mbit/s. > > After 1-2 minutes of execution, master process starts to increase it's memory allocation and network traffic becomes low. > This memory allocation & network traffic slow down continues until fail of MPI, > without core file save. This might depend on the ulimit: core file size > My program doesn't allocate memory. Can you please explain this behaviour. > How can I cause MPI to stop sending slaves if Master can't serve such traffic, instead of memory allocation and fail? Can you please try with the latest MPICH 3.0.4 to check whether it behaves in the same way. -- Reuti > Thank you, > Anatoly. > > P.S. > On my stand alone test, I simulate similar behaviour, but with single thread on each process (master & hosts). > When I start stand alone test, master stops slaves until it completes accumulated data processing and MPI doesn't increase memory allocation. > When Master is free slaves continue to send data. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI memory allocation.
by Anatoly G 05 Dec '13

05 Dec '13

Hello. I"m using MPICH2 1.5. My system contains master and 16 slaves. System uses number of communicators. The single communicator used for below scenario: Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits using MPI_Wait. Master starts with MPI_Irecv to each slave Then in endless loop: MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. Another communicator used for broadcast communication (commands between master + slaves), but it's not used in parallel with previous communicator, only before or after data transfer. The system executed on two computers linked by 1Gbit/s Ethernet. Master executed on first computer, all slaves on other one. Network traffic is ~800Mbit/s. After 1-2 minutes of execution, master process starts to increase it's memory allocation and network traffic becomes low. This memory allocation & network traffic slow down continues until fail of MPI, without core file save. My program doesn't allocate memory. Can you please explain this behaviour. How can I cause MPI to stop sending slaves if Master can't serve such traffic, instead of memory allocation and fail? Thank you, Anatoly. P.S. On my stand alone test, I simulate similar behaviour, but with single thread on each process (master & hosts). When I start stand alone test, master stops slaves until it completes accumulated data processing and MPI doesn't increase memory allocation. When Master is free slaves continue to send data.

1 0

Re: [mpich-discuss] mpifort wrapper compiler?
by Pavan Balaji 04 Dec '13

04 Dec '13

Hi Jeff, On Dec 4, 2013, at 6:02 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Per our prior discussions, do have an estimated timeline for when MPICH will provide an "mpifort" wrapper compiler? Hmm. Sorry, I totally dropped the ball on that. I guess it’s simple enough that we could do it for the 3.1 release, but I’m not sure. Are you guys in the process of making that the preferred compiler wrapper for Open MPI? — Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

mpifort wrapper compiler?
by Jeff Squyres (jsquyres) 04 Dec '13

04 Dec '13

Pavan -- Per our prior discussions, do have an estimated timeline for when MPICH will provide an "mpifort" wrapper compiler? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] reading from a read only directory
by Geoffrey Irving 03 Dec '13

03 Dec '13

On Tue, Dec 3, 2013 at 2:10 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On Tue, Dec 03, 2013 at 12:58:53PM -0800, Geoffrey Irving wrote: >> > Mea culpa. MPI_File_read_ordered is implemented in "stupid mode" >> > right now, and no one ever complained -- we don't pay much attention >> > to shared file pointers. Tell me more about how you are relying on >> > ordered mode, please. >> >> I'm dividing a large file into nearly equal size chunks and slurping >> the entire file into RAM. Then I do a bunch of rearrangement in >> memory. I could just as easily use MPI_File_read_at_all, but had >> assumed that ordered was better because my known intervals happen to >> be ordered and contiguous. Should I be using MPI_File_read_at_all >> instead, and treating the ordered routine as vestigial? > > Yeah, the ordered mode routines, to borrow the "self-consistent MPI" > model, are more strict than MPI_File_read_at_all -- not only are all > processes participating, but they are participating in I/O that must > end up in the file in rank-order. Thanks, I'll switch to ..._read_at_all. > This is probably not a significant portion of your runtime, but have > you considered combining mpi memory datatypes with > mpi_file_read_at_all to get the read and the memory rearrangement in > one go? The rearrangement step is 2 orders of magnitude faster than the I/O, so it doesn't matter in this case, but datatypes do seem cleaner for future versions of similar code. >> Okay, I was basically assuming that if I used a collective routine, it >> would be smart about accessing the file system as few times as >> possible. If this isn't true I should unqueue my job now and switch >> to MPI_File_read_at_all. Is this right? Should I also use >> MPI_File_write_at_all instead of the shared fp collective write >> routine? > > Yeah, MPI_File_write_at_all and MPI_File_read_at_all will be the smart > clever routines you are looking for -- though on the Cray they may > need a bit of tuning. I *think* "good mode" is now the default, but I > don't have a lot of experience with Cray's modifications to ROMIO. Good to know. Thanks! Geoffrey

1 0

Re: [mpich-discuss] reading from a read only directory
by Rob Latham 03 Dec '13

03 Dec '13

On Tue, Dec 03, 2013 at 12:58:53PM -0800, Geoffrey Irving wrote: > > Mea culpa. MPI_File_read_ordered is implemented in "stupid mode" > > right now, and no one ever complained -- we don't pay much attention > > to shared file pointers. Tell me more about how you are relying on > > ordered mode, please. > > I'm dividing a large file into nearly equal size chunks and slurping > the entire file into RAM. Then I do a bunch of rearrangement in > memory. I could just as easily use MPI_File_read_at_all, but had > assumed that ordered was better because my known intervals happen to > be ordered and contiguous. Should I be using MPI_File_read_at_all > instead, and treating the ordered routine as vestigial? Yeah, the ordered mode routines, to borrow the "self-consistent MPI" model, are more strict than MPI_File_read_at_all -- not only are all processes participating, but they are participating in I/O that must end up in the file in rank-order. This is probably not a significant portion of your runtime, but have you considered combining mpi memory datatypes with mpi_file_read_at_all to get the read and the memory rearrangement in one go? > Okay, I was basically assuming that if I used a collective routine, it > would be smart about accessing the file system as few times as > possible. If this isn't true I should unqueue my job now and switch > to MPI_File_read_at_all. Is this right? Should I also use > MPI_File_write_at_all instead of the shared fp collective write > routine? Yeah, MPI_File_write_at_all and MPI_File_read_at_all will be the smart clever routines you are looking for -- though on the Cray they may need a bit of tuning. I *think* "good mode" is now the default, but I don't have a lot of experience with Cray's modifications to ROMIO. ==rob > > I wrote a little ordered mode shim you can link into your program. > > Looks like I last seriously did anything with it back in 2008 ? I > > cleaned up some of the warnings and made it use MPI_Count (remove > > HAVE_MPI_TYPE_SIZE_X from the makefile if you're on an old system). > > Caveat: I think the last time I ran this code was on Argonne's Blue > > Gene /L. Bear in mind the adage about tested code and broken code. > > Thanks, though in my case switching away from the shared file pointer > routine is easy, so I don't need the shim. > > Geoffrey -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] reading from a read only directory
by Geoffrey Irving 03 Dec '13

03 Dec '13

On Tue, Dec 3, 2013 at 8:20 AM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On Sat, Nov 30, 2013 at 08:35:39AM -0600, Jed Brown wrote: >> Geoffrey Irving <irving(a)naml.us> writes: >> > It's not an issue for me personally anymore: chmod +w . is fine, and >> > symlinks would also solve the problem. The question was more about >> > the motivation for the code and whether it's an issue anything thinks >> > is worth fixing (or fixable at all). That and Jed asked me several >> > times to start the thread. :) >> >> It's not intuitive and does not appear to be documented. Reading from a >> read-only directory is a reasonable thing to try and people should not >> have to wait through a queue to find out that it doesn't work. > > Mea culpa. MPI_File_read_ordered is implemented in "stupid mode" > right now, and no one ever complained -- we don't pay much attention > to shared file pointers. Tell me more about how you are relying on > ordered mode, please. I'm dividing a large file into nearly equal size chunks and slurping the entire file into RAM. Then I do a bunch of rearrangement in memory. I could just as easily use MPI_File_read_at_all, but had assumed that ordered was better because my known intervals happen to be ordered and contiguous. Should I be using MPI_File_read_at_all instead, and treating the ordered routine as vestigial? > The technical issues here: the shared file pointer and ordered mode > collectives keep as global state the location of the shared file > pointer. Where does it keep this state? Why, in the file system of > course. Because using a file system as a message-passing medium is > exactly what file systems are good at.... uh, well, maybe not, but > it's the most portable option. > > You can also do this with RMA windows (Rob Ross and I published some > papers on this topic back in 05 and 06). Has a few drawbacks but it's > got to be better than the file system. > > Even without the RMA windows, we can still do a better job for ordered > mode with MPI_SCAN -- it's just we need some globally-available place > to get the old shared file pointer value and stash the new one. > > In user-space, you'll outperform MPI-IO implementations if you manage > the offset yourself. Okay, I was basically assuming that if I used a collective routine, it would be smart about accessing the file system as few times as possible. If this isn't true I should unqueue my job now and switch to MPI_File_read_at_all. Is this right? Should I also use MPI_File_write_at_all instead of the shared fp collective write routine? > I wrote a little ordered mode shim you can link into your program. > Looks like I last seriously did anything with it back in 2008 ? I > cleaned up some of the warnings and made it use MPI_Count (remove > HAVE_MPI_TYPE_SIZE_X from the makefile if you're on an old system). > Caveat: I think the last time I ran this code was on Argonne's Blue > Gene /L. Bear in mind the adage about tested code and broken code. Thanks, though in my case switching away from the shared file pointer routine is easy, so I don't need the shim. Geoffrey

1 0

Re: [mpich-discuss] MPI_init very slow with more than 3 nodes
by Pavan Balaji 03 Dec '13

03 Dec '13

One thing you could try is to install mpich on a non-NFS directory such as /tmp and run a program that is also not on NFS to see if the problem persists. You can build mpich on an NFS directory with --prefix=/tmp/mpich and then do “make install” on each machine. Similarly for the “application”, you can run “/bin/hostname” or something similar. — Pavan On Dec 3, 2013, at 12:06 PM, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: > > Yes, it may be a good idea to look into that. Hopefully somebody in this list with NFS knowledge will give you a hint. > > Antonio > > > On 12/03/2013 09:48 AM, Bixente BODO GOMEZ wrote: >> I send a extract of tcpdump. It seems that the problem is in the file attribues o locks of nfs. I changed fstab on clients to add these options: >> >> nfsvers=3,rw,bg,noac,rsize=8192,wsize=8192 >> >> but nothing has improved. >> >> "Antonio J. Peña" <apenya(a)mcs.anl.gov> escribió: >> >>> I don't think this is helping. Maybe tcpdump / wireshark captures could be helpful. >>> >>> Antonio >>> >>> >>> On 12/02/2013 11:44 AM, Bixente BODO GOMEZ wrote: >>>> The attachment... >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> -- >>> Antonio J. Peña >>> Postdoctoral Appointee >>> Mathematics and Computer Science Division >>> Argonne National Laboratory >>> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >>> Argonne, IL 60439-4847 >>> (+1) 630-252-7928 >>> apenya(a)mcs.anl.gov >>> www.mcs.anl.gov/~apenya >> >> >> >> >> _______________________________________________ >> discuss mailing list >> discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0