December 2007 - Mpich2-core - mailman3.cels.anl.gov

Re: [MPICH] MPI_Reduce_scatter
by Dave Goodell 23 Dec '07

23 Dec '07

[moving to mpich2-core, adding David Gingold, dropping Calin] On Dec 21, 2007, at 9:31 AM, Calin Iaru wrote: > I am using PALLAS to stress MPI_Reduce_scatter. The error reported > after millions of inner loops is: > > 3: MPI error 875666319 occurred > 3: Other MPI error, error stack: > 3: MPI_Reduce_scatter(1201): MPI_Reduce_scatter(sbuf=0x2aaaabdfb010, > rbuf=0x2aaaac1fc010, rcnts=0x176e1850, MPI_INT, MPI_SUM, > comm=0x84000000) > failed > 3: MPIR_Reduce_scatter(372): > 3: MPIC_Send(48): > 3: MPIC_Wait(321): > 3: MPIDI_CH3_Progress(115): Unable to make message passing progress > 3: handle_read(280): > 3: MPIDI_CH3U_Handle_recv_pkt(250): failure occurred while > allocating memory > for a request object > 3: aborting job: > 3: application called MPI_Abort(MPI_COMM_WORLD, 875666319) - process 3 > > The library is 1.0.2p1 and I would like to know if there are some > changes that would fix this issue. David from SiCortex described a case like this to me when he was here for the workshop. He said that he had a fairly simple micro- benchmark that called MPI_Reduce_scatter in a tight loop among 4 processes. He witnessed memory on two of the processes growing rapidly and seemingly without bound. He conjectured that two of the processes (one being the root of the reduce) had greater work to do than the other two processes and that the overloaded processes would fall behind. So receive queues would build in these overworked processes, which would increase the memory usage and may further slow the receive matching, which in turn overloads the process further. (David: hopefully I characterized your example & hypothesis correctly) I created an example program that illustrates this behavior (see the bottom of this email). I ran it with 4 processes on a 4 core linux system (one of the breadboard nodes) and saw two of the processes progressively eat more and more memory. It got up to 115MB RSS each before I killed it. If you un-comment the MPI_Barrier at the end of the while loop then the memory problem goes away. What advice do we give to users in this case? I believe that the program below is valid (although admittedly pathological), yet the memory usage appears to grow without bound. Is this a failing in the MPI spec (that it doesn't specify flow control for collectives) or a problem with the MPICH2 implementation? Looks like someone raised the collective flow-control issue in the forum back in 1995 and it doesn't appear that he got an answer: http://www.mpi-forum.org/archives/mail/mpi-coll/0025.html -Dave ================= t.c ========================= #include "mpi.h" #include <stdio.h> #include <stdlib.h> int main( int argc, char **argv ) { int err = 0; int *sendbuf, *recvbuf, *recvcounts; int size, rank, i, sumval; MPI_Comm comm; MPI_Init( &argc, &argv ); comm = MPI_COMM_WORLD; MPI_Comm_size( comm, &size ); MPI_Comm_rank( comm, &rank ); recvbuf = (int *) malloc( (size * sizeof(int)) / 2 ); sendbuf = (int *) malloc( size * sizeof(int) ); for (i=0; i<size; i++) sendbuf[i] = rank + i; recvcounts = (int *)calloc( size, sizeof(int) ); for (i=0; i<size/2; i++) recvcounts[i] = size/2; while (1) { MPI_Reduce_scatter( sendbuf, recvbuf, recvcounts, MPI_INT, MPI_SUM, comm ); /* MPI_Barrier(comm); */ } /* never reached */ MPI_Finalize( ); return err; } ================= t.c =========================

4 4

MapReduce/Hadoop stuff
by Rajeev Thakur 20 Dec '07

20 Dec '07

We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6. Rajeev http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/

5 5

Re: [MPD] Question on the scalability of mpd
by Rusty Lusk 18 Dec '07

18 Dec '07

Hi Yann, I believe the largest system we ran the mpd ring on was around 4800 hosts at NCSA. We still use it and it is still a ring, although we are in the process of designing another process manager, and a new version of PMI that will work better with existing process managers like SLURM, SGE, etc. We remain interested in this area of MPI implementation. I look forward to meeting you in January. Regards, Rusty Lusk On Dec 18, 2007, at 10:01 AM, Yann K. wrote: > Mister Lusk, > > I present myself briefly. I'm working for Bull, a hardware and > software > business company, which deals HPC machines (www.bull.com/hpc) I'm a > technical responsible of the parallel computing departement. We do > a mpi > library, based on MPICH2, and a lot around mpi as well. I've seen > you in > Europvm this year and had not the issues I have today with some of the > software you might be dealing with, especially the mpd launching > subsystem. I have currently two questions for you if you have time to > answer: > > + how scalable is the mpd subsystem (>=1.90) on a huge machine ? > What's > the biggest configuration you've seen working with it ? I have > doubt on > its ring scalability (perhaps it changed its algorithm since I last > checked mpd). > > + PMI is one of the greatest thing invented after MPI in the domain > and > unfortunately not a lot of people realize how it could help them get > more flexibility in their system if all resource manager where > speaking > the PMI protocol. This one got out of the SciDac SSS work if I > remember. > When I talk to Al Geist, and asked about how those SSS interfaces did > not get more attention, he was not able to answer. I have proposed to > review chapter 5 as well for MPI, and especially the 5.2, where a > lot of > fog remains. I saw you were reviewing this as well. I've been > promoting > PMI for years *2 now :) even to OpenMPI guys who didn't know about > it)*. > I like to think there is place for more descriptions of the MPI > external > interactions, especially with upper layers systems. > > > I'll be in Chicago at the MPI2.2/3.0 meetings. > > Thanks for your time > > Yann K. >

1 0

mpich2-1.0.6p2
by Pavan Balaji 17 Dec '07

17 Dec '07

Hi all, Based on some recent bug fixes in the MPICH2 code, we are planning to release a second patch version for 1.0.6. So far, the plan is to include: 1. The fix to the gather problem that was recently reported. I'm still waiting for the user to confirm that the patch works on this. 2. The connection problem with sock on large number of processes, which Dave is working on (req. 3669). 3. The mpich2version problem that Anthony reported. Anthony, did you already fix this? 4. The datatype corruption bug in req. 3769 that Darius is fixing. Darius, any read on this problem? I just took a second look at the bug request, and I have a strange feeling that it might have something to do with the manual loop unrolling code that I added in for 1.0.6. If you like, you can just assign it to me. Any other bugs that have been fixed that should go into 1.0.6p2? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

3 8

http://www.mcs.anl.gov/mpi/www/
by Anthony Chan 17 Dec '07

17 Dec '07

Is there anyway to update the above URL with info from MPICH2 and move the above URL to http://www.mcs.anl.gov/mpi/mpich1/www ? A.Chan

4 3

sicortex mpi guy
by robl＠mcs.anl.gov 13 Dec '07

13 Dec '07

I'm sure most of you know already but David Gingold, the "MPI guy" for Sicortex, will be here today for the applications workshop. I'm going to be bugging him about their MPI-IO status. ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B

1 0

ext32
by Darius Buntinas 10 Dec '07

10 Dec '07

On the Intel concall this morning, Alexander asked about whether mpich2 has ext32 support. Does anyone know what the status of this is? Thanks, -d

4 4

Euro PVM/MPI 08
by Rajeev Thakur 07 Dec '07

07 Dec '07

The web site for Euro PVM/MPI 08 is up: http://pvmmpi08.ucd.ie/ . The deadline is a month earlier than usual: April 6th. We should start planning some papers for this. Rajeev

1 0

mpich2 website
by Dave Goodell 05 Dec '07

05 Dec '07

Where are we wrt finishing the mpich2 website? Is anybody currently working on it, or scheduled to work on it in the near future? Same question goes for getting Trac setup... While in some respects the new website is more visually attractive, it has a lot of problems that make me believe it is actually a step backwards from the old website. Many links don't work, "coming soon", "still in development", and "under construction" litter the site, and the home page scores very poorly wrt accessibility and search engine optimization. Lots of useful information that was on the old site is absent from the new site. If the goal is to present mpich2 as a modern, high-quality piece of software, then I think we have failed. Because the site is in this half-baked state, I believe that our users and potential users will believe that our software is also half-baked. I'm willing to work on it, but I don't want to duplicate effort if someone is in the middle of fixing it up. -Dave

3 3

Re: [MPICH2 Req #3592] Re: aborttest
by Dave Goodell 05 Dec '07

05 Dec '07

[moving to mpich2-core@ for discussion] OK, Pavan showed me where this test lives (the mpich1 test suite). The current test harness does the following (in very rough pseudo- shell): # test for which ps args to use below as $some_args before=`ps $some_args | grep $aborttest_instances_pattern` mpiexec -n 2 aborttest sleep 5 # give prog time to die after=`ps $some_args | grep $aborttest_instances_pattern` if [ after_lines - before_lines == 0 ] ; then # success else # fail fi I believe that the reason for this crude grep-and-diff operation is due to the lack of a standard format for 'ps' output. Given that this test harness already assumes some form of unix (bourne shell and 'ps' are used), what about a different approach? If we assume that getpid() is portable, then I could rewrite the test to output all the pids to a file and read them in afterwards. This allows multiple tests to be run concurrently and we should always be able to tell if the processes are alive or not (send signal 0 to each pid). My question to you folks is can we rely on getpid()? It's specified in a number of standards (POSIX.1-2001, 4.3BSD, SVr4, possibly others) but as we all know that doesn't guarantee anything for any given platform. Obviously I can leave the current method and use it if a getpid() capability test fails, but I'd rather just gut it to reduce the complexity if I can. -Dave On Nov 29, 2007, at 11:06 AM, Dave Goodell wrote: > Pavan, > > I can't seem to find any test named "aborttest" or anything > similarly named. Can you point me towards it so I can take a shot > at fixing it? > > Thanks, > -Dave > > On Aug 6, 2007, at 3:52 PM, Pavan Balaji wrote: > >> >> Sending this to mpich2-maint as a placeholder. >> >> -- Pavan >> >> On 07/26/2007 12:40 PM, William Gropp wrote: >>> Yes, unfortunately this is a feature caused by the lack of a >>> reliable, standard form of ps that works on all varieties of >>> Unix :( . Please just ignore these failures. >>> Bill >>> On Jul 26, 2007, at 12:29 PM, Pavan Balaji wrote: >>>> Bill, >>>> >>>> "aborttest" in the mpich test suite is failing for ch3:sock/ >>>> shared on crunch. But it looks like the failure detection is not >>>> correct, i.e., it finds some other MPI processes on crunch and >>>> thinks that the test didn't finish correctly. >>>> >>>> -- Pavan >

2 3