Yes, it is not clear how this stuff is useful for anything other than task parallel and embarassingly parallel. Yet there is a growing group of people who think it is the next best thing since sliced bread. I found a simple tutorial here: http://code.google.com/edu/parallel/mapreduce-tutorial.html Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Wednesday, December 19, 2007 10:11 AM To: [email protected] Cc: Pete Beckman; Rick Stevens; Kamil Iskra; Kazutomo Yoshii; Sam Lang; Ralph Butler Subject: Re: [mpich2-core] MapReduce/Hadoop stuff There was a talk about mapreduce during a IBM visit here and I asked about scalability and what sort of algorithms were used. The model is really most appropriate for task parallel problems and (because of the reduce) integrates the integration of the results. Fault tolerance is a major issue, so the "usual" collective algorithms aren't usable out-of-the-box. Still, understanding how to implement mapreduce in a highly scaleable and heterogeneous environment would be an interesting project, as would understanding how to optimize across mapreduce programs (compiling the programs to minimize power or data motion, combined with runtime adaptivity). Bill On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote: We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6. Rajeev http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/ William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign