We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6. Rajeev http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/
This is relevant to the Programming Models Center too. I have heard these words a lot but I don't know exactly what they refer to. Is there a tutorial about this? Do you know what it means? On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote:
We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6.
Rajeev
http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/
http://wiki.apache.org/lucene-hadoop/HadoopMapReduce Is this what you are looking for ? On Wed, 19 Dec 2007, Rusty Lusk wrote:
This is relevant to the Programming Models Center too. I have heard these words a lot but I don't know exactly what they refer to. Is there a tutorial about this? Do you know what it means?
On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote:
We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6.
Rajeev
http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/
There was a talk about mapreduce during a IBM visit here and I asked about scalability and what sort of algorithms were used. The model is really most appropriate for task parallel problems and (because of the reduce) integrates the integration of the results. Fault tolerance is a major issue, so the "usual" collective algorithms aren't usable out-of-the-box. Still, understanding how to implement mapreduce in a highly scaleable and heterogeneous environment would be an interesting project, as would understanding how to optimize across mapreduce programs (compiling the programs to minimize power or data motion, combined with runtime adaptivity). Bill On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote:
We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6.
Rajeev
http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
Yes, it is not clear how this stuff is useful for anything other than task parallel and embarassingly parallel. Yet there is a growing group of people who think it is the next best thing since sliced bread. I found a simple tutorial here: http://code.google.com/edu/parallel/mapreduce-tutorial.html Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Wednesday, December 19, 2007 10:11 AM To: [email protected] Cc: Pete Beckman; Rick Stevens; Kamil Iskra; Kazutomo Yoshii; Sam Lang; Ralph Butler Subject: Re: [mpich2-core] MapReduce/Hadoop stuff There was a talk about mapreduce during a IBM visit here and I asked about scalability and what sort of algorithms were used. The model is really most appropriate for task parallel problems and (because of the reduce) integrates the integration of the results. Fault tolerance is a major issue, so the "usual" collective algorithms aren't usable out-of-the-box. Still, understanding how to implement mapreduce in a highly scaleable and heterogeneous environment would be an interesting project, as would understanding how to optimize across mapreduce programs (compiling the programs to minimize power or data motion, combined with runtime adaptivity). Bill On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote: We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6. Rajeev http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/ William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
My tentative plan has been for the I/O team to build up an infrastructure of communicating persistent MPI processes for use in storage systems. That infrastructure might be a good environment for playing with the MapReduce ideas, if we didn't want to build on top of the Java Hadoop implementation. Rob On Dec 19, 2007, at 11:31 PM, Rajeev Thakur wrote:
Yes, it is not clear how this stuff is useful for anything other than task parallel and embarassingly parallel. Yet there is a growing group of people who think it is the next best thing since sliced bread.
I found a simple tutorial here: http://code.google.com/edu/parallel/ mapreduce-tutorial.html
Rajeev
From: [email protected] [mailto:owner-mpich2- [email protected]] On Behalf Of William Gropp Sent: Wednesday, December 19, 2007 10:11 AM To: [email protected] Cc: Pete Beckman; Rick Stevens; Kamil Iskra; Kazutomo Yoshii; Sam Lang; Ralph Butler Subject: Re: [mpich2-core] MapReduce/Hadoop stuff
There was a talk about mapreduce during a IBM visit here and I asked about scalability and what sort of algorithms were used. The model is really most appropriate for task parallel problems and (because of the reduce) integrates the integration of the results. Fault tolerance is a major issue, so the "usual" collective algorithms aren't usable out-of-the-box. Still, understanding how to implement mapreduce in a highly scaleable and heterogeneous environment would be an interesting project, as would understanding how to optimize across mapreduce programs (compiling the programs to minimize power or data motion, combined with runtime adaptivity).
Bill
On Dec 18, 2007, at 11:36 PM, Rajeev Thakur wrote:
We need to start looking at this MapReduce/Hadoop programming model from Google/Yahoo. It is getting big and Google, Yahoo, IBM are investing big money in it and in educating students about it. Google and Yahoo are two new sponsors of HiPC this year. Google has sent 12 people (from 0 in previous years). The head of Yahoo Research is here. In reference to Microsoft's investment of $500 million in data centers in Chicago and Ireland, he said "we measure data centers in watts not dollars." I heard DARPA is starting a new program on software/programming models for multicore. There was a meeting about it in San Mateo on Dec 6.
Rajeev
http://labs.google.com/papers/mapreduce.html http://lucene.apache.org/hadoop/
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
participants (5)
-
Anthony Chan -
Rajeev Thakur -
Rob Ross -
Rusty Lusk -
William Gropp