December 2010 - Esc-team - mailman3.cels.anl.gov

Exascale Software Center: Collecting Application Data
by Pete Beckman 04 Jan '11

04 Jan '11

Hello. The first version of the Survey, borrowing heavily from the material Bronis provided and the CScADS survey, has been checked into the repo. I also started the ball rolling with the co-design centers, sending it the co-design liaisons. It would be great if the NNSA labs could coordinate how they will engage the key NNSA mission application teams that are expected to be clear exascale candidates. I'll be out of the office until the evening of the 7th, but we can probably move forward improving the survey and getting responses via phone calls as soon as folks return from the break. -Pete Begin forwarded message: > From: Pete Beckman <beckman(a)mcs.anl.gov> > Date: December 29, 2010 11:41:59 AM CST > To: James P Ahrens <ahrens(a)lanl.gov>, David Martin <dem(a)alcf.anl.gov>, Alice Koniges <AEKoniges(a)lbl.gov>, Anshu Dubey <dubey(a)flash.uchicago.edu>, Rajeev Thakur <thakur(a)mcs.anl.gov>, "Ewing L. Lusk" <lusk(a)mcs.anl.gov>, Andrew Siegel <siegela(a)mcs.anl.gov>, Edoardo Apra <aprae(a)ornl.gov>, Curtis Janssen <cljanss(a)sandia.gov> > Cc: Jack Dongarra <dongarra(a)eecs.utk.edu>, Rob Ross <rross(a)mcs.anl.gov> > Subject: Exascale Software Center: Collecting Application Data > > Hello, > > We have assembled the first version of a simple survey that can help everyone share information about the directions the computational science teams for exascale are exploring. > > It is preliminary, but we would like your help both preparing to answer the questions, as well as improve the survey. I think the best thing is to have short phone interviews with each team after the New Year, so longer conversation-style responses can be capture. > > Please see the attached survey, prepare answers and discuss ways to improve the survey. After the break, we will followup with phone conversations. > > Thanks for your help. > > PS. I'll be out of the office until the late afternoon of the 7th, so some other folks will be pushing this forward while I'm out. > > -Pete > > >

2 1

ESC Next Steps Planning Meeting - Location Confirmed
by Faith Ruppert 22 Dec '10

22 Dec '10

Good afternoon all: For many Santa Fe is a difficult and far away place to get to, even though Pete really wanted red chili.....Chicago is the preferred location. The meeting will take place at ANL, January 27-28, I have included the ANL Guest House link http://www.anlgh.org/ , the room rate is ~ $70, and per diem for this area is $92. If you have any questions or comments, please let me know by COB tomorrow - as it's my last day before the holiday, I return January 4th. I will send additional meeting logistics after we return. Enjoy your Christmas and New Year! Faith Faith Ruppert Argonne National Laboratory 9700 S. Cass Ave., ETCi/240 Argonne, IL 60439 (630) 252-1351 ruppert(a)anl.gov

1 1

ESC Meeting Location
by Pete Beckman 21 Dec '10

21 Dec '10

Ok, we have three choices that won't charge us for internet, meeting space, etc. Here are the choices 1) Chicago: Major hub, lots of flights, requires taxi/shuttle, Chicago pizza 2) La Fonda, Santa Fe: requires taxi/shuttle, excellent food within walking 3) Atlanta Airport Hilton: free shuttle, long flights for west/northwest coasters We will probably collect a small registration fee to cover little things along the way, since DOE meetings like this are so hard to budget. If you have a strong preference for one of these, please email me directly as a reply to this message. Otherwise, we will pick the location based on best deal, flight availability, etc. by the end of the day. PS... (please do not suggest additional locations. Faith has already done plenty of phone calls and negotiating, and we don't have time for more options this time... maybe next time). -Pete

1 0

Reserve the dates...
by Pete Beckman 20 Dec '10

20 Dec '10

ESC-Team, We are looking at January 27 and 28th for the ESC planning meeting. The plan would be to fly in the 26th evening and end about 4:00 on the 28th so some people could catch flights home. We are still looking at several locations, based on cost and weather. Currently, ANL is by far the cheapest, but we are also looking at a couple hub cities that get less snow and New Mexico, which is convenient for west coasters. Faith will have something picked before the Holiday break. More details to follow, but please reserve those dates... -Pete

1 0

Notes from the Friday Meeting
by Pete Beckman 14 Dec '10

14 Dec '10

Hello, The meeting last Friday with ASCR and NNSA went very well. I've checked the slides into the repo, and the next step is some very hard work gathering information and planning. Sorry... but expect more mail and to-do items. Here are some notes from the meeting that Jack, Paul Messina, Rick, and I put together: ((in no particular order, just notes on the conversation)) * Folks liked the concept that modest hardware architecture/functionality changes could simplify some of the system software that has to be developed and that it is realistic to believe that hardware vendors would consider implementing such suggestions. The topic came up while we were discussing Blue Gene and some changes we had pushed to allow diagnostics that could be run as backfill jobs in the queue. * There was a discussion about "mortgages". NNSA has invested in several technologies that have become widely used by application scientists at NNSA labs (Open Speedshop, Totalview, Lustre, etc). NNSA now carries the "mortgage" on these packages since they don't have a large enough market or user base to get key enhancements from the community -- meaning to carry them forward for each successive platform requires rather significant funds. Thuc pointed out that whatever the ESC plan, we should carefully consider the mortgages created. * In a similar discussion, Thuc pointed out that data analysis software and viz tools were largely up to the applications. Since the data analysis and viz is still funded, the issue really seemed to be more of an accounting issue --- which funding bucket does the work come from... the apps or the system software. Thuc seemed to indicate having the apps fund it helped her, but I'm not sure how. * We discussed the new world of checkpointing... Dan agreed that standard checkpointing to a filesystem would not scale, and that in-system storage could support more sophisticated error recovery (and probably not full-system checkpointing). We discussed for example, that with in-system storage, maybe a "file" was not the right abstraction, but that saving out persistent arrays and objects to NVRAM may be more suitable. * Along the same lines, we discussed data analysis, and that pushing all the data to a filesystem for later exploration would not scale well. Dan gave the analogy of the LHC... there is some filtering that has to happen between the experiment (detector) and the disks. Likewise, he agreed that doing in-situ analysis, and supporting the framework for that (even if the specific tool was supported via the application team) would be good. * Both of these topics (checkpointing and viz/analysis) would require more sophisticated run-time systems. * Everyone suggested we start with a narrow scope, and then grow it with funds and success. * HQ would like an observer at the January meeting, and by then we should have a good estimate of version 1.0 * It is important that at the March review we have a good communication plan, one that includes discussion of education, web sites, blogs, workshops, etc. * We do not need to track DARPA carefully. Folks said it was a "long story", but not to worry about aligning too closely. Nevertheless, it seems we need to be prepared to answer questions and explain any overlapping work. * We had a very animated discussion about virtual machines and emulation environments. Environments like Google/Android and the iPhone have development environments that can be downloaded. (I demonstrated virtualbox and my ubuntu and winxp images running on my mac). The notion that we might want to have a simple to download virtual machine environment like ARM and x86 have was well received. We discussed the idea that the hardware RFP might just want to specify that as part of the system. It could help test different ideas as well as support QA and even education. * At the review in the Spring, it was suggested that we prioritize things kind of like a travel guide book... one '$' or '$$' or '$$$' depending on the money a portfolio may require. * In the spring, ASCR is going to have a meeting where they will showcase the research they are funding. Any ESC plan should understand that effort. * We agreed that the ESC was focused on Open Source, and not commercialization or productization. We should (as explained above) have a plan for the "mortgage". However, from what I could gather from the discussion, I didn't get the impression that folks thought vendors reduced the mortgage. * Folks believe BSP is dead and that task and graph methods of programming may be more scalable. Someone said "BSP does not scale to 1 billion cores", and everyone agreed. * The notion of a unified run-time system that can support composition of programming models so folks could move away from their current model in an incremental fashion was well received. -Pete

1 0

ESC Next Steps for planning
by Pete Beckman 13 Dec '10

13 Dec '10

Hello, In preparation for the meeting with ASCR and NNSA to report on ESC planning efforts, we thought it might be helpful to focus on explaining 3 big areas: 1) The group who is working together, our goals, basic schedule 2) The "wiring diagram" of the process, how ESC will work with co-design efforts, hw platforms, vendors, and making co-design decisions 3) Background on some of the technical choices, evolution/revolution, etc. So... we will need some help with material. Based on our phone conversation last week, I've mapped the NNSA technical areas to a bit broader set, still based on what we have called "system software": Programming models --> Programming models (PM) Systems software --> OS, Run-tume systems, Cybersec, SysMgmt (ORCS) I/O, net, and storage --> I/O, Networking, and Storage (IONS) Tools --> Tools (TL) Viz/data analysis --> Data Analysis and Visualization (DAV) --> Math libraries, algorithms, and frameworks (MAF) To help launch things quickly, I've populated the 6 groups with some folks, based on what I thought their interests might be, and where we needed coverage. Not everyone on this email list was stuck in a group, we will have plenty of time to shift, add, and rearrange folks. Also, I've left off some of the managers, so they can help coordinate across several areas -- so it's just a quick bootstrapping. Notes: * Jim Ahrens has volunteered to help bring together some data analysis and visualization folks together to help build the plan. More names to follow as he builds that. * Rob Ross has volunteered to bring together the IONS group to discuss how to organize that part of the plan. Initial draft of working groups: PM: Rajeev Thakur, Pat McCormick, Barbara Chapman, Pavan Balaji, Dan Quinlan, Bill Gropp ORCS: Ron Brightwell, Pete Beckman, Ron Minnich, Kamil Iskra, Marc Snir, Bill Kramer IONS: Rob Ross, Gary Grider, Mark Seager, Shane Canon, Galen Shipman, Lee Ward TL: Bronis de Supinski, Jeff Vetter, Al Malony, John Mellor-Crummy DAV: Jim Ahrens MAF: Jack Dongarra, Mike Heroux, Jim Demmel There are probably cross-cutting topics like Fault/Resilience and Performance that should be wired across the several groups. We will get to those, but the groups above are probably enough to get started. So... here are the specific items we need help with: * Sections 1 and 2, above. I'll work on a draft today, and then circulate. If folks have ideas for material, please, please send it! For example, someone suggested a good description of part of the wiring could be: CoDesign/Mission Apps --> Capabilities --> Technology --> Software --> Teams/Packages. In other words, trying to capture the discussion from Friday, we need to have a fairly open-ended conversation about capabilities required before selecting technology and software starting points that could lead to actual folks working on the software. * Technical choices: It would be very useful for our conversations in DC to have some concrete examples from each group above on evolution/revolution, and current discussions. All slides or ASCII text is welcome. Rick suggested that showing where we are almost certain a particular approach will not scale, and where a new technique is required would help the program office understand some of the key areas. For example, I know Rajeev, Rusty, and Bill Gropp, within the context of both CS research and MPI3, have mapped out what MPI calls will simply be unscalable, based on O(p) -- showing where new invention will be required. They have some very concrete examples that demonstrate what can't continue to scale well as we move to billions of threads. Similarly, I believe the file system folks have mapped out some operations that have semantics that make it impossible to scale to billions of clients. Likewise, there has been some discussion about what would change dramatically with in-system storage. Given the previous email from Jim and Edgar, there are things they have been examining that indicate ways new algorithms can greatly improve performance. So, it would be very helpful if each of the groups could organize a top evo/revo list... with the understanding that rigorous defense will be needed at some point. In other words, we probably can't say "bulk synchronous programming can't continue to scale", even though we may all believe that, without more supporting explaination. I think pointing out some clear examples and why they end up needing significant re-thinking of the solution would help the program office understand the scope of some of the issues (just a sampling). * As you have seen, we have opened up a conversation with the co-design centers. The Combustion folks and Climate folks have also offered up folks to partner with the ESC, even though they are still forming proposals, etc. Our next step is to followup with what Bronis had as the basis for their interviews, and the xls inventory, and the attached PPT (checked into the repo) that we used for 4 summers at the CScADS workshops, when we asked SciDAC and INCITE users about their current software and techniques. So... could someone volunteer to synthesize the material we have so far into single (compact) document that we can use for discussion with the DOE Mission apps and CoDesign centers? Also, we will need the xls updated for each of the major DOE computing platforms. Finally, we need to pick the Jan/Feb meeting times quickly, since schedules are filling up. Before we open up a Doodle... if you have suggested times or obvious big conflicts for large groups of people (conference, etc), please email Faith. She will set up the Doodle later this week and get the ball rolling on a venue. -Pete

3 2

Final (hopefully) slide set
by Pete Beckman 10 Dec '10

10 Dec '10

Thanks for everyone's comments and improvements. We have rolled in as many as possible. Attached is the final deck we will be presenting. I'll check them into the repot later today. -Pete

1 0

Current Draft of slides for tomorrow
by Pete Beckman 09 Dec '10

09 Dec '10

Jack, Paul Messina, Rick, and I are still editing these, but if you have suggestions, please send them. We will have to send the final version very very late tonight or early tomorrow, so NNSA labs on the phone will have the copy. -Pete

1 0

Fwd: Software requirements for Exascale System Software
by Pete Beckman 07 Dec '10

07 Dec '10

FYI, I sent this email to the esc-co-design contacts, as we gear up to discuss their current approaches and needed system software. > Hello, > > As we continue planning for Exascale Software, we intend to refresh a set of software and capability inventories that were constructed some time ago for large systems, and then work with your co-design center to sketch out predicted future needs. We realize that what your codes are using now is only a snapshot (off by at least a 1000x) from what you might be doing several years from now, but the exercise will provide a framework for describing where we are now, and where your code expects to go as it relates to system software. > > Included below are the original spreadsheets, just to help you get a flavor for part of the work we are trying to do. > > So, here is a quick sketch of what we are trying to put together (there is nothing to do yet --- we will try and schedule some phone calls to discuss and walk through pieces, to make it as easy as possible to discuss things) > > * A quick inventory of what system software is currently required/used by the codes your application will use as a basis for exascale (see examples in spreadsheet) > > * Your best guess of what you will need in the future. Understanding this from a "capability" perspective, rather than a currently used library or package perspective, would be very helpful. For example, while you might be using a certain math library or I/O library today, the path to exascale might mean a different approach will be needed... So we would like to be as forward looking as possible with respect to what system software capabilities might be helpful for your application. > > * Finally, there are a series of questions that the LLNL folks have been using to open a dialog with their code teams regarding their future needs. For example, what is your model for parallelism, do you have issues with load balancing, what programming model do you expect to move toward (MPI+OpenMP, MPI+Pthreads, UPC, Global Arrays, etc, etc), what do you expect to do for I/O, etc. We will be tuning up those questions, with your help, so the Exascale community can have a shared, high-level understanding of application expectations for system software. > > > So... we will try and start setting up some phone calls. However, if your code team has already done some of this, for example, for your proposal, it would be very helpful if you could forward the information to us, so we can use it to build the questions and inventory. Any and all suggestions are welcome. > > Thanks, > > -Pete

1 0

Testing esc-team email list
by Pete Beckman 06 Dec '10

06 Dec '10

This is a test. 1, 2, 3, ... 10e18 -Pete

1 0