January 2011 - X-store - mailman3.cels.anl.gov

fault handling text (section 3)
by Phil Carns 04 Feb '11

04 Feb '11

The text is still rough, but I reorganized section 3 (operating in a faulty environment) to hopefully flow a little better. This is the approximate outline: 3.0 - general intro, why revolution is needed 3.1 "efficiently identifying failures" - intro to problem - 2 solutions: - no proactive fault detection, clients detect and put/locate data elsewhere - proactive fault detection using epidemic algorithms 3.2 "redundant storage for scientific data" - intro to problem, why synchronous replication is inefficient - propose asynchronous replication in conjunction with background erasure coding (no alternatives listed, but there is a lot of wiggle room here) 3.3 "reconstruction strategies" - talk about policy options, such as replica placement, replication level, and when to reconstruct - 2 solutions: - deterministic placement (consistent hashing), constant replication level, and immediate reconstruction - place replicas anywhere, fluid replication level, deferred reconstruction There is a draft figure in 3.2 to help visualize the difference in replication schemes. Lee, make sure to check the first solution described in section 3.1. I constructed that by gluing together some of your comments, but it could use a little more detail. It seemed logical to explain the "no explicit failure detector" approach first, because it seemed awkward to expound on the wonders of epidemic protocols only to come back later to say we might not use them at all :) I forgot who is working on the placement section right now, but I commandeered the erasure code discussion from there because it made more sense in the fault tolerance section. -Phil

3 19

Proposal name
by Bradley W. Settlemyer 01 Feb '11

01 Feb '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I wan thinking of a name for the proposal. Came up with ROStAr (Replicated Object Storage Architecture) which could become ROXStAr (Replicated Object eXascale Storage Architecture) but that's too tongue in cheek probably. Which is all based on looking at something ending in StAr, and that isn't On-Star. Oh well, things to do when you're out sick. Also thought about something like Osterizer -- you know, like the blender. Cheers, - -- Brad Settlemyer Research Associate Oak Ridge National Laboratory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJNOaDnAAoJELH6JBMGOjyZEBYIAJUCBcjoFk6Zj11Vxqp7g5O4 5e6x+T8q79stMTwLjL5Ozk0aps73BSt6vG5zABXRTfj1mV8Exwo0bZYQs4k5a8vv I99G0SA/qFyimSFZQi/aYUKYV3srcgP8xxL9d/sNQSt+ThddxSzhwCc21X3+l6Tq 1gIzhJZmOBaYvjjQ5QebqZTmF9RKGh/L8l1q7TfzLUbpfo59jYFrZoPmA/9asn4a u8ihK2khnXlq8tCQLxotogiiSfQ4Ffq5PByqTTIQWqIyQtxx0vD3JyfCYFGfN9+N kgfFlf00e1MlUCz2OWJV3W1Isol1cm/X3qubrpQ5FhBvQAJetbbpooNBlDwvKhA= =ygDV -----END PGP SIGNATURE-----

2 1

Namespace strawman
by Settlemyer, Bradley W. 31 Jan '11

31 Jan '11

Sorry about not getting this out earlier. The ORNL VPN is acting up for me. Having to use the webmail interface: Hello So here is a strawman for object store namespace operations: So probably we should have a set of attributes that are automatically appended to every object: ObjectId (uint64) Version (uint64) JobId (uint64) CreationTime (uint64) ModificationTime (uint64) CreateUserid (uint32) Mutable (boolean) (The type sizes are guesses - due to index packing concerns, 48 bits may make a lot more sense than 64 where possible, and 8 bytes is an upper limit on what you need to store a date). Then, for sanity's sake it might be nice to add more attributes via the high level software: ObjectName ApplicationName Some kind of signature hash to locate the binary this object associates was generated by -- this isn't trivial to my little brain in the general case, but fairly easy to accomplish on the XT5, and likely easy on the BlueGene. Okay, its not hard to come up with more attributes to add. So then for a set of interfaces, we might want the following type operations for the easy stuff: OID object_begin(attrs*) int object_write(OID, size_t, char*) int object_read(OID, size_t, char*) int object_commit(OID) int object_load(size_t, OID*) Really, object loading should be asynchronous, with the option to check an object list for completed objects -- at least to go fast in a general type way. Then for namespace operations: int object_setattr(OID, key_type, value_type) int object_iter_setattr(size_t, OID*, key_type, value) int object_getattr(size_t, OID*, key_type, value_type*) int object_iter_get_attr(size_t, OID*, key_type, value_type*) OID* object_find(key_type, value) OID* object_mfind(size_t, key_type*, value*) In this case, the object iterator is just a shallow container, not the full object. From an implementation point of OID* is likely the wrong type, you probably want an object iterator. You could implement POSIX on top of that as long as the iterator was persistent across reboots. This is just a strawman, but with the object iterator, you can construct better interfaces for querying the namespace. Bear with my loose syntax here: OID* object_filter(size_t, OID*, int (filter_func*)(OID)) That could be used to perform searches for attributes within ranges or meeting some conditional. So what's missing? What cannot be implemented efficiently? Obviously the namespace needs a consistency model. And assuming the namespace is distributed, these functions do an awful lot of work in the background to go trolling around that distributed space. Should we expose more of that distribution info. I would lean toward yes actually. Maybe even do that pNFS style with a mapping function, to find the namespace distribution. Though since namespaces are mutable, the map can be stale. Which is an argument not to do that. Another thing to consider is an interface for constructing indices: int object_create_index(size_t, OID*, size_t, key_type*) That can really speed up those find and mfind operations. A start? Cheers, -- Brad Settlemyer Research Associate Oak Ridge National Laboratory

5 10

6.3 Software Management committed
by Atchley, Scott 25 Jan '11

25 Jan '11

Hi all, I have added text for 6.3. I did not edit 6.1 or 6.2. I believe section 6 needs a brief intro and and I can take a crack at it once more is filled in. Section 6.2 has a note to add text from the CODES description, AESOP, etc. Who has the CODES text? Also, is AESOP from CMU? Is it this or something else? http://www.cs.cmu.edu/afs/cs/project/able/www/aesop/ Scott

4 6

intro hacking done
by Rob Ross 21 Jan '11

21 Jan '11

hey all, i made a pass over the intro. i think it's better, have a look and make comments. i think we will want to adjust the headings of the research sections a bit as we continue to iterate on the topics in them...maybe how i describe them on p.4 is closer to where we will end up? thanks, rob

1 0

Updated text in 4.4 and 5.2
by Bradley W. Settlemyer 21 Jan '11

21 Jan '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Attached the new doc. Its still a work in progress. Was planning to add in bib tex entries next and then re-edit text. I'm uncertain on length -- of basically anything. I presume the final length is desired to be 20 pages. At present its 26.5. I hate to add too much text when its already over. Do we need to set page budgets or something similar. Personally, I think the text I put in the core abstraction part could move into the intro, as its practically object store motivating -- but there is no detailed motivation part. Just the assertion that a new storage system is needed. Here are a few reasons I think a new storage system is needed that are not necessarily captured in the proposal: 1. Large numbers of clients break existing file systems by making everything look random at the disk array. 2. Accessing existing file systems from user space requires way too many memory copies. 3. File systems don't expose anything about the underlying devices. And so users go slow portably -- except that things aren't even portable!!! 4. Posix I/O apis are broken. Even new ones like async I/O were done wrong, much less older ones like access and readdir. 5. File systems load balance in ways that mystify HPC users!!! 20% full disks worry about becoming overfull rather than providing QOS to clients. 6. All that computational power in the storage system, and no way to leverage it! I could probably come up with more if pressed. But should any of these things be in the intro? Instead of the old disks and cpus are increasing in perf at different rates? Cheers, - -- Brad Settlemyer Research Associate Oak Ridge National Laboratory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJNN1ZpAAoJELH6JBMGOjyZN20IALJndZriMBCfsGMZCtGatfx9 CxQOREUdpAW+gQKnxbNuLu7UALNzaBmejLBQCsfExmxbFchtICCAGIReg5RqjEGI 5Tm/F/wrYu9Loq03nwNhGAOpnf0R2ho08G3XGDlpLRquAyWuAqNm/vbh+bzay3J4 scsOgUGfiK8eVlULx5sA+Lidc7fJP1P47E5hRk82CHLzVB1/VXsoR1mgnU+UZ7f3 XX2JBPCWDrLdJut2bZnq1gmU6aAKYTpsliEDWVv1dxf1VYUzGum1XejfPqejr4Br AJA/iBoZwZOAiv3iguEHG+oMl2HcYCzxaTxWVFW+piV4jFDNogqlFdsU714ly1s= =0WCn -----END PGP SIGNATURE-----

5 16

notes on the current proposal outline etc.
by Rob Ross 21 Jan '11

21 Jan '11

Hi all, Phil and I walked through the proposal this morning and assigned some section lengths, thought about the high-level organization, etc. Let's chat about this today. Also, I finally have some time after the call today during which I can concentrate on this, so let's think about where I could best contribute. Thanks, Rob --- Executive Summary (1 page, doesn't count) 1 Introduction (3 pages) 1.1 Architectural Change 1.2 Exascale Storage Characteristics 1.3 Exascale Storage Systems 1.4 Proposed Work Notes: - Proposed Work -> Proposed Work and Impact. Augment the bold statement that is already there to say more about the impact. - just one of the Agg. I/O throughput graphs - Architectural Change -> Exascale Architectures and Implications on Storage - merge 1.2 into 1.1 - merge 1.3 into 1.4, so we're more clear about *our* plan - need an overall architecture diagram of some sort so reviewers have a mental picture of the system - set up that core section is infrastructure that we will all use, that hard core research is in Sections 3-5. - in areas requiring revolutionary solutions, the team will pursue two design paths (?) - steal diagram from CODES proposal 2 Core components (3 pages) 2.1 Portable network abstraction layer 2.2 Local storage abstraction 2.3 Security model 2.4 Programming and Productivity Notes: - core set of components that form the infrastructure for a storage system - many examples of these around (LNET, BMI, Trove, Portals, OSD (?), ...) - based on preliminary research, we will develop replacements for these components adapted to their role in Exascale storage - need text from Dries - need text from Sam - client/server component figure here? this is software, while the S.1 figure is HW? or can we somehow get all this onto one diagram in the intro? - API? 3 Operating in a faulty environment (4 pages) 3.1 Tolerating client failures and network data loss 3.2 Efﬁciently identifying failures 3.3 Distributing data and reconstructing after loss Notes: - each major section should have at least one diagram. this is cheezy i know, but important esp. if reviewer knowledge is low... - idempotency is probably too low-level, definitely too much material. could just cut this section entirely i think? - S3.2 should be first subsection. - The team will investigate two possible (promising?) solutions. Solution A, Solution B. - discuss at end of subsection, provide short motivation for both - rationale: - not two teams - no server A, server B - consider splitting how we distribute and how we reconstruct into two (short) sections 4 Data placement and organization (4 pages) 4.1 Coordinating placement of new data with job execution 4.2 Mitigating the cost of inefﬁcient placement 4.3 Utilizing mixed resources Notes: - S4.1 needs to be reorganized after various insertions - S4.3 title should have "heterogeneous" in it? - S4.3 needs some "what we will do" text... 5 Data models and name spaces (Productivity?) (4 pages) 5.1 Enabling exascale science data models 5.2 Enabling scalable and fault tolerant name spaces 5.3 Discovery of data 5.4 Advanced data analysis Notes: - needs better intro - logically organizing data in a manner that closely matches to science data organization - mapping this organization onto storage efficiently - helping scientists find their data more quickly and easily - aiding in the processing of data, minimizing data movement 6 Support Infrastructure (2 pages) 6.1 Testing, Debugging, and Monitoring 6.2 Simulation 6.3 Software management 7 Management and Measure (2 pages) 7.1 Management 7.2 Milestones 7.3 Long term measure 7.4 Risk Mitigation Notes: - get rid of the risk mitigation section - reiterate the impact in the long term measure section Appendices (xx pages, don't count)

1 0

core storage text, name space plans
by Phil Carns 20 Jan '11

20 Jan '11

I committed the changes that I wanted for local storage abstraction text. I tried to say something about striking a balance between useful semantics at that level vs. extracting device performance. I'll let Brad fill in the latter part. We can cut the size of what I wrote later as needed. ... I'm going to take a cut at the "scalable and fault tolerant namespaces" part next (its just a random pile of notes right now). I think from the ANL point of view we have a few general ideas to pursue: 1) exploring API/semantics beyond POSIX 2) storing namespace information (and metadata in general) in the object storage system itself, rather than designating a separate metadata infrastructure 3) tuning the above, so that objects holding namespace and metadata information have appropriate redundancy, latency, and data models Lee, do you need some space in there to put ideas that your team wants to pursue? Or would you rather just see what I drop in there first? I know that you guys have already have a name server of some sort up and running. thanks, -Phil

2 3

moving stuff around
by Phil Carns 20 Jan '11

20 Jan '11

I just moved around a few things in the proposal: - moved the core section to be right after the intro (like we discussed earlier), but I didn't make any text adjustments to reflect that yet. I just wanted to make sure I remembered what topics are likely to be introduced where. - Per Brad's note in the core/local-storage section that we didn't address reads, I made a note to move some ideas that were awkwardly mentioned in the placement section (that we want to reorganize our local log structure during idle time in order to improve read/analysis performance) into the core/local-storage discussion instead. - To support the above (and various other places where we talk about burstiness), I moved the jaguar/intrepid burstiness figures all the way up into the intro. I was trying to figure out where to put them so... - I added a section in the intro called "Exascale storage characteristics". It has a bullet list of (at a high level) what we think the think the key properties of exascale storage are. I borrowed from Brad's list and added a few things. One of the properties is the burstiness as a result of exascale systems being capability machines so I referenced those two figures there. This stuff needs to be cleaned up, I'm just dumping ideas. I think that new section in the intro can be used to address some of Brad's comments (that we need to more concrete motivation for why we are building something new) and possibly some of Lee's comments (that our likely reviewers might need some gentle education to set the stage since we've cornered a lot of the storage expertise already). -Phil

2 1

mailing list
by Phil Carns 18 Jan '11

18 Jan '11

Hi all, We can use this mailing list for discussion on the x-store proposal moving forward. Here are the current subscribers: Rob Ross (ANL) Phil Carns (ANL) Sam Lang (ANL) Dries Kimpe (ANL) Lee Ward (Sandia) Galen Shipman (ORNL) Brad Settlemyer (ORNL) Scott Atchley (ORNL) thanks, -Phil

3 2