February 2011 - X-store - mailman3.cels.anl.gov

comments on 5.2, discovery of data
by Phil Carns 23 Aug '11

23 Aug '11

I don't know if there will end up being room or not, but when I was reading this I was thinking that it would be neat to have a figure there to illustrate the idea of constructing multiple name spaces out of attribute/provenance information. Maybe something with three panels; the first would list some objects and their attributes just in a plain table. Something like this, following the examples discussed in the text (maybe labeling the categories of standard provenance vs. domain-specific attributes): object jobid owner ocean_mod_resolution -------------------------------------------- A 123 user1 5000 B 123 user1 5000 C 222 user1 N/A D 245 user2 4000 E 245 user2 4000 F 245 user2 4000 The other two panels would show a couple of name spaces created from those objects; one with a tree hierarchy that organizes the objects by owner and then job id, and the other with a tree that organizes by ocean model resolution (and leaves out object C, which was from a different problem domain). Might help illustrate how its possible to have multiple hierarchical name spaces overlaying the same data. I'd be happy to help draw something like that if needed (and if anyone else thinks it would be a good idea). BTW, I have no idea what range of values make sense for ocean model resolution :) My only comment on the text itself is that I think the last couple of sentences of the first paragraph (about the implementation challenge of how to maintain indexes) could be moved to the last paragraph instead. I think that point fits in better with the other implementation-y stuff at the end than it does with the introductory text. Also makes sure that the idea sounds interesting before hinting that it might be hard to build :) -Phil

2 5

Sandia team changes have been deposited
by Ward, Lee 28 Feb '11

28 Feb '11

All, I have deposited the Sandia team's changes. Missing a couple of CVs but those are not likely to produce conflicts so I believe we can proceed. A couple of members on my team provided some approximate budget information. I did not modify the document to actually include that content. I dropped these budget files into the bib directory. Doesn't seem the right place but it's where I found budget info from others. I'm going to send a tar-ball of what I checked in back to my team for a double-check. I'm confident I got everything, though, and don't expect any oopsies. If so, I'll handle fixes amidst the nromal flurry of activity. In short, I believe we can remove our virtual lock on the document. --Lee

3 2

build broken?
by Phil Carns 18 Feb '11

18 Feb '11

Anyone know why the pdf won't build at the moment? -Phil

2 2

Fwd: Epidemic routing
by Rob Ross 16 Feb '11

16 Feb '11

FYI. Epidemic stuff from the other team. I haven't read it yet, but sounds relevant and like we have another fan of the approach. -- Rob Begin forwarded message: > From: Richard Brooks <rrb(a)ACM.ORG> > Date: February 16, 2011 6:55:25 PM CST > To: P2PFS-L(a)CLEMSON.EDU > Subject: Epidemic routing > Reply-To: PEER-TO-PEER FILE SYSTEM DISCUSSION <P2PFS-L(a)CLEMSON.EDU> > > Attached is an IEEE Trans SMC paper and Chapter 11 from > my Security Book (Disruptive Security Technologies with > Mobile Code and Peer-to-peer networks, by R. R. Brooks, > CRc Press, Boca Raton, 2005). > > I am most happy with the SMC paper, but included the Chapter > because it deals explicitly with the continuum: > > Flooding -- Gossip -- Random walk(s) > > There is a similar chapter in my distributed sensor > networks book. > > Basically, random graph theory in math mainly looks for the > giant component threshold in graphs as the number of nodes > becomes infinite. The physics counterpart finds the percolation > threshold in a regular grid as the size of the grid becomes > infinite. > > You can use similar principles for finite systems. The mobile > net SMC paper shows how to do some of this. For gossip > protocols and n random walkers you can calculate the probability > distributions. The percolation thresholds I find are pretty robust, > and we have done this to do a number of interesting things: > clone detection, key server assignment, etc. > > Got to go back to proposal writing. Would be glad to expound on > this approach until everyone else falls asleep. > -- > =================== > R. R. Brooks > > Associate Professor > Holcombe Department of Electrical and Computer Engineering > Clemson University > > 313-C Riggs Hall > PO Box 340915 > Clemson, SC 29634-0915 > USA > > Tel. 864-656-0920 > Fax. 864-656-5910 > email: rrb(a)acm.org > web: http://www.clemson.edu/~rrb >

1 0

Lee's "full pass"
by Ward, Lee 15 Feb '11

15 Feb '11

I did as I said I would. Changed very little. Some small sentence wording, due to awkward construction. Changed a little to use less, um, judgemental. I note that the sections at the end, like risk mitigation, have had no love yet. As I put in my comment, there, I don't think we want this in the proposal. --Lee

2 1

Can't tag the repository :(
by Ward, Lee 15 Feb '11

15 Feb '11

I cannot tag the proposal. It doesn't work the way CVS does and our repository does not appear to be set up to support the equivalent, copy, operation. Instead, I recorded the revision. I shipped revision number 196. --Lee

2 1

assignments
by Rob Ross 15 Feb '11

15 Feb '11

Getting close, thanks! -- Rob --- RobR: pass over 7.x, take page counts out, cut todos that i can. Phil: pass over 7.2 milestones, no more team language Sam: pass over 6.x Galen: pass over Exec. Summary Lee: full pass Phil: pass over 4.x Phil: improve titles of 4.1 and 4.2 to be more clear about reads and writes. Phil: heterogeneous into 4.3 title. Scott: pass over 4.x after Phil All: Get CVs in. Fix refs.

2 1

assignments
by Rob Ross 15 Feb '11

15 Feb '11

RobR: 5.0 (match with 3.0 and 4.0 tone), 5.1 (include geodesic figure here), 2.4 RobR: 4.0, title change: Placing data to improve performance Dries: 2.3 Sam: 3.1, active and passive models, may need some more information from Lee Lee: 3.3, 4.3, look, improve. Brad: 5.3 Scott: 3.2, trim down some, maybe "redundant storage for scientific workloads" Phil: 3.3 Galen, Phil: Read through, get a feel for level of detail, etc. --- Also, Friday concall at 2-3pm Eastern. Rob

4 35

Notes from today's concall about the collaborative proposal
by Ward, Lee 14 Feb '11

14 Feb '11

All, We spoke a lot, today, about the collaborative proposal and our pass over it. So much information, I thought I would send out a summary... Timeline: a) We get the proposal tomorrow, 2/15, or Wednesday 2/16. The other team will still be doing some editing but are supposed to be mostly done. b) We are to be done by 2/27, midnight. There will be no extensions. Your contribution: a) We are not the final pass, in fact what we do will be seriously edited for brevity. So... I suggest you try to find existing language and attempt a change that retains the original meaning and adds what you need. If you need to add anew, well you just do but (from the crystal ball) I would think discussion about what we will do that is briefly and well integrated or with only small qualification is much more likely to survive Dr. Ross's notoriously draconian edits. b) We won't have a whole lot of latitude after the editing for brevity pass, or at least I would expect that the criteria for changes will be very specific an difficult to satisfy. c) We won't have much time after the editing for brevity pass. Try to get your input in during this pass. Action item for Tony: i) Start setting up the svn repository now. You were going to call Jill to get the current mailing list to use for that. Action item for Lee: 1) Send Tony a tar-ball of the doc Tue or Wed 2) Make a PDF version and send it to the p2p mailing list at the same time as (1) 3) Send Tony Rob Ross's email address so Tony can add him to the repository as well. 4) Invite Rob Ross to our telecons. What have I forgotten? --Lee

2 5

some papers
by Rob Ross 11 Feb '11

11 Feb '11

During a break I was looking through these papers. Onix is interesting as it relates to the idea of an underlying infrastructure for distributing and managing system state. Very different in *how* they do things from what we've talked about in Triton so far, though. Percolator is an interesting use case for Triton and could serve as a motivator for the triggers that Brad is discussing in the proposal. Sits atop BigTable. The Ford paper has a bunch of availability data from Google clusters. This data could be helpful in replacing Garth's availability data with something more modern. All useful. Thanks for bringing these up! Rob On Sep 24, 2010, at 10:36 AM, Seung Son wrote: > > You're right. Google has another MR paper in OSDI: > > 1) Large-scale Incremental Processing Using Distributed Transactions > and Notifications, by Daniel Peng and Frank Dabek, Google, Inc. > > Looks like what they mainly talk about is Percolator although they > mentioned Caffeine in the paper. > > They also two additional papers in OSDI this year, and one of them > talks about availability in storage systems. > > 2) Availability in Globally Distributed Storage Systems, by Daniel > Ford, François Labelle, Florentina I. Popovici, Murray Stokely, Van- > Anh Truong, Luiz Barroso, Carrie Grimes, and Sean Quinlan, Google, > Inc. > > 3) Onix: A Distributed Control Platform for Large-scale Production > Networks, by Teemu Koponen, Martin Casado, Natasha Gude, and Jeremy > Stribling, Nicira Networks; Leon Poutievski, Min Zhu, and Rajiv > Ramanathan, Google; Yuichiro Iwata, Hiroaki Inoue, and Takayuki > Hama, NEC; Scott Shenker, International Computer Science Institute > (ICSI) and UC Berkeley > > Seung > > > <Ford.pdf><Peng.pdf><Koponen.pdf> > > > On Sep 20, 2010, at 5:29 PM, Sam Lang wrote: > >> >> It sounds like Google will have a paper in OSDI describing >> Caffeine, their "next thing" after Map/Reduce. There's some slides >> here about it and Colossus, which is their GFS version 2. There's >> not a whole lot on Colossus, but it looks like they do erasure >> codes rather than mirroring, even across clusters. Replication is >> now client driven rather than forwarding to a server and having it >> do replication. Colossus does distributed metadata now too. >> >> One thing I thought was interesting from the slide deck, after >> seeing a failure, they wait for a bit before doing recovery, >> because they sometimes see correlated failures coming in bursts. >> >> Slide deck: >> >> http://research.google.com/university/relations/facultysummit2010/storage_a… >> >> -sam >

2 2