March 2012 - Asg - mailman3.cels.anl.gov

ANL status
by Dries Kimpe 12 Mar '12

12 Mar '12

- Kevin has been working on the SSM resource. He already committed part today, and will email the list once the resource is complete. - Overall, we've mainly been preparing for the call on Thursday - Thinking about what we would like to have and need in the API - Evaluating how we would map PNetCDF on top of an object API - We're also finishing the aesop paper, which we will submit in a couple of days to NAS 2012. Dries

1 0

Wiki update: User:Reddy
by Wiki update: Asg-private: Carns 12 Mar '12

12 Mar '12

1 0

Wiki update: User:Reddy
by Wiki update: Asg: Carns 12 Mar '12

12 Mar '12

1 0

P2P progress
by Seok Bae Yun 12 Mar '12

12 Mar '12

All, Here is progress update. I wrote a draft document about the security verification methodologies of authentication and authorization model. Dr. Brooks is reviewing it. Maybe that needs to be reviewed with Dr. Brooks back and forth. This week I will add the part of performance verification methodology into the document. For the next meeting, do we need to make a call to this number next week? 505-844-9545 or is it for the Thursday meeting? Best regards, Seok Bae Yun

1 0

Call cancelled today
by Curry, Matthew Leon 12 Mar '12

12 Mar '12

Lee has cancelled the call, as someone on the call was in a noisy environment and would not mute their phone. It is unproductive to have the call when it is difficult to hear and understand one another, so we all have to cooperate and mute our phones when we're not speaking. Sorry to those of you who were being considerate, we'll try again next week. Matthew

1 0

aesop Repository branch, master, updated. a6f7ec75f3cd713ec8c4cc42660a75cd4cc353e1
by noreply＠mcs.anl.gov 12 Mar '12

12 Mar '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via a6f7ec75f3cd713ec8c4cc42660a75cd4cc353e1 (commit) via e779c443ee92503de78c9b9c5cd28805a6d5e1a8 (commit) from c201024b87ec9440827db4cf77b3df1a111f2593 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit a6f7ec75f3cd713ec8c4cc42660a75cd4cc353e1 Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Mon Mar 12 14:25:04 2012 -0500 This add the ability to reinitialize a slot by checking the state is already empty or unitialized. It will protect against a slot that is in the armed state from being messed up if it was directly initialized. This doesn't solve a possible race if two path both reinitialized (as both can be successful) but only one should reach the armed state. commit e779c443ee92503de78c9b9c5cd28805a6d5e1a8 Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Mon Mar 12 14:17:04 2012 -0500 Add minimal SSM configure option and macros to build ----------------------------------------------------------------------- Summary of changes: Makefile.in | 7 +++++++ configure.ac | 11 +++++++++++ resources/resourcebuilder/resourcebuilder.c | 14 ++++++++++++++ resources/resourcebuilder/resourcebuilder.hae | 2 ++ 4 files changed, 34 insertions(+), 0 deletions(-) Diff of changes: diff --git a/Makefile.in b/Makefile.in index 4e7aa24..5246c55 100644 --- a/Makefile.in +++ b/Makefile.in @@ -62,6 +62,7 @@ BUILD_CC = @BUILD_CC@ BUILD_LD = @BUILD_LD@ BUILD_CFLAGS = @BUILD_CFLAGS@ BUILD_LDFLAGS = @BUILD_LDFLAGS@ +BUILD_SSM = @BUILD_SSM@ STRICT_CFLAGS = @STRICT_CFLAGS@ VALGRIND_CLEAN = @VALGRIND_CLEAN@ @@ -71,6 +72,12 @@ ifneq (0,$(VALGRIND_CLEAN)) CPPFLAGS += -DVALGRIND_CLEAN -DVALGRIND endif +ifeq ($(BUILD_SSM),1) + CPPFLAGS += -I@SSMPATH@/common + LDFLAGS += -L@SSMPATH@/lib + LIBS += -lssm_ref -lssmp_mpi +endif + CFLAGS= -I$(srcdir)/src/c-utils -I$(srcdir)/include -I. -I src/c-utils -I$(srcdir) @CFLAGS@ $(CPPFLAGS) # enable relocatable diff --git a/configure.ac b/configure.ac index 530ffa1..d5db947 100644 --- a/configure.ac +++ b/configure.ac @@ -181,6 +181,17 @@ else fi dnl ========================================================================== +dnl ==== SSM ================================================================= +dnl ========================================================================== +AC_ARG_WITH([ssm], + [AS_HELP_STRING([--with-ssm@<:@=DIR@:>@], [location of SSM])], + [BUILD_SSM=1 + SSMPATH="$withval"], + [BUILD_SSM=0]) +AC_SUBST(SSMPATH) +AC_SUBST(BUILD_SSM) + +dnl ========================================================================== dnl ==== Documentation ======================================================= dnl ========================================================================== diff --git a/resources/resourcebuilder/resourcebuilder.c b/resources/resourcebuilder/resourcebuilder.c index 79897b7..5125ed1 100644 --- a/resources/resourcebuilder/resourcebuilder.c +++ b/resources/resourcebuilder/resourcebuilder.c @@ -26,6 +26,20 @@ void rb_slot_initialize (rb_slot_t * slot) #endif } +/* + * returns true if the slot was: + * moved from STATUS_UNINITIALIZED to STATUS_EMPTY. + * already in STATUS_EMPTY. + */ +int rb_slot_reinitialize (rb_slot_t * slot) +{ + const int newstatus = STATUS_EMPTY; + int status; + + status = OPA_cas_int (&slot->status, STATUS_UNINITIALIZED, newstatus); + return ((status == STATUS_UNINITIALIZED) || (status == STATUS_EMPTY)); +} + void rb_slot_clear (rb_slot_t *slot) { OPA_store_int (&slot->status, STATUS_UNINITIALIZED); diff --git a/resources/resourcebuilder/resourcebuilder.hae b/resources/resourcebuilder/resourcebuilder.hae index 7b96bed..4a6bdce 100644 --- a/resources/resourcebuilder/resourcebuilder.hae +++ b/resources/resourcebuilder/resourcebuilder.hae @@ -29,6 +29,8 @@ typedef struct rb_slot_t rb_slot_t; */ void rb_slot_initialize (rb_slot_t * slot); +int rb_slot_reinitialize (rb_slot_t * slot); + /** * Clears the state of the slot and * returns it to the uninitialized state hooks/post-receive -- aesop Repository

1 0

Absence Notice
by Seok Bae Yun 12 Mar '12

12 Mar '12

All, Dr. Brooks can't attend today's teleconference since he is out of town at the GENI conference in Los Angeles this week. Thank you. Best regards, Seok Bae Yun

2 1

Thursday Object API call
by Dries Kimpe 09 Mar '12

09 Mar '12

As there has clearly been some confusion on the mailing list recently, please allow me to place the following discussing in the proper context. What follows is for the external API to our storage system. This is the topic for the concall next Thursday. [This is in contrast with the storage backend API - which has also been discussed on the list, which is the /internal/ access API to persistent storage. In this case, the user of the API is the storage system itself (Sirocco, Triton). That's a whole other discussion.] During the last triton meeting, we were trying to imagine how such an API would look like, trying to satisfy the following goals: - minimal: if something can be implemented on top of it, in some other software layer, without too many disadvantages than it should not be in our API. - Provide enough functionality to enable implementing POSIX, key-value store and other storage models (for example HDF5/NetCDF like API) on top of it. Unfortunately, this also includes models which we don't know yet. This API would be used to implement existing (posix, NetCDF, key-value store, ...) and new storage models on top of it. The first order of business for the call next Thursday will be to define the terminology. There's already been some effort in this area, but it is crucial for a productive discussion that everybody agrees what we are talking about. The next thing will be to define the model. Granted, terminology will at least partly depend on the model. No need for actual function prototypes. No need to discuss completion notification, callbacks, or anything like that at this stage. I would like to try to answer some/all of the following questions during the concall: - Will the model expose a location concept? - Is the user of the API responsible for locating objects? - Does the API pick identifiers (object id) or does the user pick a set, or both? - How would we implement name spaces? (with posix directory as a first test) - Is there a direct group/collection concept supported by the API? - How are we going to deal with consistency? - API offers: no support, some support, per-object, ... - When creating a new object, no matter who picks the location, we know we need some control over location properties: - distance to another object (for a given distance function) or a client. - reliability - speed - health - current load - free space We also need to record a set of use cases which we already know we need to cater for: - POSIX: - implies directories, symlinks, links. - question for later: does that include posix consistency, delete-while-open behaviour? - MPI-IO (different consistency model than POSIX) - Key value store - Striping - For reasons of performance and/or reliability - Higher level library: damsel/pnetcdf/... As a reminder: Time of the call: 2pm central Conference Access Number: 760-984-1000 Conference Passcode: 692481 # Below is a summary of the concepts we explored during the meeting. This is not a proposal, simply a list of the ideas that came up during the discussion. Terminology: - fork: byte stream - object: collection of forks - location: group of objects - location has properties (load, speed, reliability, free space, ...) * Forks or attributes? -> Is there a need for two concepts? (cfr. implementing attributes using fork + TRUNCATE_ON_WRITE) -> The storage system could, with or without some annotation help, optimize certain forks. (For example, store in a database if size < x bytes, maintain a local index for a certain fork) * Do we expose layout? A 'find' or 'search' operation would then return a handle which contains both a location and a persistent object identifier (where the latter stays if the object is replicated/moved/whatever operation we support). A location search function which returns a set of locations matching a given criteria is useful in this case. * Does the storage system help us find a set of free object ids or does the API user pick an id? -> Can be both. -> If the API does pick one *and* object-id implies location, we'll need a way to specify criteria (speed,distance,free space,reliability). It is possible that no matching locations can be found. -> How to handle collision? -> if the system returns set of IDs, precreate might help -> Alternative: on collision, try searching again. * HyperDex: certain forks of an object determine location (compared to the traditional approach of having no or a single property of the object (for example object id) determine location). -> Benefit: efficient search for all forks used to determine location * Implementing name spaces: * Option: construct name space on demand by queryable forks -> Benefit: generic functionality, can be used in other scenarios -> Indexing of forks: -> Can this be done by a middleware library? -> If so, how do we manage synchronization -> Store conditional or other synchronization primitives -> we want to avoid locks -> Does the location provide a local index? -> What kind of queries are required? -> Is equivalence sufficient? Interesting paper: An OSD-based Approach to Managing Directory Operations in Parallel File Systems; Ali et al. -> They add atomic operations to T10 attributes (compare and swap) -> They store directory entries in T10 attributes by hashing the file name to a attribute(fork) number. Problem: name collision. -> They build a lock on top of the atomics and implement a hash chaining to resolve the collision. Clearly, the lock is a hack. Maybe extending the atomics to support list operations to avoid the collision problem? (cfr. our `Interfaces for coordinated access in the file system` paper) Dries

1 0

back-end local storage API
by Phil Carns 09 Mar '12

09 Mar '12

Hi all, I've been thinking about the back-end, local storage API used by Sirocco and Triton a little bit to see what we might be able to do to converge the two. Maybe we can bounce around some ideas on the mailing list? To clarify, there are two object storage API's layers that the ASG needs to sort out in general: 1) the back-end local storage API: this is the layer between a storage server and its local storage devices. It is an internal API. No applications, users, or system libraries will ever see this API. It's purpose in life is to provide fast access to storage device and present some building blocks to make a server's job easier. We aren't really bound by any external expectations. 2) the system-level storage API: this is the client-side fundamental object interface. It is the API that you use to interact with the storage service as a whole. Adventurous applications could use it directly or we can build things like POSIX interfaces or high level libraries atop it. We will be talking about this interface on Thursday of next week. In this email thread I'd like to bring up 1) for discussion. Comparing the Sirocco and Triton implementations, both do object-based storage, track some sort of version information with byte granularity, and automatically resolve conflicts between overlapping extents with different versions. There are some differences though. Here are some that aren't too hard to spot: 1) idempotency (is there a "create" operation or not) 2) how metadata is associated with an extent (version/transaction/update ids and more...) 3) how to find objects, or find extents within objects (probe/query/enumerate/etc.) 4) object identifiers (object and fork IDs vs. segment and container IDs) 5) are transactions exposed at this level (to group multiple write/stores as an atomic unit) I'm going to skip issue 1). To me that seems easy to resolve. The Triton/TOSD interface could provide wrappers that implicitly create on write (continuing with the write if the create returns 0 or EEXIST). Likewise I imagine that the Sirocco local storage API could split its write into distinct create and write steps (presumably it has a create step under the covers for the first write). So either team could support both models without much work. Delete/purge would work similarly- just ignore the error code if you try to purge something that doesn't exist. Idempotency gets a little more interesting in terms of the implications at the storage service layer. It doesn't seem like a big deal to local storage. So on to 2), which might be a more interesting R&D topic. Right now we put update ID numbers on each extent that we write, and track that ID persistently. That's just one example of things that one might want to lump onto extent metadata though, some of which have come up recently in feedback from Steve and the DOD. Here are some candidates: a) update ID (this is currently called a version number in Sirocco and a transaction number in Triton, but they are the same) b) checksum information (either for end-to-end checksumming, or for silent error scrubbing, or both) c) byte-granular provenance (for a DOD use cases where you want to be able to say where a specific block of data originated) d) a request identifier (Triton uses this to protect non-idempotent operations; duplicate requests will have the same identifier and we track it in the object to see if we already serviced the request or not) e) an identifier for atomic operations (like if you wanted to do a compare and swap on some data- something opaque that you can use to tell if a region has changed since you last looked at it. Maybe I'm must overloading a) here) So in Triton we have done a) and d) already, but d) is somewhat grafted on (both in terms of API and underlying DB organization), and from a software engineering perspective I'm not really looking forward to grafting on b) and c) as well, even though from a technical point of view we have a great vehicle to provide that functionality. Is there a way to generalize how we express extent-based metadata so that we can support all of a) through e) (or other things we haven't thought of) without breaking the local store each time we make those changes? This might be an interesting research contribution as well. If you have a generalized framework for efficiently tracking extent-based metadata, then you can do a study/model to show how much it costs each time you add incrementally more attributes, and we have some highly relevant use cases on the table already to demonstrate the value of the functionality. To my knowledge no storage APIs are doing this in a general way right now. Any thoughts on that idea? I'm going to pretend like this topic is interesting and barrel ahead with API thoughts while I'm typing :) We would want some assumptions to narrow down the model. For the purposes of discussion temporarily I'm going to refer to each type of metadata that you might tack onto an extent as an "attribute". I'm not sure that's a good terminology here, but I need a name. Here are some straw man assumptions: 1) there may be multiple attributes, but each one is a fixed size. So for example, you can say you want to track version information on some operations, but if so, that version data better be the same size on every operation. The provenance attribute may be larger, but it is likewise the same size as all other provenance attribute entries. 2) the user of this API has to register the kinds of attributes/metadata it wants to track before using them so that the storage layer can plan ahead on its database organization. You can't wait until write time to make up a new attribute and included it in the write. 3) attributes are optional on each write. You can register a checksum attribute, for example, but still make write operations for which that attribute is NULL (it would still be up to the implementation to decide if it wants/needs to allocate space for the attribute anyway- this is mainly for the caller's benefit to say "I don't care" whether the storage takes advantage of that or not) 4) you can apply attributes to any operation that permutes the storage, not just writes. So if we have a create or delete operation, then those support these same attributes. So it isn't really just for extents, although those are the most interesting use case. 5) just as there is a general way to set attributes on extents at write (or permutation) time, there need to be some hooks to get it back out in useful ways. You might want to read an arbitrary extent and get a list of its attributes along the way, you might want to find extents matching a certain attribute, you might want to find the size/bounds of an extent that encompasses a particular byte with an attribute, etc. So maybe there is a function to register a new attribute type: my_attr_id = storage_extent_attribute_register("foo", 16); - foo is a descriptive name for the attribute - 16 is the fixed size in bytes of its value - my_attr_id is just some opaque integer that the caller can use to reference that type of attribute in future operations without passing around a string And then a write (or other permutation) function has extra arguments that let the caller pass in multiple attributes: ret = write(...normal write arguments..., int attr_count, attr_id* attr_array, void** attr_value_array); - attr_count is the number of attributes - attr_array is a list of attr_ids from the register step - attr_value_array is the value of each attr being passed in There are other ways to do the above depending on how much work you want the caller to do. I'm not picky there, as long as you can pass multiple attributes at once. I'll also note that the version attribute is "magic" in some sense. That one is not just opaque data- the storage can (and should) actually poke at that one to make decisions about how to organize conflicting writes and so forth. That doesn't mean it can't share an API with the more general opaque attributes, though, I don't think. Retrieving querying the attributes is a little tougher, so I'll stop here and request some comments before trying to sketch that part out. Any thoughts? Just to reinforce again, I'm not advocating for this functionality in the system interface. That is probably a bridge too far. I think this might be useful internally to the system as a building block, though. -Phil

4 22

notes on ssi interface discussion
by Klundt, Ruth 08 Mar '12

08 Mar '12

Here's some notes recording what we discussed today regarding an storage system interface. shred as needed... Ruth storage system internal attrs: 1. sliders - persistence settings, number/type of replicas - raid stripe etc. a. initial value b. final value 2. security attr: to be used in authorization once set the object is 'claimed' by the client 3. object type: data store, key/val store type change causes reset on data attr definition: key/val pairs keys specified by a 64 bit mask values variable length but known, fixed for each attr each attr will have update id attr api: attrset( mask, iovec ) mask specifies attrs requested, iovec provides space for requested values attrget( mask, iovec ) mask is value return, iovec provides enough space for all of them user attrs: system will support extended user attrs via a key/val store object. user attr api: will use key/val store api data store api: ssi_op(ssi_cmd_t for control stuff, iovec for data) ops defined as commands in a union all async, return handle ssi_poll(handle, time ) block on null, zero time try again, non-zero time check timeout ssi_cancel(handle) attempts to cancel the op, not guaranteed to work. operation: probe( cid, oid, addr range, iovec, next cookie ) unset cid is illegal unset oid returns list of oids in given cid unset addr range returns list of frags for given cid/oid cid/oid/addr_range returns list of frags within the addr range operation: store( cid, oid, update_id, iovecs, offset ) operation: retrieve( cid, oid, iovecs, count, offset ) operation: punch( cid, oid, update_id, size, offset ) operation: uset( cid, oid, update_id, size, offset ) sets the update id to a new value operation: reset( cid, oid ) bumps the epoch number internally not user available all associated data is gone reconciliation uses epoch numbers key/val store api: key is 64 bits unsigned probe: ( cid, oid, key range, iovecs, next cookie ) store: ( cid, oid, key, iovec ) retrieve: ( cid, oid, key, iovec ) punch - like store uset - like store

4 3