August 2012 - Asg - mailman3.cels.anl.gov

Cengiz Karakoyunlu - Namespace work
by Dries Kimpe 05 Sep '12

05 Sep '12

Hi, attached is the work from my summer intern, Cengiz Karakoyunlu. He looked into how namespace operations can be mapped on the current ASG storage model. Please let us know if this can be released publicly. Thanks, Dries

3 2

will miss call today
by Rob Ross 27 Aug '12

27 Aug '12

I have another meeting that I shouldn't miss. -- Rob

3 3

Fwd: Fw: Request for public release
by Stephen.W.Poole 26 Aug '12

26 Aug '12

Sent to the wrong email. I reviewed all of them. These must not be funded under the ORNL/DoD program. Best Steve... -------- Original Message -------- Return-Path: spoole(a)ornl.gov Received: from linzimmt02o.imo.intelink.gov (LHLO mta02.ugov.gov) (10.2.188.64) by ww3.ugov.gov with LMTP; Fri, 24 Aug 2012 22:34:37 +0000 (GMT+00:00) Received: from iron-ugovb-out.osis.gov (mailout-ugov.osis.gov [172.16.5.12]) by mta02.ugov.gov (Postfix) with ESMTP id 7A80150D28B for <Stephen.w.Poole(a)ugov.gov>; Fri, 24 Aug 2012 22:34:35 +0000 (GMT) X-IronPort-AV: E=Sophos;i="4.80,307,1344225600"; d="pdf'?scan'208,223";a="434601" Received: from ghost-b.center.osis.gov (HELO fw1-b.osis.gov) ([172.16.6.242]) by iron-ugovb-in.osis.gov with ESMTP; 24 Aug 2012 18:31:25 -0400 Received: from mta01.ornl.gov (mta01.ornl.gov [128.219.14.70]) by fw1-b.osis.gov with ESMTP id q7OMXYr2016415 for <Stephen.w.Poole(a)ugov.gov>; Fri, 24 Aug 2012 18:33:34 -0400 (EDT) X-SG: RELAYLIST X-IronPort-AV: E=Sophos;i="4.80,307,1344225600"; d="pdf'?scan'208,223";a="42773292" Received: from ironport-5600-src.ens.ornl.gov (HELO exchhub2.ornl.gov) ([10.1.86.80]) by iron1.ornl.gov with ESMTP/TLS/RC4-MD5; 24 Aug 2012 18:34:14 -0400 Received: from EXCHMBB.ornl.gov ([192.168.3.201]) by exchhub2.ornl.gov ([160.91.2.112]) with mapi; Fri, 24 Aug 2012 18:34:12 -0400 From: Poole, Stephen W. <spoole(a)ornl.gov> To: 'Stephen.w.Poole(a)ugov.gov' <Stephen.w.Poole(a)ugov.gov> Date: Fri, 24 Aug 2012 18:34:11 -0400 Subject: Fw: Request for public release Thread-Topic: Request for public release Thread-Index: Ac2BdvLZF3gD4hyYQhas1DuSIa89ugA0aItS Message-ID: <A3308EF44527A547957CEBFAB3AACE2E96A4205402(a)EXCHMBB.ornl.gov> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/mixed; boundary="_004_A3308EF44527A547957CEBFAB3AACE2E96A4205402EXCHMBBornlgo_" MIME-Version: 1.0 >From my ORNL Crack Berry. ----- Original Message ----- From: Dries Kimpe [mailto:[email protected]] Sent: Thursday, August 23, 2012 05:31 PM To: Poole, Stephen W. Cc: asg(a)mcs.anl.gov <asg(a)mcs.anl.gov>; Judicael ZOUNMEVO <zounmevoj(a)mcs.anl.gov> Subject: Request for public release Attached are 3 presentations which we would like to make public. - judicael-ssm-talk.pdf & judicael-ssm-uab-talk.pdf Presentation given (internally) at Argonne today and at UAB (earlier this week) by Judicael as a summary of the work he performed this summer, regarding his efforts to port Triton over to the SSM networking layer. - judicael-ssm-uab-discussion.pdf An informal collection of slides used to illustrate the discussion Judicael had with the SSM team to get feedback regarding the use of SSM in Triton. We are sharing this as it might contain helpful information for the Sirocco team. Please let us know if there are any objections to sharing this outside of the group. Thanks, Dries -- ======================> Steve Poole Computer Science and Mathematics Division Chief Scientist / Director of Special Programs Director Extreme Scale Systems Center Oak Ridge National Laboratory 865.574.9008 "Wisdom is not a product of schooling, but of the lifelong attempt to acquire it" Albert Einstein =====================> =====================> Stephen W. Poole Senior Technical Director High Performance Computing - SPO Department of Defense (443) 654-9735 =====================> This document contains information exempt from mandatory disclosure under the FOIA. Exemption 3 applies. The information contained herein that is marked U//FOUO is for the exclusive use of the DoD, other U.S. government and U.S. contractor personnel with a need-to-know for this program's information. Such information is specifically prohibited from posting on unrestricted bulletin boards or other unlimited access applications, and to an e-mail alias.

2 1

Request for public release
by Dries Kimpe 23 Aug '12

23 Aug '12

Attached are 3 presentations which we would like to make public. - judicael-ssm-talk.pdf & judicael-ssm-uab-talk.pdf Presentation given (internally) at Argonne today and at UAB (earlier this week) by Judicael as a summary of the work he performed this summer, regarding his efforts to port Triton over to the SSM networking layer. - judicael-ssm-uab-discussion.pdf An informal collection of slides used to illustrate the discussion Judicael had with the SSM team to get feedback regarding the use of SSM in Triton. We are sharing this as it might contain helpful information for the Sirocco team. Please let us know if there are any objections to sharing this outside of the group. Thanks, Dries

1 0

SSM unlink semantics change
by Matthew Farmer 23 Aug '12

23 Aug '12

All, Per the call on Aug 13 the semantics for unlinking in SSM have changed. This change is now live in the beta branch of our repo, so anyone working directly from the beta branch will see this new behavior. I have been coordinating with Judicael at Argonne to minimize impact on his work; I am not aware of anyone else working from directly from the branch (which is discouraged outside of really bleeding-edge stuff). In summary, the change has the following effects: - New return codes SSM_REMOVE_OK, SSM_REMOVE_BUSY, SSM_REMOVE_INVALID - ssm_unlink returns immediately either indicating that the match entry was successfully unlinked (returns SSM_REMOVE_OK) or that it could not be unlinked (SSM_REMOVE_INVALID or SSM_REMOVE_BUSY). No events are generated and no deferred unlinks will take place. - ssm_drop has been added; it takes an ssm_mr and removes it from the buffer list of a match entry. If successful, it returns SSM_REMOVE_OK and the MR is unposted. Otherwise, it returns SSM_REMOVE_INVALID. - The SSM_LINK_AUTO_UNLINK flag has been added. If a match entry matches its last buffer and has this flag set, it unlinks and an unlink event is generated indicating the automatic unlinking. The documentation on the ASG private wiki will be updated to reflect this when the drop for beta 3 is pushed to the list, following implementation of two minor feature requests pending. Hopefully from there we can move to RC, get documented, and release 0.6. I'll have a roadmap for that once beta 3 is out. If you're using the beta 2 drop this change doesn't affect you, yet and you may consider this fair warning :) Shane UAB

1 0

Papers and presentations
by Ward, Lee 23 Aug '12

23 Aug '12

All, I believe I need to clarify our procedure for public presentation of materials we develop relative to this project. Any material you wish to make public must be submitted to Poole for review and approval, first. This can't be done concurrently with a submission to a workshop, conference, or journal for instance. It must be done *prior* to any kind of release. You must get approval in writing before proceeding to release. Everyone on this project is subject to this, whether your funding is from ORNL or Sandia makes no difference. If it's about this project, it needs to be reviewed and approved before you share. Pretty please? --Lee

3 3

Re: [ASG] [EXTERNAL] Re: source code repositories
by Ward, Lee 22 Aug '12

22 Aug '12

Dries, let's just let it drop? Thank you for creating the repositories. We can just let people integrate as schedules permit? --Lee -----Original Message----- From: Dries Kimpe [dkimpe(a)mcs.anl.gov<mailto:[email protected]>] Sent: Wednesday, August 22, 2012 01:12 PM Mountain Standard Time To: Anthony Skjellum Cc: asg(a)mcs.anl.gov Subject: [EXTERNAL] Re: [ASG] source code repositories * Anthony Skjellum <tony(a)cis.uab.edu> [2012-08-22 14:06:40]: > We are not ready to proceed : we do object ..sorry Sure, no problem. Would you mind sharing your objections? This was discussed in January at Clemson and also posted on the mailing list a week ago, without objections/response. I don't have strong feelings about this; However, it was discussed on the Monday call two weeks ago and I'm just proceeding as discussed there. Thanks, Dries

2 1

source code repositories
by Dries Kimpe 22 Aug '12

22 Aug '12

As mentioned on the concall last week (and the week before), I'd like to start collecting all repositories at Argonne (as discussed on during the kick-off meeting). Argonne currently has a (helpdesk supported) git hosting solution at git.mcs.anl.gov (accessible from the outside world) without account restrictions (anybody who has a public key can be added to the system - no need to fill out forms). More importantly, once a key is added to the system, there is no need to contact the helpdesk to grant/revoke access to repositories or to create/remove repositories. At Argonne, we've been using the system for all triton/aesop related repositories. Instead of going with one large ASG repository, we decided to go with several smaller repositories. There are a number of reasons for this: 1) Creating/removing repositories is easy and can be done by us directly 2) Git makes it easy to split or combine repositories while preserving history. 3) Git (like subversion) has a submodule concept which allows linking of repositories. When checking out the repository, any linked repositories will also be checked out automatically. Currently, we have the following repositories: asg/asg - holds documents, concall notes asg/aesop - Aesop source-to-source translator (*) asg/triton - Triton-specific source code (*) asg/c-utils - Shared (triton/sirocco/aesop) C source code asg/shared - Shared (triton/sirocco) aesop source code This includes aesop resources used by both sirocco and triton. This would eventually also include aesop code for components we share (such as localstore). asg/reference - Local library implementation of ASG system interface. asg/client - Programs/tools using the ASG system interface. Example: posix client, namespace support library, ... (*): These exist on git.mcs.anl.gov but not yet under the asg/ prefix. (they will be moved at some later point. Except changing the URL in git, this rename/move should not affect things) Using submodules, the following can be done: asg/triton --> asg/aesop --> asg/c-utils --> asg/shared --> asg/c-utils asg/client -- asg/reference -- asg/triton (so the depending on configure, the client code can build against triton's implementation of the ASG System API or against the simple reference implementation) When checking out asg/triton, all of the others are checkout out transparently. Each link points to a specific revision in the destination repository, so that any changes made to asg/aesop will not be visible in the asg/triton checkout until we manually update the link in asg/triton to point to the newer revision in asg/aesop. This is actually a good thing, since a chance made to asg/aesop will not suddenly break asg/triton or other repositories depending asg/aesop. Here is what I propose: * creating asg/ssm * creating asg/sirocco For asg/sirocco, using submodules, the following could be a possible layout: asg/sirocco --> asg/aesop --> asg/shared --> asg/ssm We can have as many repositories as we want, and creating/removing one (or granting access) is fast and easy (send me an email), so further modularisation (for example within sirocco) is possible. Comments/feedback? When moving over from subversion to git, it is possible to preserve history (if so desired). If somebody emails me to let me know which parts of the Nebula repository need to be moved over, I'd be happy to do the migration. I've tried to update the Nebula repository, but it seems it is broken at this point (I'm getting errors about missing externals: URL 'https://svn.cis.uab.edu/hpcl/Nebula/SSI/libidb/tags/libidb-2' at revision 1270 doesn't exist). Dries

4 7

non-durable IDB and Berkeley DB experiments
by Phil Carns 22 Aug '12

22 Aug '12

I've spent a little bit of time looking at the IDB code to understand better how it relates to the TOSD code from Triton. For some context, my main motivation at the local storage level right now is to figure out if there is a reason to pursue modifying the TOSD to adopt a common API with the Sirocco localstore, or if that time would be better spent just adopting the Sirocco localstore. It isn't clear yet if the two approaches will be good at different things or not. I think that we might have to delay that overall comparison a little longer, but in the mean time we can at least look at the IDB component by itself. The IDB is the piece of the Sirocco code that tracks metadata about version numbers and extent ranges (it does not store file data). My first thought was to compare the IDB to the TOSD by stripping out parts of the TOSD that handle things like log structure, data movement, etc., but that ended up being a tall order. The TOSD does not clearly modularize its metadata activity in a way that looks quite like what the IDB is doing. What I ended up doing instead was hacking up a variant of the IDB code that uses Berkeley DB (BDB) under the covers to store extents. I didn't modify the IDB API or semantics. I also didn't try to implement the whole thing. I just did enough to insert extents with the same semantics as the IDB, and enough of the query (aka "stab") functionality to do some simple correctness tests. BDB by default makes all transactions durable (full ACID semantics), but I set a flag to disable durability (but maintaining the other ACI semantics) for the purposes of fair comparison against the IDB, which doesn't yet provide full durability. We can turn the durability back on later if we want to similar comparisons again when IDB journaling is available. Aside from that flag I did not set any tuning parameters for BDB. It is operating in its out-of-the box mode as far as its other tunable settings go. The BDB version puts data for an object in a separate DB like the original IDB code does. You could still have transactions span objects with this approach, as long as all DBs are placed in the same Berkeley DB environment so that they can share transaction logs. I didn't look at that behavior. I've attached two tar balls with the source code that I used for comparison if anyone wants to have a look. libidb-mod.tgz is the same libidb code that was posted to the list earlier, but with the following changes: - added simple instrumentation to the "idb" test program to print elapsed time and insertion rate - added an extra argument to the idb test program to toggle the type of workload that it generates (more on that below) - added an idb-ver test program to look at insertion semantics The libidb-mod.tgz code has the same thing, but implemented with berkeley DB as the back end. There is also one extra test program called idb-threaded which is just a multi-threaded variant of the existing idb test program. The multi-threaded version is very simple, it just divides the workload up into N even sized pieces and measures the time of the slowest thread. There is no load balancing or anything like that. Regarding workloads, the original idb test program inserts completely randomized extents, meaning that they often overlap. This forces the DB to frequently delete and/or adjust conflicting extents. This is definitely an interesting workload, but we were also interested in the case where extents _do not_ overlap, which is the more likely scenario from a traditional HPC app. So we added two additional workloads: one in which a collection of non-overlapping extents are inserted in random order (what one might expect from multiple applications processes concurrently writing different portions of a data set) and one in which a collection of non-overlapping extents are inserted in linear order (what one might expect from a single application process writing all data in order). Here are some quick results below if anyone is curious. These experiments were run on my laptop, which has 4 cores and 6 GiB of RAM. I inserted 10 million extents to a single object in all tests. Before each test I deleted all data files (so each test started with a non-existent object) and flushed the buffer cache on my computer (using echo 3 > /proc/sys/vm/drop_caches as root). 10 million extents isn't nearly enough to fill the buffer cache on my laptop, but it does have an Intel SSD if anyone is curious. I don't think the disk played any role in these experiments. The file system might have, though. It is an ext4 volume with the noatime mount option. I looked at three cases for each workload: the stock IDB code ("IDB"), the Berkeley DB variant ("IDB-BDB"), and the Berkeley DB variant with 4 threads making concurrent updates ("IDB-BDB(4-threads)"). Here are some results, showing the insertion rate in ops/s for each case: IDB IDB-BDB IDB-BDB(4 threads) random, overlapping: 90,847 65,876 64,057 random, non-overlapping: 32,364 74,725 93,205 linear, non-overlapping: 301,619 229,303 154,129 *** *** the threaded, linear workload is kind of nonsensical, but I included it for completeness. Each thread is servicing a different portion of the workload in linear order. Here is the same table, but showing the resulting DB size in MiB for each case: IDB IDB-BDB IDB-BDB(4 threads) random, overlapping: 17 74 74 random, non-overlapping: 1228 615 616 linear, non-overlapping: 1228 432 752 I suspect that the threaded berkeley DB results will be more interesting when we look at durable semantics, simply because adding some disk activity will mean that threads will have some other work to occupy themselves with besides contending over an in-memory data structure. In the TOSD we also added an optimization to coalesce disk updates so that the disk isn't thrashed as much in that case, but I didn't try to implement that in this code. At any rate, I think the short story from these preliminary results is that the IDB and BDB appear to be good at different kinds of non-durable workloads. BDB has a big edge in random, non-overlapping updates, while IDB has the edge in overlapping and linear cases. It will be interesting to revisit this with durability and/or within the context of storing actual object data. -Phil

1 1

Wiki update: User:Afilby
by Wiki update: Asg-private: Carns 21 Aug '12

21 Aug '12

1 0