May 2014 - Asg - mailman3.cels.anl.gov

Weekly report
by guoxinl＠clemson.edu 19 May '14

19 May '14

Dear all, In the last week, we were working on implementing a simulator to simulate a storage datacenter including racks and servers, where our hierarchical metadata storage will work on. Best Regards! Guoxin Liu PHD Candidate ECE, Clemson University

1 0

20140519 - Sandia Status
by Danielson, Geoffrey Charles 19 May '14

19 May '14

All, Last week, Sandia hosted our UAB colleagues to discuss the future of SSM, contract details, and thesis proposals. We also continued in our work on localstore and mercury. Regards, Geoff Danielson Geoffrey.Danielson(a)hp.com gcdanie(a)sandia.gov Cell: (505) 263-0751

1 0

weekly update
by jingxuan 12 May '14

12 May '14

Hi, everyone, last two weeks, we were working on integrating and testing our prototype. We have finished manipulating multiple clients and servers and setting up a daemon as both client and server at the same time using mercury with SSM. Right now we have two issues. First,we tried to set up virtual machines as our first step test environments, but right now, the code which works in the host can not run in VM due to some popen allocate memories issues. The second is that the communication between two hosts using mercury SSM are very slow( more than 1 minutes). we will try to figure out them and keep on modifying our prototype this week. Also, we will begin to write our next deliverable due the end of MAY. Jingxuan SUN Holcombe Department of Electrical and Computer Engineering Clemson University Tel:864-643-8210 Email: jingxus(a)g.clemson.edu

1 0

Weekly report
by guoxinl＠clemson.edu 12 May '14

12 May '14

Dear all, In the last week, we were working on the design of the metadata storage prototype, defining all functions needed to support data anywhere storage and movement. Best Regards! Guoxin Liu PHD Candidate ECE, Clemson University

1 0

TAMU Presentation Material
by Gim Jongmin 12 May '14

12 May '14

The attached file is updated a graph. Sincerely, Jongmin -- Jongmin Gim Post-Doctoral Research Associate Dept. of Elecrical and Computer Enginerring Texas A&M University 333D Wisenbaker College Station, TX 77843-3259 Phone: 979-587-8311

1 0

TAMU Status Update and Presentation Material
by Gim Jongmin 12 May '14

12 May '14

Dear All, Last week, we implemented a prototype of newly designed directory cache and prepared the talk on today. The attached one is a presentation material for the talk. Sincerely, Jongmin -- Jongmin Gim Post-Doctoral Research Associate Dept. of Elecrical and Computer Enginerring Texas A&M University 333D Wisenbaker College Station, TX 77843-3259 Phone: 979-587-8311

1 0

20140512 - Sandia Status
by Danielson, Geoffrey Charles 12 May '14

12 May '14

All, Last week, Sandia worked up a schedule for continued development on Sirocco, as well as experimenting with latency measures on AESOP BDB and bug reporting on Mercury. Regards, Geoff Danielson Geoffrey.Danielson(a)hp.com gcdanie(a)sandia.gov Cell: (505) 263-0751

1 0

Re: [ASG] Asking a delay of one week for a presentation
by Phil Carns 09 May '14

09 May '14

Thanks Jongmin, that works for me. Let's cancel today's meeting for sure. As far as when to hold the next one, how about we just have folks answer if the 12th *doesn't* work for some reason? If we don't hear any objections by the end of the day then let's plan on meeting on the 12th. thanks, -Phil On 05/05/2014 09:24 AM, Gim Jongmin wrote: > Dear all, > > Last three weeks, we have been trying to improve the performance of > metadata operation, and we only remain a debugging work now. I would > like to handle this issue on our presentation so that I was wondering > if we could have a telcon 1 week later (12th May). > > I look forward to hearing from all of you. > > Sincerely, > Jongmin > > -- > Jongmin Gim > Post-Doctoral Research Associate > Dept. of Elecrical and Computer Enginerring > Texas A&M University > 333D Wisenbaker > College Station, TX 77843-3259 > Phone: 979-587-8311 >

3 2

Updated ASG API (was Re: Client API discussion)
by Phil Carns 08 May '14

08 May '14

The git repo for the ASG API reference implementation ([email protected]:asg/reference) has a new branch available called "api-1.1.0" that includes the updates below. In addition to the two changes listed earlier (version->update_id, new arguments to asg_read()), I also added ASG_VERSION_{MAJOR/MINOR/SUB} defines so that moving forward we'll have version numbers to make it easier tell exactly what revision of the API is expected or being provided by someone's implementation. I did not actually *implement* the asg_read changes; it is just an API update. In the reference implementation the two new arguments record_buf and record_buf_transferred should be set to NULL when calling asg_read. The only change that I made to the implementation was to add an assertion to make sure that the former is NULL so that no one accidentally tries to use that functionality in the reference implementation before it is available. The existing "make check" tests still pass as they did before. This is in a branch for now just to make sure it doesn't disrupt anyone at an inconvenient time, but if everyone is comfortable with the update then we should merge it back to master soon. thanks, -Phil On 03/19/2014 09:42 AM, Carns, Philip H. wrote: > On 03/18/2014 05:29 PM, Curry, Matthew Leon wrote: >> * Phil - update asg.h to reflect API changes >> > Proposed updated header file is attached, based on the lastest copy in > asg/reference/include/asg.h. > > There were only two concrete changes from the meeting: rename "version" > to "update_id", and add probe parameters to asg_read(). > > The former one was just a search and replace, and the latter one meant > adding two new arguments to asg_read(): > > asg_record_info_t * record_buf, > asg_size_t * record_buf_transferred, > > ... which is a pointer to the record info to be filled in by the call > (can be NULL if you don't want the record information) and an output > argument telling you how many records were filled in, if any. > > I'm curious about a few things after looking at this: > > 1) Does the asg_size_t* argument of asg_probe_fork() (which is the > convention that I used for the record_buf_transferred argument above) > represent the number of record_info_t structs that were returned, or the > volume of memory that they take up? > > 2) Should we pepper this api with doxygen or something to help document > it? I'm not sure to what degree things like "this argument can be null > if you don't care" or "this is a count, not a memory buffer size" type > of stuff is supposed to be folded into a separate API document or put in > comments here. > > 3) Should we have some mechanism to indicate what revision of the API > the header refers to? I have a feeling that we aren't done tweaking > it. I don't know how headers usually handle this (I guess a #define for > now? We don't have a vast collection of external applications that we > need to maintain backwards compatibility with or anything, we just need > some safety regarding what API someones code is using) but it might be a > good thing to go ahead and throw in. Whenever we bump the revision we > can also add a comment explaining what changed from the last one. > > thanks, > -Phil

1 0

Re: [ASG] [EXTERNAL] Re: Aesop questions
by Phil Carns 06 May '14

06 May '14

On 05/06/2014 07:15 PM, Kimpe, Dries Leon Tom wrote: > * Curry, Matthew Leon <mlcurry(a)sandia.gov> [2014-05-06 22:56:03]: > > [...] > >>>> 2. We discussed spawning extra threads in the server so that we can >>>> have real parallelism while also using Aesopï¿½s concurrency model. I >>>> seem to remember that this is possible, but we never discussed in >>>> detail. How would we accomplish this? >>> You can use the aethread resource, which causes the statements following >>> the call to the resource to be executed on a dedicated thread. >>> That thread will be able to call blocking calls without blocking aesop >>> threads. >> aethreads work until a blocking call into a resource is executed, at which >> point control goes back into the aesop scheduler, correct? What if we want >> to have, say, one aesop-aware thread per core alive through the whole >> server execution? > That's very difficult to do, since aesop tends to take threads and keep > them for a while. In other words, if any of your other threads call a > blocking function that thread might not be released any time soon, and > some other thread might continue executing code following the blocking > call. > > There might be something that can be done about this, but it could be > complicated. Depending on the specifics of the use case (what exactly you want the threads to do) it might be easier to just spawn threads like you normally would. When it comes down to it you can do this in .ae code like you would in .c code for the most part. In triton, for example, we keep a some dedicated threads around throughout execution to drive progress on Mercury at all times. In that case that was the easiest route to get the behavior that we wanted. I might be getting off in the weeds at this point, but we can share that chunk of code (the part that drives Mercury) if you guys are interested. It has several triton-isms in it (types, error conventions, etc.) but it might be a helpful example or we consider cleaning it up into a more general-purpose shared component. -Phil >>> However, if you simply want more aesop concurrency, then you would use a >>> lonely pbranch. >>> I.e. a pbranch without an enclosing pwait. >> I assume that a lonely pbranch is still scheduled serially onto one core >> with all the other pbranches. > Correct, though in most cases it won't be serially. Any available thread > in aesop (from resources, or aethread calls) can be used to execute any > runnable code. So in most instances, there are usually a number of threads > busy executing pbranches. > > Dries

1 0