February 2012 - Asg - mailman3.cels.anl.gov

Re: [ASG] [EXTERNAL] Re: [SSM 0.6 preview r931] SSM branch preview release (see inside)
by Dries Kimpe 20 Feb '12

20 Feb '12

The bounce mail talked about a message being rejected as it was over the 30MB limit; I saw that the attachments were pretty small but I wasn't sure if one if the attachments had been stripped or not. Better safe than sorry! "Curry, Matthew Leon" <mlcurry(a)sandia.gov> wrote: Yeah, we had that issue before, too. I think I've gotten HTML mails that were bigger than Shane's attachments, though. :) Matthew On Feb 20, 2012, at 9:13 AM, Dries Kimpe wrote: > Just a note: I got some messages from the mailing list software that the > clemson server was refusing emails because of large attachments. > It's probably best not to send large attachments to the mailing list... > Dries > * Matthew Farmer <shanefarmer22(a)gmail.com> [2012-02-20 10:06:06]: >> All, >> I am running a very high fever and am basically out of commission. This >> has somewhat hampered my development efforts. >> I am almost done with getting libssm+ssmtests+transports cleaned up, >> tested, and tagged, but in my present state I just can't do it, and frankly >> I need to be horizontal and unconscious. So in the interest of >> productivity, I'm attaching the 90% version of libssm itself and the >> libssmp_localmem transport. (TCP and MPI are pending. InfiniBand is * >> really* pending.) The API is now stable and the general order of >> operations is settled. After testing and documentation this will become >> the 0.6 tag release. >> This code release (and the drop it's leading up to) includes all discussed >> features to date: >> - "normal" iovec memory semantics (no more custom iovecs or "chains" -- >> multiple buffers can be added directly to match entries) >> - much stricter memory management (no layer violations) and the >> associated scaffolding >> - get/put offests (both read and write) >> - local-side-only scatter-gathers (SSM_ME_VECTOR, implies that when >> initiating a get or a put all buffers instead of the first should be used >> for initiator only) >> - the buffer matching Lee discussed (i.e. size only matched against the >> first buffer, and special event results SSM_RESULT_NO_BUFFERS) >> - the never-unlink behavior (currently the default -- more on this later) >> - everything needed for ssm_cancel to function (although it doesn't >> technically function yet) >> - better naming and more sensible namespaces >> - fairly extensive refactoring that cut down on LOC and removed a lot of >> indirection (no more formal modules, just some custom SSM data types) >> - support for non-concurrent (i.e. non-threaded) SSM that only makes >> progress on ssm_wait (this is actually the *only* option right now, but the >> core is written to make adding some other options e.g. pthreads trivial). >> For interested parties, the public interface is in common/ssm.h, which in >> turn includes the stuff from common/ssm/main. Those files in main are the >> top-level API functions that you use with SSM. Comments and documentation >> are sparse right now, but this is just meant to give everyone a look while >> I get it finalized and tested (despite which I'm sure Matthew Curry will >> immediately find a bug). Other files in common/ssm are either internal or >> legacy; the relevant ones are included from ssm.h and everything else is >> doomed. >> This is a versioned export, not a tag, and can be reproduced like so: >> svn export -r 931 >> https://svn.cis.uab.edu/hpcl/Nebula/SSM/libssm_ref/branches/libssm_ref-buff… >> svn export -r 931 >> https://svn.cis.uab.edu/hpcl/Nebula/SSM/libssmp_localmem/branches/libssmp_l… >> PLEASE NOTE THAT THIS IS NOT A FORMAL TAG/DROP AND IS UNTESTED, BUGGY CODE. >> PLEASE DON'T RELY ON THIS WORKING 100%, BUT PLAN ON A STABLE VERSION VERY >> SOON. The API is stable, so building against is futureproof, but there has >> been literally one test case ran on the code. This is just meant to show >> you the interface, and to peek at the implementation if something isn't >> clear. Everything should build -- please tell me if it doesn't -- but it >> would be brash to link and start calling functions. >> When I do the 0.6 drop I'll write a brief changelog and a general usage >> guide. >> I will try to be awake for the call -- if I'm not there, let this function >> as my status report. I would greatly appreciate if any discourse regarding >> this code release was via e-mail, as it causes me no small discomfort to >> speak, whereas typing just requires persistence. Moving forward, I would >> estimate only a handful of hours to the 0.6 drop I keep promising, but that >> is contingent upon my health and ability to write code without earning the >> bitter distrust of gcc. (warning: user is writing gibberish) >> Anyway, take a look, and any questions or comments please let me know. >> Guys at ANL, I expect this to be a very close approximation of a final >> interface, so please give me any comments. I apologize for the delay in >> tested+stable product. >> If any of this e-mail sounds rambling or doesn't make sense I apologize >> *grin* >> I'm going back to sleep. Godspeed, all. >> Shane >>_____________________________________________ >> ASG mailing list >> ASG(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/asg >_____________________________________________ > ASG mailing list > ASG(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/asg _____________________________________________ ASG mailing list ASG(a)lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/asg

1 0

Notification
by Seok Bae Yun 20 Feb '12

20 Feb '12

All, I am sorry that I cannot attend today's conference call. Here is my status update. For the security system, I made a draft version of integration diagram, and also an architecture design document. These will be reviewed and revised with Dr. Brooks in this week. After the review, we will also share and review the idea and the document with Dr. Kimpe. Best regards, Seok Bae Yun

1 0

[SSM 0.6 preview r931] SSM branch preview release (see inside)
by Matthew Farmer 20 Feb '12

20 Feb '12

All, I am running a very high fever and am basically out of commission. This has somewhat hampered my development efforts. I am almost done with getting libssm+ssmtests+transports cleaned up, tested, and tagged, but in my present state I just can't do it, and frankly I need to be horizontal and unconscious. So in the interest of productivity, I'm attaching the 90% version of libssm itself and the libssmp_localmem transport. (TCP and MPI are pending. InfiniBand is * really* pending.) The API is now stable and the general order of operations is settled. After testing and documentation this will become the 0.6 tag release. This code release (and the drop it's leading up to) includes all discussed features to date: - "normal" iovec memory semantics (no more custom iovecs or "chains" -- multiple buffers can be added directly to match entries) - much stricter memory management (no layer violations) and the associated scaffolding - get/put offests (both read and write) - local-side-only scatter-gathers (SSM_ME_VECTOR, implies that when initiating a get or a put all buffers instead of the first should be used for initiator only) - the buffer matching Lee discussed (i.e. size only matched against the first buffer, and special event results SSM_RESULT_NO_BUFFERS) - the never-unlink behavior (currently the default -- more on this later) - everything needed for ssm_cancel to function (although it doesn't technically function yet) - better naming and more sensible namespaces - fairly extensive refactoring that cut down on LOC and removed a lot of indirection (no more formal modules, just some custom SSM data types) - support for non-concurrent (i.e. non-threaded) SSM that only makes progress on ssm_wait (this is actually the *only* option right now, but the core is written to make adding some other options e.g. pthreads trivial). For interested parties, the public interface is in common/ssm.h, which in turn includes the stuff from common/ssm/main. Those files in main are the top-level API functions that you use with SSM. Comments and documentation are sparse right now, but this is just meant to give everyone a look while I get it finalized and tested (despite which I'm sure Matthew Curry will immediately find a bug). Other files in common/ssm are either internal or legacy; the relevant ones are included from ssm.h and everything else is doomed. This is a versioned export, not a tag, and can be reproduced like so: svn export -r 931 https://svn.cis.uab.edu/hpcl/Nebula/SSM/libssm_ref/branches/libssm_ref-buff… svn export -r 931 https://svn.cis.uab.edu/hpcl/Nebula/SSM/libssmp_localmem/branches/libssmp_l… PLEASE NOTE THAT THIS IS NOT A FORMAL TAG/DROP AND IS UNTESTED, BUGGY CODE. PLEASE DON'T RELY ON THIS WORKING 100%, BUT PLAN ON A STABLE VERSION VERY SOON. The API is stable, so building against is futureproof, but there has been literally one test case ran on the code. This is just meant to show you the interface, and to peek at the implementation if something isn't clear. Everything should build -- please tell me if it doesn't -- but it would be brash to link and start calling functions. When I do the 0.6 drop I'll write a brief changelog and a general usage guide. I will try to be awake for the call -- if I'm not there, let this function as my status report. I would greatly appreciate if any discourse regarding this code release was via e-mail, as it causes me no small discomfort to speak, whereas typing just requires persistence. Moving forward, I would estimate only a handful of hours to the 0.6 drop I keep promising, but that is contingent upon my health and ability to write code without earning the bitter distrust of gcc. (warning: user is writing gibberish) Anyway, take a look, and any questions or comments please let me know. Guys at ANL, I expect this to be a very close approximation of a final interface, so please give me any comments. I apologize for the delay in tested+stable product. If any of this e-mail sounds rambling or doesn't make sense I apologize *grin* I'm going back to sleep. Godspeed, all. Shane

3 2

Narasimha Reddy will be joining us
by Rob Ross 16 Feb '12

16 Feb '12

Hi all, After some discussion between Lee, Steve, and myself, we've asked Narasimha Reddy (TAMU) to join the group and pick up the tasks that had originally been allocated to O'Keefe's team. He has agreed to join us. He will start by building a POSIX(-like) client for us to use in demonstrations and comparative studies with existing parallel file systems. Reddy has a great deal of experience in the storage space that will be helpful in our activities. I'm personally quite happy to have him on the team and look forward to working with him. Reddy will be joining us on Monday's concall, and soon after we will be adding him to this list. Regards, Rob

1 0

Re: [ASG] [aesop] #10: Mac OS X build problem with fix attached
by aesop 15 Feb '12

15 Feb '12

#10: Mac OS X build problem with fix attached -----------------------------+---------------------------------------------- Reporter: lionkov@… | Owner: somebody Type: defect | Status: closed Priority: minor | Component: component1 Version: | Resolution: fixed Keywords: | -----------------------------+---------------------------------------------- Changes (by dkimpe): * status: new => closed * resolution: => fixed Comment: Fixed in 025246d0d7015004b34fa4e77e44bafd341bf832 Thanks for the patch! -- Ticket URL: <http://trac.mcs.anl.gov/projects/aesop/ticket/10#comment:1> aesop <https://trac.mcs.anl.gov/projects/aesop> The Aesop Language

1 0

aesop Repository branch, master, updated. 025246d0d7015004b34fa4e77e44bafd341bf832
by noreply＠mcs.anl.gov 15 Feb '12

15 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 025246d0d7015004b34fa4e77e44bafd341bf832 (commit) from b24e4a2943cff168708d4110b41ca45f6c58bc83 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 025246d0d7015004b34fa4e77e44bafd341bf832 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Wed Feb 15 21:58:33 2012 -0600 Patch from ticket #10. Fixes #10 ----------------------------------------------------------------------- Summary of changes: bin/aecc-utils.in | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) Diff of changes: diff --git a/bin/aecc-utils.in b/bin/aecc-utils.in index 7f14db2..9c708ed 100644 --- a/bin/aecc-utils.in +++ b/bin/aecc-utils.in @@ -56,8 +56,8 @@ addOption() if test "${takesArg}" = "yes"; then msg="${msg} ${argString}" fi - len=$(expr 40 - length "${msg}") - sp=$(expr substr "${SPACES}" 1 $len) + len=$(expr 40 - $(echo "${msg}" | wc -c)) + sp=$(echo "${SPACES}" | cut -c 1-${len}) HELPMESSAGE="${HELPMESSAGE}${msg}${sp}${message}\n" } @@ -494,7 +494,7 @@ runCmd() verbose="$1" ; shift error="$1" ; shift cmd=$@ - compilerout=$(mktemp) + compilerout=$(mktemp tmp.XXXXXXXX) if test "$verbose" = "yes"; then echo "$cmd" fi hooks/post-receive -- aesop Repository

1 0

Re: [ASG] [aesop] #7: integrate library linking and paths with aecc
by aesop 15 Feb '12

15 Feb '12

#7: integrate library linking and paths with aecc ---------------------------+------------------------------------------------ Reporter: carns | Owner: somebody Type: defect | Status: new Priority: major | Component: code generator Version: git repository | Keywords: ---------------------------+------------------------------------------------ Changes (by dkimpe): * version: => git repository * component: component1 => code generator -- Ticket URL: <http://trac.mcs.anl.gov/projects/aesop/ticket/7#comment:2> aesop <https://trac.mcs.anl.gov/projects/aesop> The Aesop Language

1 0

Re: [ASG] [aesop] #7: integrate library linking and paths with aecc
by aesop 15 Feb '12

15 Feb '12

#7: integrate library linking and paths with aecc -------------------+-------------------------------------------------------- Reporter: carns | Owner: somebody Type: defect | Status: new Priority: major | Component: component1 Version: | Keywords: -------------------+-------------------------------------------------------- Comment(by dkimpe): Likewise, there should be no need to add -Iinclude to make aecc find the aesop.h header. -- Ticket URL: <http://trac.mcs.anl.gov/projects/aesop/ticket/7#comment:1> aesop <https://trac.mcs.anl.gov/projects/aesop> The Aesop Language

1 0

[aesop] #16: Usage of goto in aesop code
by aesop 14 Feb '12

14 Feb '12

#16: Usage of goto in aesop code ---------------------------+------------------------------------------------ Reporter: dkimpe | Owner: Type: defect | Status: new Priority: minor | Component: code generator Version: git repository | Keywords: ---------------------------+------------------------------------------------ The translator should issue errors when an attempt is made to use goto to jump past blocking calls. Right now, incorrect code is issued. -- Ticket URL: <http://trac.mcs.anl.gov/projects/aesop/ticket/16> aesop <https://trac.mcs.anl.gov/projects/aesop> The Aesop Language

1 0

aesop Repository branch, master, updated. b24e4a2943cff168708d4110b41ca45f6c58bc83
by noreply＠mcs.anl.gov 14 Feb '12

14 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via b24e4a2943cff168708d4110b41ca45f6c58bc83 (commit) via fb9d6a547936c1f0be3cab1213f0f39374e9098e (commit) via 4bdc5b8a5cf97a4a2d3a3aac58165817f9c88cee (commit) from 740dd98e4720d8c6cf15b5003d5236d7ed858919 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit b24e4a2943cff168708d4110b41ca45f6c58bc83 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Tue Feb 14 14:05:03 2012 -0600 Update cancel document based on meeting discussion - Make the cancel flag sticky to: - avoid race between aesop cancelling and the underlying operation cancel - Allow the resource to return success if the cancel request was received by the resource but the operation completed at the same time. By having a sticky cancel flag, we no longer use the return code of the resource as the notification of a request to cancel, since the next blocking call will instead deliver the notification (as it will do immediate completion indicating cancel) commit fb9d6a547936c1f0be3cab1213f0f39374e9098e Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Feb 10 19:53:48 2012 -0600 No need for aecc-utils to be executable commit 4bdc5b8a5cf97a4a2d3a3aac58165817f9c88cee Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Feb 10 18:29:05 2012 -0600 Aesop user guide and performance guide. ----------------------------------------------------------------------- Summary of changes: configure.ac | 3 +- doc/aesop-pbranch-cancel.txt | 26 +++++++++------ doc/aesop-user-guide.txt | 72 ++++++++++++++++++++++++++++++++++++++++++ doc/aesop.txt | 33 ------------------- doc/module.mk.in | 4 ++ 5 files changed, 94 insertions(+), 44 deletions(-) create mode 100644 doc/aesop-performance.txt create mode 100644 doc/aesop-user-guide.txt delete mode 100644 doc/aesop.txt create mode 100644 doc/module.mk.in Diff of changes: diff --git a/configure.ac b/configure.ac index 6912ff4..bc69217 100644 --- a/configure.ac +++ b/configure.ac @@ -287,7 +287,7 @@ AC_SUBST(BLOCKSOPT) AC_SUBST(LIBCFLAGS) AC_SUBST(THREAD_LIB) -AC_CONFIG_FILES([bin/aecc-utils], [chmod +x bin/aecc-utils]) +AC_CONFIG_FILES([bin/aecc-utils], []) AC_CONFIG_FILES([bin/aecc], [chmod +x bin/aecc]) AC_OUTPUT([Makefile @@ -305,5 +305,6 @@ parser/module.mk parser/tests/blocking/module.mk src/socket/module.mk src/socket/test/module.mk +doc/module.mk ]) diff --git a/doc/aesop-pbranch-cancel.txt b/doc/aesop-pbranch-cancel.txt index 712f5f2..61b9ce1 100644 --- a/doc/aesop-pbranch-cancel.txt +++ b/doc/aesop-pbranch-cancel.txt @@ -62,11 +62,17 @@ Problems with the current implementation: === Semantics of the aesop-level cancel: * totally asynchronous (no longer blocks until pbranches exit) * cancel signal to pbranch is guaranteed to be delivered exactly once -** will affect current resource operation (if present) or the next one to - be posted -** cancel status is cleared for a pbranch once it has impacted one blocking - resource call (so pbranch can issue blocking cleanup functions safely if - it needs to) +* Cancel sets a flag in the execution context of the pbranch +** will affect current resource operation (if present), calling the cancel + function of the resource. +** If no resource call is active, it will simply set the flag in the execution +context. +** The cancel flag remains in the pbranch execution context. Currently we do +not plan to provide a way to clear the flag. +** The resources are responsible for checking the flag in the post functions, +and return immediately with an appropriate error code if they detect the +cancel flag is set. + * cancel will return true if it succeeded in delivering the cancel request, false if it detects that another pbranch *at the same scope or higher* already issued a cancel @@ -83,7 +89,7 @@ Problems with the current implementation: question (and the operation is guaranteed to return ECANCEL or whatever is appropriate for the resource) * returns false if it failed to cancel the operation in question -* cancel function can no longer issue op callback directly (this complicates +* cancel function is no longer allowed to call the op callback directly (this complicates some possible deadlock scenarios) === Aesop internal implementation details: @@ -101,13 +107,13 @@ Problems with the current implementation: ** proceed recursively, setting flag in child pbranches in the same manner === Semantics of the resource-level post functions: -* will have hook to check (and clear) parent ctrl structure cancel flag +* will have hook to check parent ctrl structure cancel flag * if detect that pbranch has cancelled, the post should immediately fail with appropriate return code for resource === Misc: -* provide helper functions is_cancelled() and clear_cancelled() so a pbranch - can check and clear it's cancelled status manually if it wants to +* provide helper functions is_cancelled() so a pbranch + can check it's cancelled status manually if it wants to * One possible implementation idea is to treat child pwait blocks as resources, so that they generate op ids and have a cancel function just like a resource call. Might make the aesop code a little cleaner in @@ -178,7 +184,7 @@ Example resource cancel semantics: triton_ret_t resource_post(void) { /* don't start op if we have a pending, undelivered cancel signal */ - if(test_and_clear_cancelled_state()) + if(test_cancelled_state()) return(ECANCELLED); /* do whatever .... */ diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt new file mode 100644 index 0000000..e69de29 diff --git a/doc/aesop-user-guide.txt b/doc/aesop-user-guide.txt new file mode 100644 index 0000000..550ff16 --- /dev/null +++ b/doc/aesop-user-guide.txt @@ -0,0 +1,72 @@ + +Aesop +===== + +== Source files + +Aesop source and header files are named *.ae and *.hae, respectively. They +are used exactly like c source and header files, except that they support +additional constructs, such as the __blocking qualifier for functions and +the pwait and pbranch constructs. + +In addition, Aesop remote code (ie, RPC functions) can be found in .aer and +.haer functions. These source files support an additional __remote +qualifier for functions to be invoked via RPC. + +== Compilers + +The configure process produces two compilers in the maint/ directory. aecc +is a compiler for .ae files, and aercc is a compiler for .aer files. Both +are shell scripts that combine preprocessing, source translation, and +compilation into one step. + +[source,c] +---- +sdafa +sdfa +sdfasdf +---- + +== Resources + +Resources are the lowest level aesop components that present Aesop +compatible intefaces for managing concurrency. The current list of +resources includes: + +* file resource +** code/src/versioned-osd/prototype/file-resource +** provides a mapping of open, close, read, write etc. functions. It uses + a thread pool by default to implement nonblocking semantics. + +* Berkeley DB resource +** code/src/versioned-osd/prototype/bdb-resource +** provides a mapping of common Berkeley DB functions. It uses a thread + pool by default to implement nonblocking semantics. + +* socket resource +** code/src/common/resources/aesocket/ +** waits for a file descriptor to be ready for read or write operations. + It does not implement actual read or write calls; it is assumed that the + caller will use this resource in conjunction with nonblocking reads and + writes. It uses libev, which in turn uses epoll() on Linux. + +* timer resource +** code/src/common/resources/timer/ +** implements timers with millisecond resolution. The underlying + implementation uses libev. + +* scheduling resource +** code/src/common/resource/scheduling/ +** Implements various synchronization primitives, for example Aesop + equivalents for condition variables. + +* branch threader resource +** code/src/common/resource/branch-threader/ +** Proof of concept that allows the developer to manually allocate a thread + for particular aesop code blocks. + +== Test programs + +A variety of tests and/or examples for Aesop functionality can be found in +the code/src/aesop/parser/tests/blocking directory. These tests use a +dummy resource to demonstrate various language features. diff --git a/doc/aesop.txt b/doc/aesop.txt deleted file mode 100644 index e1e3d7d..0000000 --- a/doc/aesop.txt +++ /dev/null @@ -1,33 +0,0 @@ -Aesop -===== - -== Source files - -Aesop source and header files are named *.ae and *.hae, respectively. They -are used exactly like c source and header files, except that they support -additional constructs, such as the __blocking qualifier for functions and -the pwait and pbranch constructs. - -In addition, Aesop remote code (ie, RPC functions) can be found in .aer and -.haer functions. These source files support an additional __remote -qualifier for functions to be invoked via RPC. - -== Compilers - -The configure process produces two compilers in the maint/ directory. aecc -is a compiler for .ae files, and aercc is a compiler for .aer files. Both -are shell scripts that combine preprocessing, source translation, and -compilation into one step. - -== Resources - -Resources are the lowest level aesop components that present Aesop -compatible intefaces for managing concurrency. An example is the timer -resource, found in aesop/resource/timer. It implements timers with -millisecond resolution. The underlying implementation uses libev. - -== Test programs - -A variety of tests and/or examples for Aesop functionality can be found in -the aesop/parser/tests/blocking directory. These tests use a -dummy resource to demonstrate various language features. diff --git a/doc/module.mk.in b/doc/module.mk.in new file mode 100644 index 0000000..979af15 --- /dev/null +++ b/doc/module.mk.in @@ -0,0 +1,4 @@ +ASCIIDOCSRC += doc/aesop-user-guide.txt \ + doc/aesop-performance.txt \ + doc/aesop-pbranch-cancel.txt + hooks/post-receive -- aesop Repository

1 0