- Asg - mailman3.cels.anl.gov

aesop Repository branch, master, updated. 538e92278ea6126d9bcfcc32780a189611cae773
by noreply＠mcs.anl.gov 26 Feb '12

26 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 538e92278ea6126d9bcfcc32780a189611cae773 (commit) via b5d0b3868d519732867f7a505f2fe920f36e4830 (commit) from 8c722638265ae580606988cd9759a19e61b61307 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 538e92278ea6126d9bcfcc32780a189611cae773 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Feb 24 18:52:27 2012 -0600 Aesop User Guide: Language Chapter commit b5d0b3868d519732867f7a505f2fe920f36e4830 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Feb 24 18:52:12 2012 -0600 Update to cancel doc; clarification ----------------------------------------------------------------------- Summary of changes: doc/aesop-pbranch-cancel.txt | 4 +- doc/aug_language.txt | 473 ++++++++++++++++++++++++++++++++++++++++++ doc/aug_standard.txt | 2 + 3 files changed, 477 insertions(+), 2 deletions(-) Diff of changes: diff --git a/doc/aesop-pbranch-cancel.txt b/doc/aesop-pbranch-cancel.txt index 61b9ce1..0196736 100644 --- a/doc/aesop-pbranch-cancel.txt +++ b/doc/aesop-pbranch-cancel.txt @@ -73,8 +73,8 @@ not plan to provide a way to clear the flag. and return immediately with an appropriate error code if they detect the cancel flag is set. -* cancel will return true if it succeeded in delivering the cancel request, - false if it detects that another pbranch *at the same scope or higher* +* cancel will return AE_SUCCESS if it succeeded in delivering the cancel request, + AE_ERR_EXIST if it detects that another pbranch *at the same scope or higher* already issued a cancel ** it is Ok if a child pbranch already is already cancelled in its own scope; just skip that one and return success anyway diff --git a/doc/aug_language.txt b/doc/aug_language.txt index 3084d2a..6b843dd 100644 --- a/doc/aug_language.txt +++ b/doc/aug_language.txt @@ -2,3 +2,476 @@ The Aesop Programming Language ============================== + +Language Basics +---------------- + +Source files +~~~~~~~~~~~~ + +Aesop source and header files are named *.ae and *.hae, respectively. They +are used exactly like C source and header files, except that they support +additional constructs, such as the `__blocking` qualifier for functions (and +function pointers) and the `pwait` and `pbranch` keywords (See +<<ref-blocking>>). + +///// +In addition, Aesop remote code (ie, RPC functions) can be found in .aer and +.haer functions. These source files support an additional __remote +qualifier for functions to be invoked via RPC. +//// + +C compatibility +~~~~~~~~~~~~~ + +Aesop is an extension to the ANSI C language. Therefore, valid ANSI C code is +also valid aesop code. + +[NOTE] +.Implementation Detail +==== +The current implementation of the aesop translator, while supporting many C99 +features, does not yet support all of the C99 functionality , in particular +those related to variable declarations in locations other than the beginning +of the function. These issues will be fixed in later releases. +==== + + +Threads and Aesop +~~~~~~~~~~~~~~~~~ + +Aesop is fully thread safe. Provided the function itself is thread safe, +it is legal to call the same function (blocking and regular) concurrently +from multiple threads. + +Aesop programs exposing concurrency through aesop's language features (see +<<ref-pbranch>> have to be thread-safe, as the implementation may choose to +use multiple threads to execute code when possible. + + +//=================================================================== +//=================================================================== +[[ref-blocking]] +Blocking Functions +------------------ +//=================================================================== +//=================================================================== + +Aesop extends the C language with a additional function type: blocking +functions. Blocking functions support concurrent +execution, and are used by aesop to introduce concurrency to +the C language. + + +The `__blocking` qualifyier +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All blocking functions must be marked as such. Below is an example of a +blocking function. + +---- +__blocking int test (void); +---- + +It is also possible to declare a pointer to a blocking function, similar to +declaring a pointer to a regular function. + +---- +__blocking int (*test) (void); +---- + +The `__blocking` qualifier is part of the type. Therefore, the following +two function pointers have distinct types and cannot be assigned to each other. + +---- +void (*func1) (int); +__blocking void (*func2) (int); +---- + +Cancellation +~~~~~~~~~~~~ + +Blocking functions also differ from regular functions in that blocking +functions can be _cancelled_. The exact results of cancelling a blocking +function differs from function to function. For example, a blocking function +implementing `read` or `write` might return `-EINTR` upon cancellation. How +blocking functions can be cancelled is described in <<ref-cancel>>. + +[[ref-whenblocking]] +When are functions blocking? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Functions can be made 'blocking' by choice, simply by adding the `__blocking` +qualifier to the function declaration and definition. + +However, a function is required to be blocking if one of the following is +true: + +* The function calls another function which is blocking, for example + `aesop_timer` (see <<ref-timer>>). +* The function uses aesop specific functionality (i.e. is not plain C). + +Since every aesop program tends to call at least one blocking function, this +means that the starting point of the program (see <<ref-main>>) is necessarily +blocking. + +Calling a blocking function is more expensive than calling a regular function. +In general, a function should not be made blocking unless there is a good +reason to do so. + +[NOTE] +.Implementation Detail +In the current aesop implementation, only blocking functions are transformed +by the aesop source to source translator. Regular C functions are emitted +unmodified. + +When should functions be blocking? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Except in the cases listed in <<ref-whenblocking>>, aesop does not attempt to +enforce if a function should be blocking or not. It is up to the programmer to +decide which functions need to be blocking. + +In general, functions that are not cpu-bound and depend on external (to the +cpu) events are best made blocking. This is done so that aesop can switch +execution to code that is ready to execute while waiting for the external +event to occur. + +For example, a function reading data from the network or a disk will typically +be blocking, since the function I/O bound. If the function is blocking, as +soon as the processor core would become idle, execution can continue +elsewhere. If the function is not blocking, the processor core would become +idle until the read completes. + + +[[ref-pbranch]] +Parallel Branches +------------------ + +This section explores the concept of parallel branches (or /pbranch/). + + +Parallel Branches +~~~~~~~~~~~~~~~~~ + + +The best way to introduce parallel branches is through an example: + +---- +1 __blocking void example (void) +2 { +3 pwait +4 { +5 pbranch +6 { +7 printf ("In branch 1\n"); +8 aesop_timer (10000); +9 } +10 pbranch +11 { +12 printf ("In branch 2\n"); +13 aesop_timer (10000); +14 } +15 } +16 printf ("After pwait\n"); +17 } +---- + +The function `aesop_timer` (discussed in <<ref-timer>>) suspends execution for +the given number of microseconds. In a way it is the aesop version of the +POSIX `usleep` function. + +The output from calling `example` will be: +---- +In branch 1 +In branch 2 +< ~10000 ms wait > +After pwait +---- + +In this example, the second `printf` (line 12) executes without waiting for +the first timer call (line 8) to complete. The `aesop_timer` call is a +blocking call, so after starting the timer, execution continued in the next +`pbranch` (line 10), as that branch was ready to execute. When the second +branch reaches the timer call (line 13) and can no longer continue, the same +thing happens. If any code in the program (even outside this function) is +ready to execute, execution will continue there. + + +The `pwait` (line 3) construct groups both branches. The `pwait` acts as a +barrier, in that execution will not continue past the `pwait` until all the +branches it encloses have completed. Therefore, in this example, only after +both timer calls completed, execution continues with the 3rd `printf` (line +16). + +IMPORTANT: The aesop language does _not guarantee_ that multiple threads will +be used to execute pbranches concurrently. However, it does guarantee that a +blocking call will not prevent execution of other branches, by temporarily +switching execution to another `pbranch`. + + +[NOTE] +.Implementation Detail +============= +Blocking functions act as preemption points. If a blocking function does not +complete immediately, execution will continue where possible (for example with +the next available `pbranch`). +============= +// not sure if we want to say the following +//However, the language does guarantee that the pbranches will _start execution_ +//in lexical ordering, i.e. the second pbranch will not start execution before +//the first one has been started. + + +Branch Flow Control +~~~~~~~~~~~~~~~~~ + +A branch can exit by completing the last statement in the branch or by +using the `pbreak` keyword. The effect of `pbreak` is similar to that of +`break` in a `for` or `while` loop. + +.Example +----- + +pbranch +{ + ... + pbreak; + printf ("will never get here... :(\n"); +} +----- + +In the example above, `printf` will never execute as the branch will exit +once the `pbreak` statement is reached. + +Branch Variables +~~~~~~~~~~~~~~~~ + +A pbranch can access the variables defined in the enclosing scope. However, in +some cases, it is desirable to give each pbranch a private variable. +To avoid having to declare a variable in each pbranch, and initialize that +variable from the original upon entering the pbranch, the `pprivate` keyword +is provided. + +The `pprivate` keyword applies to a variable declaration, and controls +the scope of the variable with respect to pbranches in the same scope. +Without `pprivate`, variables are shared between pbranches. Declaring a +variable as `pprivate` gives each pbranch a private copy of the variable. At +the start of a pbranch, the private copy is initialized using the current +value of the shared variable. + +The `pprivate` keyword is especially relevant for pbranches created +within `for` and `while` loops. The code below shows an example. + +---- +pwait +{ + pprivate int i; + for (i=0; i<n; ++i) + { + pbranch + { + /* use i */ + } + } +} +----- + +In the example above, each pbranch will get a private copy of the loop +variable `i`. Since the next iteration of the loop does not depend on the +completion of the pbranch, `pprivate` is required here to prevent changes to +the value of `i` in a pbranch while the branch is still executing. +In this case, `pprivate` required, as manually assigning `i` to a private copy +would be prone to a race condition. + + +[[ref-cancel]] +Cancelling branches +~~~~~~~~~~~~~~~~~~~~ + +WARNING: This section describes the new cancel model, which significantly +differs from the model in older aesop versions. + +It is possible for a pbranch to cancel the other branches in a `pwait` +construct, by calling `ae_cancel_branches()`. +This function will only attempt to cancel pbranches belonging to the same +pwait as the branch of the caller. + +Though `ae_cancel_branches` can only be called from within a pbranch, *it is +not a blocking function*. `ae_cancel_branches` will signal all other +pbranches, but will not wait for them to exit. +The `ae_cancel_branches` function returns `AE_SUCCESS` if it was able to +signal one or more branches, and returns `AE_ERR_EXIST` if all other +branches in the pwait group already had an active cancel signal. + +As only blocking functions support cancellation, cancelling a pbranch that is +not currently executing a blocking function will not have any immediate +effect. However, the cancel signal is persistent, and will affect all +subsequent blocking functions in the branch. This means that every branch will +eventually learn of the cancellation request (because it calls a blocking +function which will immediately return indicating cancel), or exit without +further calling a blocking function. + +It is not possible to clear a cancel signal in a pbranch. The recommended +practice is to exit a the pbranch as soon as a cancellation request is +detected. Any blocking functions needed to restore a consistent internal state +should be called after the `pwait` scope ends. + +A pbranch can check the status of the cancel signal using the `ae_is_cancelled +()` function, which returns non-zero if the calling branch has the cancel +signal set, and 0 otherwise. + + +.Example +---- +__blocking void cancel_example (void) +{ + pwait + { + pbranch + { + if (aesop_timer (10000) == 0) + { + // timer expired + ae_cancel_branches (); + } + } + pbranch + { + do_something (); + ae_cancel_branches (); + } + } +} +---- + +Using pbranches and cancellation, it is reasonably straightforward to +implement a time-out using aesop. In the example above, `do_something` will +be cancelled if it does not complete within 10000 ms. After 10000 ms, +`aesop_timer` will return 0 (indicating success), causing the first pbranch to +cancel the second branch. However, if `do_something` completes before the +timer, the second branch will cancel the timer. + +Lonely pbranches +~~~~~~~~~~~~~~~~~ + +It is possible to use a `pbranch` outside of `pwait`. In this case, the +pbranch executes as normal but it not possible to wait for the branch +complete. + + +---- +__blocking example_lonely (void) +{ + pbranch { + printf ("in pbranch"); + aesop_timer (1000); + } + printf ("returning\n"); +} +---- + +In the example above, a call to the function `example_lonely` might return +*before* the pbranch it contains completes. + +[TIP] +Lonely pbranches most closely resemble detached threads. They execute +independently of the main control flow, and can no longer be synchronized +with. In addition, lonely pbranches cannot be cancelled. + + +Miscellaneous & Hello World +--------------------------- + + + +Including aesop.h +~~~~~~~~~~~~~~~~~ + +All aesop source files must include `aesop.h`. +Failing to do so will cause a compilation error. + + +[[ref-main]] +The aesop main function +~~~~~~~~~~~~~~~~~~~~~~~ + +The `main` function for an aesop program is identified using `aesop_set_main`. +The named function must be a blocking function, and must take the same +arguments as the standard C `main` function. + +---- +int aesop_main (int argc, char ** args) +{ +} + +aesop_main_set(aesop_main); +---- + +[TIP] +==== +Sometimes it is useful to be able to execute some code _before_ aesop +initializes. For this, `aesop_main_set_with_init` can be used instead of +`aesop_main_set`. The former takes an additional parameter naming a _regular_ +C function which will be called _before_ initializing any aesop component. +After initializing aesop, the blocking aesop main function will be called as +usual. + + +.Example: +---- +aesop_main_set_with_init (pre_init, aesop_main); +---- +==== + + +Hello World +~~~~~~~~~~~ + +Below is an example aesop program. + +---- +#include <aesop.h> +#include <timer.hae> +#include <stdio.h> +#include <stdlib.h> + +__blocking int test_main(int argc, char** argv) +{ + printf ("Before pwait.\n"); + pwait{ + pbranch { + aesop_timer(10000); + } + pbranch { + printf("pbranch2"); + } + } + + printf("After pwait.\n"); + return (EXIT_SUCCESS); +} + +aesop_main_set (test_main); +---- + +Some observations: + +* All aesop programs have to include `aesop.h`. +* The starting point of the program is indicated using `aesop_main_set`. +* The starting function is `__blocking`. + +The output of this program should be: + +---- +Before pwait. +pbranch2 +After pwait. +---- + +There will be approximately 10000 ms between outputting `pbranch2` and `After +pwait.`, as execution will not continue past the `pwait` until all of its +branches have completed. The first branch will take at least 10000 ms to +complete, due to the `aesop_timer` call. + + diff --git a/doc/aug_standard.txt b/doc/aug_standard.txt index 6334a23..2056919 100644 --- a/doc/aug_standard.txt +++ b/doc/aug_standard.txt @@ -1,4 +1,5 @@ +[[ref-library]] The Aesop Standard Library ========================== @@ -50,6 +51,7 @@ Standard Aesop modules The following functions are bundled with the aesop distribution. +[[ref-timer]] Timer ~~~~~~ hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. 8c722638265ae580606988cd9759a19e61b61307
by noreply＠mcs.anl.gov 26 Feb '12

26 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 8c722638265ae580606988cd9759a19e61b61307 (commit) from ee2173105cff65cae2a7e9e86805f92ef601da8c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 8c722638265ae580606988cd9759a19e61b61307 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Sun Feb 26 20:58:58 2012 -0500 edit remaining aesop performance graphs ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 140 +++++++++++++++++++++++++++------------------ 1 files changed, 84 insertions(+), 56 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index a26b69c..e182ba7 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -7,7 +7,7 @@ performance, memory efficiency, programmer productivity, and compile time performance. Aesop is a programming language and programming model designed -to implement distributed system software with developer productivity and run +to implement distributed system software with high development productivity and run time efficiency. Further details about the languange and development environment can be found in the Aesop User's Guide. @@ -261,11 +261,11 @@ previous runs. Secondly, it allowed us to cycle between servers as described above without any risk of one daemon interfering with the performance of another daemon. -On fusion we determined we could use 16 client processes per physical node. Using +On fusion we determined that we could use 16 client processes per physical node. Using more clients per node caused the bottleneck of the test to become the client nodes instead of the server. Therefore, for the largest scale tests shown in this study we utilized 65 total nodes. One node acted as the server, -while 64 nodes executed up to 1024 client processes. When scaling the +while 64 nodes executed up to a total of 1024 client processes. When scaling the number of client processes, we also scaled the number of physical client nodes in the same manner. Thus the 16 process case used one physical client node, the 128 process case used 8 physical client nodes, and so on. @@ -325,9 +325,9 @@ mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-request Aesop is intended to improve developer productivity while preserving scalable performance for system software workloads. The goal of the raw performance comparison therefore is not to show that Aesop offers superior -performance to traditional approaches in all scenarios. Instead, we intend -to show that Aesop offers competitive performance that is sufficient to -consider Aesop as a viable programming model based on its productivity +performance to traditional approaches in all scenarios. Instead, we want to +determine if Aesop offers sufficient performance for consideration +as a viable programming model based on its productivity merits. We will elaborate on the productivity aspect of the comparison later in <<sec-productivity>>. @@ -370,52 +370,73 @@ in terms of its relative performance. ==== Network I/O -In this set of tests, we had specifically developed the thread-per-client-nb -to evalute the performance difference between blocking and non-blocking sockets.It is clear that the non-blocking sockets do not perform as well blocking -sockets. We did this test because aesop uses non-blocking sockets and were -trying to determine the performance difference between aesop and -the thread-per-client server. <<fig-readnull>> and <<fig-writenull>> cases show -that aesop mirrors the thread-per-client-nb server as expected. +The write-null and read-null experiments were conducted in the same manner +as the write and read tests. The difference in this case is that no disk +access was perfomed. In the write case, incoming data was discarded by the +server. In the read case, the server transmitted uninitialized data. + +.Runtime Performance for Write-Null Test +[[fig-writenull]] +image::fig/write-null-hist.png[] .Runtime Performance for Read-Null Test [[fig-readnull]] image::fig/read-null-hist.png[] -.Runtime Performance for Write-Null Test -[[fig-writenull]] -image::fig/write-null-hist.png[] +Aesop is competitive with (and in most cases better than) the other +implementations except for the thread-per-client server in both the +write-null (<<fig-writenull>>) and read-null (<<fig-readnull>>) evaluation. We were unable to +isolate a concrete reason for this discrepancy in profiling. One notable +difference in the two implementations, however, is that the +thread-per-client server uses blocking socket operations, while the aesop +socket resource uses non-blocking operations. Based on this observation, we implemented the +thread-per-client-nb server to isolate the impact of non-blocking socket +operations on performance. The thread-per-client-nb implementation is +identical to the thread-per-client implementation except that each socket +uses non-blocking operations and polling to transmit and receive data. As +seen in these tests, the use of non-blocking operations slows down the +thread-per-client server to the point that it is practically equivalent to +the Aesop server at scale. + +Another notable observation in these graphs is that the Aesop server is +competative at small scale, and in fact is the fastest implementation in the +16 client process read-null test and nearly the fastest in the 16 client +process write-null test. This supports the observation from the previous +section that poor Aesop performance at small scale is likely a tuning flaw +in the file resource used in the Aesop standard library, rather than a +fundamental programming language problem. ==== Fairness -Absolute performance is not the only metric that one might look at so here -the fairness of the various server types is examined. The bars on the -graphs shows the total time for a client to complete all of its requests. +This section evaluates the fairness of the various server implementations in +terms of service times across clients. We illustrate the fairness of each +server by plotting the difference between the fastest and slowest total time +for a client to complete all of its requests. The fastest client is the bottom of the bar and the slowest is the top of bar. These results are from the 1024 client size using the same iteration that was selected for the maximum runtime performance above. -In <<fig-readtime>> and <<fig-writetime>> we see that the event server is -the only fair server. During our initial experimentation, aesop was configured -to be a fair system, however this fairness degraded the overall total -performance so we instead configured the file resource to be unfair by design. -This is demostrated in the <<fig-readtime>> and <<fig-writetime>>, showing -that aesop is extremely unfair. Thread-pool is also unfair because we applied -the same change to this server implementation. +.Fastest and Slowest Total Client Runtime for Write Test +[[fig-writetime]] +image::fig/write-time.png[] .Fastest and Slowest Total Client Runtime for Read Test [[fig-readtime]] image::fig/read-time.png[] -.Fastest and Slowest Total Client Runtime for Write Test -[[fig-writetime]] -image::fig/write-time.png[] - -The <<fig-readnulltime>> and <<fig-writenulltime>> show similar fairness between -all server types except thread-pool in the case of the write-null test case. -Here the aesop network resource is not using an unfair design and approximates -the fairness of the other thread-per-* models. The thread-pool server does -have the design such that is is unfair in how it services requests, -<<fig-writenulltime>> shows this clearly. +In <<fig-readtime>> and <<fig-writetime>> we see that the event server is +the only fair server. However, it achieves this fairness by sacrificing +overall throughput. During our initial experimentation, aesop was configured +to be a fair system as it favored executing all disk and network operations +in FIFO order if possible. We discovered that this behavior was detrimental +to throughput in practice; the server performed better if it made progress +on requests that had most recently completed a network or disk operation +rather than requests that had been waiting the longest. We observed the +same behavior in the thread-pool server as well and made the same adjustment +for it to favor "hot" requests as opposed to FIFO ordering. + +Interestingly, we see that the thread-per-client, thread-per-client-nb, and +thread-per-op servers are more fair on the write test than on the read test. .Fastest and Slowest Total Client Runtime for Read-Null Test [[fig-readnulltime]] @@ -425,43 +446,50 @@ image::fig/read-null-time.png[] [[fig-writenulltime]] image::fig/write-null-time.png[] +The <<fig-readnulltime>> and <<fig-writenulltime>> show similar fairness between +all server types except thread-pool in the case of the write-null test case. +The thread-pool server is the only one that shows a notable difference in +fairness due to its scheduling approach. + ==== Latency +.Per Operation Latency Write Test (Min/Max/First Quartile/Third Quartile) +[[fig-writelat]] +image::fig/write-lat.png[] + +.Per Operation Latency Read Test (Min/Max/First Quartile/Third Quartile) +[[fig-readlat]] +image::fig/read-lat.png[] + The last client metric is an examination of per operation latency. The -client collects all the operation latency and the computes the minimum and +client measures the latency of each individual request and the computes the minimum and maximum latency, the first quartile latency and third quartile latency. These metrics are graphed using a box and whiskers plot. The box represents the -first and thrid quartiles and the whiskers are the minimum and maximum +first and third quartiles and the whiskers are the minimum and maximum values. The following results are for the 1024 client size selected from the same iteration as the maximum runtime graphs. <<fig-readlat>> and <<fig-writelat>> show the aesop offers vary comparitve latency performance as the other configurations and only noteably thread-per-op and event are -signficantly worse. +signficantly worse. An interesting observation is that the overall fairness +across clients shown in the previous section does not appear to correlate in +any way with individual request latency. -.Per Operation Latency Read Test (Min/Max/First Quartile/Third Quartile) -[[fig-readlat]] -image::fig/read-lat.png[] +.Per Operation Latency Write-Null Test (Min/Max/First Quartile/Third Quartile) +[[fig-writenulllat]] +image::fig/write-null-lat.png[] -.Per Operation Latency Write Test (Min/Max/First Quartile/Third Quartile) -[[fig-writelat]] -image::fig/write-lat.png[] +.Per Operation Latency Read-Null Test (Min/Max/First Quartile/Third Quartile) +[[fig-readnulllat]] +image::fig/read-null-lat.png[] In <<fig-readnulllat>> and <<fig-writenulllat>> we see again that aesop has very good latency metrics compared to the other servers. In this case the thread-per-op and event are noticeably worse than the other server implementations. -.Per Operation Latency Read-Null Test (Min/Max/First Quartile/Third Quartile) -[[fig-readnulllat]] -image::fig/read-null-lat.png[] - -.Per Operation Latency Write-Null Test (Min/Max/First Quartile/Third Quartile) -[[fig-writenulllat]] -image::fig/write-null-lat.png[] - ==== Summary -The runtime analysis shows that aesop is very competitive with the most +The runtime analysis shows that aesop is competitive with the most optimal alternate server implementations at the 1024 client size. In the _read_ test, aesop is 4.5% slower that the fastest implementation, threadpool. Aesop was the fastest implementation in the _write_ test. @@ -471,10 +499,10 @@ however, as we showed above, the difference in performance was due to the native performance difference in synchronous and asynchronous sockets. If we look at the performance comparison to the thread-per-client-nb, aesop was only 4.7% slower in the _read null_ test and was faster in the _write null_ -test. If aesop is using native asynchronous transport such as Infiniband verbs -or SSM, it will perform as well as a thread-per-client implementation. +test. If aesop were using a native asynchronous transport such as Infiniband verbs +or SSM, we believe that it would perform as well as a thread-per-client implementation. -Another interesting factor was that their was no one fastest server +Another interesting factor was that there was no one fastest server implementation. Each workload presents a different challenge and building a single tuned server is difficult. A key advantage of aesop is the ability to change the underyling implementation/turning for resources or the aesop hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. ee2173105cff65cae2a7e9e86805f92ef601da8c
by noreply＠mcs.anl.gov 26 Feb '12

26 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via ee2173105cff65cae2a7e9e86805f92ef601da8c (commit) from 748836087ad6bf360bb804ef5d8b4d9b361dd040 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit ee2173105cff65cae2a7e9e86805f92ef601da8c Author: Phil Carns <carns(a)mcs.anl.gov> Date: Sun Feb 26 08:18:55 2012 -0500 edit evaluation text up through first 2 graphs ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 123 +++++++++++++++++++++++++++++--------------- 1 files changed, 81 insertions(+), 42 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index fac5cdc..a26b69c 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -31,6 +31,7 @@ server that can write or read data from local files. It is expected to process requests from multiple clients simultaneously. The description of the test server and client are given in the following subsections. +[[sec-client-design]] === Client Design Each of the example servers used for comparison in this document implement @@ -215,55 +216,64 @@ network and disk. [[runtime-perf]] == Runtime Performance Evaluation and Analysis -The evaluation of runtime performance was done by executing a series of tests -using each server implementation type with the same client as discussed above. -We then compare the results for each of the server implementations against -aesop determine the overall effiecency of aesop compared to hand-tuned -solutions. The four test types were selected to evaluate two models. The -first two test types (read,write) examine real disk I/O. Aesop is inteneded -for use in a storage server so we want to examine multiplexing of disk and -network I/O. The second set (read-null,write-null) eliminates disk I/O and -only has network I/O. This removes the storage bottleneck and examines how -well the different server types handle concurrency. +The evaluation of runtime performance was performed by executing a series +of tests using an identical client test harness (described in +<<sec-client-design>>) for each server implementation. We used small +request sizes in order to stress concurrent, latency-bound requests in an +effort to highlight the ability of each server to quickly multiplex between +small operations. The read and write tests used the O_DIRECT file access +mode in order to bypass the Linux buffer cache and insure that the disk is +involved in each I/O transfer. The read-null and write-null tests are +included to stress the pure network performance of each server and insure +that disk I/O is _not_ involved in those cases. === Experiment -We ran our tests on the Argonne Fusion cluster which is a standard Linux HPC -cluster. The experiment was done by running each client mode (read, write, -read-null, write-null) against all six server types. - -The testing was done by running a job on Fusion that executed one of the client -test types against all six server implementations scaling the clients from -16 to 1024. This kept the wall time of the job reasonable and allows a -consistent comparison between server types within a test type. The network -is still a shared resource on the cluster or each individual test was run -five times and the results below are the median from the five iterations, -based on the maximum runtime. - ==== Experiment System -Fusion is a cluster run by the Argonne Laboratory Computing Resource Center +All experiments were executed on the Fusion cluster managed by +the Argonne Laboratory Computing Resource Center (LCRF). Fusion is a IBM iDataPlex dx360 M2 system. It features 320 compute -nodes which consist of two Intel Nehalem 2.6 GH Xeon processors and 36 GB +nodes which each consist of two Intel Nehalem 2.6 GH Xeon processors and 36 GB of RAM. The compute nodes have hyper threading disabled. The cluster has an Infiniband QDR interconnect. Each compute node also a single SATA 7200 RPM hard disk for local scratch storage. ==== Experiment Details -The experiment consisted of four separate jobs on Fusion. The top level unit -is the test type: read, write, read-null, write-null. For each test type, -the number of clients are scaled up from 16 to 1024. At each client scale, -each server implementation is tested in sequence. The individual test is -executed five times. On each iteration, the server is started then the MPI -client program is started. The server is restarted for each iteration so -that the memory statistics are not polluted by a previous run. - -On fusion we determined we could use 16 clients per physical node. Using +The tests were performed by +submitting one job for each request type (read, write, read-null, or +write-null). Each job executed a series of scripts that launched each +server implementation in turn to service workloads that scaled +the number of clients from +16 to 1024. This methodology insured that exactly the same clients and +server nodes were used used when making comparisons across implementations +for a given workload. Fusion is a shared resource and may experience +increased network contention at times. We therefore executed each +test case five times and cycled between server implementations in a +round-robin fashion to minimize the possibility of any given server +execution being unfairly penalized by external contention. We show the +median result in all graphs unless +otherwise noted. The server daemon was restarted on each +iteration for two reasons. First, this approach allowed us to capture memory statistics independently for each +run and insure that results were not affected by resources left over from +previous runs. Secondly, it allowed us to cycle between servers as +described above without any risk of one daemon interfering with the +performance of another daemon. + +On fusion we determined we could use 16 client processes per physical node. Using more clients per node caused the bottleneck of the test to become the client -nodes instead of the server. The clients were scaled up as follows: 16, 128, 256, 512 and 1024 clients. - -The clients connected to the server using the IB network with IPoIB. +nodes instead of the server. Therefore, for the largest scale tests shown +in this study we utilized 65 total nodes. One node acted as the server, +while 64 nodes executed up to 1024 client processes. When scaling the +number of client processes, we also scaled the number of physical client +nodes in the same manner. Thus the 16 process case used one physical client +node, the 128 process case used 8 physical client nodes, and so on. + +The clients connected to the server using the IB network with IPoIB. Note +that MPI is only used in the client test harness for coordination and timing +among client processes. All communication between clients and servers is +performed with TCP/IP sockets. ===== Read @@ -310,6 +320,17 @@ data to be discarded by the server. mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 4096 --size 4096 --type-wn ********************* +=== Expectations + +Aesop is intended to improve developer productivity while preserving +scalable performance for system software workloads. The goal of the raw +performance comparison therefore is not to show that Aesop offers superior +performance to traditional approaches in all scenarios. Instead, we intend +to show that Aesop offers competitive performance that is sufficient to +consider Aesop as a viable programming model based on its productivity +merits. We will elaborate on the productivity aspect of the comparison later in +<<sec-productivity>>. + === Evaluation Here the runtime results are presented from the experimentation. All graphs @@ -317,17 +338,35 @@ are shown in log scale. ==== Disk I/O -In <<fig-readhist>> and <<fig-writehist>>, we see that aesop fairs -favorably to the other server implementations. In general the various -servers show similar scaling. +.Runtime Performance for Write Test +[[fig-writehist]] +image::fig/write-hist.png[] + +<<fig-writehist>> shows the overall run time of the concurrent write +workload for each server implementation as the number of client processes +is scaled from 16 to 1024. In this graph we see that Aesop does not +perform as well as the other servers for small workloads (taking 3.2 +seconds at the smallest scale, verses 1.9 seconds for the thread-per-op +server). However, Aesop is the fastest server at the largest scale +(taking 119.8 seconds verses 130.8 seconds for the nearest competitors +in thread-per-client and threhad-per-client-nb). .Runtime Performance for Read Test [[fig-readhist]] image::fig/read-hist.png[] -.Runtime Performance for Write Test -[[fig-writehist]] -image::fig/write-hist.png[] +<<fig-readhist>> shows the results of the read experiment. Aesop performs +more favorably at small scale for this workload than in the previously shown +write workload. At the largest scale, +Aesop completes the test in 48.3 seconds verses 46.1 seconds for the fastest +server (thread-pool). The event server performs particularly poorly in all +cases, ultimately running the largest scale test in 77.2 seconds. + +The small scale results for Aesop may indicate that additional tuning +is needed to improve latency for small test runs. The issue is likely +isolated to the write path of the file I/O resource in the Aesop standard +library, as we see assymetric results in the read and write tests for Aesop +in terms of its relative performance. ==== Network I/O hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. 748836087ad6bf360bb804ef5d8b4d9b361dd040
by noreply＠mcs.anl.gov 26 Feb '12

26 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 748836087ad6bf360bb804ef5d8b4d9b361dd040 (commit) from 95f048bed626d0a6075a1d2211dadff3f6eff5ac (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 748836087ad6bf360bb804ef5d8b4d9b361dd040 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Sun Feb 26 07:12:39 2012 -0500 misc. editing of intro and case study description ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 153 +++++++++++++++++++++++++++++++-------------- 1 files changed, 105 insertions(+), 48 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index 696bda7..fac5cdc 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -2,23 +2,46 @@ == Introduction -We examine the performance of Aesop based on several factors including -runtime performance, memory effieceny, programmer productivity and -compile time performance. We evaluated aesop by building a simple TCP server -which would recieve requests from clients and then either recieve or send -data which was either generated, discarded, read from a file or written to -a file. This server design was then implemented in aesop and then in generic -C code using other concurrency models for comparison. The description of -the test server and client are below and these serve as the basis for the -evaluations in the rest of the text. We also explore some micro-benchmarks -to analyze how core features of aesop impact performance. +This document examines the performance of Aesop in terms of runtime +performance, memory efficiency, programmer productivity, and compile time +performance. + +Aesop is a programming language and programming model designed +to implement distributed system software with developer productivity and run +time efficiency. +Further details about the languange and development environment can be found +in the Aesop User's Guide. + +The remainder of this document is organized as follows. +<<sec-case-study>> and <<runtime-perf>> describe a network service +case study and use it to evaluate the performance of Aesop +relative to more traditonal server architectures. +<<sec-memory>> provides a more detailed analysis of memory efficiency in +Aesop. <<sec-productivity>> evaluates the productivity of the Aesop +language using the original case study as an example. +<<sec-compile>> concludes by discussing Aesop compile-time code translation +performance. + +[[sec-case-study]] +== Case study description + +We will use a simple network serivce case study for quantitative +evaluation of the Aesop programming language. The case study is a TCP +server that can write or read data from local files. It is expected to +process requests from multiple clients simultaneously. The description +of the test server and client are given in the following subsections. === Client Design -The client is a basic C program that uses TCP sockets to send messages to -the server. The client will run in a loop generating a specifed number of -operations to the server. The general flow is that client send a request -to the server that contains and optional payload. The client then waits for +Each of the example servers used for comparison in this document implement +an identical request protocol and are therefore evaluated using the same +client test harness. The client is a basic C program that uses TCP sockets to send messages to +the server. It use MPI to coordinate processes and generate a highly +concurrent workload. + +The client will execute in a loop generating a specifed number of +operations to the server. The general flow is that each client process sends a request +to the server that contains an optional payload. The client then waits for the server to send an acknowledgement that also contain an optional payload. ==== Request Types @@ -39,12 +62,16 @@ acknowledgement to client. ===== Read-Null +This request is identical to the *Read* request, except that the server +sends uninitialized data rather than performing any file I/O. The client sends a request with a size to the server. The server then allocates a buffer for the response based on the size the client requested. The server then sends an acknowledgement with this buffer as the payload. ===== Write-Null +This request is identical to the *Write* request, except that the server +discards incoming data rather than performing any file I/O. The client sends a request with a size and a payload. The server recieves the request but then simply discards the payload. The server sends an acknowledgement back to the client. @@ -52,18 +79,27 @@ acknowledgement back to the client. [float] ==== Implementation -The client has a set of command line parameters which control selecting the -test type, the number of requets and the size of the request. The client is -also a MPI program. This allows starting an arbitray number of clients and -then synchronizing startup. The clients all barrier until they are ready to -connect to the server. Clients exit the barrier, connect to the server -and begin sending requests in a loop. When the client completes it waits -at another barrier and then reports statistics about the run. +The client test harness provides command line parameters to control the +request type, the number of requets, and the size of the request. Each +process barriers until they are all ready to +connect to the server. Clients then exit the barrier, connect to the server, +and begin sending requests in a loop. Once each process completes its +requests, it waits +at another barrier and then reports statistics about the run. Each process +records the total amount of time taken to execute its workload (beginning +before the initial connection and ending after receipt of the last +acknowledgement). The time taken by the slowest process is reported as the +aggregate run time. Each process also records the time needed to service each +individual request (from before the request is sent until after the +acknowledgement is received) in order to calculate statistics about +individual request latencies. === Server Design The server supports the client request types listed above. Each server -implementation accepts client connections and then waits for requests. +implementation accepts client connections and then waits for requests on +those connections. The following diagram illustrates the steps performed by +each request type. .Server Request Handling [graphviz] @@ -109,12 +145,14 @@ digraph G } -------- -The servers share a common base of code for handling the requests. -This insures that the variation in performance is primarily due -to concurrency within the server. There is one exception to this which is -the use of blocking or non-blocking sockets. This will be ellaborated on -more in the analysis. All event loops are implemented using libev. <<libev>> -The thread implementations all use `pthread` threading library. +We implemented the same request protocol in multiple server daemons +in order to contrast different approaches to concurrent +request processing. Each server uses the same fundamental coding style to +the degree possible. One server is implemented using the Aesop language, +while all other servers are implemented in C. The `pthread` library was +used in all cases that required explicit threading, while the `libev` +library was used in all cases that required an explicit event loop +<<libev>>. ==== Server Implementations @@ -123,43 +161,55 @@ a different type of concurrency model. ===== Aesop -The Aesop server is done using Aesop of course. The server uses a +The Aesop server is implemented in the Aesop programming language. The server uses a 'lonely pbranch' to service each client. All operations for a client are -handled within a single pbranch. The underlying socket resource uses -non-blocking sockets with a thread pool of 16 threads. The file resource uses +handled within a single pbranch. The socket and file operations are +performed with blocking Aesop functions that are provided by the Aesop +standard library. The underlying socket resource uses +non-blocking sockets with a thread pool of 12 threads. The file resource uses synchronous IO and a thread pool with 4 threads. ===== Thread-per-client The thread-per-client server spawns a thread for each client connection. All -requests for the client are handled within the same thread. This model uses -blocking sockets. The thread remains alive until the client disconnects. +requests for a given client are handled within the same thread. This model uses +blocking socket operations and standard file read and write operations. +Each thread executes until the corresponding client disconnects. ===== Thread-per-client-nb -The thread-per-client-nb server is identical to the thread-per-client server -except that is uses non-blocking sockets instead of blocking sockets. We -implemented this version to investigate the possible performance difference -between the synchronous and asynchronous socket calls. +The thread-per-client-nb server is identical to the thread-per-client +server, except that is uses non-blocking socket calls in place of blocking +socket calls. For example, in order to send a message, a thread will +perform non-blocking sends until it encounters the EWOULDBLOCK error code. +It then polls the socket until it is ready and continues sending data. +We implemented this version to investigate the possible performance difference +between the synchronous and asynchronous socket calls in a scenario where +all other factors are held constant. ===== Thread-per-operation -The thread-per-operation server uses and event loop to watch all sockets for -activity and when requests are available a thread is spawned and the request +The thread-per-operation server uses an event loop to watch all sockets for +activity. When a new request is available, a thread is spawned and the request is handled completely from within that thread. When the request is complete -the thread is destroyed. Blocking sockets are used in this implementation. +the thread is destroyed. Blocking socket operations and standard file read +and write functions are used in this implementation. ===== Thead-pool -The thread-pool server uses and event loop to watch all sockets for activity. -When requests are available, the event loop puts the request on a queue and +The thread-pool server uses an event loop to watch all sockets for activity. +When a new request is available, the event loop puts the request on a queue and wakes up a thread from the thread pool. The request is handled completely -from within a single thread of the thread pool. Blocking sockets are used -in this implementation. +from within a single thread of the thread pool. Blocking sockets and +standard file read and write functions are used in this implementation. The +thread pool was implemented following best practices for scalable condition +variable performance as described in <<hp-cond-variable>>. ===== Event -The event server uses a execution context to handle all clients and requests. -The event loop watches all sockets and handles each request in a callback. The +The event server uses an event loop not only to detect incoming requests, +but to service them as well. Each request processing step is executed from +an event loop callback function. The event server uses non-blocking sockets and asynchronous file I/O. Note that -the though the operating system can still use multiple cores to drive the +although this implementation does not use any explicit threads, the +operating system can still use multiple cores to drive both the network and disk. [[runtime-perf]] @@ -495,6 +545,7 @@ The results for these are shown in the +tcmalloc+ column. The progam used to obtain these results is in the repository: +tests/blocking-overhead.ae+. +[[sec-memory]] == Runtime Memory Efficiency Another aspect of the overall performance is the memory efficiency of each @@ -588,7 +639,8 @@ So, in the example above, even though the test will return without waiting for the pbranch to complete, its stack variables (`var` in this case) will consume memory until the pbranch returns. -== Productivity Performance +[[sec-productivity]] +== Productivity The core design element of aesop is to make programming of a concurrent server easier, so the trade off for memory and runtime performance should @@ -650,6 +702,7 @@ servicing a write operation requires 5 disconnected event handlers. Although the event model appears less complex than the thread-pool model according to CC and Mod. CC, qualitatively it is significantly more challenging to develop. +[[sec-compile]] == Compile Time Performance Currently aesop imposes some overhead when compiling aesop source. This is @@ -681,6 +734,10 @@ can be mitigated by constructing a Makefile that supports parallel make. - [[[Bame]]] Paul Bame. pmccabe. http://parisc-linux.org/~bame/pmccabe/ - [[[Wheeler]]] David A. Wheeler. sloccount. http://www.dwheeler.com/sloc/ - [[[libev]]] Marc Lehmann. http://software.schmorp.de/pkg/libev.html +- [[[hp-cond-variable]]] Hewlet Packard Development Company L.P.. "Techniques for + Improving the Scalability of Applications Using POSIX Thread + Condition Variables." + http://h21007.www2.hp.com/portal/download/files/unprot/hpux/MakingCondition… == Appendix A - Raw Data hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. 95f048bed626d0a6075a1d2211dadff3f6eff5ac
by noreply＠mcs.anl.gov 26 Feb '12

26 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 95f048bed626d0a6075a1d2211dadff3f6eff5ac (commit) via d28a39b3a7faa8513933345a622cbce64b437b00 (commit) from d74952042d90e5b4ef086b1e5c4e00c4419f713b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 95f048bed626d0a6075a1d2211dadff3f6eff5ac Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Sun Feb 26 02:57:18 2012 -0600 Updated PNG images commit d28a39b3a7faa8513933345a622cbce64b437b00 Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Sun Feb 26 02:56:25 2012 -0600 Aesop performance paper updates ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 588 ++++++++++++++++++++++++++++++------------ doc/fig/read-hist.png | Bin 4950 -> 6163 bytes doc/fig/read-lat.png | Bin 0 -> 4823 bytes doc/fig/read-mem.png | Bin 4564 -> 6666 bytes doc/fig/read-null-hist.png | Bin 4877 -> 6091 bytes doc/fig/read-null-lat.png | Bin 0 -> 5182 bytes doc/fig/read-null-mem.png | Bin 5055 -> 5753 bytes doc/fig/read-null-time.png | Bin 0 -> 4671 bytes doc/fig/read-null.png | Bin 6700 -> 8137 bytes doc/fig/read-time.png | Bin 0 -> 5261 bytes doc/fig/read.png | Bin 5818 -> 7353 bytes doc/fig/write-hist.png | Bin 4795 -> 5507 bytes doc/fig/write-lat.png | Bin 0 -> 5150 bytes doc/fig/write-mem.png | Bin 4562 -> 6575 bytes doc/fig/write-null-hist.png | Bin 5022 -> 5605 bytes doc/fig/write-null-lat.png | Bin 0 -> 5189 bytes doc/fig/write-null-mem.png | Bin 4594 -> 6522 bytes doc/fig/write-null-time.png | Bin 0 -> 5001 bytes doc/fig/write-null.png | Bin 6410 -> 7703 bytes doc/fig/write-time.png | Bin 0 -> 5233 bytes doc/fig/write.png | Bin 5494 -> 6847 bytes 21 files changed, 420 insertions(+), 168 deletions(-) create mode 100644 doc/fig/read-lat.png create mode 100644 doc/fig/read-null-lat.png create mode 100644 doc/fig/read-null-time.png create mode 100644 doc/fig/read-time.png create mode 100644 doc/fig/write-lat.png create mode 100644 doc/fig/write-null-lat.png create mode 100644 doc/fig/write-null-time.png create mode 100644 doc/fig/write-time.png Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index 9dab84b..696bda7 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -21,33 +21,28 @@ operations to the server. The general flow is that client send a request to the server that contains and optional payload. The client then waits for the server to send an acknowledgement that also contain an optional payload. -[float] ==== Request Types The client supports the following request types. -[float] ===== Read The client sends a request with a file name and a size. The server will then open the file, read the contents up to the size specified. The server returns the data with the acknowledgement of the operation. -[float] ===== Write The client sends a request with a file name, size and payload. The server will then create the file and write the payload. The server then sends an acknowledgement to client. -[float] ===== Read-Null The client sends a request with a size to the server. The server then allocates a buffer for the response based on the size the client requested. The server then sends an acknowledgement with this buffer as the payload. -[float] ===== Write-Null The client sends a request with a size and a payload. The server recieves @@ -126,7 +121,6 @@ The thread implementations all use `pthread` threading library. We implemented six server types including Aesop. Each server type examines a different type of concurrency model. -[float] ===== Aesop The Aesop server is done using Aesop of course. The server uses a @@ -135,14 +129,12 @@ handled within a single pbranch. The underlying socket resource uses non-blocking sockets with a thread pool of 16 threads. The file resource uses synchronous IO and a thread pool with 4 threads. -[float] ===== Thread-per-client The thread-per-client server spawns a thread for each client connection. All requests for the client are handled within the same thread. This model uses blocking sockets. The thread remains alive until the client disconnects. -[float] ===== Thread-per-client-nb The thread-per-client-nb server is identical to the thread-per-client server @@ -150,14 +142,12 @@ except that is uses non-blocking sockets instead of blocking sockets. We implemented this version to investigate the possible performance difference between the synchronous and asynchronous socket calls. -[float] ===== Thread-per-operation The thread-per-operation server uses and event loop to watch all sockets for activity and when requests are available a thread is spawned and the request is handled completely from within that thread. When the request is complete the thread is destroyed. Blocking sockets are used in this implementation. -[float] ===== Thead-pool The thread-pool server uses and event loop to watch all sockets for activity. When requests are available, the event loop puts the request on a queue and @@ -165,7 +155,6 @@ wakes up a thread from the thread pool. The request is handled completely from within a single thread of the thread pool. Blocking sockets are used in this implementation. -[float] ===== Event The event server uses a execution context to handle all clients and requests. The event loop watches all sockets and handles each request in a callback. The @@ -173,6 +162,7 @@ event server uses non-blocking sockets and asynchronous file I/O. Note that the though the operating system can still use multiple cores to drive the network and disk. +[[runtime-perf]] == Runtime Performance Evaluation and Analysis The evaluation of runtime performance was done by executing a series of tests @@ -225,7 +215,6 @@ nodes instead of the server. The clients were scaled up as follows: 16, 128, 256 The clients connected to the server using the IB network with IPoIB. -[float] ===== Read The read test had clients each issue 16 requests asking for 4 KiB from @@ -239,7 +228,6 @@ client in the local storage of the server. mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 16 --size 4096 --type-r ********************* -[float] ===== Write The write test had clients each issue 16 requests sending 4 KiB of data to @@ -252,7 +240,6 @@ containing all the files is deleted between each test iteration. mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 16 --size 4096 --type-w ********************* -[float] ===== Read-Null The read-null test had clients each issue 4096 requests requesting 4 KiB of @@ -263,7 +250,6 @@ data to be returned from the server. mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 4096 --size 4096 --type-rn ********************* -[float] ===== Write-Null The write-null test had clients each issue 4096 requests sending 4 KiB of @@ -316,18 +302,104 @@ Absolute performance is not the only metric that one might look at so here the fairness of the various server types is examined. The bars on the graphs shows the total time for a client to complete all of its requests. The fastest client is the bottom of the bar and the slowest is the top of bar. - -//.Fastest and Slowest Total Client Runtime for Read Test -//[[fig-readtime]] -//image::fig/read-time.png[] - -//.Fastest and Slowest Total Client Runtime for Write Test -//[[fig-writetime]] -//image::fig/write-time.png[] - -=== Runtime Analysis - -Aesop compares favorably to other concurrency models but will of course lose +These results are from the 1024 client size using the same iteration that +was selected for the maximum runtime performance above. + +In <<fig-readtime>> and <<fig-writetime>> we see that the event server is +the only fair server. During our initial experimentation, aesop was configured +to be a fair system, however this fairness degraded the overall total +performance so we instead configured the file resource to be unfair by design. +This is demostrated in the <<fig-readtime>> and <<fig-writetime>>, showing +that aesop is extremely unfair. Thread-pool is also unfair because we applied +the same change to this server implementation. + +.Fastest and Slowest Total Client Runtime for Read Test +[[fig-readtime]] +image::fig/read-time.png[] + +.Fastest and Slowest Total Client Runtime for Write Test +[[fig-writetime]] +image::fig/write-time.png[] + +The <<fig-readnulltime>> and <<fig-writenulltime>> show similar fairness between +all server types except thread-pool in the case of the write-null test case. +Here the aesop network resource is not using an unfair design and approximates +the fairness of the other thread-per-* models. The thread-pool server does +have the design such that is is unfair in how it services requests, +<<fig-writenulltime>> shows this clearly. + +.Fastest and Slowest Total Client Runtime for Read-Null Test +[[fig-readnulltime]] +image::fig/read-null-time.png[] + +.Fastest and Slowest Total Client Runtime for Write-Null Test +[[fig-writenulltime]] +image::fig/write-null-time.png[] + +==== Latency + +The last client metric is an examination of per operation latency. The +client collects all the operation latency and the computes the minimum and +maximum latency, the first quartile latency and third quartile latency. These +metrics are graphed using a box and whiskers plot. The box represents the +first and thrid quartiles and the whiskers are the minimum and maximum +values. The following results are for the 1024 client size selected from +the same iteration as the maximum runtime graphs. <<fig-readlat>> and +<<fig-writelat>> show the aesop offers vary comparitve latency performance +as the other configurations and only noteably thread-per-op and event are +signficantly worse. + +.Per Operation Latency Read Test (Min/Max/First Quartile/Third Quartile) +[[fig-readlat]] +image::fig/read-lat.png[] + +.Per Operation Latency Write Test (Min/Max/First Quartile/Third Quartile) +[[fig-writelat]] +image::fig/write-lat.png[] + +In <<fig-readnulllat>> and <<fig-writenulllat>> we see again that aesop has +very good latency metrics compared to the other servers. In this case the +thread-per-op and event are noticeably worse than the other server +implementations. + +.Per Operation Latency Read-Null Test (Min/Max/First Quartile/Third Quartile) +[[fig-readnulllat]] +image::fig/read-null-lat.png[] + +.Per Operation Latency Write-Null Test (Min/Max/First Quartile/Third Quartile) +[[fig-writenulllat]] +image::fig/write-null-lat.png[] + +==== Summary + +The runtime analysis shows that aesop is very competitive with the most +optimal alternate server implementations at the 1024 client size. +In the _read_ test, aesop is 4.5% slower that the fastest implementation, +threadpool. Aesop was the fastest implementation in the _write_ test. +The thread-per-client implementation was fastest in the _read null_ and +_write null_ tests. Here aesop is significantly slower at around 25-30%, +however, as we showed above, the difference in performance was due to the +native performance difference in synchronous and asynchronous sockets. +If we look at the performance comparison to the thread-per-client-nb, aesop +was only 4.7% slower in the _read null_ test and was faster in the _write null_ +test. If aesop is using native asynchronous transport such as Infiniband verbs +or SSM, it will perform as well as a thread-per-client implementation. + +Another interesting factor was that their was no one fastest server +implementation. Each workload presents a different challenge and building +a single tuned server is difficult. A key advantage of aesop is the ability +to change the underyling implementation/turning for resources or the aesop +runtime without changing the aesop server source. During our investigation +we experimented with different underlying thread models for the network and +file resources of aesop. The aesop resources can be configured at runtime +to use the different thread models or tuning options. With the knowledge +that building a specific server implementation that is optimal for a generic +workload, aesop becomes powerful because these types of changes can be made +without ever changing the server source. + +=== Analysis + +As we've seen aesop compares favorably to other concurrency models but loses some performance compared to best case hand tuned version. Here we examine the overheads associated with aesop. The first item to examine is the cost associated with an aesop blocking call. @@ -339,9 +411,6 @@ blocking calls, when compared to a plain C function call, have extra overhead due to the way they are transformed by the aesop compiler. The following section highlights the sources of this overhead. -===== Understanding Blocking Call Overhead - -[float] ===== State Management Most of the overhead is caused by the need to preserve the state @@ -353,7 +422,6 @@ variables. As allocating heap memory is much more time consuming than allocating space on the stack, calling a blocking function is more expensive than calling a regular function. -[float] ===== Synchronization Overhead A second source of overhead originates from the multi-threaded nature of aesop @@ -385,12 +453,11 @@ Currently, the aesop compiler uses a combination of atomic operations and mutexes to maintain thread-safety. There is an ongoing effort to convert to atomic operations where possible. -[float] ===== Quantifying Blocking Call Overhead -For this test, a regular and a blocking function are called in a loop. -By timing the total time required to complete the loop, an estimate of the -time needed to execute the function call is obtained. +For this test, a regular C function and a aesop blocking function are +called in a loop. By timing the total time required to complete the loop, +an estimate of the time needed to execute the function call is obtained. The results were obtained on an intel i7 CPU running at 2.7GHz, using gcc 4.5.3 (using +-O2+), glibc 2.13-r4 and kernel 3.2.5. @@ -398,7 +465,7 @@ using gcc 4.5.3 (using +-O2+), glibc 2.13-r4 and kernel 3.2.5. There are a number of different test configurations: .Test Results -[width="20%",cols="h,^,^,^,^,^,<,<",valign="middle",frame="topbot",options="header"] +[cols="h,^,^,^,^,^,<,<",valign="middle",frame="topbot",options="header"] |===== 1.2+<.^| Test 5+| Options 2+^.^| seconds/call ^d| regular | blocking | malloc/free | mutex | opa | malloc | tcmalloc @@ -410,9 +477,8 @@ There are a number of different test configurations: |===== For test 1, a simple regular C function (i.e. not using `__blocking`) taking 2 -arguments is used. -Test 2 uses the same function, but this time the function is marked as -`__blocking`. +arguments is used. Test 2 uses the same function, but this time the +function is marked as `__blocking`. Tests 3-5 were added to provide a better context for understanding the magnitude of the blocking call overhead. For test 3, the function from test 1 @@ -423,17 +489,111 @@ a mutex. Test 5 replaces the mutex by a single atomic operation As a way to study the effect of the malloc implementation, these tests were also executed using a +tcmalloc+, an alternative memory allocator library. -The results for these are shown in the tcmalloc column. +The results for these are shown in the +tcmalloc+ column. -[TIP] +[NOTE] The progam used to obtain these results is in the repository: +tests/blocking-overhead.ae+. +== Runtime Memory Efficiency + +Another aspect of the overall performance is the memory efficiency of each +server implementation. We compare aesop to the other server implementations +as we did for the runtime performance. + +=== Experiment + +During the experiment detailed in the <<runtime-perf>> section. We recorded +the VmHWM stat from the server when the client test was completed. The VmHWM +stat is recorded by Linux during an applications runtime and represents the +peak resident set size (RSS). RSS represents the amount of paged-in memory. +If we looked at the VmPeak which is the total required virtual memory, this +would punish the models which use numerous threads. + +=== Evaluation + +The following graphs show the memory usage in KiB in log scale. In +<<fig-readmem>> and <<fig-writemem>> we see that thread-pool limits the +memory usage as the client work load increases because the thread-pool +by design limits the number of requests that can be in progress at once. The +other server implementations scale as the number of clients increase. + +<<fig-readnullmem>> and <<fig-writenullmem>> show a similar result as the +disk I/O tests. Aesop demonstrates that it is no worse then any of the thread +models. + +.Memory Usage Read Test +[[fig-readmem]] +image::fig/read-mem.png[] -=== Code Complexity +.Memory Usage Write Test +[[fig-writemem]] +image::fig/write-mem.png[] -As a measure for productivity, we investigated the code of each server using a -set of complexity metrics. +.Memory Usage Read-Null Test +[[fig-readnullmem]] +image::fig/read-null-mem.png[] + +.Memory Usage Write-Null Test +[[fig-writenullmem]] +image::fig/write-null-mem.png[] + +=== Analysis + +Aesop does not exhibit worse scaling in terms of memory usage than any of +the other thread-per models but it does add a cost in memory overhead for +each blocking call. + +=== Function arguments and stack variables of blocking calls + +Aesop, in order to implement the additional functionality provided by blocking +calls, rewrites blocking calls when translating the aesop code to C code. +This translation introduces a certain amount of overhead, both in memory usage +and execution performance. + +The main memory overhead incurred by blocking functions originates from the +need to protect the logical state of the function while temporarily switching +to other functions. + +For example, stack variables are moved to the heap. As long as the blocking +function does not complete, the memory for these variables is not released. +Arguments to the function need to be relocated to the heap as well, +and so does the type returned from the function (if not void). + +In a normal C program, the items listed above consume stack space. In blocking +functions, these consume heap space instead. In addition, aesop internally +maintains a number of control structures. Pointers to these structures are +passed as function arguments to the blocking function, and consequently +consume stack space. Currently, aesop adds about 4 pointers and 2 integers to +each blocking function call. + +=== Lonely pbranches + +A lonely pbranch will keep the enclosing scope alive (up to the function +scope) until the pbranch exits. + +.lonely pbranch scope +[source, C] +---- +__blocking int test (void) +{ + int var[10000]; + pbranch { + ... + } +} +---- + +So, in the example above, even though the test will return without waiting for +the pbranch to complete, its stack variables (`var` in this case) will +consume memory until the pbranch returns. + +== Productivity Performance + +The core design element of aesop is to make programming of a concurrent +server easier, so the trade off for memory and runtime performance should +be worth it. To evaluate this we examine the code complexity of each of +server implementations. .Implementation complexity analysis. [[table-complex]] @@ -448,24 +608,6 @@ set of complexity metrics. | event | 28 | 23 | 341 |============================ -////// -\begin{table} -\small -\begin{center} -\caption{Complexity analysis for example servers} -\begin{tabular}{lrrr} -\hline -& CC & mod. CC & SLOC \\ -\hline -\hline -\end{tabular} -\label{tab:complexity} -\end{center} -\normalsize -\vspace{-.2in} -\end{table} -///// - <<table-complex>> compares the code complexity of each server implementation using McCabe Cyclomatic Complexity (CC) <<McCabe>>, Modified McCabe Cyclomatic Complexity (Mod. CC), and Source Lines of Code @@ -484,8 +626,9 @@ provides a binding between Aesop and the standard POSIX socket API as part of its default functionality. The focus of this comparison is on the core logic defining the server implementation. -The Aesop and thread-per-client servers are very similar in terms of complexity. The -slight increase in complexity for the thread-per-client server results from +The Aesop and thread-per-client servers are very similar in terms of +complexity. The slight increase in complexity for the thread-per-client +server results from the additional function calls needed to create and join threads. The nonblocking version of the thread-per-client server (thread-per-client-nb) uses two additional lines of code to place each socket into non-blocking @@ -498,137 +641,246 @@ thread-per-client or aesop model. In the case of the thread-pool server, this additional complexity arises from not only the queueing and thread management logic, but also the event loop which is necessary to detect incoming requests and dispatch them to the queue. The event server -complexity arises from the necessity of dividing servicing routines into multiple -sub-functions and manually tracking state between those functions. -An additional complexity of the event +complexity arises from the necessity of dividing servicing routines +into multiple sub-functions and manually tracking state between +those functions. An additional complexity of the event model which is not captured by these metrics is the fact that control flow is not preserved across the processing of a given request. For example, servicing a write operation requires 5 disconnected event handlers. Although the event model appears less complex than the thread-pool model according to CC and Mod. CC, qualitatively it is significantly more challenging to develop. -=== Performance Evaluation - -We evaluated the performance of our server implementation. -The same client was used for all server implementations. - -.Performance writing to server memory. -[[fig-write]] -image::fig/write-hist.png["Write"] - -.Performance reading from server memory. -[[fig-read]] -image::fig/read-hist.png["Read"] - -.Performance writing to disk. -[[fig-write-null]] -image::fig/write-null-hist.png["Writing to disk"] - -.Performance reading from disk. -[[fig-read-null]] -image::fig/read-null-hist.png["Reading from disk"] - -.Memory usage during the write test -[[fig-write-mem]] -image::fig/write-mem.png["Write test server memory usage"] - -.Memory usage during the read test -[[fig-read-mem]] -image::fig/read-mem.png["Read test server memory usage"] - -.Memory usage during the write-null test -[[fig-write-mem]] -image::fig/write-null-mem.png["Write-null test server memory usage"] - -.Memory usage during the read-null test -[[fig-read-mem]] -image::fig/read-null-mem.png["Read-null test server memory usage"] - -== Runtime Memory Efficiency - -This section examines the memory efficiency of the translated aesop code. +== Compile Time Performance -=== Function arguments and stack variables of blocking calls +Currently aesop imposes some overhead when compiling aesop source. This is +demomnstrated in a simple micro-benchmark. The test takes an existing aesop +source fill with 4200 lines or source and is about 116 KB in size and compiles +it to an object file. The same source file is then renamed to a .c file and +four +#define+ are added which redefine the aesop keywords to nothing. -Aesop, in order to implement the additional functionality provided by blocking -calls, rewrites blocking calls when translating the aesop code to C code. -This translation introduces a certain amount of overhead, both in memory usage -and execution performance. This section focuses on memory overhead, -deferring the discussion of execution overhead to <<ref-blocking-runtime>>. +The test is executed on a dual processor Intel Xeon E5620 running at 2.4 GHz +with 24 GB of RAM. -The main memory overhead incurred by blocking functions originates from the -need to protect the logical state of the function while temporarily switching -to other functions. +.Compile Time Comparison +[[table-compile]] +[width="50%",cols="1,1>", options="header"] +|============================ +| Type | Time (seconds) +| aesop | 112.02 +| C | 1.22 +|============================ -For example, stack variables are moved to the heap. As long as the blocking -function does not complete, the memory for these variables is not released. -Arguments to the function need to be relocated to the heap as well, -and so does the type returned from the function (if not void). +The performance penalty is significant but this only effects development. It +can be mitigated by constructing a Makefile that supports parallel make. -In a normal C program, the items listed above consume stack space. In blocking -functions, these consume heap space instead. In addition, aesop internally -maintains a number of control structures. Pointers to these structures are -passed as function arguments to the blocking function, and consequently -consume stack space. Currently, aesop adds about 4 pointers and 2 integers to -each blocking function call. +== Bibliography -=== Lonely pbranches +[bibliography] +- [[[McCabe]]] McCabe, T.J. A Complexity Measure. In IEEE Transactions on + Software Engineering, vol.SE-2, no.4, pp. 308- 320, Dec. 1976. +- [[[Bame]]] Paul Bame. pmccabe. http://parisc-linux.org/~bame/pmccabe/ +- [[[Wheeler]]] David A. Wheeler. sloccount. http://www.dwheeler.com/sloc/ +- [[[libev]]] Marc Lehmann. http://software.schmorp.de/pkg/libev.html -A lonely pbranch will keep the enclosing scope alive (up to the function -scope) until the pbranch exits. +== Appendix A - Raw Data -.lonely pbranch scope -[source, C] ----- -__blocking int test (void) -{ - int var[10000]; - pbranch { - ... - } -} ----- - -So, in the example above, even though the test will return without waiting for -the pbranch to complete, it's stack variables (`var` in this case) will -consume memory until the pbranch returns. +.Read Runtime Data +[[table-readhist]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 0.794610 | 0.706736 | 0.775575 | 2.135912 | 0.861456 | 1.284919 +| 128 | 6.740836 | 3.755610 | 3.865248 | 3.395985 | 5.688868 | 9.602547 +| 256 | 12.352416 | 12.936459 | 13.051936 | 12.383918 | 11.434637 | 19.542018 +| 512 | 23.641337 | 27.395763 | 26.318174 | 24.776392 | 22.944439 | 39.058706 +| 1024 | 48.309275 | 54.852327 | 55.658511 | 47.564368 | 46.179671 | 77.216961 +|============================ +.Write Runtime Data +[[table-writehist]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +|clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 3.200944 | 2.306964 | 2.157996 | 1.934971 | 2.684861 | 2.512977 +| 128 | 18.213375 | 14.991499 | 15.802083 | 16.594684 | 18.065047 | 17.075809 +| 256 | 33.515213 | 32.899976 | 32.548540 | 32.662463 | 35.354696 | 33.900419 +| 512 | 67.640246 | 66.168558 | 66.324414 | 67.188763 | 71.184339 | 69.184007 +| 1024 | 119.799388 | 130.895169 | 130.829158 | 140.270790 | 141.208043 | 140.464092 +|============================ +.Read-Null Runtime Data +[[table-readnull]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1.122259 | 1.688565 | 1.845739 | 1.511863 | 1.220004 | 1.225304 +| 128 | 3.640566 | 4.558670 | 5.159914 | 9.884832 | 4.002515 | 7.252784 +| 256 | 7.087344 | 5.400444 | 6.604424 | 19.875843 | 7.448785 | 14.500385 +| 512 | 13.661447 | 10.603367 | 12.862459 | 40.884105 | 14.549549 | 28.967834 +| 1024 | 27.013576 | 21.625077 | 25.814349 | 82.675118 | 29.376645 | 50.822263 +|============================ -== Compile Time Performance +.Write-Null Runtime Data +[[table-writenull]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1.007547 | 0.932891 | 1.012248 | 1.561492 | 1.368119 | 1.372057 +| 128 | 4.844869 | 3.680892 | 4.771543 | 10.240339 | 16.074748 | 7.923767 +| 256 | 9.263294 | 6.997207 | 9.135328 | 20.342675 | 29.219365 | 15.994714 +| 512 | 18.293856 | 14.069904 | 18.382822 | 41.136291 | 59.712499 | 32.640966 +| 1024 | 37.081757 | 28.534744 | 39.618136 | 83.716303 | 116.833184 | 58.592679 +|============================ -=== Parallel Make +.Read Time Data +[[table-readtime]] +[cols="1>,1>,1>,1>", options="header"] +|============================ +| server | min | max | x +| aesop | 0.041914 | 48.309275 | 1.0 +| thread-per-client | 0.055891 | 54.852327 | 2.0 +| thread-per-client-nb | 0.067599 | 55.658511 | 3.0 +| thread-per-op | 0.667692 | 47.564368 | 4.0 +| thread-pool | 0.073988 | 46.179671 | 5.0 +| event | 63.155711 | 77.216961 | 6.0 +|============================ -It is possible to speed up the translation of aesop files by using the `-jn` -option to make, replacing `n` by the desired number of concurrent jobs. +.Write Time Data +[[table-writetime]] +[cols="1>,1>,1>,1>", options="header"] +|============================ +| server | min | max | x +| aesop | 0.415026 | 119.799388 | 1.0 +| thread-per-client | 67.926019 | 130.895169 | 2.0 +| thread-per-client-nb | 77.042924 | 130.829158 | 3.0 +| thread-per-op | 77.181582 | 140.270790 | 4.0 +| thread-pool | 0.915993 | 141.208043 | 5.0 +| event | 119.316831 | 140.464092 | 6.0 +|============================ -=== CCache +.Read-Null Time Data +[[table-readnulltime]] +[cols="1>,1>,1>,1>", options="header"] +|============================ +| server | min | max | x +| aesop | 16.165011 | 27.013576 | 1.0 +| thread-per-client | 9.041217 | 21.625077 | 2.0 +| thread-per-client-nb | 15.326102 | 25.814349 | 3.0 +| thread-per-op | 63.169438 | 82.675118 | 4.0 +| thread-pool | 20.796985 | 29.376645 | 5.0 +| event | 35.460415 | 50.154710 | 6.0 +|============================ -At this time, there are a number of issues blocking the use of ccache in -combination with the aesop source to source translator (either to cache the -translation or to cache the compilation of the generated C source code). +.Write-Null Time Data +[[table-writenulltime]] +[cols="1>,1>,1>,1>", options="header"] +|============================ +| server | min | max | x +| aesop | 26.075373 | 37.081757 | 1.0 +| thread-per-client | 20.131762 | 28.534744 | 2.0 +| thread-per-client-nb | 29.589322 | 39.618136 | 3.0 +| thread-per-op | 69.875107 | 83.716303 | 4.0 +| thread-pool | 11.849568 | 116.833184 | 5.0 +| event | 51.039087 | 58.592679 | 6.0 +|============================ -A first issue is related to incorrect handling of compiler names in the build -system, causing the build to fail if the compiler is set to `ccache gcc`. +.Read Latency Data +[[table-readlat]] +[cols="1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| server | first | min | max | third | x +| aesop | 0.000257 | 0.000251 | 0.000268 | 0.000266 | 1.0 +| thread-per-client | 0.000250 | 0.000242 | 0.000267 | 0.000265 | 2.0 +| thread-per-client-nb | 0.000249 | 0.000242 | 0.000265 | 0.000264 | 3.0 +| thread-per-op | 0.000462 | 0.000340 | 0.000495 | 0.000489 | 4.0 +| thread-pool | 0.000256 | 0.000239 | 0.000278 | 0.000275 | 5.0 +| event | 0.016989 | 0.006868 | 0.034917 | 0.028945 | 6.0 +|============================ -The second issue stems from the fact that aesop introduces additional -dependencies which are not understood by ccache. Therefore, subtly failures -would be introduced when aesop is updated and the cache is not manually -cleared. +.Write Latency Data +[[table-writelat]] +[cols="1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| server | first | min | max | third | x +| aesop | 0.003235 | 0.002687 | 0.004293 | 0.003931 | 1.0 +| thread-per-client | 0.005229 | 0.002843 | 0.008247 | 0.008217 | 2.0 +| thread-per-client-nb | 0.004103 | 0.001798 | 0.008219 | 0.008124 | 3.0 +| thread-per-op | 0.017350 | 0.011734 | 0.037548 | 0.030839 | 4.0 +| thread-pool | 0.003174 | 0.002525 | 0.004169 | 0.003877 | 5.0 +| event | 0.045139 | 0.011745 | 0.103348 | 0.103275 | 6.0 +|============================ -Given these issues, at this point it is not recommended to use ccache in -combination with aesop. However, both issues can be resolved in a later -aesop release (see ticket #137 and #200 in the triton repository). +.Read-Null Latency Data +[[table-readnulllat]] +[cols="1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| server | first | min | max | third | x +| aesop | 0.000066 | 0.000041 | 0.000083 | 0.000077 | 1.0 +| thread-per-client | 0.000062 | 0.000041 | 0.000074 | 0.000070 | 2.0 +| thread-per-client-nb | 0.000067 | 0.000044 | 0.000086 | 0.000082 | 3.0 +| thread-per-op | 0.000117 | 0.000062 | 0.000331 | 0.000257 | 4.0 +| thread-pool | 0.000078 | 0.000048 | 0.000102 | 0.000096 | 5.0 +| event | 0.000077 | 0.000044 | 0.000135 | 0.000116 | 6.0 +|============================ +.Write-Null Latency Data +[[table-writenulllat]] +[cols="1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| server | first | min | max | third | x +| aesop | 0.000070 | 0.000040 | 0.000101 | 0.000091 | 1.0 +| thread-per-client | 0.000065 | 0.000042 | 0.000079 | 0.000076 | 2.0 +| thread-per-client-nb | 0.000064 | 0.000042 | 0.000081 | 0.000077 | 3.0 +| thread-per-op | 0.000110 | 0.000062 | 0.000379 | 0.000302 | 4.0 +| thread-pool | 0.000049 | 0.000036 | 0.000053 | 0.000052 | 5.0 +| event | 0.000074 | 0.000046 | 0.000184 | 0.000133 | 6.0 +|============================ +.Read Memory Data +[[table-readmem]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1720 | 840 | 864 | 928 | 860 | 984 +| 128 | 2720 | 2748 | 2600 | 2832 | 1040 | 2988 +| 256 | 4700 | 5256 | 5064 | 5076 | 1160 | 5520 +| 512 | 5284 | 9316 | 9564 | 9352 | 1344 | 8428 +| 1024 | 10276 | 18216 | 18668 | 18288 | 1656 | 19248 +|============================ +.Write Memory Data +[[table-writemem]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1692 | 844 | 864 | 944 | 876 | 956 +| 128 | 3152 | 2912 | 2852 | 2932 | 1044 | 2080 +| 256 | 3912 | 5436 | 5152 | 5104 | 1192 | 3292 +| 512 | 6764 | 9844 | 9960 | 9760 | 1316 | 5604 +| 1024 | 14700 | 18648 | 18940 | 18920 | 1652 | 19284 +|============================ -== Bibliography +.Read-Null Memory Data +[[table-readnullmem]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1580 | 836 | 844 | 864 | 920 | 640 +| 128 | 2060 | 1972 | 1788 | 1000 | 1104 | 696 +| 256 | 2244 | 2992 | 2868 | 1000 | 1212 | 768 +| 512 | 2868 | 5200 | 5004 | 1124 | 1388 | 904 +| 1024 | 3900 | 9752 | 9292 | 4948 | 1744 | 1128 +|============================ -[bibliography] -- [[[McCabe]]] McCabe, T.J. A Complexity Measure. In IEEE Transactions on - Software Engineering, vol.SE-2, no.4, pp. 308- 320, Dec. 1976. -- [[[Bame]]] Paul Bame. pmccabe. http://parisc-linux.org/~bame/pmccabe/ -- [[[Wheeler]]] David A. Wheeler. sloccount. http://www.dwheeler.com/sloc/ -- [[[libev]]] Marc Lehmann. http://software.schmorp.de/pkg/libev.html +.Write-Null Memory Data +[[table-writenullmem]] +[cols="1>,1>,1>,1>,1>,1>,1>", options="header"] +|============================ +| clients | aesop | thread | thread-nb | thread-per-op | thread-pool | event +| 16 | 1600 | 856 | 816 | 852 | 916 | 680 +| 128 | 2876 | 2632 | 2652 | 1728 | 1144 | 744 +| 256 | 4588 | 5176 | 4904 | 1036 | 1248 | 808 +| 512 | 8036 | 9728 | 9604 | 7184 | 1436 | 1336 +| 1024 | 14084 | 19060 | 18764 | 11068 | 1808 | 5632 +|============================ diff --git a/doc/fig/read-hist.png b/doc/fig/read-hist.png index cf14290..a1adbfb 100644 Binary files a/doc/fig/read-hist.png and b/doc/fig/read-hist.png differ diff --git a/doc/fig/read-lat.png b/doc/fig/read-lat.png new file mode 100644 index 0000000..d1684eb Binary files /dev/null and b/doc/fig/read-lat.png differ diff --git a/doc/fig/read-mem.png b/doc/fig/read-mem.png index 9e76e8c..f698098 100644 Binary files a/doc/fig/read-mem.png and b/doc/fig/read-mem.png differ diff --git a/doc/fig/read-null-hist.png b/doc/fig/read-null-hist.png index 048f988..7a392e2 100644 Binary files a/doc/fig/read-null-hist.png and b/doc/fig/read-null-hist.png differ diff --git a/doc/fig/read-null-lat.png b/doc/fig/read-null-lat.png new file mode 100644 index 0000000..c74c5ca Binary files /dev/null and b/doc/fig/read-null-lat.png differ diff --git a/doc/fig/read-null-mem.png b/doc/fig/read-null-mem.png index 105f664..e1736b7 100644 Binary files a/doc/fig/read-null-mem.png and b/doc/fig/read-null-mem.png differ diff --git a/doc/fig/read-null-time.png b/doc/fig/read-null-time.png new file mode 100644 index 0000000..f137f44 Binary files /dev/null and b/doc/fig/read-null-time.png differ diff --git a/doc/fig/read-null.png b/doc/fig/read-null.png index 1f75427..6aacf17 100644 Binary files a/doc/fig/read-null.png and b/doc/fig/read-null.png differ diff --git a/doc/fig/read-time.png b/doc/fig/read-time.png new file mode 100644 index 0000000..e948032 Binary files /dev/null and b/doc/fig/read-time.png differ diff --git a/doc/fig/read.png b/doc/fig/read.png index bed1524..ccbd16c 100644 Binary files a/doc/fig/read.png and b/doc/fig/read.png differ diff --git a/doc/fig/write-hist.png b/doc/fig/write-hist.png index 2db69be..250a6af 100644 Binary files a/doc/fig/write-hist.png and b/doc/fig/write-hist.png differ diff --git a/doc/fig/write-lat.png b/doc/fig/write-lat.png new file mode 100644 index 0000000..8c3997f Binary files /dev/null and b/doc/fig/write-lat.png differ diff --git a/doc/fig/write-mem.png b/doc/fig/write-mem.png index 7947b19..69829cf 100644 Binary files a/doc/fig/write-mem.png and b/doc/fig/write-mem.png differ diff --git a/doc/fig/write-null-hist.png b/doc/fig/write-null-hist.png index e6bb934..4f50dae 100644 Binary files a/doc/fig/write-null-hist.png and b/doc/fig/write-null-hist.png differ diff --git a/doc/fig/write-null-lat.png b/doc/fig/write-null-lat.png new file mode 100644 index 0000000..8b4a776 Binary files /dev/null and b/doc/fig/write-null-lat.png differ diff --git a/doc/fig/write-null-mem.png b/doc/fig/write-null-mem.png index 137a995..a019e31 100644 Binary files a/doc/fig/write-null-mem.png and b/doc/fig/write-null-mem.png differ diff --git a/doc/fig/write-null-time.png b/doc/fig/write-null-time.png new file mode 100644 index 0000000..8c0f210 Binary files /dev/null and b/doc/fig/write-null-time.png differ diff --git a/doc/fig/write-null.png b/doc/fig/write-null.png index 4a1e53d..295bc3d 100644 Binary files a/doc/fig/write-null.png and b/doc/fig/write-null.png differ diff --git a/doc/fig/write-time.png b/doc/fig/write-time.png new file mode 100644 index 0000000..f6b66da Binary files /dev/null and b/doc/fig/write-time.png differ diff --git a/doc/fig/write.png b/doc/fig/write.png index e095fe0..c0494e7 100644 Binary files a/doc/fig/write.png and b/doc/fig/write.png differ hooks/post-receive -- aesop Repository

1 0

Re: [ASG] [EXTERNAL] PROM Review
by Ward, Lee 25 Feb '12

25 Feb '12

Hi Jen, Please find it, attached. I did not put it in a template, figured you had one you probably preferred and we are many institutions so none that we have is appropriate. See you Tuesday. Thanks! Let me know if there is something more we can do? --Lee On Feb 24, 2012, at 12:28 PM, Williams, Jennifer L. wrote: All, Just a reminder, I need to get your presentation by tomorrow. Thanks, Jen From: Williams, Jennifer L. Sent: Friday, February 10, 2012 6:38 PM To: lee(a)sandia.gov<mailto:[email protected]>; Rob Ross (rross(a)mcs.anl.gov<mailto:[email protected]>) Cc: stephen.w.poole(a)ugov.gov<mailto:[email protected]> Subject: PROM Review Hi Lee and Rob, Here is the logistics for the meeting on 02/28/2012: The HSM Storage/Filesystem task is set to be discussed at 1:15. 1 hour has been slated for this. The meeting is set to take place at Hotel at Arundel Preserve, 7795 Arundel Mills Boulevard • Hanover, MD 21076. Below is a link for the room block. https://secure.thehotelarundel.com/V1WebControls/Custom/index.aspx Group code: 303 Password: 676000274 I will also need to get your presentation material by 02/25/2012 to allow me enough time to prepare the materials (books for the sponsor) for the review. If you have any questions, please feel free to give me a call. Thanks, Jen Jennifer Williams Project Management Assistant/Project Controls High Performance Computing Program Oak Ridge National Laboratory Bldg 5002, Room 204 Office: (865) 574-2209 williamsjl(a)ornl.gov<mailto:[email protected]>

3 4

aesop Repository branch, master, updated. d74952042d90e5b4ef086b1e5c4e00c4419f713b
by noreply＠mcs.anl.gov 25 Feb '12

25 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via d74952042d90e5b4ef086b1e5c4e00c4419f713b (commit) from f34732449e03b2e06195b92896688d45d0843f71 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d74952042d90e5b4ef086b1e5c4e00c4419f713b Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Sat Feb 25 20:41:35 2012 -0600 Updates to performance document. ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 295 +++++++++++++++++++++++++-------------------- 1 files changed, 167 insertions(+), 128 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index 4b51e59..9dab84b 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -1,4 +1,4 @@ -= Aesop Performance Evaluation and Analysis += Aesop Performance Analysis == Introduction @@ -119,6 +119,7 @@ This insures that the variation in performance is primarily due to concurrency within the server. There is one exception to this which is the use of blocking or non-blocking sockets. This will be ellaborated on more in the analysis. All event loops are implemented using libev. <<libev>> +The thread implementations all use `pthread` threading library. ==== Server Implementations @@ -139,20 +140,22 @@ synchronous IO and a thread pool with 4 threads. The thread-per-client server spawns a thread for each client connection. All requests for the client are handled within the same thread. This model uses -blocking sockets. +blocking sockets. The thread remains alive until the client disconnects. [float] ===== Thread-per-client-nb The thread-per-client-nb server is identical to the thread-per-client server -except that is uses non-blocking sockets instead of blocking sockets. +except that is uses non-blocking sockets instead of blocking sockets. We +implemented this version to investigate the possible performance difference +between the synchronous and asynchronous socket calls. [float] ===== Thread-per-operation The thread-per-operation server uses and event loop to watch all sockets for activity and when requests are available a thread is spawned and the request -is handled completely from within that thread. Blocking sockets are used -in this implementation. +is handled completely from within that thread. When the request is complete +the thread is destroyed. Blocking sockets are used in this implementation. [float] ===== Thead-pool @@ -164,9 +167,11 @@ in this implementation. [float] ===== Event -The event server uses a single thread to handle all clients and requests. The -event loop watches all sockets and handles each request in a callback. The -event server uses non-blocking sockets and asynchronous file I/O. +The event server uses a execution context to handle all clients and requests. +The event loop watches all sockets and handles each request in a callback. The +event server uses non-blocking sockets and asynchronous file I/O. Note that +the though the operating system can still use multiple cores to drive the +network and disk. == Runtime Performance Evaluation and Analysis @@ -174,23 +179,169 @@ The evaluation of runtime performance was done by executing a series of tests using each server implementation type with the same client as discussed above. We then compare the results for each of the server implementations against aesop determine the overall effiecency of aesop compared to hand-tuned -solutions. +solutions. The four test types were selected to evaluate two models. The +first two test types (read,write) examine real disk I/O. Aesop is inteneded +for use in a storage server so we want to examine multiplexing of disk and +network I/O. The second set (read-null,write-null) eliminates disk I/O and +only has network I/O. This removes the storage bottleneck and examines how +well the different server types handle concurrency. === Experiment We ran our tests on the Argonne Fusion cluster which is a standard Linux HPC -cluster. The experiment was done +cluster. The experiment was done by running each client mode (read, write, +read-null, write-null) against all six server types. + +The testing was done by running a job on Fusion that executed one of the client +test types against all six server implementations scaling the clients from +16 to 1024. This kept the wall time of the job reasonable and allows a +consistent comparison between server types within a test type. The network +is still a shared resource on the cluster or each individual test was run +five times and the results below are the median from the five iterations, +based on the maximum runtime. + +==== Experiment System + +Fusion is a cluster run by the Argonne Laboratory Computing Resource Center +(LCRF). Fusion is a IBM iDataPlex dx360 M2 system. It features 320 compute +nodes which consist of two Intel Nehalem 2.6 GH Xeon processors and 36 GB +of RAM. The compute nodes have hyper threading disabled. The cluster has +an Infiniband QDR interconnect. Each compute node also a single SATA 7200 RPM +hard disk for local scratch storage. + +==== Experiment Details + +The experiment consisted of four separate jobs on Fusion. The top level unit +is the test type: read, write, read-null, write-null. For each test type, +the number of clients are scaled up from 16 to 1024. At each client scale, +each server implementation is tested in sequence. The individual test is +executed five times. On each iteration, the server is started then the MPI +client program is started. The server is restarted for each iteration so +that the memory statistics are not polluted by a previous run. + +On fusion we determined we could use 16 clients per physical node. Using +more clients per node caused the bottleneck of the test to become the client +nodes instead of the server. The clients were scaled up as follows: 16, 128, 256, 512 and 1024 clients. + +The clients connected to the server using the IB network with IPoIB. + +[float] +===== Read + +The read test had clients each issue 16 requests asking for 4 KiB from +disk. Each client specifies a unique file to be read on on each request. All +clients specifiy unique files. The files are first generated by a script +that runs before the read test starts. The script generates files for every +client in the local storage of the server. + +.Execution Parameters +********************* +mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 16 --size 4096 --type-r +********************* + +[float] +===== Write + +The write test had clients each issue 16 requests sending 4 KiB of data to +be written to disk. Each client specifies a unique file name for each request +and all clients specifiy unique files from each other. The directory +containing all the files is deleted between each test iteration. + +.Execution Parameters +********************* +mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 16 --size 4096 --type-w +********************* + +[float] +===== Read-Null + +The read-null test had clients each issue 4096 requests requesting 4 KiB of +data to be returned from the server. + +.Execution Parameters +********************* +mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 4096 --size 4096 --type-rn +********************* + +[float] +===== Write-Null + +The write-null test had clients each issue 4096 requests sending 4 KiB of +data to be discarded by the server. -[[ref-blocking-runtime]] -=== Performance Implications of Blocking Calls +.Execution Parameters +********************* +mpirun -np <procs> echo-client --ip <ip> --port 9999 --path <path> --num-requests 4096 --size 4096 --type-wn +********************* + +=== Evaluation + +Here the runtime results are presented from the experimentation. All graphs +are shown in log scale. + +==== Disk I/O + +In <<fig-readhist>> and <<fig-writehist>>, we see that aesop fairs +favorably to the other server implementations. In general the various +servers show similar scaling. + +.Runtime Performance for Read Test +[[fig-readhist]] +image::fig/read-hist.png[] + +.Runtime Performance for Write Test +[[fig-writehist]] +image::fig/write-hist.png[] + +==== Network I/O + +In this set of tests, we had specifically developed the thread-per-client-nb +to evalute the performance difference between blocking and non-blocking sockets.It is clear that the non-blocking sockets do not perform as well blocking +sockets. We did this test because aesop uses non-blocking sockets and were +trying to determine the performance difference between aesop and +the thread-per-client server. <<fig-readnull>> and <<fig-writenull>> cases show +that aesop mirrors the thread-per-client-nb server as expected. + +.Runtime Performance for Read-Null Test +[[fig-readnull]] +image::fig/read-null-hist.png[] + +.Runtime Performance for Write-Null Test +[[fig-writenull]] +image::fig/write-null-hist.png[] + +==== Fairness + +Absolute performance is not the only metric that one might look at so here +the fairness of the various server types is examined. The bars on the +graphs shows the total time for a client to complete all of its requests. +The fastest client is the bottom of the bar and the slowest is the top of bar. + +//.Fastest and Slowest Total Client Runtime for Read Test +//[[fig-readtime]] +//image::fig/read-time.png[] + +//.Fastest and Slowest Total Client Runtime for Write Test +//[[fig-writetime]] +//image::fig/write-time.png[] + +=== Runtime Analysis + +Aesop compares favorably to other concurrency models but will of course lose +some performance compared to best case hand tuned version. Here we examine +the overheads associated with aesop. The first item to examine is the +cost associated with an aesop blocking call. + +==== Performance Implications of Blocking Calls While this isn't immediately visible from looking at the aesop source code, blocking calls, when compared to a plain C function call, have extra overhead due to the way they are transformed by the aesop compiler. The following section highlights the sources of this overhead. -==== Understanding Blocking Call Overhead +===== Understanding Blocking Call Overhead +[float] ===== State Management Most of the overhead is caused by the need to preserve the state @@ -202,8 +353,7 @@ variables. As allocating heap memory is much more time consuming than allocating space on the stack, calling a blocking function is more expensive than calling a regular function. - - +[float] ===== Synchronization Overhead A second source of overhead originates from the multi-threaded nature of aesop @@ -230,15 +380,13 @@ pwait { } ---- - [NOTE] Currently, the aesop compiler uses a combination of atomic operations and mutexes to maintain thread-safety. There is an ongoing effort to convert to atomic operations where possible. - -==== Quantifying Blocking Call Overhead - +[float] +===== Quantifying Blocking Call Overhead For this test, a regular and a blocking function are called in a loop. By timing the total time required to complete the loop, an estimate of the @@ -282,115 +430,6 @@ The progam used to obtain these results is in the repository: +tests/blocking-overhead.ae+. - -//========================================================================= -== Case study: Implementing a small network server -//========================================================================= - -While micro-benchmarks can be useful, they often fail to capture the -complexity found in real applications. - -To provide a higher level evaluation, we compared the performance of a simple -network server programmed in aesop to that of the same server implemented in -C. We also quantified the code complexity of the aesop server, compared to the -different C versions. - -=== Server Description - -The example server listens on a TCP socket for incoming client connections. -Once a client connects, the server waits until a request is received or until -the client closes the connection. - -The server recognizes four different request types: - -*READ and WRITE*:: The server reads or writes a file specified by the client. - -*READ-NULL and WRITE-NULL*:: The same as READ and WRITE respectively, except -that the file read or write operations are omitted. The requested data -is still transferred over the network. - -The read-null and write-null cases attempt to determine the maximum network -bandwidth the server can sustain, by ensuring the disk operations are not a -bottleneck. -==== Implementation Details - -The server as described above was implemented in 5 different ways. - -===== Explicit Threading - -The threaded server uses manual thread management to explicitly created or -destroy a thread in response to an incoming connection or request. -We used the `pthread` threading library. -For the threaded implementation, we distinguish between 4 different -variations. - - -[width="80%",cols=">.^1h,4",frame="none",grid="none"] -|========================================================= -| thread-per-client | -A thread is created when a client connects, and this -thread is dedicated to the connection. All requests from this connection will -be handled by the same thread. - -| | - - -| thread-per-client-nb | -The same as above, but in this case the thread calls -the _asynchronous_ versions of the `read` and `write` system calls. By -including this option, performance differences related to the asynchronous -nature of the system calls are highlighted. - -| | - -| thread-per-op | -In this mode, a thread is created for every incoming -request. After the request is completed, the thread is destroyed. - -| | - -|thread-pool | -As above, but when a thread has finished executing a request -it is returned to a pool and reused when a new request arrives. - -|============================================================= - -===== Explicit Event Handling - -For this version, _libev_ is used to implement an event-driven server. -At any given time, a number of different events can occur, the event loop -waits for one of the following events to occur: - -[horizontal] -*Accept*:: An new client connected to the server. The server will start an -attempt to read a request from the connection. -*Read*:: A read from a connection completed. -*Write*:: A write operation (writing data to a client) completed. - -In response to one of these events, the next step in handling the connection -will be started using an asynchronous call before going back to the main event -loop to wait for another event to occur. - -Note that this server implementation does not use create any threads, and only -utilizes a single core (though the operating system can still use multiple -cores to drive the network and disk). - -==== Aesop - -The aesop version, from a code point of view, most closely resembles the -_thread_per_client_ code. However, after translation, the resulting C code can -support the event model as well as a threaded model. - -The actual result will depend on the actual resource implementation. -(For more information about resources, see the aesop user guide). -It is important to point out that the choice between an event driven or -threaded approach is limited to the resource implementation, and that no -changes to the actual server code are required. - -The aesop code uses a lonely pbranch when a client connects. -The code within the lonely pbranch is a direct implementation of the flow -chart provided with the server description. - === Code Complexity As a measure for productivity, we investigated the code of each server using a hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. f34732449e03b2e06195b92896688d45d0843f71
by noreply＠mcs.anl.gov 25 Feb '12

25 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via f34732449e03b2e06195b92896688d45d0843f71 (commit) from 532859b80da020901b9df4f2a2b6ad918f279a62 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit f34732449e03b2e06195b92896688d45d0843f71 Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Sat Feb 25 16:37:57 2012 -0600 In-progress updates to performance doc. ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 352 ++++++++++++++++++++++++++++++--------------- 1 files changed, 234 insertions(+), 118 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index 96e7845..4b51e59 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -1,99 +1,185 @@ - -Aesop Performance Tuning Guide -============================== += Aesop Performance Evaluation and Analysis == Introduction -This document describes a number of tips and techniques to speed up code -development using aesop. This includes the time required to execute the source -to source translator as well as the runtime efficiency of the generated code. +We examine the performance of Aesop based on several factors including +runtime performance, memory effieceny, programmer productivity and +compile time performance. We evaluated aesop by building a simple TCP server +which would recieve requests from clients and then either recieve or send +data which was either generated, discarded, read from a file or written to +a file. This server design was then implemented in aesop and then in generic +C code using other concurrency models for comparison. The description of +the test server and client are below and these serve as the basis for the +evaluations in the rest of the text. We also explore some micro-benchmarks +to analyze how core features of aesop impact performance. -//========================================================================= -== Translation Performance -//========================================================================= +=== Client Design -=== Parallel Make +The client is a basic C program that uses TCP sockets to send messages to +the server. The client will run in a loop generating a specifed number of +operations to the server. The general flow is that client send a request +to the server that contains and optional payload. The client then waits for +the server to send an acknowledgement that also contain an optional payload. -It is possible to speed up the translation of aesop files by using the `-jn` -option to make, replacing `n` by the desired number of concurrent jobs. +[float] +==== Request Types -=== CCache +The client supports the following request types. -At this time, there are a number of issues blocking the use of ccache in -combination with the aesop source to source translator (either to cache the -translation or to cache the compilation of the generated C source code). +[float] +===== Read -A first issue is related to incorrect handling of compiler names in the build -system, causing the build to fail if the compiler is set to `ccache gcc`. +The client sends a request with a file name and a size. The server will then +open the file, read the contents up to the size specified. The server returns +the data with the acknowledgement of the operation. -The second issue stems from the fact that aesop introduces additional -dependencies which are not understood by ccache. Therefore, subtly failures -would be introduced when aesop is updated and the cache is not manually -cleared. +[float] +===== Write -Given these issues, at this point it is not recommended to use ccache in -combination with aesop. However, both issues can be resolved in a later -aesop release (see ticket #137 and #200 in the triton repository). +The client sends a request with a file name, size and payload. The server will +then create the file and write the payload. The server then sends an +acknowledgement to client. +[float] +===== Read-Null -//========================================================================= -== Runtime Memory Efficiency -//========================================================================= +The client sends a request with a size to the server. The server then allocates +a buffer for the response based on the size the client requested. The server +then sends an acknowledgement with this buffer as the payload. -This section examines the memory efficiency of the translated aesop code. +[float] +===== Write-Null -=== Function arguments and stack variables of blocking calls +The client sends a request with a size and a payload. The server recieves +the request but then simply discards the payload. The server sends an +acknowledgement back to the client. -Aesop, in order to implement the additional functionality provided by blocking -calls, rewrites blocking calls when translating the aesop code to C code. -This translation introduces a certain amount of overhead, both in memory usage -and execution performance. This section focuses on memory overhead, -deferring the discussion of execution overhead to <<ref-blocking-runtime>>. +[float] +==== Implementation -The main memory overhead incurred by blocking functions originates from the -need to protect the logical state of the function while temporarily switching -to other functions. +The client has a set of command line parameters which control selecting the +test type, the number of requets and the size of the request. The client is +also a MPI program. This allows starting an arbitray number of clients and +then synchronizing startup. The clients all barrier until they are ready to +connect to the server. Clients exit the barrier, connect to the server +and begin sending requests in a loop. When the client completes it waits +at another barrier and then reports statistics about the run. -For example, stack variables are moved to the heap. As long as the blocking -function does not complete, the memory for these variables is not released. -Arguments to the function need to be relocated to the heap as well, -and so does the type returned from the function (if not void). +=== Server Design -In a normal C program, the items listed above consume stack space. In blocking -functions, these consume heap space instead. In addition, aesop internally -maintains a number of control structures. Pointers to these structures are -passed as function arguments to the blocking function, and consequently -consume stack space. Currently, aesop adds about 4 pointers and 2 integers to -each blocking function call. +The server supports the client request types listed above. Each server +implementation accepts client connections and then waits for requests. +.Server Request Handling +[graphviz] +-------- +digraph G +{ + subgraph I { + rank = same; + incoming [label="client connects", shape="box"]; + close [label="close connection"]; + } + wait [label="receive request"]; + incoming -> wait; + wait -> close [label="client closes connection"]; + wait -> read_1 [label = "READ"]; + wait -> readn_1 [label = "READ NULL"]; + wait -> write_1 [label = "WRITE"]; + wait -> writen_1 [label = "WRITE NULL"]; -=== Lonely pbranches + subgraph R { + read_1 [label="read from file"]; + read_2 [label="send data"]; + read_1 -> read_2 -> wait; + } -A lonely pbranch will keep the enclosing scope alive (up to the function -scope) until the pbranch exits. + subgraph W { + write_1 [label="receive data"]; + write_2 [label="write to file"]; + write_1 -> write_2 -> wait; + } -.lonely pbranch scope -[source, C] ----- -__blocking int test (void) -{ - int var[10000]; - pbranch { - ... + subgraph RN { + readn_1 [label="send data"]; + readn_1 -> wait; + } + + subgraph WN { + writen_1 [label="receive data"]; + writen_1 -> wait; } } ----- - -So, in the example above, even though the test will return without waiting for -the pbranch to complete, it's stack variables (`var` in this case) will -consume memory until the pbranch returns. +-------- - -//========================================================================= -== Runtime Execution Speed -//========================================================================= +The servers share a common base of code for handling the requests. +This insures that the variation in performance is primarily due +to concurrency within the server. There is one exception to this which is +the use of blocking or non-blocking sockets. This will be ellaborated on +more in the analysis. All event loops are implemented using libev. <<libev>> + +==== Server Implementations + +We implemented six server types including Aesop. Each server type examines +a different type of concurrency model. + +[float] +===== Aesop + +The Aesop server is done using Aesop of course. The server uses a +'lonely pbranch' to service each client. All operations for a client are +handled within a single pbranch. The underlying socket resource uses +non-blocking sockets with a thread pool of 16 threads. The file resource uses +synchronous IO and a thread pool with 4 threads. + +[float] +===== Thread-per-client + +The thread-per-client server spawns a thread for each client connection. All +requests for the client are handled within the same thread. This model uses +blocking sockets. + +[float] +===== Thread-per-client-nb + +The thread-per-client-nb server is identical to the thread-per-client server +except that is uses non-blocking sockets instead of blocking sockets. + +[float] +===== Thread-per-operation +The thread-per-operation server uses and event loop to watch all sockets for +activity and when requests are available a thread is spawned and the request +is handled completely from within that thread. Blocking sockets are used +in this implementation. + +[float] +===== Thead-pool +The thread-pool server uses and event loop to watch all sockets for activity. +When requests are available, the event loop puts the request on a queue and +wakes up a thread from the thread pool. The request is handled completely +from within a single thread of the thread pool. Blocking sockets are used +in this implementation. + +[float] +===== Event +The event server uses a single thread to handle all clients and requests. The +event loop watches all sockets and handles each request in a callback. The +event server uses non-blocking sockets and asynchronous file I/O. + +== Runtime Performance Evaluation and Analysis + +The evaluation of runtime performance was done by executing a series of tests +using each server implementation type with the same client as discussed above. +We then compare the results for each of the server implementations against +aesop determine the overall effiecency of aesop compared to hand-tuned +solutions. + +=== Experiment + +We ran our tests on the Argonne Fusion cluster which is a standard Linux HPC +cluster. The experiment was done [[ref-blocking-runtime]] === Performance Implications of Blocking Calls @@ -226,56 +312,6 @@ is still transferred over the network. The read-null and write-null cases attempt to determine the maximum network bandwidth the server can sustain, by ensuring the disk operations are not a bottleneck. - -.Server Request Handling -[graphviz] --------- -digraph G -{ - subgraph I { - rank = same; - incoming [label="client connects", shape="box"]; - close [label="close connection"]; - } - - wait [label="receive request"]; - - incoming -> wait; - wait -> close [label="client closes connection"]; - wait -> read_1 [label = "READ"]; - wait -> readn_1 [label = "READ NULL"]; - wait -> write_1 [label = "WRITE"]; - wait -> writen_1 [label = "WRITE NULL"]; - - - - subgraph R { - read_1 [label="read from file"]; - read_2 [label="send data"]; - read_1 -> read_2 -> wait; - } - - subgraph W { - write_1 [label="receive data"]; - write_2 [label="write to file"]; - write_1 -> write_2 -> wait; - } - - subgraph RN { - readn_1 [label="send data"]; - readn_1 -> wait; - } - - subgraph WN { - writen_1 [label="receive data"]; - writen_1 -> wait; - } - - -} - --------- - ==== Implementation Details The server as described above was implemented in 5 different ways. @@ -469,6 +505,85 @@ image::fig/write-null-mem.png["Write-null test server memory usage"] [[fig-read-mem]] image::fig/read-null-mem.png["Read-null test server memory usage"] +== Runtime Memory Efficiency + +This section examines the memory efficiency of the translated aesop code. + +=== Function arguments and stack variables of blocking calls + +Aesop, in order to implement the additional functionality provided by blocking +calls, rewrites blocking calls when translating the aesop code to C code. +This translation introduces a certain amount of overhead, both in memory usage +and execution performance. This section focuses on memory overhead, +deferring the discussion of execution overhead to <<ref-blocking-runtime>>. + +The main memory overhead incurred by blocking functions originates from the +need to protect the logical state of the function while temporarily switching +to other functions. + +For example, stack variables are moved to the heap. As long as the blocking +function does not complete, the memory for these variables is not released. +Arguments to the function need to be relocated to the heap as well, +and so does the type returned from the function (if not void). + +In a normal C program, the items listed above consume stack space. In blocking +functions, these consume heap space instead. In addition, aesop internally +maintains a number of control structures. Pointers to these structures are +passed as function arguments to the blocking function, and consequently +consume stack space. Currently, aesop adds about 4 pointers and 2 integers to +each blocking function call. + +=== Lonely pbranches + +A lonely pbranch will keep the enclosing scope alive (up to the function +scope) until the pbranch exits. + +.lonely pbranch scope +[source, C] +---- +__blocking int test (void) +{ + int var[10000]; + pbranch { + ... + } +} +---- + +So, in the example above, even though the test will return without waiting for +the pbranch to complete, it's stack variables (`var` in this case) will +consume memory until the pbranch returns. + + + + +== Compile Time Performance + +=== Parallel Make + +It is possible to speed up the translation of aesop files by using the `-jn` +option to make, replacing `n` by the desired number of concurrent jobs. + +=== CCache + +At this time, there are a number of issues blocking the use of ccache in +combination with the aesop source to source translator (either to cache the +translation or to cache the compilation of the generated C source code). + +A first issue is related to incorrect handling of compiler names in the build +system, causing the build to fail if the compiler is set to `ccache gcc`. + +The second issue stems from the fact that aesop introduces additional +dependencies which are not understood by ccache. Therefore, subtly failures +would be introduced when aesop is updated and the cache is not manually +cleared. + +Given these issues, at this point it is not recommended to use ccache in +combination with aesop. However, both issues can be resolved in a later +aesop release (see ticket #137 and #200 in the triton repository). + + + == Bibliography @@ -477,3 +592,4 @@ image::fig/read-null-mem.png["Read-null test server memory usage"] Software Engineering, vol.SE-2, no.4, pp. 308- 320, Dec. 1976. - [[[Bame]]] Paul Bame. pmccabe. http://parisc-linux.org/~bame/pmccabe/ - [[[Wheeler]]] David A. Wheeler. sloccount. http://www.dwheeler.com/sloc/ +- [[[libev]]] Marc Lehmann. http://software.schmorp.de/pkg/libev.html hooks/post-receive -- aesop Repository

1 0

Slides for our review next week
by Ward, Lee 25 Feb '12

25 Feb '12

All, Please find, attached, much of the slide deck I intend we use for our review next week. Please check it and give me comments? Please check it for your contribution to make sure that is what you really want to say and that I haven't screwed up? Note that the ANL and ORNL contributions are not yet present. I needed to do this deck before they could really contribute. We will be "tag-teaming" the presentation so they needed some idea of the flow before they could craft their parts. Note, also, there is no pretty template. I am required to use the Sandia template but that just cannot happen in this case, it's solely my presentation, it's all of us here. I suspect Joan, at ORNL, will end up sticking all of this in a template to the customer's liking. For everyone who decided they could go edge-to-edge-to-edge on their contribution, the outside borders will either be covered or your content grouped and ruthlessly shrunk to fit, making it nearly unreadable. You might want to rethink what you really want there and get me an update by COB tomorrow, 2/23. You'll note that this deck is too long, I'll keep working to make it smaller but in the end I may be racing through the slides. I won't have the time to linger long on any slide. A short, powerful, message will rule the day. --Lee

4 5

aesop Repository branch, master, updated. 532859b80da020901b9df4f2a2b6ad918f279a62
by noreply＠mcs.anl.gov 24 Feb '12

24 Feb '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 532859b80da020901b9df4f2a2b6ad918f279a62 (commit) from b1a353910dc7653896225a342edcf76a37bdab22 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 532859b80da020901b9df4f2a2b6ad918f279a62 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Feb 24 17:33:07 2012 -0500 fill in more text for complexity analysis ----------------------------------------------------------------------- Summary of changes: doc/aesop-performance.txt | 70 +++++++++++++++++++++++++++------------------ 1 files changed, 42 insertions(+), 28 deletions(-) Diff of changes: diff --git a/doc/aesop-performance.txt b/doc/aesop-performance.txt index 850df4a..96e7845 100644 --- a/doc/aesop-performance.txt +++ b/doc/aesop-performance.txt @@ -373,8 +373,6 @@ set of complexity metrics. | event | 28 | 23 | 341 |============================ -// @TODO: Need to update the table with other forms of threading - ////// \begin{table} \small @@ -393,32 +391,46 @@ set of complexity metrics. \end{table} ///// -<<table-complex>> compares the code complexity of each implementation using McCabe -Cyclomatic Complexity (CC) <<McCabe>>, Modified McCabe Cyclomatic Complexity -(Mod. CC), and Source Lines of Code (SLOC). - -//TODO: reference the -//metrics and the tools that we used to collect them (pmccabe and sloccount). - -To simplify the comparison, error handling was excluded for all servers except -for assertions on expected return codes. The protocol definition (ie, -request and acknowledgement structs) as well as helper functions to loop -over send and receive were not counted in any of the implementations, as these -were similar in all four. - -Aesop and thread are very similar in terms of complexity, with the slight -increase in the thread model due to function calls needed to create and join -threads. - -Thread pool and event model are both much more complex than the thread or -aesop model. An additional complexity of the event model which is not -captured by these metrics is the fact that control flow is not preserved -across the processing of a given request. For example, servicing a write -operation requires 5 disconnected event handlers. So although the event -model appears less complex according to CC and mod. CC, qualitatively it is -significantly more challenging to develop. - - +<<table-complex>> compares the code complexity of each server +implementation using McCabe Cyclomatic Complexity (CC) <<McCabe>>, +Modified McCabe Cyclomatic Complexity (Mod. CC), and Source Lines of Code +(SLOC). The CC and Mod. CC metrics were measured using the pmccabe tool, +version 2.6, created by Paul Bame <<Bame>>, while the SLOC metrics were +measured using the sloccount tool, version 2.26, +created by David A. Wheeler <<Wheeler>>. + +To simplify the comparison, error handling was excluded for all servers +except for assertions on expected return codes. The protocol definition +(ie, request and acknowledgement structs) as well as helper functions to +loop over send and receive were not counted in any of the implementations, +as these were similar in all four. Also note that the Aesop version +does not include the Aesop standard library, which +provides a binding between Aesop and the standard POSIX socket API as part +of its default functionality. The focus +of this comparison is on the core logic defining the server implementation. + +The Aesop and thread-per-client servers are very similar in terms of complexity. The +slight increase in complexity for the thread-per-client server results from +the additional function calls needed to create and join threads. The +nonblocking version of the thread-per-client server (thread-per-client-nb) +uses two additional lines of code to place each socket into non-blocking +mode. The remaining code logic needed to manage the nonblocking socket +calls is implemented in helper functions which are not included in the +analysis. + +The thread-pool and event models are both much more complex than the +thread-per-client or aesop model. In the case of the thread-pool server, +this additional complexity arises from not only the queueing and thread +management logic, but also the event loop which is necessary to detect +incoming requests and dispatch them to the queue. The event server +complexity arises from the necessity of dividing servicing routines into multiple +sub-functions and manually tracking state between those functions. +An additional complexity of the event +model which is not captured by these metrics is the fact that control flow +is not preserved across the processing of a given request. For example, +servicing a write operation requires 5 disconnected event handlers. +Although the event model appears less complex than the thread-pool model according to CC and +Mod. CC, qualitatively it is significantly more challenging to develop. === Performance Evaluation @@ -463,3 +475,5 @@ image::fig/read-null-mem.png["Read-null test server memory usage"] [bibliography] - [[[McCabe]]] McCabe, T.J. A Complexity Measure. In IEEE Transactions on Software Engineering, vol.SE-2, no.4, pp. 308- 320, Dec. 1976. +- [[[Bame]]] Paul Bame. pmccabe. http://parisc-linux.org/~bame/pmccabe/ +- [[[Wheeler]]] David A. Wheeler. sloccount. http://www.dwheeler.com/sloc/ hooks/post-receive -- aesop Repository

1 0