This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 67e973c40dde1c203a0ea63f989c543740e4fd53 (commit) from ff4af2f590c9c49cc21c9e89299f8cc26214d77e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 67e973c40dde1c203a0ea63f989c543740e4fd53 Author: Kevin Harms <[email protected]> Date: Wed Oct 31 11:29:52 2012 -0500 Add information on file size and object generation to simulation deliverable ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/figs/histogram-data.csv | 24 ++++ .../simulation-2012-10/figs/histogram-plot.pdf | Bin 0 -> 4780 bytes .../simulation-2012-10/simulation-2012-10.txt | 114 ++++++++++++++++++-- 3 files changed, 130 insertions(+), 8 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/figs/histogram-data.csv create mode 100644 code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv b/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv new file mode 100644 index 0000000..49e781b --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv @@ -0,0 +1,24 @@ +0,1023,1967524 +1024,2047,650490 +2048,4095,805129 +4096,8191,1381084 +8192,16383,583050 +16384,32767,358091 +32768,65535,241314 +65536,131071,154327 +131072,262143,101811 +262144,524287,51701 +524288,1048575,40943 +1048576,2097151,29999 +2097152,4194303,21557 +4194304,8388607,26912 +8388608,16777215,12352 +16777216,33554431,4677 +33554432,67108863,3402 +67108864,134217727,1504 +134217728,268435455,492 +268435456,536870911,352 +536870912,1073741823,195 +1073741824,2147483647,70 +2147483648,4294967295,73 +4294967296,34540134780,32 diff --git a/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf b/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf new file mode 100644 index 0000000..b5c2789 Binary files /dev/null and b/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf differ diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index a47e249..c7f587c 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -288,14 +288,112 @@ behavior of a rebuild algorithm, it is important that the distribution of these objects reflect the real-world distribution of data on production storage systems. We therefore based the distribution of objects on the characteristics of the GPFS file system on Intrepid, a 557 teraflop IBM Blue -Gene/P system operated by the Argonne Leadership Computing Facility... - -TODO: fill in the following material: - -* Information about Intrepid and its file system -* How data was captured from Intrepid, what date it was captured -* Summarize data (histogram? total files? total size?) -* Method used to generate representative objects based on Intrepid file distribution (including striping strategy) +Gene/P system operated by the Argonne Leadership Computing Facility. + +The distribution of objects is based on the /intrepid-fs0 scratch file system +of Intrepid. The /intrepid-fs0 file system has 4.5 PiB of space which is backed +by sixteen DDN 9900 S2A storage systems which are directly attached via DDR +InfiniBand to 128 file servers. There are eight file servers per each DDN +unit. The storage backend exports 1152 LUNs which in turn are represented +as 1152 GPFS NSDs. The /intrepid-fs0 file system has a 4 MiB block size. GPFS +does not officially disclose thier placement policy of blocks, but it is +roughly done in a round-robin fashion where blocks are evenly distributed +over all NSDs. GPFS takes into account space utilization and so will maintain +a generally equal space distribution amoung all NSDs. The general placement +will also select blocks from "relatively" different NSDs, meaning not in +sequential NSD order so that file system leverages the maximum number of +storage controllers and file servers for a given file. The /intrepid-fs0 +file system is also configured with the GPFS scatter allocation algorithm +which means file system blocks on disk are randomized, so all I/O is +essentially random. + +We collected data from the /intrepid-fs0 GPFS file system by using the +'mmapplypolicy' GPFS tool which will scan the system metadata and allow +collection of any metadata metrics. Using this data collection we can extract +the file size of every file on the system. The data we present here was +collected on June 25th, 2012. The scan shows we have a total of 4377.7 TiB +of data on the system contained within 411,292,738 files. We took the list +of file sizes and processed it using R. We computed a histogram of the +file sizes. The bin sizes we selected for the histogram are powers of 2. +This results in bins of: 2^0^, 2^10^, 2^11^, .., 2^32^, maximum file size. +We chose these bin sizes after experimenting with different possible sizes. +Selection of the bins is a bit difficult because there are many small files +and a few very large files. <<fs0-hist-data>> and <<fs0-hist-plot>> show +the histogram that was generated. + +[[fs0-hist-data]] +.Histogram Data +[format="csv",width="50%",cols=">,>,>",options="header"] +|======= +Bin Start,Bin End,Count +include::figs/histogram-data.csv[] +|======= + +[[fs0-hist-plot]] +.Histogram Plot +image::figs/histogram-plot.pdf[scaledwidth="75%"] + +==== Generation of Objects + +Given the histogram data above, we used this to generate a list of file +sizes for use in our simulated system since we did not want to simulate a +system with 400 million files at this time. The generation of objects involves +three steps. Step 1, generate the file size. Step 2, strip file over multiple +objects. Step 3, generate object IDs for each object. + +===== File Size Generation +We converted the histogram into a cumulative distribution function and then +generate a random floating point number between [0, 1). We then use this to +determine which bin to pull from. Example: + + bin1 = 0.3 + bin2 = 0.4 + bin3 = 0.43 + bin4 = 0.48 + x = 0.457 + + The random value of 0.457 selects bin3. + +For each bin, we use the midpoint of the bin to generate the file size. Based +on <<fs0-hist-data>> above, the first bin has the midpoint 512, so if the +first bin is selected, we generate a file size of 512 bytes. + +===== File Striping +We wanted to emulate the file striping of GPFS in some manner because it +would not be realistic to have extrememly large files only hosted as a +single object on one server. We went with a method similar to GPFS so +we used the 4 MiB file system block size to break files into multiple objects. +However, we did not want to create single 4 MiB objects because that would not +be how the object storage is used. We took any generated file size greater +than 4 MiB and chopped it into 4 MiB pieces. These peices then get assigned +to a "server" and chunks on the same server are combined. The examples +below illustrate this process. + + Example: + + # of servers = 4 + file size = 16 MiB + result: + server 1 = 4 MiB + server 2 = 4 MiB + server 3 = 4 MiB + server 4 = 4 MiB + + # of server = 4 + file size = 22 MiB + result: + server 1 = 8 MiB + server 2 = 6 MiB + server 3 = 4 MiB + server 4 = 4 MiB + +===== Object ID Generation +Once we generate one or more objects for a given file size, we randomly +generate an object ID for each object. This means that the group of objects +we generated for a file do not get placed how they would normally but this +still results in an even distribution of objects over the system with +representitive sizes. At this point we have a list of objects for the +system and we notify each server of its list of objects. [[sec-params]] === Representative system and parameters hooks/post-receive --