April 2013 - Triton-commits - mailman3.cels.anl.gov

branch, master, updated. c26ebeb94ea9535d3aaf51aeac79dd33942c0f94
by noreply＠mcs.anl.gov 19 Apr '13

19 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via c26ebeb94ea9535d3aaf51aeac79dd33942c0f94 (commit) from 55e5316dbd6ecb26a1507b5d03f52c113278e045 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c26ebeb94ea9535d3aaf51aeac79dd33942c0f94 Author: cengiz <cengiz.k(a)uconn.edu> Date: Fri Apr 19 15:04:14 2013 -0400 More changes to the related work and introduction ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 17 +++++++++++++++++ papers/asg-osd/paper.tex | 5 ++++- 2 files changed, 21 insertions(+), 1 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index 71f46b5..6e93a3d 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -601,3 +601,20 @@ title = {{Design of an Intelligent Object-based Storage device}}, title = {{Librados API documentation}}, url = {http://ceph.com/docs/master/api/librados/} } + +@book{Lovelace:2003:VD:1406022, + author = {Lovelace, Dave and Ayyar, Rama and Sala, Alvaro and Sokal, Valeria}, + title = {{Vsam Demystified}}, + year = {2003}, + isbn = {0-7384-5323-4}, + edition = {First}, + publisher = {IBM Corp.}, + address = {Riverton, NJ, USA}, +} + +@misc{russon2004ntfs, + title={NTFS documentation}, + author={Russon, Richard and Fledel, Yuval}, + journal={2009-06-13). http://www. scribd. com/doc/2187280/NTFS-Documentation}, + year={2004} +} diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 2882929..c84c1a8 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -845,7 +845,10 @@ Ohio Supercomputing Center looked at mapping Parallel Virtual File System~\cite{ existing object-based storage emulation~\cite{1306916, Devulapalli07integratingparallel}. This mapping moved the functionality of the common components of a traditional storage system; such as I/O, directory or metadata servers to OSDs and improved the performance of the overall system due to the capabilities of -the object-based storage devices ~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14}. +the object-based storage devices~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14}. +VSAM~\cite{Lovelace:2003:VD:1406022} supports both fixed and variable sized records depending on the +application. Forks in NTFS~\cite{russon2004ntfs} are similar to records in ASG storage model; they are +byte streams storing file data and auxiliary information; such as metadata and security settings. A number of studies form the technical basis of our work. Transactional Object Storage Device (TOSD) ~\cite{carns2010object} shows that object-based storage is a common component of many parallel file hooks/post-receive --

1 0

branch, master, updated. 55e5316dbd6ecb26a1507b5d03f52c113278e045
by noreply＠mcs.anl.gov 07 Apr '13

07 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 55e5316dbd6ecb26a1507b5d03f52c113278e045 (commit) from aabe2c73d336fc95e2a56956b19f4e4ee173a793 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 55e5316dbd6ecb26a1507b5d03f52c113278e045 Author: cengiz <cengiz.k(a)uconn.edu> Date: Sun Apr 7 15:47:12 2013 -0400 Done with most of the changes recommended in the meeting * Still need to modify related work and intro ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 10 +- papers/asg-osd/paper.tex | 268 ++++++++++++++++++++++++--------------------- papers/asg-osd/usenix.sty | 95 ---------------- 3 files changed, 148 insertions(+), 225 deletions(-) delete mode 100644 papers/asg-osd/usenix.sty Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index faac40c..71f46b5 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -584,11 +584,6 @@ title = {{Design of an Intelligent Object-based Storage device}}, address = {Berkeley, CA, USA}, } -@misc{librados-doc, - title = {{Librados API documentation}}, - url = {http://ceph.com/docs/master/api/librados/} -} - @inproceedings{weil:pdsw07, address = {Reno, NV}, author = {Weil, Sage A. and Leung, Andrew and Brandt, Scott A. and Maltzahn, Carlos}, @@ -601,3 +596,8 @@ title = {{Design of an Intelligent Object-based Storage device}}, title = {{RADOS: A Fast, Scalable, and Reliable Storage Service for Petabyte-scale Storage Clusters}}, year = {2007} } + +@misc{librados-doc, + title = {{Librados API documentation}}, + url = {http://ceph.com/docs/master/api/librados/} +} diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 9d6baa3..2882929 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -63,8 +63,8 @@ % should be used if it is desired that the figures are to be displayed in % draft mode. % -\documentclass[10pt, conference, compsocconf]{IEEEtran} -% Add the compsocconf option for Computer Society conferences. +\documentclass[conference]{IEEEtran} +% Add the compsoc option for Computer Society conferences. % % If IEEEtran.cls has not been installed into the LaTeX system files, % manually specify the path to it like: @@ -356,21 +356,24 @@ % author names and affiliations -% use a multiple column layout for up to two different +% use a multiple column layout for up to three different % affiliations - -%\author{\IEEEauthorblockN{Authors Name/s per 1st Affiliation (Author)} -%\IEEEauthorblockA{line 1 (of Affiliation): dept. name of organization\\ -%line 2: name of organization, acronyms acceptable\\ -%line 3: City, Country\\ -%line 4: Email: name(a)xyz.com} +%\author{\IEEEauthorblockN{Michael Shell} +%\IEEEauthorblockA{School of Electrical and\\Computer Engineering\\ +%Georgia Institute of Technology\\ +%Atlanta, Georgia 30332--0250\\ +%Email: http://www.michaelshell.org/contact.html} +%\and +%\IEEEauthorblockN{Homer Simpson} +%\IEEEauthorblockA{Twentieth Century Fox\\ +%Springfield, USA\\ +%Email: homer(a)thesimpsons.com} %\and -%\IEEEauthorblockN{Authors Name/s per 2nd Affiliation (Author)} -%\IEEEauthorblockA{line 1 (of Affiliation): dept. name of organization\\ -%line 2: name of organization, acronyms acceptable\\ -%line 3: City, Country\\ -%line 4: Email: name(a)xyz.com} -%} +%\IEEEauthorblockN{James Kirk\\ and Montgomery Scott} +%\IEEEauthorblockA{Starfleet Academy\\ +%San Francisco, California 96678-2391\\ +%Telephone: (800) 555--1212\\ +%Fax: (888) 555--1212}} % conference papers do not typically use \thanks and this command % is locked out in conference mode. If really needed, such as for @@ -408,6 +411,7 @@ \begin{abstract} +%\boldmath Distributed object-based storage models are an increasingly popular alternative to traditional block-based or file-based storage abstractions in large-scale storage systems. Object-based storage models store and access data @@ -416,21 +420,26 @@ easier data sharing, locality and better security compared to traditional storag Although many large-scale storage systems share common goals of performance, scalability and fault tolerance, their underlying storage models are typically tailored to specific classes of applications through incremental evolution of previous standards. -In this paper, we propose a new distributed object-based storage API -that seeks to combine data access features (i.e. concurrent reads\&writes, versioning, +In this paper, we propose a new distributed object-based storage model, ASG, +that seeks to combine data access features (i.e. concurrent reads and writes, versioning, conditional operations, failure handling) into a unified API that is suitable for use as a base model in a wide range of contexts, including file systems, cloud storage, databases, HPC systems and big data. We identify the common requirements of these data models, identify key -features of previous object-based storage interfaces, and explore how they can be unified effectively +features of previous object-based storage models and explore how they can be unified effectively without sacrificing usability. We validate the effectiveness of our approach with case studies -that evaluate how the storage API we are proposing can be used to construct a namespace and a -column-store database. +that evaluate how ASG storage model can be used to construct a namespace and a +column-store database and to support concurrent write accesses. \end{abstract} +% IEEEtran.cls defaults to using nonbold math in the Abstract. +% This preserves the distinction between vectors and scalars. However, +% if the conference you are submitting to favors bold math in the abstract, +% then you can use LaTeX's standard command \boldmath at the very start +% of the abstract to achieve this. Many IEEE journals/conferences frown on +% math in the abstract anyway. + +% no keywords -%\begin{IEEEkeywords} -%component; formatting; style; styling; -%\end{IEEEkeywords} % For peer review papers, you can put extra information on the cover @@ -453,41 +462,42 @@ Modern applications demand high-performance, security and portability from the s computers and this requires the storage units to redefine the abstraction to store and manage data. In order to meet these demands, a new storage model called \textit{object-based storage} has been developed ~\cite{1222722, osd-3}. Storage devices that store and access data according to the object-based storage -model are called as \textit{object-based storage devices}. Storage system developers prefer to build -software implementations of object-based storage model on block-based devices; since there is no hardware -implementation of an object-based storage device yet. +model are called as \textit{object-based storage devices}. %\textit{properties of the object-based storage model} An \textit{object} is an ordered logical collection of bytes with a numerical identifier. Objects consist -of data, user settable attributes and device managed metadata including security policies that protect data +of data, attributes and device managed metadata including security policies that protect data ~\cite{1222722, Welch08scalableperformance}. They have variable sizes and they can be used to store any -kind of data in the storage system. Having data organized as objects makes it possible for a -disk to do data management operations on behalf of a user~\cite{5944457} and offloads the storage -management task from the host operating system to the object-based storage. User-settable attributes can -be further enriched to set performance goals, manage replicas or handle concurrency in object-based -storage model to enable self-management capabilities and they give the user an effective control of data -on the disk~\cite{gibson:storage, 5944457}. Object-based storage is also suitable for realizing -\textit{active storage}~\cite{5944457,Acharya:1998:ADP:291006.291026, riedel:thesis}; because objects -provide more features to encapsulate the computations compared to traditional blocks. +kind of data in the storage system. Having data organized as objects enables offloading storage +management tasks; such as block remapping and space management from the host operating system to the +object-based storage. Object attributes can be further enriched to set performance goals, manage replicas +or handle concurrency in object-based storage model to enable self-management capabilities and they give +the user an effective control of data~\cite{gibson:storage, 5944457}. Object-based storage is +also suitable for realizing \textit{active storage}~\cite{5944457,Acharya:1998:ADP:291006.291026, +riedel:thesis}; because objects provide more features to encapsulate the computations compared to +traditional blocks. Although several object-based storage models have already been implemented and used as the basis for the popular storage and file systems~\cite{Weil06ceph:a, Beaver:2010:FNH:1924943.1924947, lustre-web}, these object-based storage models are often tailored to a particular use case or data model. %\textit{Requirements of common data models (file systems, cloud storage, databases and big data)} On the other hand, common data applications; such as file systems, cloud storage, databases and big data -require the underlying storage models to support various features; such as concurrency, reliability, +need the underlying storage models to meet specific requirements; such as concurrency, reliability, scalability and flexible data structures. %Although object-based storage model has been investigated in many research studies and several %optimizations have been introduced for this model, we believe that the object-based storage model %primitives need to be revisited. -In this work we propose a new object-based storage model, known as the ASG interface, that seeks to unify +In this work we propose a new object-based storage model, known as ASG, that seeks to unify the features necessary to support the data models outlined above without compromising usability or -limiting implementation flexibility. ASG interface intends to bring versioning, conditional operation, -fork\&record structure and unified data\&attribute features together. The rest of the paper is organized -as follows; Section 2 talks about related work in object-based storage systems. Section 3 describes the -ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the -advantages\&disadvantages of each model. We give sample usage scenarios of ASG model in Section 5 followed -by the conclusion and possible directions for future development of ASG in Section 6. +limiting implementation flexibility. Object-based storage model was developed to be implemented in +hardware originally~\cite{osd-3}; however storage system developers has preferred to implement object-based +storage as a software layer~\cite{1306916, Devulapalli07integratingparallel, librados-doc, lustre-web} and +ASG storage model follows this design decision too. ASG storage model brings versioning, conditional +operation, fork and record structure and unified data and attribute features together. The +rest of the paper is organized as follows; Section~\ref{primitives} describes the ASG architecture in +details and presents example use cases. Section~\ref{relatedwork} talks about related work in object-based +storage systems. We finish with conclusion and possible directions for future development of ASG in +Section~\ref{conclusion}. %\textit{Describe the sections in the remainder of the paper} %We believe an object-based storage system can offer a lot more sophisticated functionalities by @@ -495,66 +505,7 @@ by the conclusion and possible directions for future development of ASG in Secti %system model, ASG, that unifies the conditional operation and versioning features previously explored in %related studies described in the previous paragraph. -\section{Related Work} -Network-Attached Secure Disk (NASD)~\cite{gibson:storage} is the primary study on -object-based storage and it led to specifications of standards~\cite{osd-3} for object-based -storage. -%It is based on the idea of integrating the disk management functions into the storage -%device and it offers a variable-length object-storage interface. -NASD introduces variable-length objects with attributes, rather than fixed-length traditional blocks, to -enable self-management and to obviate the need to know about the host operating system. Moving data -management to the storage disks increases the networking, security and space management capabilities. - -There have been numerous studies in literature with similar scope to our work. -OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar -to the one specified by the OSD standard~\cite{osd-3} except for the addition of dedicated directory -objects. The directory objects in OSD+ store file names and attributes and support metadata related -operations. -%Each OSD+ stores a single directory object. -%In order to guarantee correctness and atomicity, OSD+ relies on the back-end file system (Ext3, Ext4) and -%three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. -Panasas File System~\cite{Welch08scalableperformance} is built on object-based storage devices. -%Panasas File System clients have direct access to the objects which are a combination of data and -%attributes. -OSD wire protocol of Panasas uses the operations from the OSD standard~\cite{osd-3} to enable byte-oriented -access to data, to manipulate attributes and to create or delete objects. -%Panasas clients use iSCSI/OSD protocol to access the object-based storage devices. -Lustre~\cite{lustre-web} is a distributed file system based on object-based storage. Object Storage -Server (OSS) in Lustre is responsible for providing access to file data stored in objects on Object -Storage Targets (OSTs). -%Lustre has two types of servers; Metadata Server (MDS) and Object Storage Server (OSS). Actual metadata is -%stored on Metadata Targets (MDTs) and file data is stored on Object Storage Targets (OSTs) as objects. MDS -%is responsible for handling metadata requests and making the metadata stored in MDTs available to the -%clients; whereas OSS is responsible for filesystem I/O and for providing access to file data stored on -%OSTs. -\textit{Ursa Minor} is a parallel file system that supports versioned writes. It keeps the existing -object-storage interface~\cite{osd-3} mostly intact except for introducing \textit{slices} which are -fragments of object data and it uses timestamps to distinguish different versions of data. -Datamods~\cite{watkinsdatamods} is a framework that exploits existing large-scale storage system services -to support complex data models and interfaces. -%Traditionally middleware is tailored to support specific data models; however many services provided by -%middleware already exist in distributed storage systems. -Datamods avoids duplicating services already provided in distributes storage systems in middleware and -improves scalability since it is not limited to single dimension at the file level. -RADOS~\cite{librados-doc, weil:pdsw07} is an object storage service forming the foundation of -Ceph~\cite{Weil06ceph:a}. It provides the clients a single logical object store and offloads object -replication, failure detection and data management tasks to the underlying object store daemons. -Ohio Supercomputing Center looked at mapping Parallel Virtual File System~\cite{pvfs-web} on top of an -existing object-based storage emulation~\cite{1306916, Devulapalli07integratingparallel}. This mapping -moved the functionality of the common components of a traditional storage system; such as I/O, directory -or metadata servers to OSDs and improved the performance of the overall system due to the capabilities of -the object-based storage devices ~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14}. - -A number of studies form the technical basis of our work. Transactional Object Storage Device (TOSD) -~\cite{carns2010object} shows that object-based storage is a common component of many parallel file -systems and it introduces three optimizations to the object-based storage model in order to serve highly -concurrent workloads better; atomicity, versioning and commutativity. Goodell et al.~\cite{goodell-pdsw12} -extended the POSIX API by organizing the storage around data objects in order to map complex data -structures to these data objects and have direct access between the data objects and applications. Carns -et al.~\cite{carns-pdsw12} investigated conditional update operations as an alternative to distributed -pessimistic locking operations in object-based storage systems. - -\section{Implementation Strategies} +\section{Proposed Model} \label{primitives} In this section we describe the ASG storage model, its fundamental building blocks and basic primitives. @@ -671,27 +622,12 @@ be retrieved. The retrieved information consists of the identifier of the first container/object/fork/record existing in the given entity, number of containers/objects/forks/records, total number of records and the version numbers of these records. -\section{Comparison with Existing Models} -Wisconson Storage System (WiSS) is a flexible storage system developed for the UNIX -environment~\cite{DBLP:journals/spe/ChouDKK85}. It supports unconventional types of data and it consists of -four distinct layers; \textit{level 0}, \textit{level 1}, -\textit{level 2} and \textit{level 3}. The basic structure units of WiSS are \textit{pages} at -\textit{level 0}. \textit{Level 0} performs I/O operations on pages. \textit{Level 1} is responsible for -maintaining a buffer pool of pages to make the access to the important pages faster. \textit{Level 2} of -WiSS carries some similarities to the ASG storage model. This level consists of \textit{records}, each of -which stored in an underlying page. A record consists of record type, record length and actual data. Record -length and actual data also exist in ASG records. Similar to the ASG storage model, each record is known by -a numerical identifier. Unlike ASG, records in WiSS have unique identifiers; whereas records in different -enclosing objects or forks etc. in ASG can have the same identifier. WiSS supports B-trees to index records -and long data items (a collection of pages) to store unlimited size of data; however in ASG there are no -such mechanisms yet. WiSS uses locking at the file granularity; whereas ASG supports versioning to provide -concurrency. \textit{Level 3} of WiSS has primitives to create or delete files using low-level primitives -which is similar to the Namespace Implementation we will discuss in Section~\ref{namespace}. - -\section{Sample Use Case: Namespace Implementation} +\subsection{Example Use Cases} \label{namespace} -In this section we show how versioning and conditional operations can be used in ASG storage model; by implementing namespace operations as an example. +In this section we show how ASG storage model can be used as a basis to construct a namespace and a column +store and to support concurrent write accesses. +\subsubsection{Namespace Implementation} A namespace can be defined as an abstract collection of unique keys. A key can be in a string or integer format and it may appear in more than one namespaces with or without having the same meaning; but it needs to be unique in each of these namespaces. The uniqueness property of @@ -736,12 +672,34 @@ namespace, update returns error. It calls underlying \textit{read} and \textit{w \end{itemize} +\subsubsection{Column Store} +Column Store use case will go here. + +\subsubsection{Concurrent Write Access} +Concurrent Write Access use case will go here. + +%\section{Comparison with Existing Models} +%Wisconson Storage System (WiSS) is a flexible storage system developed for the UNIX +%environment~\cite{DBLP:journals/spe/ChouDKK85}. It supports unconventional types of data and it consists of +%four distinct layers; \textit{level 0}, \textit{level 1}, +%\textit{level 2} and \textit{level 3}. The basic structure units of WiSS are \textit{pages} at +%\textit{level 0}. \textit{Level 0} performs I/O operations on pages. \textit{Level 1} is responsible for +%maintaining a buffer pool of pages to make the access to the important pages faster. \textit{Level 2} of +%WiSS carries some similarities to the ASG storage model. This level consists of \textit{records}, each of +%which stored in an underlying page. A record consists of record type, record length and actual data. Record +%length and actual data also exist in ASG records. Similar to the ASG storage model, each record is known by +%a numerical identifier. Unlike ASG, records in WiSS have unique identifiers; whereas records in different +%enclosing objects or forks etc. in ASG can have the same identifier. WiSS supports B-trees to index records +%and long data items (a collection of pages) to store unlimited size of data; however in ASG there are no +%such mechanisms yet. WiSS uses locking at the file granularity; whereas ASG supports versioning to provide +%concurrency. \textit{Level 3} of WiSS has primitives to create or delete files using low-level primitives +%which is similar to the Namespace Implementation we will discuss in Section~\ref{namespace}. + % no \IEEEPARstart %This demo file is intended to serve as a ``starter file'' %for IEEE conference papers produced under \LaTeX\ using %IEEEtran.cls version 1.7 and later. -%All manuscripts must be in English. These guidelines include complete descriptions of the fonts, spacing, and related information for producing your proceedings manuscripts. Please follow them and if you have any questions, direct them to the production editor in charge of your proceedings at Conference Publishing Services (CPS): Phone +1 (714) 821-8380 or Fax +1 (714) 761-1784. % You must have at least 2 lines in the paragraph with the drop letter % (should never be an issue) @@ -838,9 +796,68 @@ namespace, update returns error. It calls underlying \textit{read} and \textit{w % footnotes above bottom floats. This can be corrected via the \fnbelowfloat % command of the stfloats package. +\section{Related Work} +\label{relatedwork} +Network-Attached Secure Disk (NASD)~\cite{gibson:storage} is the primary study on +object-based storage and it led to specifications of standards~\cite{osd-3} for object-based +storage. +%It is based on the idea of integrating the disk management functions into the storage +%device and it offers a variable-length object-storage interface. +NASD introduces variable-length objects with attributes, rather than fixed-length traditional blocks, to +enable self-management and to obviate the need to know about the host operating system. Moving data +management to the storage disks increases the networking, security and space management capabilities. +There have been numerous studies in literature with similar scope to our work. +OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar +to the one specified by the OSD standard~\cite{osd-3} except for the addition of dedicated directory +objects. The directory objects in OSD+ store file names and attributes and support metadata related +operations. +%Each OSD+ stores a single directory object. +%In order to guarantee correctness and atomicity, OSD+ relies on the back-end file system (Ext3, Ext4) and +%three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. +Panasas File System~\cite{Welch08scalableperformance} is built on object-based storage devices. +%Panasas File System clients have direct access to the objects which are a combination of data and +%attributes. +OSD wire protocol of Panasas uses the operations from the OSD standard~\cite{osd-3} to enable byte-oriented +access to data, to manipulate attributes and to create or delete objects. +%Panasas clients use iSCSI/OSD protocol to access the object-based storage devices. +Lustre~\cite{lustre-web} is a distributed file system based on object-based storage. Object Storage +Server (OSS) in Lustre is responsible for providing access to file data stored in objects on Object +Storage Targets (OSTs). +%Lustre has two types of servers; Metadata Server (MDS) and Object Storage Server (OSS). Actual metadata is +%stored on Metadata Targets (MDTs) and file data is stored on Object Storage Targets (OSTs) as objects. MDS +%is responsible for handling metadata requests and making the metadata stored in MDTs available to the +%clients; whereas OSS is responsible for filesystem I/O and for providing access to file data stored on +%OSTs. +\textit{Ursa Minor} is a parallel file system that supports versioned writes. It keeps the existing +object-storage interface~\cite{osd-3} mostly intact except for introducing \textit{slices} which are +fragments of object data and it uses timestamps to distinguish different versions of data. +Datamods~\cite{watkinsdatamods} is a framework that exploits existing large-scale storage system services +to support complex data models and interfaces. +%Traditionally middleware is tailored to support specific data models; however many services provided by +%middleware already exist in distributed storage systems. +Datamods avoids duplicating services already provided in distributes storage systems in middleware and +improves scalability since it is not limited to single dimension at the file level. +RADOS~\cite{librados-doc, weil:pdsw07} is an object storage service forming the foundation of +Ceph~\cite{Weil06ceph:a}. It provides the clients a single logical object store and offloads object +replication, failure detection and data management tasks to the underlying object store daemons. +Ohio Supercomputing Center looked at mapping Parallel Virtual File System~\cite{pvfs-web} on top of an +existing object-based storage emulation~\cite{1306916, Devulapalli07integratingparallel}. This mapping +moved the functionality of the common components of a traditional storage system; such as I/O, directory +or metadata servers to OSDs and improved the performance of the overall system due to the capabilities of +the object-based storage devices ~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14}. + +A number of studies form the technical basis of our work. Transactional Object Storage Device (TOSD) +~\cite{carns2010object} shows that object-based storage is a common component of many parallel file +systems and it introduces three optimizations to the object-based storage model in order to serve highly +concurrent workloads better; atomicity, versioning and commutativity. Goodell et al.~\cite{goodell-pdsw12} +extended the POSIX API by organizing the storage around data objects in order to map complex data +structures to these data objects and have direct access between the data objects and applications. Carns +et al.~\cite{carns-pdsw12} investigated conditional update operations as an alternative to distributed +pessimistic locking operations in object-based storage systems. \section{Conclusions and Future Work} +\label{conclusion} ASG storage model is good and can be further improved to support snapshotting, dynamic namespaces etc. % conference papers do not normally have an appendix @@ -851,7 +868,8 @@ ASG storage model is good and can be further improved to support snapshotting, d %The authors would like to thank... -%more thanks here + + % trigger a \newpage just before the given reference @@ -884,7 +902,7 @@ ASG storage model is good and can be further improved to support snapshotting, d %\end{thebibliography} -\bibliographystyle{plain} +\bibliographystyle{IEEEtran} \bibliography{pario,REFERENCES,paper} % that's all folks diff --git a/papers/asg-osd/usenix.sty b/papers/asg-osd/usenix.sty deleted file mode 100644 index debc97b..0000000 --- a/papers/asg-osd/usenix.sty +++ /dev/null @@ -1,95 +0,0 @@ -% usenix.sty - to be used with latex2e for USENIX. -% To use this style file, look at the template usenix_template.tex -% -% $Id: usenix.sty,v 1.2 2005/02/16 22:30:47 maniatis Exp $ -% -% The following definitions are modifications of standard article.sty -% definitions, arranged to do a better job of matching the USENIX -% guidelines. -% It will automatically select two-column mode and the Times-Roman -% font. - -% -% USENIX papers are two-column. -% Times-Roman font is nice if you can get it (requires NFSS, -% which is in latex2e. - -\if@twocolumn\else\input twocolumn.sty\fi -\usepackage{mathptmx} % times roman, including math (where possible) - -% -% USENIX wants margins of: 1" sides, 1" bottom, and 1" top. -% 0.25" gutter between columns. -% Gives active areas of 6.5" x 9" -% -\setlength{\textheight}{9.0in} -\setlength{\columnsep}{0.25in} -\setlength{\textwidth}{6.50in} - -\setlength{\topmargin}{0.0in} - -\setlength{\headheight}{0.0in} - -\setlength{\headsep}{0.0in} - -% Usenix wants no page numbers for camera-ready papers, so that they can -% number them themselves. But submitted papers should have page numbers -% for the reviewers' convenience. -% -% -% \pagestyle{empty} - -% -% Usenix titles are in 14-point bold type, with no date, and with no -% change in the empty page headers. The whole author section is 12 point -% italic--- you must use {\rm } around the actual author names to get -% them in roman. -% -\def\maketitle{\par - \begingroup - \renewcommand\thefootnote{\fnsymbol{footnote}}% - \def\@makefnmark{\hbox to\z@{$\m@th^{\@thefnmark}$\hss}}% - \long\def\@makefntext##1{\parindent 1em\noindent - \hbox to1.8em{\hss$\m@th^{\@thefnmark}$}##1}% - \if@twocolumn - \twocolumn[\@maketitle]% - \else \newpage - \global\@topnum\z@ - \@maketitle \fi\@thanks - \endgroup - \setcounter{footnote}{0}% - \let\maketitle\relax - \let\@maketitle\relax - \gdef\@thanks{}\gdef\@author{}\gdef\@title{}\let\thanks\relax} - -\def\@maketitle{\newpage - \vbox to 2.5in{ - \vspace*{\fill} - \vskip 2em - \begin{center}% - {\Large\bf \@title \par}% - \vskip 0.375in minus 0.300in - {\large\it - \lineskip .5em - \begin{tabular}[t]{c}\@author - \end{tabular}\par}% - \end{center}% - \par - \vspace*{\fill} -% \vskip 1.5em - } -} - -% -% The abstract is preceded by a 12-pt bold centered heading -\def\abstract{\begin{center}% -{\large\bf \abstractname\vspace{-.5em}\vspace{\z@}}% -\end{center}} -\def\endabstract{} - -% -% Main section titles are 12-pt bold. Others can be same or smaller. -% -\def\section{\@startsection {section}{1}{\z(a)}{-3.5ex plus-1ex minus - -.2ex}{2.3ex plus.2ex}{\reset@font\large\bf}} - hooks/post-receive --

1 0

branch, master, updated. aabe2c73d336fc95e2a56956b19f4e4ee173a793
by noreply＠mcs.anl.gov 05 Apr '13

05 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via aabe2c73d336fc95e2a56956b19f4e4ee173a793 (commit) from 582aba6c290f317d87978cd3e1d3e5270fd8c7b2 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit aabe2c73d336fc95e2a56956b19f4e4ee173a793 Author: cengiz <cengiz.k(a)uconn.edu> Date: Fri Apr 5 13:28:41 2013 -0400 Modified related work section * Added summary for datamods and rados * Shortened related work section ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 25 +++++++++ papers/asg-osd/paper.tex | 122 +++++++++++++++++++++++++++------------------- 2 files changed, 96 insertions(+), 51 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index 8704ffe..faac40c 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -223,6 +223,14 @@ and Klundt, R. and Danielson, G. and Karakoyunlu, C. and Chandy, J. and Settlemy organization={IEEE} } +@inproceedings{watkinsdatamods, + title={{DataMods: Programmable File System Services}}, + author={Watkins, Noah and Maltzahn, Carlos and Brandt, Scott and Manzanares, Adam}, + booktitle={Proceedings of 2012 Parallel Data Storage Workshop (PDSW 2012)}, + year={2012}, + organization={IEEE} +} + @inproceedings{pvfs-bgp-sc09, author = {Lang, Samuel and Carns, Philip and Latham, Robert and Ross, Robert and Harms, Kevin and Allcock, William}, title = {{I/O} performance challenges at leadership scale}, @@ -576,3 +584,20 @@ title = {{Design of an Intelligent Object-based Storage device}}, address = {Berkeley, CA, USA}, } +@misc{librados-doc, + title = {{Librados API documentation}}, + url = {http://ceph.com/docs/master/api/librados/} +} + +@inproceedings{weil:pdsw07, + address = {Reno, NV}, + author = {Weil, Sage A. and Leung, Andrew and Brandt, Scott A. and Maltzahn, Carlos}, + booktitle = {Proceedings of the 2007 ACM Petascale Data Storage Workshop (PDSW 07)}, + citeulike-article-id = {6765406}, + keywords = {randomized\_data\_placement}, + month = nov, + posted-at = {2010-11-05 08:33:52}, + priority = {0}, + title = {{RADOS: A Fast, Scalable, and Reliable Storage Service for Petabyte-scale Storage Clusters}}, + year = {2007} +} diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 0095f5f..9d6baa3 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -453,8 +453,8 @@ Modern applications demand high-performance, security and portability from the s computers and this requires the storage units to redefine the abstraction to store and manage data. In order to meet these demands, a new storage model called \textit{object-based storage} has been developed ~\cite{1222722, osd-3}. Storage devices that store and access data according to the object-based storage -model are called as \textit{object-based storage devices}. Storage system developers prefer to build software -implementations of object-based storage model on block-based devices; since there is no hardware +model are called as \textit{object-based storage devices}. Storage system developers prefer to build +software implementations of object-based storage model on block-based devices; since there is no hardware implementation of an object-based storage device yet. %\textit{properties of the object-based storage model} @@ -471,24 +471,23 @@ on the disk~\cite{gibson:storage, 5944457}. Object-based storage is also suitabl provide more features to encapsulate the computations compared to traditional blocks. Although several object-based storage models have already been implemented and used as the basis for the -popular storage and file systems ~\cite{Weil06ceph:a, Beaver:2010:FNH:1924943.1924947, lustre-web}, these object-based storage models are often tailored to a particular use case or data model. +popular storage and file systems~\cite{Weil06ceph:a, Beaver:2010:FNH:1924943.1924947, lustre-web}, these +object-based storage models are often tailored to a particular use case or data model. %\textit{Requirements of common data models (file systems, cloud storage, databases and big data)} On the other hand, common data applications; such as file systems, cloud storage, databases and big data -require the underlying storage models to support various features; such as concurrency, reliability, scalability and flexible data structures. +require the underlying storage models to support various features; such as concurrency, reliability, +scalability and flexible data structures. %Although object-based storage model has been investigated in many research studies and several %optimizations have been introduced for this model, we believe that the object-based storage model -%primitives need to be revisited. Transactional Object Storage Device -%(TOSD) work~\cite{carns2010object} shows that object-based storage is a common component of many parallel -%file systems and it introduces three optimizations to the object-based storage model in order to serve -%highly concurrent workloads better; atomicity, versioning and commutativity. Goodell et al. -%~\cite{goodell-pdsw12} extended the POSIX API by organizing the storage around data objects in order to map -%complex data structures to these data objects and have direct access between the data objects and -%applications. Carns et al. ~\cite{carns-pdsw12} investigated conditional update operations as an -%alternative to distributed pessimistic locking operations in object-based storage systems. +%primitives need to be revisited. In this work we propose a new object-based storage model, known as the ASG interface, that seeks to unify -the features necessary to support the data models outlined above without compromising usability or limiting -implementation flexibility. ASG interface intends to bring versioning, conditional operation, fork\&record -structure and unified data\&attribute features together. The rest of the paper is organized as follows; Section 2 talks about related work in object-based storage systems. Section 3 describes the ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the advantages\&disadvantages of each model. We give sample usage scenarios of ASG model in Section 5 followed by the conclusion and possible directions for future development of ASG in Section 6. +the features necessary to support the data models outlined above without compromising usability or +limiting implementation flexibility. ASG interface intends to bring versioning, conditional operation, +fork\&record structure and unified data\&attribute features together. The rest of the paper is organized +as follows; Section 2 talks about related work in object-based storage systems. Section 3 describes the +ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the +advantages\&disadvantages of each model. We give sample usage scenarios of ASG model in Section 5 followed +by the conclusion and possible directions for future development of ASG in Section 6. %\textit{Describe the sections in the remainder of the paper} %We believe an object-based storage system can offer a lot more sophisticated functionalities by @@ -497,42 +496,63 @@ structure and unified data\&attribute features together. The rest of the paper i %related studies described in the previous paragraph. \section{Related Work} - -Network-Attached Secure Disk (NASD) ~\cite{gibson:storage} from Carnegie Mellon is the primary study on -object-based storage systems and it led to specifications of standards~\cite{osd-3} for object-based -storage devices. It is based on the idea of integrating the disk management functions into the storage -device and it offers a variable-length object-storage interface. NASD introduces variable-length objects -with attributes, rather than fixed-length traditional blocks, to enable self-management and to obviate the -need to know about the host operating system. Moving data management to the storage disks increases the -networking, security and space management capabilities. - -OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar to -the one specified by the OSD standard ~\cite{osd-3} except for the addition of dedicated directory objects. -The directory objects in OSD+ store file names and attributes and support metadata related operations. Each -OSD+ stores a single directory object. In order to guarantee correctness and atomicity, OSD+ relies on the -back-end file system (Ext3, Ext4) and three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. - -Panasas File System~\cite{Welch08scalableperformance} is built on object-based storage devices. Panasas -File System clients have direct access to the objects which are a combination of data and attributes. OSD -wire protocol of Panasas uses the operations from the OSD standard~\cite{osd-3} to enable byte-oriented -access to data, to manipulate attributes and to create or delete objects. Panasas clients use iSCSI/OSD -protocol to access the object-based storage devices. - -Lustre ~\cite{lustre-web} is a distributed file system based on object-based storage. Lustre has two types -of servers; Metadata Server (MDS) and Object Storage Server (OSS). Actual metadata is stored on Metadata -Targets (MDTs) and file data is stored on Object Storage Targets (OSTs) as objects. MDS is responsible -for handling metadata requests and making the metadata stored in MDTs available to the clients; whereas -OSS is responsible for filesystem I/O and for providing access to file data stored on OSTs. - -\textit{Ursa Minor} is a parallel file system supporting write versioning; however, \textit{Ursa Minor} keeps -the existing object-storage interface~\cite{osd-3} mostly intact except for introducing \textit{slices} which -are fragments of object data and it uses timestamps to distinguish different versions of data. - -Other related work~\cite{Devulapalli07integratingparallel} from Ohio Supercomputing Center looked at -mapping Parallel Virtual File System~\cite{pvfs-web} on top of an existing object-based storage -emulation~\cite{1306916}. This mapping moved the functionality of the common components of a traditional -storage system; such as I/O, directory or metadata servers to OSDs and improved the performance -of the overall system due to the capabilities of the object-based storage devices.~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14} +Network-Attached Secure Disk (NASD)~\cite{gibson:storage} is the primary study on +object-based storage and it led to specifications of standards~\cite{osd-3} for object-based +storage. +%It is based on the idea of integrating the disk management functions into the storage +%device and it offers a variable-length object-storage interface. +NASD introduces variable-length objects with attributes, rather than fixed-length traditional blocks, to +enable self-management and to obviate the need to know about the host operating system. Moving data +management to the storage disks increases the networking, security and space management capabilities. + +There have been numerous studies in literature with similar scope to our work. +OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar +to the one specified by the OSD standard~\cite{osd-3} except for the addition of dedicated directory +objects. The directory objects in OSD+ store file names and attributes and support metadata related +operations. +%Each OSD+ stores a single directory object. +%In order to guarantee correctness and atomicity, OSD+ relies on the back-end file system (Ext3, Ext4) and +%three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. +Panasas File System~\cite{Welch08scalableperformance} is built on object-based storage devices. +%Panasas File System clients have direct access to the objects which are a combination of data and +%attributes. +OSD wire protocol of Panasas uses the operations from the OSD standard~\cite{osd-3} to enable byte-oriented +access to data, to manipulate attributes and to create or delete objects. +%Panasas clients use iSCSI/OSD protocol to access the object-based storage devices. +Lustre~\cite{lustre-web} is a distributed file system based on object-based storage. Object Storage +Server (OSS) in Lustre is responsible for providing access to file data stored in objects on Object +Storage Targets (OSTs). +%Lustre has two types of servers; Metadata Server (MDS) and Object Storage Server (OSS). Actual metadata is +%stored on Metadata Targets (MDTs) and file data is stored on Object Storage Targets (OSTs) as objects. MDS +%is responsible for handling metadata requests and making the metadata stored in MDTs available to the +%clients; whereas OSS is responsible for filesystem I/O and for providing access to file data stored on +%OSTs. +\textit{Ursa Minor} is a parallel file system that supports versioned writes. It keeps the existing +object-storage interface~\cite{osd-3} mostly intact except for introducing \textit{slices} which are +fragments of object data and it uses timestamps to distinguish different versions of data. +Datamods~\cite{watkinsdatamods} is a framework that exploits existing large-scale storage system services +to support complex data models and interfaces. +%Traditionally middleware is tailored to support specific data models; however many services provided by +%middleware already exist in distributed storage systems. +Datamods avoids duplicating services already provided in distributes storage systems in middleware and +improves scalability since it is not limited to single dimension at the file level. +RADOS~\cite{librados-doc, weil:pdsw07} is an object storage service forming the foundation of +Ceph~\cite{Weil06ceph:a}. It provides the clients a single logical object store and offloads object +replication, failure detection and data management tasks to the underlying object store daemons. +Ohio Supercomputing Center looked at mapping Parallel Virtual File System~\cite{pvfs-web} on top of an +existing object-based storage emulation~\cite{1306916, Devulapalli07integratingparallel}. This mapping +moved the functionality of the common components of a traditional storage system; such as I/O, directory +or metadata servers to OSDs and improved the performance of the overall system due to the capabilities of +the object-based storage devices ~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14}. + +A number of studies form the technical basis of our work. Transactional Object Storage Device (TOSD) +~\cite{carns2010object} shows that object-based storage is a common component of many parallel file +systems and it introduces three optimizations to the object-based storage model in order to serve highly +concurrent workloads better; atomicity, versioning and commutativity. Goodell et al.~\cite{goodell-pdsw12} +extended the POSIX API by organizing the storage around data objects in order to map complex data +structures to these data objects and have direct access between the data objects and applications. Carns +et al.~\cite{carns-pdsw12} investigated conditional update operations as an alternative to distributed +pessimistic locking operations in object-based storage systems. \section{Implementation Strategies} \label{primitives} hooks/post-receive --

1 0

branch, master, updated. 582aba6c290f317d87978cd3e1d3e5270fd8c7b2
by noreply＠mcs.anl.gov 04 Apr '13

04 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 582aba6c290f317d87978cd3e1d3e5270fd8c7b2 (commit) from 8a1778de45a2d25395c43fff868bb155032dbf8b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 582aba6c290f317d87978cd3e1d3e5270fd8c7b2 Author: cengiz <cengiz.k(a)uconn.edu> Date: Thu Apr 4 05:02:21 2013 -0400 Some updates the introduction section ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 15 +++++++++ papers/asg-osd/paper.tex | 72 ++++++++++++++++++++++++++++------------------ 2 files changed, 59 insertions(+), 28 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index c641865..8704ffe 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -561,3 +561,18 @@ title = {{Design of an Intelligent Object-based Storage device}}, bibsource = {DBLP, http://dblp.uni-trier.de} } +@inproceedings{Beaver:2010:FNH:1924943.1924947, + author = {Beaver, Doug and Kumar, Sanjeev and Li, Harry C. and Sobel, Jason and Vajgel, Peter}, + title = {Finding a needle in Haystack: facebook's photo storage}, + booktitle = {Proceedings of the 9th USENIX conference on Operating systems design and implementation}, + series = {OSDI'10}, + year = {2010}, + location = {Vancouver, BC, Canada}, + pages = {1--8}, + numpages = {8}, + url = {http://dl.acm.org/citation.cfm?id=1924943.1924947}, + acmid = {1924947}, + publisher = {USENIX Association}, + address = {Berkeley, CA, USA}, +} + diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 0b73a8d..0095f5f 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -421,7 +421,7 @@ that seeks to combine data access features (i.e. concurrent reads\&writes, versi conditional operations, failure handling) into a unified API that is suitable for use as a base model in a wide range of contexts, including file systems, cloud storage, databases, HPC systems and big data. We identify the common requirements of these data models, identify key -features of previous object storage interfaces, and explore how they can be unified effectively +features of previous object-based storage interfaces, and explore how they can be unified effectively without sacrificing usability. We validate the effectiveness of our approach with case studies that evaluate how the storage API we are proposing can be used to construct a namespace and a column-store database. @@ -446,48 +446,55 @@ column-store database. \section{Introduction} - +%\textit{why did object-based storage model come out?} Storage devices have seen a lot of improvement in terms of performance and complexity in the last decades; but they could not catch the development rate of the computation and processing units of the computers. Modern applications demand high-performance, security and portability from the storage systems of the -computers and this requires the storage units to redefine the abstraction to store and manage the data. +computers and this requires the storage units to redefine the abstraction to store and manage data. In order to meet these demands, a new storage model called \textit{object-based storage} has been developed -~\cite{1222722, osd-3}. Storage devices that store and access the data according to the object-based storage +~\cite{1222722, osd-3}. Storage devices that store and access data according to the object-based storage model are called as \textit{object-based storage devices}. Storage system developers prefer to build software implementations of object-based storage model on block-based devices; since there is no hardware -implementation of object-based storage devices yet. +implementation of an object-based storage device yet. +%\textit{properties of the object-based storage model} An \textit{object} is an ordered logical collection of bytes with a numerical identifier. Objects consist of data, user settable attributes and device managed metadata including security policies that protect data ~\cite{1222722, Welch08scalableperformance}. They have variable sizes and they can be used to store any -kind of data in the storage system. Having data organized as objects makes it possible for the -disk to do data management operations on behalf of the user ~\cite{5944457} and offloads the storage -management from the host operating system to the object-based storage. User-settable attributes can -be further enriched to set performance goals, manage replicas, or handle concurrency in object-based +kind of data in the storage system. Having data organized as objects makes it possible for a +disk to do data management operations on behalf of a user~\cite{5944457} and offloads the storage +management task from the host operating system to the object-based storage. User-settable attributes can +be further enriched to set performance goals, manage replicas or handle concurrency in object-based storage model to enable self-management capabilities and they give the user an effective control of data on the disk~\cite{gibson:storage, 5944457}. Object-based storage is also suitable for realizing \textit{active storage}~\cite{5944457,Acharya:1998:ADP:291006.291026, riedel:thesis}; because objects provide more features to encapsulate the computations compared to traditional blocks. -Although object-based storage model has been investigated in many research studies and several -optimizations have been introduced for this model, we believe that the object-based storage model -primitives need to be revisited. Transactional Object Storage Device -(TOSD) work~\cite{carns2010object} shows that object-based storage is a common component of many parallel -file systems and it introduces three optimizations to the object-based storage model in order to serve -highly concurrent workloads better; atomicity, versioning and commutativity. Goodell et al. -~\cite{goodell-pdsw12} extended the POSIX API by organizing the storage around data objects in order to map -complex data structures to these data objects and have direct access between the data objects and -applications. Carns et al. ~\cite{carns-pdsw12} investigated conditional update operations as an -alternative to distributed pessimistic locking operations in object-based storage systems. - -We believe an object-based storage system can offer a lot more sophisticated functionalities by -implementing versioning and conditional operations and we introduce a new object-based storage -system model, ASG, that unifies the conditional operation and versioning features previously explored in -related studies described in the previous paragraph. The rest of the paper is organized as follows; -Section 2 talks about related work in object-based storage systems. Section 3 describes the ASG -architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the -advantages\&disadvantages of each model. We give a sample usage scenario of ASG model in Section 5 followed -by the conclusion and possible directions for future development of ASG in Section 6. +Although several object-based storage models have already been implemented and used as the basis for the +popular storage and file systems ~\cite{Weil06ceph:a, Beaver:2010:FNH:1924943.1924947, lustre-web}, these object-based storage models are often tailored to a particular use case or data model. +%\textit{Requirements of common data models (file systems, cloud storage, databases and big data)} +On the other hand, common data applications; such as file systems, cloud storage, databases and big data +require the underlying storage models to support various features; such as concurrency, reliability, scalability and flexible data structures. +%Although object-based storage model has been investigated in many research studies and several +%optimizations have been introduced for this model, we believe that the object-based storage model +%primitives need to be revisited. Transactional Object Storage Device +%(TOSD) work~\cite{carns2010object} shows that object-based storage is a common component of many parallel +%file systems and it introduces three optimizations to the object-based storage model in order to serve +%highly concurrent workloads better; atomicity, versioning and commutativity. Goodell et al. +%~\cite{goodell-pdsw12} extended the POSIX API by organizing the storage around data objects in order to map +%complex data structures to these data objects and have direct access between the data objects and +%applications. Carns et al. ~\cite{carns-pdsw12} investigated conditional update operations as an +%alternative to distributed pessimistic locking operations in object-based storage systems. +In this work we propose a new object-based storage model, known as the ASG interface, that seeks to unify +the features necessary to support the data models outlined above without compromising usability or limiting +implementation flexibility. ASG interface intends to bring versioning, conditional operation, fork\&record +structure and unified data\&attribute features together. The rest of the paper is organized as follows; Section 2 talks about related work in object-based storage systems. Section 3 describes the ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the advantages\&disadvantages of each model. We give sample usage scenarios of ASG model in Section 5 followed by the conclusion and possible directions for future development of ASG in Section 6. + +%\textit{Describe the sections in the remainder of the paper} +%We believe an object-based storage system can offer a lot more sophisticated functionalities by +%implementing versioning and conditional operations and we introduce a new object-based storage +%system model, ASG, that unifies the conditional operation and versioning features previously explored in +%related studies described in the previous paragraph. \section{Related Work} @@ -531,6 +538,15 @@ of the overall system due to the capabilities of the object-based storage device \label{primitives} In this section we describe the ASG storage model, its fundamental building blocks and basic primitives. +\textit{separate section describing asg} +You can devote a separate section in the paper to describing what +technical features were needed to support the data models you listed, +and how that (and the usability constraint) led to the design we are +proposing. Point out right here that these are (mostly) not novel ideas +independently: the contribution here is a model for how to combine them +in a reasonable way. This material is going to be somewhat dense- you +don't want to have to repeat this stuff elsewhere in the paper. + \subsection{Architecture} The main architecture of ASG storage model is shown in Figure \ref{architecture}. hooks/post-receive --

1 0

C-Utils Repository branch, master, updated. 127da042e1712a01c754559872b9c2d2f902cca9
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, master has been updated via 127da042e1712a01c754559872b9c2d2f902cca9 (commit) from cb49623cabcfb334fdbb729b49ac455898c49798 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 127da042e1712a01c754559872b9c2d2f902cca9 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Tue Apr 2 15:43:30 2013 -0500 Cleanup include/ public headers on make clean Fixes make distcheck complaint re leftover files. ----------------------------------------------------------------------- Summary of changes: Makefile.am | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) Diff of changes: diff --git a/Makefile.am b/Makefile.am index 2399f49..fc3cfb1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -46,6 +46,14 @@ header-stamp: $(public_headers) clean-local: -rm header-stamp + -( \ + for a in $(public_headers); do\ + base=`echo $$a | sed -e 's,.*/,,'` ;\ + rm -f $(builddir)/include/$${base} ;\ + done ;\ + ) + + #----------------------------------------------------- hooks/post-receive -- C-Utils Repository

1 0

C-Utils Repository branch, master, updated. cb49623cabcfb334fdbb729b49ac455898c49798
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, master has been updated via cb49623cabcfb334fdbb729b49ac455898c49798 (commit) via 8bd3f9e2b0f8cb35c1e9237bddd11f101fc8f6bc (commit) via 737372146487fa8fbd5177c1e870800b9a18a280 (commit) via 43f2eba5fecfe9d4b69c3a47afc347158177f7d0 (commit) from f07401b62b45ba683fc336e6c143b1464159926f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit cb49623cabcfb334fdbb729b49ac455898c49798 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 16:19:15 2013 -0400 clean up .tap files commit 8bd3f9e2b0f8cb35c1e9237bddd11f101fc8f6bc Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 16:04:58 2013 -0400 tell automake about tap redirector script commit 737372146487fa8fbd5177c1e870800b9a18a280 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 16:03:29 2013 -0400 fix warning commit 43f2eba5fecfe9d4b69c3a47afc347158177f7d0 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 16:03:03 2013 -0400 add test-tap.h to automake is there a cleaner way to add headers used by multiple programs? ----------------------------------------------------------------------- Summary of changes: Makefile.am | 4 +++- tests/Makefile.subdir | 4 ++++ tests/test-tap.h | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-) Diff of changes: diff --git a/Makefile.am b/Makefile.am index d01762a..2399f49 100644 --- a/Makefile.am +++ b/Makefile.am @@ -4,6 +4,7 @@ AUTOMAKE_OPTIONS = parallel-tests LOG_COMPILER=$(top_srcdir)/maint/tap-script.sh TESTS = +CLEANFILES = noinst_HEADERS = check_PROGRAMS = @@ -14,7 +15,7 @@ include_HEADERS = AM_CPPFLAGS = -EXTRA_DIST = COPYRIGHT TODO +EXTRA_DIST = COPYRIGHT TODO maint/tap-script.sh nobase_include_HEADERS= @@ -74,3 +75,4 @@ include src/Makefile.subdir include libev/Makefile.subdir include tests/Makefile.subdir +CLEANFILES+=$(TESTS:%=%.tap) diff --git a/tests/Makefile.subdir b/tests/Makefile.subdir index 8238ab0..09f5543 100644 --- a/tests/Makefile.subdir +++ b/tests/Makefile.subdir @@ -13,5 +13,9 @@ tests_testlist_LDADD = $(testlib) tests_testhash_LDADD = $(testlib) tests_test_uint128_LDADD = $(testlib) +tests_testlist_SOURCES = tests/testlist.c tests/test-tap.h +tests_testhash_SOURCES = tests/testhash.c tests/test-tap.h +tests_test_uint128_SOURCES = tests/test-uint128.c tests/test-tap.h + #TESTSRC += tests/testlist.c tests/testhash.c tests/test-uint128.c diff --git a/tests/test-tap.h b/tests/test-tap.h index 7640fbb..6858ce9 100644 --- a/tests/test-tap.h +++ b/tests/test-tap.h @@ -24,7 +24,7 @@ do { \ }\ else {\ printf("not ok %d - %s\n", __testcount, __name); \ - printf(" # failed condition: " #__expr "\n"); \ + printf(" # failed condition: " #__expr "\n"); \ __failcount++; \ }\ __testcount++; \ hooks/post-receive -- C-Utils Repository

1 0

C-Utils Repository branch, master, updated. f07401b62b45ba683fc336e6c143b1464159926f
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, master has been updated via f07401b62b45ba683fc336e6c143b1464159926f (commit) via 841c05cb1962e8a3351faccb1fb26a7dae15e68a (commit) via 62d1d50f28446af6f9746d7d496527e9ebddfd0c (commit) via 0d72587ef7d0b1a8ccb52042923bc5a52e01db5f (commit) via 9fb01725a3f9f8f6e7673ac4a28c60120e562185 (commit) from 909b3f77cabebd176824de712059b5d919e52071 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit f07401b62b45ba683fc336e6c143b1464159926f Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 15:31:53 2013 -0400 update testhash to generate tap output commit 841c05cb1962e8a3351faccb1fb26a7dae15e68a Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 15:23:21 2013 -0400 show failed condition on failure commit 62d1d50f28446af6f9746d7d496527e9ebddfd0c Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 15:14:00 2013 -0400 update testlist to output in tap format commit 0d72587ef7d0b1a8ccb52042923bc5a52e01db5f Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 15:01:12 2013 -0400 convert uint128 test to tap output commit 9fb01725a3f9f8f6e7673ac4a28c60120e562185 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 15:00:26 2013 -0400 automake test harness changes turn on parallel-tests option and add a wrapper that will put test output in .tap files ----------------------------------------------------------------------- Summary of changes: Makefile.am | 3 +++ maint/tap-script.sh | 5 +++++ tests/test-tap.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/test-uint128.c | 23 ++++++++++++++++++----- tests/testhash.c | 27 +++++++++++++++++++++------ tests/testlist.c | 43 ++++++++++++++++++++++++++++--------------- 6 files changed, 123 insertions(+), 26 deletions(-) create mode 100755 maint/tap-script.sh create mode 100644 tests/test-tap.h Diff of changes: diff --git a/Makefile.am b/Makefile.am index ffd7bcd..d01762a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -1,5 +1,8 @@ ACLOCAL_AMFLAGS = -I m4 +AUTOMAKE_OPTIONS = parallel-tests +LOG_COMPILER=$(top_srcdir)/maint/tap-script.sh + TESTS = noinst_HEADERS = diff --git a/maint/tap-script.sh b/maint/tap-script.sh new file mode 100755 index 0000000..2db6a27 --- /dev/null +++ b/maint/tap-script.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +$* 2>&1 | tee $1.tap + +exit ${PIPESTATUS[0]} diff --git a/tests/test-tap.h b/tests/test-tap.h new file mode 100644 index 0000000..7640fbb --- /dev/null +++ b/tests/test-tap.h @@ -0,0 +1,48 @@ +/* + * (C) 2009 The University of Chicago + * + * See COPYRIGHT in top-level directory. + */ + +#ifndef __TEST_TAP +#define __TEST_TAP + +#include <stdio.h> +#include <assert.h> + +#define tap_begin(__testcount, __failcount) \ +do {\ + __testcount = 1; \ + __failcount = 0; \ +}while(0) + +#define tap(__name, __expr, __testcount, __failcount) \ +do { \ + assert(__testcount > 0); \ + if(__expr) { \ + printf("ok %d - %s\n", __testcount, __name); \ + }\ + else {\ + printf("not ok %d - %s\n", __testcount, __name); \ + printf(" # failed condition: " #__expr "\n"); \ + __failcount++; \ + }\ + __testcount++; \ +}while(0) + +#define tap_end(__testcount, __failcount) \ +do {\ + printf("1..%d\n", __testcount-1); \ +}while(0) + + +#endif /* __TEST_TAP */ + +/* + * Local variables: + * c-indent-level: 4 + * c-basic-offset: 4 + * End: + * + * vim: ts=8 sts=4 sw=4 expandtab + */ diff --git a/tests/test-uint128.c b/tests/test-uint128.c index 29eda67..6508831 100644 --- a/tests/test-uint128.c +++ b/tests/test-uint128.c @@ -8,29 +8,42 @@ #include <stdlib.h> #include "triton-uint128.h" +#include "test-tap.h" int main(int argc, char *argv[]) { uint128_t big; char big_str[TRITON_UINT128_STRLEN]; + int testcase, failcount; + + tap_begin(testcase, failcount); big = triton_uint128_from_uint64(0); triton_uint128_to_string(big_str, TRITON_UINT128_STRLEN, big); - printf("initialize to zero: %s\n", big_str); + tap("triton_uint128_from_uint64(0)", strcmp(big_str, "0:0") == 0, + testcase, failcount); big = triton_uint128_inc(big); triton_uint128_to_string(big_str, TRITON_UINT128_STRLEN, big); - printf("increment by one: %s\n", big_str); + tap("triton_uint128_inc(0)", strcmp(big_str, "0:1") == 0, + testcase, failcount); big = triton_uint128_from_uint64(UINT64_MAX); triton_uint128_to_string(big_str, TRITON_UINT128_STRLEN, big); - printf("initialize to UINT64_MAX: %s\n", big_str); + tap("triton_uint128_from_uint64(UINT64_MAX)", big.l == UINT64_MAX, + testcase, failcount); big = triton_uint128_inc(big); triton_uint128_to_string(big_str, TRITON_UINT128_STRLEN, big); - printf("increment by one: %s\n", big_str); + tap("triton_uint128_inc(UINT64_MAX)", strcmp(big_str, "1:0") == 0, + testcase, failcount); + + tap_end(testcase, failcount); - return(0); + if(failcount) + return(-1); + else + return(0); } /* diff --git a/tests/testhash.c b/tests/testhash.c index 61d366d..a7d7bf1 100644 --- a/tests/testhash.c +++ b/tests/testhash.c @@ -10,6 +10,7 @@ #include "triton-hash.h" #include "triton-string.h" #include "triton-uint128.h" +#include "test-tap.h" #define DEFAULT_TEST_SIZE 1000 @@ -107,6 +108,7 @@ int main(int argc, char *argv[]) int i, j, testsize, hash; triton_string_t *string; struct triton_hash_table *ktable; + int testcount, failcount; if(argc == 1) testsize = DEFAULT_TEST_SIZE; @@ -120,6 +122,8 @@ int main(int argc, char *argv[]) assert(testsize > 100); + tap_begin(testcount, failcount); + es = malloc(sizeof(*es) * testsize); /* Begin integer tests... */ @@ -141,8 +145,10 @@ int main(int argc, char *argv[]) { result = triton_hash_search(table, &i); entry = triton_list_get_entry(result, struct e, link); - assert(entry->v == i); + if(entry->v != i) + break; } + tap("triton_hash_search (integers)", i==testsize, testcount, failcount); triton_hash_destroy_and_finalize(table, struct e, link, e_destroy); @@ -153,7 +159,7 @@ int main(int argc, char *argv[]) string = malloc(sizeof(triton_string_t)); triton_string_init(string, "string: _%d_", i); hash = triton_hash_string_hash(string, 23); - printf("hash(\"%s\") -> %i\n", string->string, hash); + /* printf("hash(\"%s\") -> %i\n", string->string, hash); */ triton_string_destroy(string); free(string); } @@ -178,9 +184,11 @@ int main(int argc, char *argv[]) assert(result != NULL); ss = triton_hash_get_entry(result, struct s, link); sscanf(ss->v->string, "[%i]", &j); - assert(i == j); + if(i != j) + break; free(string); } + tap("triton_hash_search (strings)", i==testsize, testcount, failcount); triton_hash_destroy_and_finalize(table, struct s, link, s_destroy); @@ -205,9 +213,10 @@ int main(int argc, char *argv[]) assert(result != NULL); ns = triton_hash_get_entry(result, struct n, link); assert(ns != NULL); - assert(ns->node->l == node.l && - ns->node->u == node.u); + if(ns->node->l != node.l || ns->node->u != node.u) + break; } + tap("triton_hash_search (128 bit)", i==3, testcount, failcount); ktable = triton_hash_init(k_compare, k_hash, 1024); assert(ktable); @@ -223,9 +232,15 @@ int main(int argc, char *argv[]) { assert(triton_hash_search(ktable, &vals[i % 10].k)); } + tap("triton_hash_search (opaque key)", 1, testcount, failcount); free(es); - return 0; + + tap_end(testcount, failcount); + if(failcount) + return(-1); + else + return(0); } /* diff --git a/tests/testlist.c b/tests/testlist.c index e02c482..66ad662 100644 --- a/tests/testlist.c +++ b/tests/testlist.c @@ -7,6 +7,7 @@ #include <stdlib.h> #include <stdio.h> #include "triton-list.h" +#include "test-tap.h" #define DEFAULT_TEST_SIZE 1000 @@ -22,6 +23,7 @@ int main(int argc, char *argv[]) struct e *es, *ep, *esc; triton_list_link_t *p, *s; int i, testsize, delcount; + int testcount, failcount; if(argc == 1) testsize = DEFAULT_TEST_SIZE; @@ -35,8 +37,9 @@ int main(int argc, char *argv[]) assert(testsize > 100); - es = malloc(sizeof(*es) * testsize); + tap_begin(testcount, failcount); + es = malloc(sizeof(*es) * testsize); triton_list_init(&l); @@ -65,7 +68,8 @@ int main(int argc, char *argv[]) } } - assert(triton_list_count(&l) == (testsize-delcount)); + tap("triton_list_del", triton_list_count(&l) == (testsize-delcount), + testcount, failcount); /* test foreach */ triton_list_for_each(p, s, &l) @@ -73,7 +77,8 @@ int main(int argc, char *argv[]) triton_list_del(p); } - assert(triton_list_empty(&l)); + tap("triton_list_for_each with triton_list_del", triton_list_empty(&l), + testcount, failcount); /* test push/pop */ for(i = 0; i < testsize; ++i) @@ -84,8 +89,10 @@ int main(int argc, char *argv[]) for(i = testsize - 1; i >= 0; --i) { p = triton_stack_pop(&l); - assert(triton_list_get_entry(p, struct e, link)->v == i); + if(triton_list_get_entry(p, struct e, link)->v != i) + break; } + tap("triton_stack_pop", i==-1, testcount, failcount); /* test enqueue/dequeue */ for(i = 0; i < testsize; ++i) @@ -93,16 +100,16 @@ int main(int argc, char *argv[]) triton_queue_enqueue(&es[i].link, &l); } - assert(triton_list_count(&l) == testsize); + tap("triton_queue_enqueue", triton_list_count(&l) == testsize, testcount, failcount); for(i = 0; i < testsize; ++i) { p = triton_queue_dequeue(&l); - assert(triton_list_get_entry(p, struct e, link)->v == i); - assert(triton_list_count(&l) == (testsize - i - 1)); + if(triton_list_get_entry(p, struct e, link)->v != i) + break; } - assert(triton_list_empty(&l)); + tap("triton_queue_dequeue", i == testsize && triton_list_empty(&l), testcount, failcount); /* test insert_before */ for(i = 0; i < 100; ++i) @@ -124,8 +131,8 @@ int main(int argc, char *argv[]) { assert(triton_list_get_entry(triton_queue_dequeue(&l), struct e, link)->v == i); } - - assert(triton_list_empty(&l)); + /* NOTE: asserts are checking this test case in greater detail above */ + tap("triton_list_insert_before", triton_list_empty(&l), testcount, failcount); /* test insert_after */ for(i = 0; i < 100; ++i) @@ -138,7 +145,7 @@ int main(int argc, char *argv[]) triton_list_insert_after(&es[i].link, &es[0].link, &l); } - assert(triton_list_get_entry(triton_queue_dequeue(&l), struct e, link)->v == 0); + tap("triton_list_insert_after", triton_list_get_entry(triton_queue_dequeue(&l), struct e, link)->v == 0, testcount, failcount); for(i = 0; i < (testsize - 100); ++i) { @@ -150,7 +157,8 @@ int main(int argc, char *argv[]) assert(triton_list_get_entry(triton_queue_dequeue(&l), struct e, link)->v == i); } - assert(triton_list_empty(&l)); + /* NOTE: asserts are checking this test case in greater detail above */ + tap("triton_list_insert_after 2", triton_list_empty(&l), testcount, failcount); for(i = 0; i < testsize; ++i) { @@ -165,11 +173,16 @@ int main(int argc, char *argv[]) assert(ep->v == i); ++i; } - - assert(triton_list_empty(&l)); + /* NOTE: asserts are checking this test case in greater detail above */ + tap("triton_list_for_each_entry", triton_list_empty(&l), testcount, failcount); free(es); - return 0; + + tap_end(testcount, failcount); + if(failcount) + return(-1); + else + return(0); } /* hooks/post-receive -- C-Utils Repository

1 0

C-Utils Repository branch, master, updated. 909b3f77cabebd176824de712059b5d919e52071
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, master has been updated via 909b3f77cabebd176824de712059b5d919e52071 (commit) via 41325a456c461bfc9b9643e58d74d3b39fd25181 (commit) from 1125ad81fbf40f246f71d976d5904ce52c1eb762 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 909b3f77cabebd176824de712059b5d919e52071 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 12:52:15 2013 -0400 update testlist to not require arguments commit 41325a456c461bfc9b9643e58d74d3b39fd25181 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Apr 2 12:49:59 2013 -0400 update testhash so it doesn't require arguments ----------------------------------------------------------------------- Summary of changes: tests/testhash.c | 14 +++++++++++--- tests/testlist.c | 15 ++++++++++++--- 2 files changed, 23 insertions(+), 6 deletions(-) Diff of changes: diff --git a/tests/testhash.c b/tests/testhash.c index 149dcbb..61d366d 100644 --- a/tests/testhash.c +++ b/tests/testhash.c @@ -11,6 +11,8 @@ #include "triton-string.h" #include "triton-uint128.h" +#define DEFAULT_TEST_SIZE 1000 + struct e { int v; @@ -106,9 +108,15 @@ int main(int argc, char *argv[]) triton_string_t *string; struct triton_hash_table *ktable; - assert(argc == 2); - - testsize = atoi(argv[1]); + if(argc == 1) + testsize = DEFAULT_TEST_SIZE; + else if(argc == 2) + testsize = atoi(argv[1]); + else + { + fprintf(stderr, "Error: invalid arguments.\n"); + return(-1); + } assert(testsize > 100); diff --git a/tests/testlist.c b/tests/testlist.c index 00a5486..e02c482 100644 --- a/tests/testlist.c +++ b/tests/testlist.c @@ -5,8 +5,11 @@ */ #include <stdlib.h> +#include <stdio.h> #include "triton-list.h" +#define DEFAULT_TEST_SIZE 1000 + struct e { int v; @@ -20,9 +23,15 @@ int main(int argc, char *argv[]) triton_list_link_t *p, *s; int i, testsize, delcount; - assert(argc == 2); - - testsize = atoi(argv[1]); + if(argc == 1) + testsize = DEFAULT_TEST_SIZE; + else if(argc == 2) + testsize = atoi(argv[1]); + else + { + fprintf(stderr, "Error: invalid arguments.\n"); + return(-1); + } assert(testsize > 100); hooks/post-receive -- C-Utils Repository

1 0

C-Utils Repository branch, cleanup, updated. 1125ad81fbf40f246f71d976d5904ce52c1eb762
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, cleanup has been updated via 1125ad81fbf40f246f71d976d5904ce52c1eb762 (commit) from 020053b2b05370082d43aa924d39bba828286132 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 1125ad81fbf40f246f71d976d5904ce52c1eb762 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Mar 29 11:05:10 2013 -0500 By default, don't build shared version of c-utils ----------------------------------------------------------------------- Summary of changes: configure.ac | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) Diff of changes: diff --git a/configure.ac b/configure.ac index c996202..4c476f2 100644 --- a/configure.ac +++ b/configure.ac @@ -15,7 +15,7 @@ AC_CONFIG_HEADER([c-utils-config.h]) AC_PROG_CC AM_PROG_CC_C_O AM_PROG_AR -AC_PROG_LIBTOOL +LT_INIT([disable-shared]) AC_PROG_AWK AC_PROG_LN_S AS_MKDIR_P @@ -26,6 +26,10 @@ dnl AC_REQUIRE_AUX_FILE([tap-driver.sh]) AC_CHECK_SIZEOF([long int]) +# We don't want to build shared libraries +# not properly setup for it (versioning etc.) +AM_DISABLE_SHARED([true]) + dnl ====================================================================== dnl bundle libev? dnl ====================================================================== hooks/post-receive -- C-Utils Repository

1 0

C-Utils Repository branch, master, updated. 1125ad81fbf40f246f71d976d5904ce52c1eb762
by noreply＠mcs.anl.gov 02 Apr '13

02 Apr '13

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "C-Utils Repository". The branch, master has been updated via 1125ad81fbf40f246f71d976d5904ce52c1eb762 (commit) via 020053b2b05370082d43aa924d39bba828286132 (commit) via 7e9cfef4ee5ecf2587ece0d8946fcd69b384cbee (commit) via 815cb2501574632a4e8122b6e58ecb9e21056e46 (commit) via 5634360b9a82376e1e9cd8c1ff58f73e371f418f (commit) via 58786f549869b1fa86acfebeeaadb5a21e0118e1 (commit) via c0aef804222203b688f633ae8039440dfd07eeb6 (commit) from dd55918cf621cb0767dd2351830e150310a4d20e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 1125ad81fbf40f246f71d976d5904ce52c1eb762 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Fri Mar 29 11:05:10 2013 -0500 By default, don't build shared version of c-utils ----------------------------------------------------------------------- Summary of changes: .gitignore | 31 +++ Makefile.am | 73 +++++++ Makefile.in | 314 ------------------------------ TODO | 10 + configure.ac | 78 ++++++-- depend.sh | 16 -- libev/Makefile.subdir | 33 +++ libev/module.mk.in | 9 - {config => m4}/libev.m4 | 0 {config => m4}/unix.m4 | 0 module.mk.in | 5 - prepare | 9 +- src/Makefile.subdir | 16 ++ c-utils-common.c => src/c-utils-common.c | 0 c-utils-common.h => src/c-utils-common.h | 0 jenkins-hash.h => src/jenkins-hash.h | 0 lookup3.c => src/lookup3.c | 0 triton-base.h => src/triton-base.h | 0 triton-hash.h => src/triton-hash.h | 0 triton-list.h => src/triton-list.h | 0 triton-string.c => src/triton-string.c | 0 triton-string.h => src/triton-string.h | 0 triton-thread.h => src/triton-thread.h | 0 triton-uint128.h => src/triton-uint128.h | 0 workqueue.c => src/workqueue.c | 0 workqueue.h => src/workqueue.h | 0 tests/Makefile.subdir | 17 ++ tests/module.mk.in | 2 - 28 files changed, 246 insertions(+), 367 deletions(-) create mode 100644 .gitignore create mode 100644 Makefile.am delete mode 100644 Makefile.in create mode 100644 TODO delete mode 100755 depend.sh create mode 100644 libev/Makefile.subdir delete mode 100644 libev/module.mk.in rename {config => m4}/libev.m4 (100%) rename {config => m4}/unix.m4 (100%) delete mode 100644 module.mk.in create mode 100644 src/Makefile.subdir rename c-utils-common.c => src/c-utils-common.c (100%) rename c-utils-common.h => src/c-utils-common.h (100%) rename jenkins-hash.h => src/jenkins-hash.h (100%) rename lookup3.c => src/lookup3.c (100%) rename triton-base.h => src/triton-base.h (100%) rename triton-hash.h => src/triton-hash.h (100%) rename triton-list.h => src/triton-list.h (100%) rename triton-string.c => src/triton-string.c (100%) rename triton-string.h => src/triton-string.h (100%) rename triton-thread.h => src/triton-thread.h (100%) rename triton-uint128.h => src/triton-uint128.h (100%) rename workqueue.c => src/workqueue.c (100%) rename workqueue.h => src/workqueue.h (100%) create mode 100644 tests/Makefile.subdir delete mode 100644 tests/module.mk.in Diff of changes: diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f10c7be --- /dev/null +++ b/.gitignore @@ -0,0 +1,31 @@ +.deps + +aclocal.m4 +autom4te.cache +configure +c-utils-config.h.in +m4/libtool.m4 +m4/ltoptions.m4 +m4/ltsugar.m4 +m4/ltversion.m4 +m4/lt~obsolete.m4 +build-aux/missing +build-aux/ltmain.sh +build-aux/install-sh +build-aux/depcomp +build-aux/config.sub +build-aux/config.guess +build-aux/compile +build-aux/ar-lib +Makefile.in +*.o +*.lo +config.h +config.log +c-utils-config.h +Makefile +libtool +stamp-h1 +.dirstamp +.libs +config.status diff --git a/Makefile.am b/Makefile.am new file mode 100644 index 0000000..ffd7bcd --- /dev/null +++ b/Makefile.am @@ -0,0 +1,73 @@ +ACLOCAL_AMFLAGS = -I m4 + +TESTS = + +noinst_HEADERS = +check_PROGRAMS = +bin_PROGRAMS = +noinst_LTLIBRARIES = +lib_LTLIBRARIES = +include_HEADERS = + +AM_CPPFLAGS = + +EXTRA_DIST = COPYRIGHT TODO + +nobase_include_HEADERS= + +#------------------------------------------------------------ + +# List public headers in autoconf @public_headers@ variable +# In addition to being installed, +# this makes sure that copies of (or links to) these headers +# will exist in the build include directory to make embedding c-utils +# easier. +#----------------------------------------------------- + +public_headers= + +include_HEADERS += $(public_headers) + +all-local: header-stamp +check-local: header-stamp + +header-stamp: $(public_headers) + -$(MKDIR_P) $(builddir)/include + -( \ + for a in $^; do\ + cp $$a $(builddir)/include ; \ + done ;\ + ) + -touch header-stamp + +clean-local: + -rm header-stamp + +#----------------------------------------------------- + +# note: requires automake 1.12 or above +# Ignored on older versions +#TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \ +# $(top_srcdir)/build-aux/tap-driver.sh + +#----------------------------------------------------- + +EXTRA_DIST += prepare + +lib_LTLIBRARIES += lib/libc-utils.la +lib_libc_utils_la_SOURCES = $(c_utils_sources) +lib_libc_utils_la_LIBADD = + +#include_HEADERS += container-features.hh + +AM_CPPFLAGS += -I$(top_srcdir)/src + +# Add things that need to go into the library to c_utils_sources +# c_utils_headers is for public headers for the library +c_utils_sources = +c_utils_headers = + +include src/Makefile.subdir +include libev/Makefile.subdir +include tests/Makefile.subdir + diff --git a/Makefile.in b/Makefile.in deleted file mode 100644 index ae9fe99..0000000 --- a/Makefile.in +++ /dev/null @@ -1,314 +0,0 @@ -############################################################### -# General documentation -# -# This is a single makefile that runs the entire c-utils build -# process. There are no makefiles in subdirectories. For a -# general introduction to this approach, please read this document -# by Peter Miller: -# -# http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html -# -# Each subdirectory contains a module.mk file that gets included -# when make is executed. These module.mk files tell make about the -# files in each subdirectory that must be built, including any -# special case rules. Make uses this information to generate a -# sinle dependency graph and orchestrate the build process from this -# top level directory. -# -# We categorize our source depending on what it will be used for. -# For example, there are lists of source files for building the -# server, building the library, building documentation, etc. -# - -############################################################### -# Generic makefile setup - -# define a few generic variables that we need to use; DESTDIR may -# be overridden on the command line during make install -DESTDIR = -srcdir = @srcdir@ -abs_srcdir = @abs_srcdir@ -builddir = @BUILD_ABSOLUTE_TOP@ -prefix = @prefix@ -datarootdir = @datarootdir@ -exec_prefix = @exec_prefix@ -includedir = $(DESTDIR)@includedir@ -mandir = $(DESTDIR)@mandir@ -sbindir = $(DESTDIR)@sbindir@ -bindir = $(DESTDIR)@bindir@ -libdir = $(DESTDIR)@libdir@ - -# subdirs that contain configure/make -# SUBDIRS = @subdirs@ - -VPATH = $(srcdir) - -CC = @CC@ -LD = @LD@ -BUILD_CC = @BUILD_CC@ -BUILD_LD = @BUILD_LD@ -BUILD_CFLAGS = @BUILD_CFLAGS@ -BUILD_LDFLAGS = @BUILD_LDFLAGS@ - -STRICT_CFLAGS = @STRICT_CFLAGS@ -VALGRIND_CLEAN = @VALGRIND_CLEAN@ -CPPFLAGS = @CPPFLAGS@ - -ifneq (0,$(VALGRIND_CLEAN)) - CPPFLAGS += -DVALGRIND_CLEAN -DVALGRIND -endif - -CFLAGS= -I$(srcdir)/include -I. -I$(srcdir) @CFLAGS@ $(CPPFLAGS) - -CWARNS= -ifneq (0,$(STRICT_CFLAGS)) - CWARNS += -Wcast-align -Wbad-function-cast - CWARNS += -Wmissing-prototypes -Wmissing-declarations - CWARNS += -Wundef -Wpointer-arith - CWARNS += -Wnested-externs - CWARNS += -Wredundant-decls - # Very noisy, too strict - CWARNS += -W -Wno-unused-label -Wno-sign-compare - CWARNS += -Wcast-qual - CWARNS += -Wshadow - CWARNS += -Wwrite-strings - CWARNS += -Wformat -endif - -# don't want to warn when we call format functions without string literal parameters -# e.g. triton_string_init(&str, myname); -CWARNS += -Wno-format-security - -LDSHARED = $(TCC) -shared - -# tell libev to use our config.h instead of its own; we can't just put this -# in the module.mk.in because we need it to take effect when the .d file is -# generated as well. -LIBCFLAGS += -DEV_CONFIG_H=\"c-utils-config.h\" -LDFLAGS += @LDFLAGS@ -LIBS += @LIBS@ - -DEPLIBS += @THREAD_LIB@ - -CPPDEFS=$(filter -D%,$(LIBCFLAGS) $(CFLAGS)) -CPPINCS=$(filter -I%,$(LIBCFLAGS) $(CFLAGS)) - -USE_DYLIBS = @USE_DYLIBS@ - -LIBSRC := -TESTSRC := - -# Eliminate all default suffixes. We want explicit control. -.SUFFIXES: - -QUIET_COMPILE = @QUIET_COMPILE@ -WARNINGS_AS_ERRORS = @WARNINGS_AS_ERRORS@ - -ifdef V - QUIET_COMPILE = 0 -endif - -ifdef W - ifeq ($(W),1) - WARNINGS_AS_ERRORS = 1 - else - WARNINGS_AS_ERRORS = 0 - endif -endif - -ifdef D - DA="-l" -else - DA= -endif - -ifeq ($(QUIET_COMPILE),1) - # say a one-line description of the action, do not echo the command - Q=@echo - E=@ - VA= -else - # do not say the short Q lines, but do echo the entire command - Q=@echo >/dev/null - E= - VA="-v" -endif - -ifeq ($(WARNINGS_AS_ERRORS),1) - EWARNS = -Werror -else - EWARNS = -endif - -# PHONY targets are targets that do not result in the generation -# of a file that has the same name as the target. Listing them -# here keeps make from accidentally doing too much work (see GNU -# make manual). -.PHONY: all clean - -################################################################ -# Find project subdirectories - -# MODULES is a list of subdirectories that we wish to operate on. -# They are identified by the presence of module.mk files (makefile -# includes). -MODULES := $(shell find . -name "*.mk" | sed -e 's/^.\///;s/module.mk//') - - -################################################################ -# Default target forward pointer, to avoid other targets in make stubs -all:: - -################################################################ -# Makefile includes - -# this is how we pull build information from all of the project -# subdirectories, make sure to catch top level module.mk as well -include $(patsubst %, %/module.mk, $(MODULES)) module.mk - -# Just like dir, but strip the slash off the end, to be pretty. -dirname = $(patsubst %/,%,$(dir $(1))) - -# Generate the canonical in-tree location of a file, given a possible out -# of tree reference. -canonname = $(patsubst $(srcdir)/%,%,$(call dirname,$(1))) - -canonfname = $(patsubst $(srcdir)/%,%,$(1)) - -# the "lastword" function is more concise, but not available on -# older versions of GNU Make -# bname = $(lastword $(subst /, ,$(1))) -bname = $(word $(words $(subst /, ,$(1))),$(subst /, ,$(1))) - -canondirname = $(subst /,_,$(1)) - -################################################################ -# Derived file lists - -# NOTES: At this point, the subdirectory makefile includes have informed -# us what the source files are. Now we want to generate some -# other lists (such as objects, executables, and dependency files) -# by manipulating the lists of source files - -# Grab any CFLAGS defined by the make stub for a particular file, and -# for the directory in which the source resides. -# Similar for ldflags. -modcflags = $(MODCFLAGS_$(call canonname,$(1))) \ - $(MODCFLAGS_$(patsubst $(srcdir)/%,%,$(1))) - -MODCFLAGS_C = $(call modcflags,$(1)) - -modldflags = $(MODLDFLAGS_$(call canonname,$(1))) \ - $(MODLDFLAGS_$(patsubst $(srcdir)/%,%,$(1))) - -MODLDFLAGS_C = $(call modldflags,$(1)) - -modlibs = $(MODLIBS_$(call canonname,$(1))) \ - $(MODLIBS_$(patsubst $(srcdir)/%,%,$(1))) - -MODLIBS_C = $(call modlibs,$(1)) - -modname = $(MODNAME_$(call canonname,$(1))) \ - $(MODNAME_$(patsubst $(srcdir)/%,%,$(1))) - -# use TCC variable so that we can keep CC as the system compiler -TCC = $(if $(MPICC),$(MPICC),$(CC)) -CPPCC = $(CC) - -modcc = $(if $(MODCC_$(1)),$(MODCC_$(1)),$(TCC)) - -MCC_C = $(call modcc,$(1)) - -# The object files for the library generated from .c files -LIBOBJS=$(patsubst %.c,%.o,$(LIBSRC)) -LIBDEPS=$(patsubst %.c,%.d,$(LIBSRC)) -TESTOBJS=$(patsubst %.c,%.o,$(TESTSRC)) -TESTDEPS=$(patsubst %.c,%.d,$(TESTSRC)) -TESTPROGS=$(patsubst %.c,%,$(TESTSRC)) - -DEPENDS=$(LIBDEPS) $(TESTDEPS) - -# for now we just use static if we're not on a mac -ifneq (,$(USE_DYLIBS)) -LIBRARIES=libc-utils.dylib -else -LIBRARIES=libc-utils.a -endif - -#################################################################### -# Rules and dependencies - -# Normal rules - -$(LIBOBJS) $(TESTOBJS): %.o: %.c - $(Q) " CC $@" - $(E)$(call MCC_C,$*) -c -o $@ \ - $(LIBCFLAGS) $(CFLAGS) $(CWARNS) \ - $(call MODCFLAGS_C,$*) $< - -$(TESTPROGS): %: %.o $(LIBRARIES) - $(Q) " LD $@" - $(E)$(call MCC_C,$*) \ - -o $@ $< $(LDFLAGS) \ - -L$(builddir) -lc-utils $(call MODLIBS_C,$*) \ - $(LIBS) - -# Auto-generated dependencies for .c files -$(LIBDEPS) $(TESTDEPS): %.d: %.c - $(Q) " DEP $@" - $(E)CC="$(call MCC_C,$*)" $(srcdir)/depend.sh $(call dirname,$*) $(LIBCFLAGS) $(CFLAGS) $(call MODCFLAGS_C,$*) $< > $@ - -libc-utils.a: $(LIBOBJS) - $(Q) " RANLIB $@" - $(E)ar rcs $@ $(LIBOBJS) - -# shared objects - -libtriton.so: $(LIBOBJS) - $(Q) " LDSO $@" - $(E)$(LDSHARED) -Wl,-soname,libtriton.so -o $@ $(LIBOBJS) - -libtriton.dylib: $(LIBOBJS) - $(Q) " DYLIB $@" - $(E)$(TCC) -o libtriton.dylib -dynamiclib -current_version 0.0.1 -install_name lib/libtriton.0.0.1.dylib \ - $(LIBOBJS) $(DEPLIBS) - -# User targets - -all:: $(LIBRARIES) - -clean:: - $(Q) " CLEAN" - $(E)rm -f $(LIBOBJS) $(LIBRARIES) $(DEPENDS) - -tests:: $(TESTPROGS) - -# this is where we include all of our automatic dependencies. -# NOTE: we wrap this in ifneq's in order to prevent the dependencies -# from being generated for special targets that don't require them -ifeq (,$(filter clean nodep,$(MAKECMDGOALS))) --include $(DEPENDS) -endif -# add this as a make goal to disable rebuilding deps -.PHONY: nodep -.nodep:; @: - -# rule for generating cscope information -cscope: - find @SRC_ABSOLUTE_TOP@ -regex ".*\.$c\|h\|ae\|hae\|aer\|haer$" -o -iname "*.sm" \ - > $(srcdir)/cscope.files - ( cd @SRC_ABSOLUTE_TOP@; cscope -be -i @SRC_ABSOLUTE_TOP@/cscope.files ) - -tags:: - find @SRC_ABSOLUTE_TOP@ -regex ".*\.$c\|h\|ae\|hae\|aer\|haer$" | ctags -L- --excmd=pattern -B --extra=+f --langmap=c:+.ae.aer.hae.haer -I pwait,pbranch -f @SRC_ABSOLUTE_TOP@/tags - -etags: -# Note: this only reads src and prunes the test directories - cd @SRC_ABSOLUTE_TOP@ ; find src \ - -name test -prune -o -name tests -prune -o \ - -regex ".*\.$c\|h\|ae\|hae\|aer\|haer$" -print \ - | xargs etags - -install:: all - install -d $(libdir) - install -m 644 libc-utils.* $(libdir) diff --git a/TODO b/TODO new file mode 100644 index 0000000..72d640a --- /dev/null +++ b/TODO @@ -0,0 +1,10 @@ +TODO Items: + +- cleanup library naming (get rid of triton) + - triton-string -> cu-string.h or something + +- check if easy/possible to rename ev.h when installing to avoid conflict + (or should we just use the public libev?) + +- fix tests so that they output TAP + diff --git a/configure.ac b/configure.ac index 38bcbf2..4c476f2 100644 --- a/configure.ac +++ b/configure.ac @@ -1,17 +1,64 @@ -dnl ************************************************************** -dnl -dnl Process this file with autoconf to produce a configure script. -dnl You may need to use autoheader as well if changing any DEFINEs - AC_PREREQ(2.50) -AC_INIT(c-utils, 0.0.1, triton-dev(a)mcs.anl.gov) +AC_INIT([c-utils], [0.0.1], + [[email protected]],[],[http://trac.mcs.anl.gov/projects/c-utils]) + +AC_CONFIG_MACRO_DIR([m4]) + +AC_CONFIG_AUX_DIR([build-aux]) + +AM_INIT_AUTOMAKE([foreign subdir-objects -Wall]) +m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) -AC_CONFIG_HEADER(c-utils-config.h) +AC_CONFIG_SRCDIR([src/c-utils-common.h]) +AC_CONFIG_HEADER([c-utils-config.h]) AC_PROG_CC +AM_PROG_CC_C_O +AM_PROG_AR +LT_INIT([disable-shared]) +AC_PROG_AWK +AC_PROG_LN_S +AS_MKDIR_P + +dnl AC_REQUIRE_AUX_FILE([tap-driver.sh]) + AC_CHECK_SIZEOF([long int]) + +# We don't want to build shared libraries +# not properly setup for it (versioning etc.) +AM_DISABLE_SHARED([true]) + +dnl ====================================================================== +dnl bundle libev? +dnl ====================================================================== + +AC_ARG_ENABLE(libev, + [AS_HELP_STRING( + [--enable-libev], + [Bundle libev into c-utils]) + ]) +BUNDLE_LIBEV= +AS_IF([test "x$enable_libev" = "xyes"], [BUNDLE_LIBEV=1]) +AC_SUBST([BUNDLE_LIBEV]) + +AM_CONDITIONAL([LIBEV],[test "x$BUNDLE_LIBEV" == "x1"]) + +AC_MSG_CHECKING([if we need to bundle libev]) + +if test "x$BUNDLE_LIBEV" == "x1"; then + AC_MSG_RESULT([yes]) + + # bring in libev tests + m4_include([m4/libev.m4]) + +else + AC_MSG_RESULT([no]) +fi + + + dnl ====================================================================== dnl Try harder to be valgrind safe dnl ====================================================================== @@ -36,7 +83,12 @@ AC_ARG_WITH(perftools, [Location of google perftools installation]), PERFTOOL_DIR="$withval",PERFTOOL_DIR="") -if test -n "$PERFTOOL_DIR" ; +if test "X${PERFTOOL_DIR}" == "Xyes"; then + # if not specified, + PERFTOOL_DIR="/usr" +fi + +if test "X${PERFTOOL_DIR}" != "Xno"; then CPPFLAGS="$CPPFLAGS -I${PERFTOOL_DIR}/include" LDFLAGS="$LDFLAGS -L${PERFTOOL_DIR}/lib" @@ -84,6 +136,7 @@ AC_SUBST(LDFLAGS) AC_SUBST(LIBS) AC_SUBST(CPPFLAGS) +dnl this can probably go AC_ARG_ENABLE(verbose, [ --enable-verbose Enables verbose output during build process], QUIET_COMPILE=0, @@ -114,11 +167,8 @@ AC_ARG_ENABLE(coverage, ) AC_SUBST(ENABLE_COVERAGE) -m4_include([config/libev.m4]) +AC_CONFIG_FILES([Makefile]) -AC_OUTPUT([Makefile -module.mk -tests/module.mk -libev/module.mk -]) +AC_CONFIG_LINKS([config.h:c-utils-config.h]) +AC_OUTPUT diff --git a/depend.sh b/depend.sh deleted file mode 100755 index e05fe68..0000000 --- a/depend.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/sh -DIR="$1" -shift -if [ -n "$DIR" ] ; then - DIR="$DIR"/ -fi -# Explanation of some arguments: -# -MG tells gcc to ignore errors from missing header files, which is -# important in Triton because some of them are autogenerated and won't exist -# yet when we run this tool. -# -MM tells gcc not to bother including system header files. Our -# build is slow enough already so it isn't worth it. -# -x c tells gcc to pretend like the input file is a c file regardless of -# filename extension. This is useful for cases where we want gcc to -# generate dependencies on raw .ae files. -exec $CC -x c -M -MM -MG "$@" | sed -e "s@^$.*$\.o:@$DIR\1.o:@" diff --git a/libev/Makefile.subdir b/libev/Makefile.subdir new file mode 100644 index 0000000..58c8e6b --- /dev/null +++ b/libev/Makefile.subdir @@ -0,0 +1,33 @@ + +if LIBEV +# build libev as convenience library so we can easily override the CFLAGS/etc. +# for this library +noinst_LTLIBRARIES += libev/libev.la + +# add it into libc-utils so we don't have to install 2 libraries +lib_libc_utils_la_LIBADD += libev/libev.la + +libev_libev_la_SOURCES = libev/ev.c \ + libev/event.c \ + libev/ev++.h \ + libev/ev_vars.h\ + libev/ev_wrap.h\ + libev/event.h + +# turn off warnings for libev compile +libev_libev_la_CFLAGS = $(AM_CFLAGS) -w + +# note: the other .c files in this directory are included automatically by +# ev.c as needed; don't compile them explicitly +noinst_HEADERS += \ + libev/ev_epoll.c \ + libev/ev_kqueue.c\ + libev/ev_poll.c\ + libev/ev_port.c\ + libev/ev_select.c\ + libev/ev_win32.c + +public_headers += \ + libev/ev.h + +endif diff --git a/libev/module.mk.in b/libev/module.mk.in deleted file mode 100644 index 7a3ec40..0000000 --- a/libev/module.mk.in +++ /dev/null @@ -1,9 +0,0 @@ -DIR := libev - -# note: the other .c files in this directory are included automatically by -# ev.c as needed; don't compile them explicitly -LIBSRC += $(DIR)/ev.c \ - $(DIR)/event.c - -# turn off warnings for libev compile -MODCFLAGS_$(DIR) = -w diff --git a/config/libev.m4 b/m4/libev.m4 similarity index 100% rename from config/libev.m4 rename to m4/libev.m4 diff --git a/config/unix.m4 b/m4/unix.m4 similarity index 100% rename from config/unix.m4 rename to m4/unix.m4 diff --git a/module.mk.in b/module.mk.in deleted file mode 100644 index a43bfe8..0000000 --- a/module.mk.in +++ /dev/null @@ -1,5 +0,0 @@ -LIBSRC += c-utils-common.c \ - triton-string.c \ - lookup3.c \ - workqueue.c - diff --git a/prepare b/prepare index a5b2ffe..9c00396 100755 --- a/prepare +++ b/prepare @@ -1,9 +1,4 @@ #!/bin/sh -if aclocal --force -I ./config && autoheader -f && autoconf -f; then - echo "configure script successfully regenerated" - exit 0 -else - echo "some part of configure regeneration failed" - exit -1 -fi +autoreconf -fi || (echo "Autogen failed!"; exit 1) + diff --git a/src/Makefile.subdir b/src/Makefile.subdir new file mode 100644 index 0000000..dc29495 --- /dev/null +++ b/src/Makefile.subdir @@ -0,0 +1,16 @@ +c_utils_sources += \ +src/c-utils-common.c\ +src/lookup3.c\ +src/triton-string.c\ +src/workqueue.c + +public_headers += src/c-utils-common.h \ + src/triton-base.h \ + src/triton-list.h \ + src/triton-string.h \ + src/triton-thread.h \ + src/triton-hash.h \ + src/triton-uint128.h \ + src/workqueue.h \ + src/jenkins-hash.h + diff --git a/c-utils-common.c b/src/c-utils-common.c similarity index 100% rename from c-utils-common.c rename to src/c-utils-common.c diff --git a/c-utils-common.h b/src/c-utils-common.h similarity index 100% rename from c-utils-common.h rename to src/c-utils-common.h diff --git a/jenkins-hash.h b/src/jenkins-hash.h similarity index 100% rename from jenkins-hash.h rename to src/jenkins-hash.h diff --git a/lookup3.c b/src/lookup3.c similarity index 100% rename from lookup3.c rename to src/lookup3.c diff --git a/triton-base.h b/src/triton-base.h similarity index 100% rename from triton-base.h rename to src/triton-base.h diff --git a/triton-hash.h b/src/triton-hash.h similarity index 100% rename from triton-hash.h rename to src/triton-hash.h diff --git a/triton-list.h b/src/triton-list.h similarity index 100% rename from triton-list.h rename to src/triton-list.h diff --git a/triton-string.c b/src/triton-string.c similarity index 100% rename from triton-string.c rename to src/triton-string.c diff --git a/triton-string.h b/src/triton-string.h similarity index 100% rename from triton-string.h rename to src/triton-string.h diff --git a/triton-thread.h b/src/triton-thread.h similarity index 100% rename from triton-thread.h rename to src/triton-thread.h diff --git a/triton-uint128.h b/src/triton-uint128.h similarity index 100% rename from triton-uint128.h rename to src/triton-uint128.h diff --git a/workqueue.c b/src/workqueue.c similarity index 100% rename from workqueue.c rename to src/workqueue.c diff --git a/workqueue.h b/src/workqueue.h similarity index 100% rename from workqueue.h rename to src/workqueue.h diff --git a/tests/Makefile.subdir b/tests/Makefile.subdir new file mode 100644 index 0000000..8238ab0 --- /dev/null +++ b/tests/Makefile.subdir @@ -0,0 +1,17 @@ + +check_PROGRAMS += tests/testlist \ + tests/testhash \ + tests/test-uint128 + +TESTS += tests/testlist \ + tests/testhash \ + tests/test-uint128 + +testlib = lib/libc-utils.la + +tests_testlist_LDADD = $(testlib) +tests_testhash_LDADD = $(testlib) +tests_test_uint128_LDADD = $(testlib) + +#TESTSRC += tests/testlist.c tests/testhash.c tests/test-uint128.c + diff --git a/tests/module.mk.in b/tests/module.mk.in deleted file mode 100644 index 4cc273c..0000000 --- a/tests/module.mk.in +++ /dev/null @@ -1,2 +0,0 @@ -TESTSRC += tests/testlist.c tests/testhash.c tests/test-uint128.c - hooks/post-receive -- C-Utils Repository

1 0