March 2012 - Asg - mailman3.cels.anl.gov

Re: [ASG] [aesop] #7: integrate library linking and paths with aecc
by aesop 30 Mar '12

30 Mar '12

#7: integrate library linking and paths with aecc ---------------------------+------------------------------------------------ Reporter: carns | Owner: Type: defect | Status: assigned Priority: major | Component: code generator Version: git repository | Keywords: ---------------------------+------------------------------------------------ Changes (by carns): * owner: carns => * status: accepted => assigned Comment: This task isn't 100% complete, but enough is done that aecc can be used cleanly outside of the original aesop build tree. I'm releasing the ticket for now, and either I or someone else can accept it later to continue polishing. * The underlying compiler is specified at configure time (rather than defaulting to gcc) * gcc and clang both work properly for all code in the aesop tree * aecc automatically works properly whether it is executed from a build tree or from an installation prefix * aecc does not require the original build or src directory to operate, only the installation directory * aecc automatically sets required CFLAGS for compilation The use of aecc for external projects is documented here: [wiki:"Using Aesop in other projects"]. The following tasks still remain if we want to polish this further: * The aecc command line arguments are still a little strange. You must separate normal C compiler arguments with a -- delimiter. * Aecc does not do linking, and therefore does not automatically link in appropriate libraries. -- Ticket URL: <https://trac.mcs.anl.gov/projects/aesop/ticket/7#comment:5> aesop <https://trac.mcs.anl.gov/projects/aesop> The Aesop Language

1 0

aesop Repository branch, master, updated. 3aa8160a8eb61d564b1cd65a3edc6cceaef3a724
by noreply＠mcs.anl.gov 30 Mar '12

30 Mar '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via 3aa8160a8eb61d564b1cd65a3edc6cceaef3a724 (commit) via a0a4d18c196918ca5e6b90003ffc7f278fd69a40 (commit) from a3a5c22c9ef1de98f051657fc32ce0a34c4a412e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 3aa8160a8eb61d564b1cd65a3edc6cceaef3a724 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 14:21:32 2012 -0400 implicitly set include path for aesop headers commit a0a4d18c196918ca5e6b90003ffc7f278fd69a40 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 13:52:36 2012 -0400 set some intrisic cflags in aecc automatically ----------------------------------------------------------------------- Summary of changes: bin/aecc-utils.in | 11 ++++++----- configure.ac | 4 ++++ 2 files changed, 10 insertions(+), 5 deletions(-) Diff of changes: diff --git a/bin/aecc-utils.in b/bin/aecc-utils.in index 8067f85..c8e7014 100644 --- a/bin/aecc-utils.in +++ b/bin/aecc-utils.in @@ -1,6 +1,7 @@ # find the location of this aecc script DIR="$( cd "$( dirname "$0" )" && pwd )" +AECC_CFLAGS="@AECC_CFLAGS@ -I$DIR/../include" # the parsers can be in one of two locations; either in the same directory # as this script (if the tools have been installed) or in the ../parser @@ -539,7 +540,7 @@ doCPP() jcppopt="-D__blocking='' -Dpbranch='' -Dpwait='' -Dpbreak=''" fi runCmd $verbose "Preprocessing step failed:" \ - "${ccpath} -DAESOP_PARSER -x c ${jcppopt} -o ${cppout} ${ccopts} ${cppinput}" + "${ccpath} -DAESOP_PARSER -x c ${jcppopt} -o ${cppout} ${AECC_CFLAGS} ${ccopts} ${cppinput}" return $? } @@ -561,7 +562,7 @@ parseAesopSource() fi runCmd $verbose "Aesop parsing step failed: " \ - "${AEPARSER} -c ${compiler} ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${compileropts}" + "${AEPARSER} -c ${compiler} ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${AECC_CFLAGS} ${compileropts}" return $? } @@ -583,7 +584,7 @@ parseAesopHeader() fi runCmd $verbose "Aesop header parsing failed: " \ - "${AEPARSER} -c ${compiler} -j ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${compileropts}" + "${AEPARSER} -c ${compiler} -j ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${AECC_CFLAGS} ${compileropts}" return $? } @@ -628,7 +629,7 @@ parseRemoteSource() done runCmd $verbose "Aesop remote parsing failed: " \ - "${AERPARSER} ${serviceopts} ${regopts} -c ${compiler} ${popts} ${namedopt} ${outopt} ${inputopt} -- ${compileropts}" + "${AERPARSER} ${serviceopts} ${regopts} -c ${compiler} ${popts} ${namedopt} ${outopt} ${inputopt} -- ${AECC_CFLAGS} ${compileropts}" return $? } @@ -645,6 +646,6 @@ doCompile() fi runCmd $verbose "Compilation step failed: " \ - "${compiler} -x c -c -o ${outfile} ${compileropts} ${infile}" + "${compiler} -x c -c -o ${outfile} ${AECC_CFLAGS} ${compileropts} ${infile}" return $? } diff --git a/configure.ac b/configure.ac index 1b6ea22..cb0f4d5 100644 --- a/configure.ac +++ b/configure.ac @@ -311,6 +311,10 @@ AC_COMPILE_IFELSE( AC_SUBST(BLOCKSOPT) +# these are cflags that will be automatically applied by aecc +AECC_CFLAGS=$LIBCFLAGS +AC_SUBST(AECC_CFLAGS) + AC_SUBST(LIBCFLAGS) AC_SUBST(THREAD_LIB) hooks/post-receive -- aesop Repository

1 0

aesop Repository branch, master, updated. a3a5c22c9ef1de98f051657fc32ce0a34c4a412e
by noreply＠mcs.anl.gov 30 Mar '12

30 Mar '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "aesop Repository". The branch, master has been updated via a3a5c22c9ef1de98f051657fc32ce0a34c4a412e (commit) via ee239226683200ec1fc2af7b8b7514801e07d8b2 (commit) via 4343eb2dc271124e0d6d538246f66c52b9c52a70 (commit) via a5023c1a5420ee439654381c267fdf32e1b48029 (commit) from 6d276d9a5f67af975494cf30561e26720058629c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit a3a5c22c9ef1de98f051657fc32ce0a34c4a412e Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 13:32:06 2012 -0400 install the underlying haskell binary parser commit ee239226683200ec1fc2af7b8b7514801e07d8b2 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 13:31:02 2012 -0400 remove unused static srcdir references commit 4343eb2dc271124e0d6d538246f66c52b9c52a70 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 13:23:19 2012 -0400 remove deprecated dependency options commit a5023c1a5420ee439654381c267fdf32e1b48029 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 30 13:20:15 2012 -0400 locate aecc-utils and parsers at run time ----------------------------------------------------------------------- Summary of changes: Makefile.in | 1 + bin/aecc-utils.in | 37 +++++++++++++++++++------------------ bin/aecc.in | 9 +++++---- 3 files changed, 25 insertions(+), 22 deletions(-) Diff of changes: diff --git a/Makefile.in b/Makefile.in index 07a34a8..01806ef 100644 --- a/Makefile.in +++ b/Makefile.in @@ -692,6 +692,7 @@ install:: all install -d $(bindir) install -d $(libdir) install -d $(includedir) + install -m 755 parser/ae-blocking-parser $(bindir) install -m 755 bin/aecc $(bindir) install -m 755 bin/aecc-utils $(bindir) install -m 644 lib/libaesop.* $(libdir) diff --git a/bin/aecc-utils.in b/bin/aecc-utils.in index 9c708ed..8067f85 100644 --- a/bin/aecc-utils.in +++ b/bin/aecc-utils.in @@ -1,11 +1,20 @@ -SRCDIR=@SRC_RELATIVE_TOP@ -BUILDDIR=@BUILD_ABSOLUTE_TOP@ +# find the location of this aecc script +DIR="$( cd "$( dirname "$0" )" && pwd )" -AEPARSER=${BUILDDIR}/parser/ae-blocking-parser -AERPARSER=${BUILDDIR}/parser/ae-remote-parser -AESRCDIR=${SRCDIR}/src/aesop +# the parsers can be in one of two locations; either in the same directory +# as this script (if the tools have been installed) or in the ../parser +# directory (if we are building in tree) +if [ -x ${DIR}/ae-blocking-parser ]; then + AEPARSER=${DIR}/ae-blocking-parser +elif [ -x ${DIR}/../parser/ae-blocking-parser ]; then + AEPARSER=${DIR}/../parser/ae-blocking-parser +else + echo Unable to find ae-blocking-parser + exit 1 +fi + SPACES=" " optind=0 @@ -514,31 +523,23 @@ runCmd() rm ${compilerout} } -# doCPP <verbose> <compiler> <input> <output> <gendeps> <named outfile> <just cpp> [<compiler opts>] +# doCPP <verbose> <compiler> <input> <output> <named outfile> <just cpp> [<compiler opts>] doCPP() { verbose="$1" ; shift ccpath="$1" ; shift cppinput="$1" ; shift cppout="$1" ; shift - gendeps="$1" ; shift namedoutfile="$1" ; shift precompile="$1" ; shift ccopts=$@ - - depargs="" - if test "${gendeps}" = "yes"; then - depoutsrc=$(echo "${cppinput}" | sed -e "s|$|.d|") - depout=$(echo "${depoutsrc}" | sed -e "s|^${SRCDIR}|${BUILDDIR}|") - depargs="-MMD -MF ${depout} -MT ${namedoutfile}" - fi jcppopt="-E" if test "${precompile}" = "yes"; then jcppopt="-D__blocking='' -Dpbranch='' -Dpwait='' -Dpbreak=''" fi runCmd $verbose "Preprocessing step failed:" \ - "${ccpath} -DAESOP_PARSER -x c ${jcppopt} -o ${cppout} ${ccopts} ${cppinput} ${depargs}" + "${ccpath} -DAESOP_PARSER -x c ${jcppopt} -o ${cppout} ${ccopts} ${cppinput}" return $? } @@ -560,7 +561,7 @@ parseAesopSource() fi runCmd $verbose "Aesop parsing step failed: " \ - "${AEPARSER} -c ${compiler} ${popts} -r ${namedinput} -o ${output} -i ${input} -- -I${AESRCDIR} ${compileropts}" + "${AEPARSER} -c ${compiler} ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${compileropts}" return $? } @@ -582,7 +583,7 @@ parseAesopHeader() fi runCmd $verbose "Aesop header parsing failed: " \ - "${AEPARSER} -c ${compiler} -j ${popts} -r ${namedinput} -o ${output} -i ${input} -- -I${AESRCDIR} ${compileropts}" + "${AEPARSER} -c ${compiler} -j ${popts} -r ${namedinput} -o ${output} -i ${input} -- ${compileropts}" return $? } @@ -627,7 +628,7 @@ parseRemoteSource() done runCmd $verbose "Aesop remote parsing failed: " \ - "${AERPARSER} ${serviceopts} ${regopts} -c ${compiler} ${popts} ${namedopt} ${outopt} ${inputopt} -- -I${AESRCDIR} ${compileropts}" + "${AERPARSER} ${serviceopts} ${regopts} -c ${compiler} ${popts} ${namedopt} ${outopt} ${inputopt} -- ${compileropts}" return $? } diff --git a/bin/aecc.in b/bin/aecc.in index 98510a0..dbbea92 100644 --- a/bin/aecc.in +++ b/bin/aecc.in @@ -1,8 +1,10 @@ #!/bin/sh -BUILDDIR=@BUILD_ABSOLUTE_TOP@ +# find the location of this aecc script +DIR="$( cd "$( dirname "$0" )" && pwd )" -. ${BUILDDIR}/bin/aecc-utils +# aecc utils should be in the same dir as aecc +. ${DIR}/aecc-utils setUsageMessage "Usage: aecc <options> <file> -- <compiler options>. Specify --help for a list of options." @@ -18,7 +20,6 @@ addOption v "show-commands" no none no empty none "Show commands run." addOption o "output-file" yes outputfile no "<output file>" none "Specifies the output file [default=a.out]." addOption i "interm-file" yes intermfile no "<interm file>" none "Specifies the intermediate file." addOption S "from-interm" no none no empty none "Start from (existing) intermediate file." -addOption M "generate-deps" no none no empty none "Generate dependency file during preprocessing step." addOption j "parse-hae" no none no empty none "Generate a C .h header from an Aesop .hae header." addOption l "pre-compile" no none no empty none "Pre-compile aesop for syntax checking." @@ -76,7 +77,7 @@ if test "${hasOpt_from_interm}" != "yes"; then fi # Step 1: first generate preprocessed file and dependency file - doCPP ${verbose} ${compiler} ${INPUTFILES} ${cppout} "${hasOpt_generate_deps}" ${outputfile} "${hasOpt_pre_compile}" ${COMPILEROPTS} + doCPP ${verbose} ${compiler} ${INPUTFILES} ${cppout} ${outputfile} "${hasOpt_pre_compile}" ${COMPILEROPTS} exitIfFailed $? # Step 2: next run parser on preprocessed file hooks/post-receive -- aesop Repository

1 0

A quick discussion of server-side data filtering work
by Settlemyer, Bradley W. 30 Mar '12

30 Mar '12

Hello So, there are a couple different approaches to how large data stores are providing query/selection functionalities. The two major ones seem to be query languages and programmatic interfaces. Summarizing everything is difficult, but here is a brief history of data store selection techniques: Pre-SQL: Databases existed, but they all had their own proprietary access interfaces. This is interesting, mainly because some of the Map-Reduce ideas are really re-imaginations of the mainframe data storage ideas from this time. SQL: The Structured Query Language. My favorite reference on this is Database Systems by Chris Date. This is the relational model to data selection, and provides a very capable language for filtering and combing data. No point getting into the details, but one key idea is the E-R model. Entities and Relations. Entities are tables that have pre-defined schema, and the relations describe the relationships between tables (many-to-one, etc.). File-based selection languages: Scripting tools we are all familiar with like sed, awk, and cut basically allow data files to be manipulated as a series of columns, or as a series of rows. Object-oriented programmatic APIs: Kindof popular in the scientific realm, object-oriented data selection may be more appropriate. One example is the Atlas EventStore for CERN (built partly at ANL I think), and then of course a lot of the Babar framework stuff at SLAC. Effectively, these ideas tend to capture an implementable subset of the Object Query Language (OQL). Obviously this could all be done with SQL, but the relationship between derived classes is a little different than one-to-one, so it makes some operations (called slimming by EventStore) more natural. Range-based data selection: This is more like where DataCutter, and I think SciDB tend to fit. Effectively they interpret data sets as long streams of data to be manipulated. NoSQL approaches: NoSQL stores are huge collections of key-value pairs, and both programming APIs and scripting languages have been designed to troll through the data. The Rob Pike approach is Sawzall, a statically typed scripting language that provides sed and awk like capabilities. Yahoo instead provides Pig-latin, which is more like a query language (Hive and others are similar). Mongo DB instead provides something that looks more like a programmatic interface, though an interpreter is provided to make ad-hoc queries possible as well. Basically, for all of these services (except the old scripting tools) a server-side middleware is acting as a type of query planner. The languages or data apis are broken down into a tree structure, and the server-side middleware tries to figure out a smart way to apply the query to the data store that prevents seeks. The key implication, is that the middleware tends to know a lot about how the data is laid out — and it must to make good decisions. In the case of an SQL data store, the query planner has to be centralized due to ACID compliance. The object-oriented query tools are trying to enhance the E-R model, so that's more of a semantic improvement, and still requires a more or less centralized query planner for the data semantic. The more recent data selection approaches have the benefit of allowing decentralized query planning. In the case of DataCutter, the middleware collaborates to buld a tree/graph of filters to perform the data filtering. SciDB and the NoSQL approaches are more shared-nothing style approaches. An initial plan is developed at a single centralized planner, but then additional work is sent to each backend, and optimized locally by the server-side selection software. Cheers, Brad

2 1

Reminder for ASG Model Concall [1pm-3pm CST]
by Dries Kimpe 30 Mar '12

30 Mar '12

[Logistics] Time : 1pm - 3pm CST Conference Access Number : 760-984-1000 Conference Passcode : 692481 # Topic : Continue discussion on Storage Model For ANL : Room 3173 [Agenda] ▰ Logistics: document access ▰ Document Update ◾ Review and update existing content ◾ Discuss noted questions/clarification points ▰ Use Cases ▰ Open Discussion

1 0

Wiki update: File:Aesop-performance-deliverable-2.3.2-03-2012.pdf
by Wiki update: Asg-private: Rross 30 Mar '12

30 Mar '12

1 0

Wiki update: File:Mpo-40001110000-report-input-03-2012.pdf
by Wiki update: Asg-private: Rross 30 Mar '12

30 Mar '12

1 0

Wiki update: Reporting
by Wiki update: Asg-private: Rross 30 Mar '12

30 Mar '12

1 0

Wiki update: File:Mpo-40001110000-deliverables-update-03-2012.pdf
by Wiki update: Asg-private: Rross 30 Mar '12

30 Mar '12

1 0

Model Concall tomorrow + Agenda / Document
by Dries Kimpe 29 Mar '12

29 Mar '12

Reminder for the concall tomorrow: [Logistics] Time : 1pm - 3pm CST Conference Access Number : 760-984-1000 Conference Passcode : 692481 # Topic : Continue discussion on Storage Model [Agenda] (please send me an email if you want to add topics) ▰ Logistics: document access ▰ Document Update ◾ Review and update existing content ◾ Discuss noted questions/clarification points ▰ Use Cases ▰ Open Discussion [Document Access] If you cannot access the ASG repository (git clone [email protected]:asg.git), please send me your public key. The document (also attached this email) can be found in directory: documents/design. Dries

1 0