FunnyQT is a model querying and model transformation library for the functional Lisp-dialect Clojure providing a rich and efficient querying and transformation API. This paper describes the FunnyQT solution to the TTC 2014 FIXML transformation case. It solves the core task of generating Java, C#, C++, and C code for a given FIXML message. It also solves the extension tasks of determining reasonable types for the fields of classes. This paper describes the FunnyQT solution of the TTC 2014 FIXML Case [LYTM14] which solves the core task of generating Java, C#, and C++ code for a given FIXML messages. It also solves the extension task of heuristically determining appropriate types for the fields of the generated classes and the extension task to generate non-object-oriented C code. The solution also sports several features that were not requested. For example, if an XML element has multiple children with the same tag, then the corresponding class or struct will have a field being an array of the type corresponding to the tag instead of multiple numbered fields. For C++ and C, proper destructors/recursive freeing functions are generated, and the classes/structs are declared in a header and defined in a separate implementation file. For all languages, proper import/include/using-statements are generated, and the code compiles without warnings using the standard compilers for the respective language (GCC, Mono, Java). The solution allows to create a data model given a single FIXML message as requested by the case description, but it can also be run on arbitrary many FIXML messages at once. The idea is that with a reasonable large number of sample messages, the transformation is able to produce a more accurate data model. By having more samples, optional attributes and child elements are more likely to be identified. Similarly, child elements which usually occur only once but may in fact occur multiple times are more likely to be identified. And finally, the heuristical detection of an appropriate field type benefits from more sample data, too. Section A in the appendix on page 6 shows the code which was generated for the FIXML position report message test2.xml. FunnyQT [Hor13] is a model querying and transformation library for the functional Lisp dialect Clojure. Queries and transformations are plain Clojure programs using the features provided by the FunnyQT API. This API is structured into several task-specific sub-APIs/namespaces, e.g., there is a namespace containing constructs for writing polymorphic functions dispatching on metamodel type, a namespace containing constructs for model-to-model transformations, etc. The solution project is available on Github1, and it is set up for easy reproduction on SHARE2. 1https://github.com/tsdh/ttc14-fixml 2http://is.ieis.tue.nl/staff/pvgorp/share/?page=ConfigureNewSession&vdi=Ubuntu12LTS_TTC14_ 64bit_FunnyQT4.vdi
1 Introduction
Solution Description In this section, the transformation specification for all three main tasks is going to be explained. 2.1
Task 1: XML to Model Since handling XML files is a common task, FunnyQT already ships with a namespace funnyqt.xmltg which contains a transformation function xml2xml-graph from XML files to a DOM-like model conforming to a detailed XML metamodel which also supports XML namespaces. This function uses Java’s Stream API for XML (StAX) under the hoods, so XML files that aren’t well-formed lead to parsing errors. 2.2
Task 2: XML Model to OO Model Core task 2 deals with transforming the XML models into models conforming to a metamodel suited for object-oriented languages. The metamodel used by the FunnyQT solution is shown in Figure 1. [1..1] type
Type [1..1] elemType
Field name : EString initialValue : EString
FunnyQT contains a feature for generating metamodel-specific APIs which is used here. The generated XML and OO APIs are referred to by the namespace aliases xml and oo in the listings below. They contain getter and setter functions for attributes (e.g., (xml/set-name! el val)), role name accessor functions (e.g., (oo/->fields cls)), and several more.
In FunnyQT, a model-to-model transformation is specified using the deftransformation macro. It receives the name of the transformation, and a vector defining input and output models plus additional parameters. In this case, there is only one single input model xml, one single output model oo. (deftransformation xml-graph2oo-model [[xml] [oo]] ...)
Inside such a transformation definition, arbitrary many rule and helper function definitions may occur. The first rule of the transformation is element2class shown in the next listing.
(^:top element2class :from [e ’[:and Element !RootElement]] :to [c (element-name2class (xml/name e))])
The ^:top annotation defines this rule as a top-level rule being applied automatically. The :from clause restricts the elements e this rule is applicable for to those of metamodel type Element but not of type RootElement. The reason is that we don’t want to create a class for the FIXML element which is the root element of any FIXML message.
The :to clause defines which elements should be created for matching elements. Usually, it would be specified as :to [x ’SomeClass] in which case x would be a new element of type SomeClass. However, in the current case, there is no one-to-one mapping between XML elements and OO classes, because the XML model may contain multiple elements with the same tag name, and there should be exactly one OO class per unique tag name. Therefore, the :to clause delegates the creation of class c to another rule element-name2class providing e’s tag name as argument.
When a rule is applied to an input element for which its :from clause matches, target elements are created according to its :to cause. The mapping from input to output elements is saved. When a rule gets applied to an element it has already been applied to, the elements that have been created by the first call are returned instead of creating new elements.
The element-name2class rule shown below receives as input a plain string, the tag-name of an element, and it creates a Class c in the target model. The name of the class corresponds to the tag-name. According to the rule semantics sketched above and the fact that this rule gets called with the tag name of any element by element2class, there will be one target class for every unique tag name.
(element-name2class :from [tag-name] :to [c ’Class {:name tag-name}] (doseq [[an at av] (all-attributes tag-name)]
(attribute2field an at av c)) (doseq [[tag max-child-no] (all-children tag-name)]
(children-of-same-tag2field tag max-child-no c)) (when-let [char-conts (seq (all-character-contents tag-name))] (character-contents2field char-conts c)))
Following the :from and :to clauses comes the rule’s body where arbitrary code may be written. Here, three other rules attribute2field, children-of-same-tag2field, and character-contents2field are called for all XML attributes, child elements, and character contents3 of element e. These rules and the helpers all-attributes, all-children, and all-character-contents are skipped for brevity but they follow the same style and mechanics.
The next listing shows the helper implementing the extension task of heuristically determining an appropriate field type from XML attribute values.
(guess-type [vals] (let [ts (set (map #(condp re-matches % #"\d\d\d\d-\d\d-\d\d.*" DATE #"[+-]?\d+\.\d+" DOUBLE #"[+-]?\d+" (int-type %)
STRING) vals))] (get-or-create-builtin-type (cond (= (count ts) 1) (first ts) (= ts #{DOUBLE INTEGER}) DOUBLE (= ts #{DOUBLE LONG}) DOUBLE (= ts #{DOUBLE LONG INTEGER}) DOUBLE (= ts #{INTEGER LONG}) LONG :else STRING))))
The guess-type function receives a collection vals. vals could either be all character contents of an XML element, or all attribute values of an attribute that occurs in many XML elements of the same tag. Every given value is checked against a regular expression that determines its type being either a timestamp in ISO 8601 notation, a double value, or an integer value. If none match, then STRING is used as its type. In case of an integer value, the function int-type further determines if the value can be represented as a 32 bit integer, or if a 64 bit long is needed.
The cond expression picks the type that can be used to represent all values. If all values are guessed to be of the very same type, then this type is chosen. For multiple numeric types, the respective “largest” type is chosen where INTEGER < LONG < DOUBLE. Else, we fall back to STRING. The picked type is then passed to the rule get-or-create-builtin-type which creates a Builtin whose type attribute is set to the type determined by the cond expression.
3The case description doesn’t demand that XML character content should be handled. However, without handling them transforming test3.xml and test4.xml would lead to several classes without any fields.
The complete xml-graph2oo-model transformation consists of 6 rules and 7 helpers amounting to 70 LOC. The result is an OO model whose field elements already have the heuristically guessed types, and where multiple-occuring XML child elements of the same type where compressed to array fields. 2.3
Task 3: OO Model to Code The last step of the overall transformation is to generate code in different programming languages from the OO model created in the previous step. In addition to the core task languages, the FunnyQT solution also generates C code as an extension.
One crucial benefit of FunnyQT being a Clojure library is that we can simply use arbitrary other Clojure and Java libraries for our needs. So for this task, we use the excellent Stencil4 library. Stencil is a Clojure templating library implementing the popular, lightweight Mustache specification5. The idea of Mustache is that one defines a template file containing placeholders which can be rendered to a concrete file by providing a map where the keys are the placeholder names and the values are the text that should be substituted. There are also placeholders for collections in which case the corresponding value of the map has to be a collection of maps. We’ll discuss the solution using the template for Java. package {{{pkg-name}}}; {{#imports}}import {{{imported-class}}};{{/imports}} class {{{class-name}}} { {{#fields}} private {{{field-type}}} {{{field-name}}}; {{/fields}} public {{{class-name}}}() { {{#fields}} this.{{{field-name}}} = {{{field-value-exp}}}; {{/fields}} } /* parametrized constructor, getters, and setters elided... */ }
So a map to feed to the Stencil templating engine needs to provide the keys :pkg-name, :imports, :class-name, etc. The values for the :imports and :fields keys need to be collections of maps representing one import or field each, e.g., a field is represented as a map with keys :field-type, :field-name, and :field-value-expression.
The templates for the other languages use the same keys (although there are some keys in the C and C++ templates that are only needed by them), so the essential job of the code generation task is to derive such a map for every class in our OO model that can then be passed to Stencil’s rendering function.
This is done using a FunnyQT polymorphic function to-mustache whose definition is given below. 1 (declare-polyfn to-mustache [el lang pkg]) 2 (defpolyfn to-mustache oo.Class [cls lang pkg] 3 {:pkg-name pkg 4 :imports (get-imports cls lang) 5 :class-name (oo/name cls) 6 :fields (mark-first-field (map #(to-mustache % lang pkg) (oo/->fields cls)))}) 7 (defpolyfn to-mustache oo.Field [f lang pkg] 8 {:field-type (field-type (oo/->type f) lang) 9 :field-name (oo/name f) 10 :field-value-exp (field-value-exp f lang) 11 :plain-field-type (let [t (oo/->type f)] 12 (type-case t 13 ’Array (oo/name (oo/->elemType t)) 14 ’Class (oo/name t) 15 nil))})
A polymorphic function in FunnyQT is a function that dispatches between several implementations based on the metamodel type of its first argument. They can be seen as a kind of object-oriented method 4https://github.com/davidsantiago/stencil 5http://mustache.github.io/ attached to metamodel classes. Line 1 declares the polymorphic function to-mustache and defines that it gets three parameters: an OO model element el, the target language lang, and the package/namespace name pkg in which the class/struct should be generated. Lines 2 to 6 then define an implementation for elements of the metamodel class Class, and lines 7-18 define an implementation for elements of metamodel class Field. Both implementations call several helper functions that query the OO model to compute the relevant values for the map’s keys which are skipped for brevity here. 3
Evaluation and Conclusion The complexity should be measured as the sum of number of operator occurrences and feature and entity type name references. The FunnyQT solution contains about 300 expressions, 24 metamodel type references, and 18 property references resulting in a complexity of 342. So it is quite complex but it does much more than what was required. A solution soving only the required tasks would have the same amount of metamodel type and property references but would be approximately one third shorter.
Accuracy should measure the degree of syntactical correctness of the generated code and the degree of how well it matches the source FIXML messages. The FunnyQT solution has a very high accuracy. The code is correct and compiles without warnings. It also matches the source FIXML messages well. The creation of one array field for multiple XML children with the same tag is better than creating several separate fields. Guessing appropriate types for the fields instead of always using string improves the usefulness of the generated code. Also, that the transformation can be run on an arbitrarily large sample of FIXML messages in one go improves the accuracy even more.
The overall development time of the solution can be estimated with about 8 person-hours for the core task and 4 more hours to generalize and extend it to the final version.
Since FunnyQT’s generic xml2xml-graph transformation uses Java’s StAX API internally, the fault tolerance is high. Documents which are not well-formed lead to parsing errors.
The execution time is good. For all provided test models, the complete transformation including parsing XML, transforming the XML model to an OO model followed by generating code in all four languages took at most 700 milliseconds on SHARE. Running the transformation on all provided and five additional FIXML messages at once took about 1.5 seconds.
The modularity of the xml-graph2oo-model is Mod = 1 dr = 1 56 = 0:16 where r is the number of rules and d is the number of dependencies between them. The code generation is implemented with 10 functions that call each other. Since some functions are recursive and called from different places 12 call dependencies can be counted. Thus, the modularity is Mod = 1 1120 = 0:2.
With respect to abstraction level, the xml-graph2oo-model transformation is quite low-level. The code generation is split into declarative templates, and functions that derive a map of template placeholder keywords to the values that have to be filled in for each class. Those functions are all pure functional. Thus, the abstraction level of the FunnyQT solution is about medium. trated.
Transformation of a Position Report Message In this section, the stepwise outcomes of transforming a position report message (test2.xml) are illusThe FIXML document itself is printed in Section A.1.
Section A.2 shows its representation as an FunnyQT XML model. This part of the overall transformation has been discussed in Section 2.1.
Section A.3 shows the OO model conforming to the metamodel shown in Figure 1 which is generated by the xml-graph2oo-model transformation discussed in Section 2.2.
Finally, the sections A.4, A.5, A.6, and A.7 show the source code files for the PosRpt class and the Util class which contains helpers for the data model classes. For Java and C#, there is only one source code file for the PosRpt class whereas for C++ and C, the class/struct is declared in a header file and its definition is held in a separate implementation file. It should be noted that all source code files are printed here exactly as produced by the transformation. No additional formatting has been done, and they all compile without warnings using standard compilers for the languages, i.e., javac from the OpenJDK project6 for Java, mcs from the Mono project7 for C#, and g++/gcc from the GNU Compiler Collection8 for C++ and C. However, the C++ code uses extended initializer lists which are new in the C++11 standard, so a -std=c++0x (or -std=c++11) has to be added to the g++ call in order not to get warnings. A.1
The Position Report as XML document
The P ositio n Report as XML Graph trentren tren ten apap ap rap tren itreub ap tA s a
H l u n t = n s e e lid h C s a H : ltoeERm scaaepNmI""LFXMlun=lun traenp e1 lircehnd l:tveeEn2m rsscaaaeeedpNmt""sPoepR=lrxefiun=lIunR= t lteeenm i:treeub10 ittrsaeub i:ttrvub12A It""aepRD=mlrsxPefiun=lIsnuRU=l"aeu51438=
e 6 :v1 lraed e=rxefi IR= lce amsP sU en saA n n n v
t ced anmsPn snU d lteeenmnnlteeenmnlteeenm leem i:treeub9H ittrsaeub ittreub t""cyTpA lun=lun ""1 t s v1A ae=mrsxPefi IsRU=laeu= A : a H : n n n v 8 e " :0 :0 0
T te u irb t te -2 irub -31 :tsaAH itrseub :v96A t""aMlun=lun "02 76 ta ae=mrsxPefi IsRU=laeu= e
n n n v te u " itrb teu PSS saA s irb I""FC lun=lun "ACO :H teub :tA e6 ir v68 ae=mrsxPefi IsRU=laeu= t a
n n n v te u itrb e l :sa5AH tseub ittrub I""D un=lnu ""KW e6 ir :v67A aen=mrsxPefin IsnRU=lvaeu= t e a t u irb sA teub Ir""cSD lun=lun J"" t :ae64H itrseub i:trv6A aen=mrsxPefin IsnRU=lvaeu= t e a t u lun= lteeenm lteeenm tenm i:ttrsaeeub36AH ittrsaeub i:ttrveub56A ""yaenS=mmlrsxPefinun=lIsunnRU=l""LvaeuAO= tenm scaep t" lee ib
r leE0 saemItrsnmlnu lu tsA " :v6 renlrcaeeddN "aen=mtenrsxPefin=IsnnRU=ten lteeenm i:ttraeeub62H ittrsaeub i:ttrveub64A tr"ckyaenSC=mlrsxPefinnu=lIsnunRU=l""vaeuSUD= ilchd leem leem saAH " : 1 2 teeu6 itrseub ittreubA ""YMMlun=lun "3021 irb ta : tA v63 ae=mrsxPefi IsRU=laeu= s a H n n n v : 0 6 e s " te te " l 0 itreub itrub itrub tr"kS nu=lun ."745 sA :v62A aen=mrsxPefin IsnRU=lvaeu= t a a H : 9 5 e te u irb t A a teu ""yTp lun=lun ""2 s lnu= lteenm :teeub43H ittrsaeub i:trvb45A aen=mrsxPefin IsnRU=lvaeu= e i r t A sa e l " ten :2H se itrub I""D nu=lun "ZZ l em e4 tu t ce amsP sU l d n n n e irb :v4A aen=mrsxPefin IsnRU=lvaeu= t a te u irb " t g :sa1H tseub ittreub ""ubS lun=lun itr"S A n tr :v3A aen=mrsxPefin IsnRU=lvaeu= e3 i e a t u i:trsabAH tse itreub I""D lnu=lun itr""nSg 0 u t t :v32A aen=mrsxPefin IsnRU=lvaeu= b 3 i e r e a t u irb t " sA e ng :aH tse itrub ""cLo lun=lun itr"S 9 ub t e2 ir t :v31A aen=mrsxPefin IsnRU=lvaeu= a s teu teu lt""uMlun=lun ""01 irb irb t t a :v6A aen=mrsxPefin IsnRU=lvaeu= 1 te u irb t " sA e M :ae75H itrseub ittrub ""yTp lun=lun "FTM tn te ta :v59A aen=mrsxPefin IsnRU=lvaeu= e u leem itrb A sa " H te u irb t sA " :ae54H itrseub ittreub ""yTp lun=lun I"SA tn te ta :v65A aen=mrsxPefin IsnRU=lvaeu= e u leem itrb A s a
H l u n = v57 raaedN "eA=rxefi=InR= l ce amsP sU d n n n n re lid h c l u n l:teeEnm sscaeepmt"mlnu lu lteeenm :e56 ittrsaeub i:ttrveub58A t""aenA=mmlrsxPefiunn=lIsunnRU=l."vaeu00= 5 e lid h C s a H : 2 5 e lid h C s a H : 8 4 e ild h C s a H : 1 4 e n re lid h c n re ild h c l u n = ten sce m sap le E : v37 raaeedNmt""yPe=lrxefinu=lIunR= teu tse itrb irub sA ta te a u :H irb 7 e te u itrb tse A u sa irb :6H ta " e " :0 :0 0 T 0 -1 9 tA t" l -0 : 3 v10 i"zBD nu=lun "02 ae=mrsxPefi IsRU=laeu= n n n v teu t"xPeS lun=lun ."00
" irb t te itrsabA itrseub :v9A aen=mrsxPefin IsnRU=lvaeu= u H t : a " 5 e teu ""cyC lun=lun "SUD irb t u s :v8A aen=mrsxPefin IsnRU=lvaeu= te itrb te saA irub :4H ta "yp e teb t"xPTeS lun=lun ""1 u ir t
A te irb s :v7 ae=mrsxPefi IsRU=laeu= u tsaAH itreub n n n v :e3 ta "p itreub "yTeqR lun=lun ""0 t te irb s :v6A ae=mrsxPefi IsRU=laeu= u t te n n n v saAH irub : t " 2 a x e teb itr"PPeS lnu=lun ."00
" u ir t te irub s :v5A ae=mrsxPefi IsRU=laeu= t te A u n n n v sa irb :1H ta e teu lt""sR lun=lun ""0 irb t s :v4A aen=mrsxPefin IsnRU=lvaeu= te u irb t a te t""cA lun=lun ""1 u irb t :v3A aen=mrsxPefin IsnRU=lvaeu= te u lce amsP sU itrb e l " lisahdC lircenhd ten lsceund=n n n lteeenm i:trsaeeub43AH ittrsaeub i:ttrvub63A I""aenD=mrsxPefinun=lIsunnRU=l"vaeuCO= :e83H l:veE34m rsaaaeepdNmt""yPe=lrxefinu=lIunR= ten :tsa3AH tse i:ttreubA ""eR=lrxefinu=lIun=""12=
m e u ilsahdC ilrcenhd lced anmsPn snU lee itrab v35 anmsPn snRU lvaeu tentn H l : l u 5 u n 3 n e lid h sC = = a ten sce :
H ten sce il:saehd16CH lircenhd l:veE13m lrscaaaeeepddNmr""aendH=mlrsxPefinun=lIsunnRU= traenptraenp li:saehd3282CH lircenhd l:eEm saepmlr"vTo lnu lu
e t n "341 leeemrp a :0 5 -0 7 :4 0 :3 9 0 T 7 -1 2 -1 1 te u irb t A : 5 v1 t""nS lun=lun "02 s ae=mrsxPefi IsRU=laeu= te irub n n n v t a " itreb "euqSNmlnu=lun ""01 u 2 t A : v14 ae=mrsxPefi IsRU=laeu= n n n v ilhd ren sC ild a h :H c ten sce f" 4 2 e ild h sC l a ce amsP sU H n d n n n :e02 ilrcehd l u n leemleeem itreub " t g :sa7H tseu ittreub ""ubS lun=lun itr"S A n t :v29A aen=mrsxPefin IsnRU=lvaeu= 2 i b e r e a t v30 lrcaaeeddN "aenD=mrsxPefin=IsnnRU= ten sA te l ng u irb t " a u lnu= leemlteenm i:treeub62H ittrsaeub i:trvb28A I""aenD=mrsxPefinnu=lIsnunRU=litr"vaeuS= e t
A l:eEm saepmlfhnBO lun lu ten :a52H tse itreub ""cLo lun=lun itr"nSg
s " v26 raaedN "eO=rxefi=InR= leem e itrub :tv27A aen=mrsxPefin IsnRU=lvaeu= a A.4
The Position Report as Java Class PosRpt.java public void setReqTyp(int ReqTyp) {
this.ReqTyp = ReqTyp; 1 package test2; 2 3 import java.text.SimpleDateFormat; 4 import java.text.ParseException; 5 import java.util.Date; 6 7 class Util { 8 private static final SimpleDateFormat dateFormat 9 = new SimpleDateFormat("yyyy-MM-dd’T’HH:mm:ssXXX"); 10 11 public static Date parseDate(String date) { 12 try { 13 return dateFormat.parse(date); 14 } catch (ParseException e) { 15 throw new RuntimeException(e); 16 } 17 } 18 }
A.5
The Position Report as C# Class
PosRpt.cs 52 53 54 55 56 57 58 59 } }
} Util.cs long getSetPxTyp(); void setSetPxTyp(long SetPxTyp); double getPriSetPx(); void setPriSetPx(double PriSetPx); long getRptID(); void setRptID(long RptID); double getSetPx(); void setSetPx(double SetPx); std::tm getBizDt(); void setBizDt(std::tm BizDt); long getAcct(); void setAcct(long Acct); Amt* getAmt_obj(); void setAmt_obj(Amt* Amt_obj); Instrmt* getInstrmt_obj(); void setInstrmt_obj(Instrmt* Instrmt_obj); Qty** getQty_objs(); void setQty_objs(Qty** Qty_objs); Pty** getPty_objs(); void setPty_objs(Pty** Pty_objs); Hdr* getHdr_obj(); void setHdr_obj(Hdr* Hdr_obj);
PosRpt.cpp 1 #ifndef _test2_Util_H_ 2 #define _test2_Util_H_ 3 4 #include <string> 5 #include <ctime> 6 7 namespace test2 { 8 class Util { 9 public: 10 static std::tm parseDate(const char* date); 11 }; 12 } 13 14 #endif // _test2_Util_H_
Util.cpp 1 #include "Util.hpp" 2 3 namespace test2 { 4 std::tm Util::parseDate(const char* date) { 5 std::tm tmp; 6 strptime(date, "%FT%TZ", &tmp); 7 return tmp; 8 } 9 }
A.7
The Position Report as C Struct
PosRpt.h 1 #ifndef _PosRpt_H_ 2 #define _PosRpt_H_ 3 4 #include "Util.h" 5 #include "Pty.h" 6 #include <time.h> 7 #include "Amt.h" 8 #include "Instrmt.h" 9 #include "Qty.h" 10 #include "Hdr.h" 11 12 typedef struct { 13 long ReqTyp; 14 char* Ccy; 15 long Rslt; 16 long AcctTyp; 17 long SetPxTyp; 18 double PriSetPx; 19 long RptID; 20 double SetPx; 1 #include "Util.h" 2 #include <stdarg.h> 3 #include <stdlib.h> 4 5 void** make_pointer_array(int size, ...) { 6 va_list ap; 7 va_start(ap, size); 8 void** ary = malloc(sizeof(void*) * size + 1); 9 int i; 10 for (i = 0; i < size; i++) { 11 ary[i] = va_arg(ap, void*); 12 } 13 ary[i] = NULL; 14 va_end(ap); 15 return ary; 16 } 17 18 struct tm parseDate(const char* date) { 19 struct tm tmp; 20 strptime(date, "%FT%TZ", &tmp); 21 return tmp; 22 }