Abstract

Any natural language expression of a set of facts - that can be represented as a knowledge graph - will more or less overtly assume a specific perspective on these facts. In this paper we see the conversion of a given knowledge graph into natural language as the construction of a narrative about the assertions made by the knowledge graph. We, therefore, propose a specific pipeline that can be applied to produce linguistic narratives from knowledge graphs using an ontological layer and corresponding rules that turn a knowledge graph into a semantic specification for natural language generation. Critically, narratives are seen as necessarily committing to specific perspectives taken on the facts presented. We show how this most commonly neglected facet of producing summaries of facts can be brought under control.

1. Introduction

In the popular noveSltranger in a Strange Land Robert Heinlein introduces a cast of people who have been trained to speak only non-subjective truths containing neither valence nor assumptions. When describing a war-like situation it might, theoretically, be possible to say that, for example, some governmental head of a country gave an order to the army to move into another country by force. Natural language renditions of corresponding states of afairs usually contain expressions suchinavsading orliberating that assume a specific perspective (taking sides) and denote some valuation of the situation at hand. In other words, rather than objective and neutral truth-sayers, we are spinning narratives out of the facts on the ground. nEvelop-O

CEUR

The concept of anarrative has migrated from its original domain in the literary sciences to a multitude of diverse and increasingly distant research fields. It has become an important element in research on game1s][or in history2[] to name a few of these domains. At long last it has also arrived in the cognitive sciences where narratives are regarded to be a central means of sense making 3[]. From there it was merely a short jump over to the field of computer science where the concept is employed to describe semantically annotated episodes of recorded activities [ 4 ] and has, subsequently, been formalized using description logics [ 5 ].

When we make assertions about people or events to capture them in a knowledge graph, we accumulate information that is supposed to represent the ground truth. In narratology this is often called thefabula [ 6 ]. This fabula can represent episodes, e.g.: • for logging an autonomous robot ftahbeula can represent trajectories of body parts and activity-specific force events [ 7 ]. • for modeling historical or current events, as done by the EventKG knowled8g]e, base [ thefabula usually consists of events and their participants.

While these fabulae contain large quantities of information they are, by themselves, not very meaningful. Only when we put them into a pragmatic context do we assign additional meaning to them. For example, we can interpret the same observed episode as ethitrohweirng something ordropping something. This diference in the narrativization, consequently, yields two distinct narratives: (1) Sherlock dropped the glass onto the floor (2) Sherlock threw the glass onto the floor

It is important to note that the knowledge graph representing these two minimal narratives can be identical. We consequently diferentiate between a factual knowledge graph, i.e. the fabula, which has not been narrativized and a (language-based) description of it,nia.err.atthivee. This pairs a situation with a selected conceptualization, i.e. interpretation, thereof and renders the latter in natural language. In addition to becoming meaningful, the description will, in turn, evoke a pragmatic stance that ascribes, for example, a specific perspective and intention to the agent(s) acting in specific roles within the narrative. The more general contribution of this work is to examine certain elements of narrative mechanics, as part of a larger efort to understand the mechanics of conflictual narrativ9e]s. [Specifically, we provide a technological scafolding for the process of constructing such narratives in order to further empirical research on how narratives emerge in the wild. In the following, we will, therefore, present a system that takes a knowledge graph, i.e. afabula, as input and converts it to a narrative.

2. Related Work

Several approaches exist for expressing formal representations of narratives. Each of these approaches is driven by the specific requirements of the given application at hand. For example, the model of Meghini et al. seeks to organize information provided in digital libraries and, therefore, models both fabulae as well as narratives as events and allows for events to feature dependent events6[]. The model uses RDF based on OWL and narrative events can have spatial or temporal relations between them, but the main purpose is to connect digitally represented entities to pertinent events, e.g. the Divine Comedy as a book and the person Dante Alighieri can be connected by a narratDivaente writes the Divine Comedy. In this approachevents are also not formally specified as no foundational framework is employed.

A diferent approach is the work described by Evans et al. that seeks to classify (partial) sensory data as a narrative that can be framed as an inductive logic programm1in0]g. task [ While their focus lies on reducing the search space by finding hypotheses – which equal narratives in their approach – that provide as simple an explanation of the observed data as possible. This approach can, therefore, be deemed compatible yet orthogonal to the one presented herein, as it focuses not on representing and using narratives via an ontology, but on a classification approach that employs such a model as a target representation.

Closer to the current task is the work described by Kroll et al. that makes a useful distinction between factual relations, as expresses by knowledge graphs, and narrative relations that constitute hypothetical relations connecting factu1a1l].oFnoers t[his, the approach needs to employ RDF* to express relations that range over relations. This approach can, therefore, be employed to postulate, for example, causation relations as a narrative that connects hitherto isolated knowledge graphs.

For our task at hand, we employ a model of narratives that provides a suitable ontological theory and subsequent model and is based on the Socio-physical Model of Activities (S5O].MA) [ SOMA already comes with the central distinction between ground and descriptive1e2n].tities [ This useful distinction is provided by the foundational layer, which in this case is the Dolce Ultra Light framework with the addition of the Descriptions and Situations Module (DUL+D&S) [ 13 ].

For generating comprehensible and appropriate natural language expressions various end-toend approaches based on some forms of machine learning exist, both for generating individual sentences based on knowledge graph1s4][ as well as for sequences of sentences for sub-graphs retrieved from larger knowledge grap1h5s].[The work presented in this paper is, however, closely related to that of constructing formal narrative structures out of distributed knowledge graphs [ 16 ]. In the symbolic approach, introduced in this work, we employ the Komet-Penman MultiLingual (KPML) system to add a natural language generation component. This system ofers a well-tested platform for grammar engineering that is specifically designed for natural language generation17[]. KPML employs large-scale grammars written within the framework of Systemic-Functional Linguistics (SFL). The employment of SFL allows us to include linguistic phenomena which are important for the generation of natural texts and which go beyond the bare propositional content that is to be expressed [ 18 ].

3. From Knowledge Graphs via Narratives to Natural Language

The method we propose and showcase in this work features two main steps that follow each other: • Firstly, a formal representation of a narrative as triples is constructed based on an existing knowledge graph of a given event (tfhabeula). For this, the events and their participants as defined in the fabula are linked to diferent narratives that describe these events and, consequently, can consider them from diferent perspectives. By filtering based on a given perspective and choosing which events and participants to include, the narrative itself is constructed. In narratology the result of this filtering is usually referredptloot [a6s].the • Secondly, the narrative - in the form of said triples - is converted into a semantic specification of the text to be produced, which is then converted into text using a natural language generation system based on KPML and systemic functional grammar[ 18 ]. In the following, we provide further details of our approach for turning knowledge graphs into natural language text using examples from the domain of everyday activities and current world afairs. 3.1. Steps

Fabula The underlying knowledge graph, which does not explicitly encode any kind of perspective on the given events, is what we consider afsabtuhlea. In the future, we aim to use existing resources, such as EventK8G], [as the basis for constructing formal narratives of events. However, since existing resources include large amounts of knowledge but also often miss important relations, this would still require a significant amount of manual work. We, therefore, present this case study on small-scale examples, for which we constructed the base knowledge graphs manually using the information available in EventKG.

Plot and Narrative The selected knowledge graph is expanded into narratives that can assume diferent perspectives with respective Events that are mapped to Tasks and Participants that are mapped to their respective Roles. This content selection and filtering based on perspective produces naarrative specification represented as triples. Figu3rgeives an overview as to what sort of entities and relationships can appear in narrative specification triples. Semantic Specification The narrative specification triples are mapped to semantic structures such as discourse relations between events (motivation, explanation, concession), action specifications, object descriptions; assembling these structures results in a semantic specification, which is a feature structure using concepts from the Generalized Upper Model (GUM) and Upper Interaction Ontology to represent its elements [ 19 ].

Tactical Generation Once a semantic specification exists, it is transferred to the language generation software KPML to produce finely-controlled natural language text. 1In this paper concepts that are terms of the SOMA ontology are denoted by setting their labels in Small Caps.

3.2. Narrativization of Knowledge Graphs

As proposed in the formal model adopted for representing narr5a]t,invaer[ratives define Tasks which are executed in the given Events, and the Roles that the events’ participants can take. Based on this formal theory of narratives, we consider several ways in which a neutral knowledge graph can be narrativized: Event - Task An event that exists in the KG can be construed in diferent tasks: for example, E1 in Figure2 represents the invasion of Ukraine by Russia, which, depending on the speaker’s point of view, might be characterized e.g. as an invasion, the launch of a special military operation, or even as a liberation.

Participant - Role A narrative also defines at least one role in the task which is taken by the participants of the event. Thus, diferent narratives might include very diferent kinds of roles, e.g. an invaderversus a liberator, as well as having diferent participants take those roles, e.g. in E2, the escalation of the Russia-Ukraine crisis, the role of the agent causing the escalation might be taken by Russia, by NATO, or by other participants, depending on the point of view being constructed by the narrative.

Terminology Terms used to refer to events or their participants can impact the surrounding narratives in diferent ways. Firstly, collections of knowledge graphs, such as provided by EventKG, which combine information from several distinct sources, provide diferent terms for the events themselves by combining events from diferent sources through the “sameAs” relationship. For example, depending on the source, the ongoing war in Ukraine is referred to as “Ukrainian War”, “Russo-Ukrainian War” or “Russian military intervention in Ukraine”; and the annexation of Crimea can be found under “Annexation of Crimea by Russian Federation”, “Crimean Crisis 2014”, or “armed political conflict surrounding Crimea”. Additionally, alternative names are also provided, including “Secession and Incorporation of Crimea” as referring to the annexation of Crimea, or “Russian Spring” which refers to the pro-Russian unrest in Ukraine in 2013. Secondly, the events included in EventKG also have a “type” - thus, the Ukrainian war might be seen not only as a war, but also as an armed conflict, a military conflict, or an “intervention”; while the aforementioned unrest in 2014 is defined e.g. as “secession”, “protest”, “civil disorder”, or “insurgency”. Thirdly, terminology can be used to characterize the participants in the event. This ranges from smaller distinctions such as the usage of a person’s full name or title (e.g. “Putin” vs. “President Vladimir Putin”, vs. “dictator Putin”), to terms that also cast participants in very specific roles regarding the task defined in the narrative: an example of this in the Twitter conversation surrounding the war in Ukraine is the casting of its invasion as a liberation - however, not only is the invasion itself characterized as aiming to liberate the country in general, but the oppressor it is to be liberated from is “the Nazis”. This term can be found in diferent contexts. In tweets talking about “liberation from the Nazis”, the latter usually refers more generally to the Ukrainian government, which thus takes the role of oppressor in the liberation task. However, when this narrative is challenged, the term is defended by refocusing it on other groups, usually the Azov regiment of the Ukrainian Army [ 9 ]. Relationships between events In EventKG, the events themselves are connected only through sub-event relationships, but not in terms of their causality or other relations. The combination of several events in textual expressions can add additional connections that are by and large contingent on the narrative perspective: an event might be caused by a diferent one, be motivated by another or happen in spite of another event that opposes it. For example, depending on the perspective of the speaker, the invasion of Ukraine by Russia (E1) might be caused by continuously escalating tensions (E2) to which Russia is reacting, or it might be started by Russia with the goal of further escalating the existing crisis.

Perspective Filtering Consequently, through the steps described above, a perspective has been chosen, that primarily involves the selection of a frame-giving conceptualization of the event. In the terminology of SOMA this corresponds to selecting a Tascklastshifieast an Event. Analogous to frame entities found in FrameN2e0]t, [each specific frame contains a frame-specific configuration of roles, e.g. invading features invaders and invaded entities whereas liberating involves the liberated and the liberators. Correspondingly, one and the same entity can be endowed with a a positive valence, e.g. the liberator, or a negative one, e.g. the invader, albeit both are instantiated by the same entity in the knowledge graph. A set of sample configurations is given in Figur2e.

3.3. From Narrative Specification to Texts

As previously mentioned, for our purposes here a narrative specification is a collection of triples asserting that some events happened and had various entities as participants. The events may have been connected to each other by relations such as: • opposition - one event should have prevented another but did not • motivation - one event happened because its agentive participants desired some other event to happen • explanation - one event happened because another one happened A summary of the available relationships and entity types is given in3F. igure

For the final part of the language generation process, i.e. the realization, we use18K]P.ML [ KPML does not accept triples as inputs, but rather semantic specifications (semspecs), which are represented as feature structures. An example such semantic specification for “the cup” is as follows:

( OBJ_0b4ax4 / | O b j e c t | : LEX CUP : d e t e r m i n e r t h e ) A semspec contains an identifier for the entity it describes, an ontological characterization in terms of a semantic type defined in the Generalized Upper Model or in the Upper Interaction Ontology, and various other lexico-semantic information such as a lexical item to use in the realization, determiners for objects, flags about identifiability and so on. Semspecs can include other semspecs, or refer to them via identifiers. In particular, a semspec for an event or object will typically include semspecs for entities playing some role for that event or object.

The conversion from a set of triples – the narrative specification – to a semspec makes use of the compositionality of KPML semspecs. That is, to generate a semantics specification for a relationship between events it is enough to generate a semspec for each event participating in the relation, and then generate a semspec for the relation according to an appropriate template which is then filled in with the semspecs for the events. Likewise, to generate a semspec for an event is to fill in an appropriate template with the semspecs produced for the participant objects and so on.

In the table following we list some example narrative specs produced for the knowlege graphs augmented as shown in Figur2e above and the corresponding KPML outputs, i.e. natural language expressions, produced from them. The semspecs themselves tend to be more verbose and we omit them for reasons of space. The interested reader can find these examples, and more, at our github repository for KPML examp2lOesn. e note about the examples: each entity also has associated a lexical item via a “hasLex” property. For space reasons, these were not listed as for these examples the entity names are enough for the reader to infer the lexical item.

2https://github.com/mpomarlan/KPML_examples. A copy of KPML, with recent patches, is available upon request. Narrative specification ( ' construedAs ' , ' invade ' , ' d i s p m a t a c t ' ) ( ' hasAgent ' , ' invade ' , ' r u s s i a ' ) ( ' h a s P a t i e n t ' , ' invade ' , ' ukraine ' ) ( ' hasTense ' , ' invade ' , ' past ' ) ( ' construedAs ' , ' l i b e r a t e ' , ' d i s p m a t a c t ' ) ( ' hasAgent ' , ' l i b e r a t e ' , ' r u s s i a ' ) ( ' h a s P a t i e n t ' , ' l i b e r a t e ' , ' ukraine ' ) ( ' hasOpponent ' , ' l i b e r a t e ' , ' n a z i s ' ) ( ' hasTense ' , ' l i b e r a t e ' , ' p r e s e n t − continuous ' ) ( ' hasDeterminer ' , ' n a z i s ' , ' the ' ) ( ' construedAs ' , ' l i b e r a t e ' , ' d i s p m a t a c t ' ) ( ' hasAgent ' , ' l i b e r a t e ' , ' r u s s i a ' ) ( ' h a s P a t i e n t ' , ' l i b e r a t e ' , ' ukraine ' ) ( ' hasSource ' , ' l i b e r a t e ' , ' n a z i s ' ) ( ' hasTense ' , ' l i b e r a t e ' , ' p r e s e n t − continuous ' ) ( ' hasDeterminer ' , ' n a z i s ' , ' the ' ) ( ' construedAs ' , ' launch ' , ' d i s p m a t a c t ' ) ( ' hasAgent ' , ' launch ' , ' r u s s i a ' ) ( ' h a s P a t i e n t ' , ' launch ' , ' o p e r a t i o n ' ) ( ' h a s D e s t i n a t i o n ' , ' launch ' , ' ukraine ' ) ( ' h a s S i z e ' , ' o p e r a t i o n ' , ' s p e c i a l ' ) ( ' hasMatQuality ' , ' o p e r a t i o n ' , ' m i l i t a r y ' ) ( ' hasTense ' , ' launch ' , ' past ' ) ( ' hasDeterminer ' , ' o p e r a t i o n ' , ' a ' )

4. Conclusion and Future Work

In this preliminary and ongoing work, we have introduced a pipeline for turning knowledge graphs into natural language texts that assume a specific perspective on the propositional content given by the respective knowledge graph. For this, a formal model of narratives provides the ontological foundation for representing the intermediate perspective-specific representation. It should also be noted that not only do natural language expressions contain biases and value judgments, but also ontological models can feature biases imported by their de2si1g]n.ers [ In the present work, however, the perspective is explicitly modeled as an integral part of the narratives at hand. This feature enables us to produce narrative-specific descriptions of a given knowledge graph that, as opposed to, for example, tweets in the wild, makes the underlying perspective openly available for inspection via the narrative specification.

Our approach, however, still needs more work to make it scalable and generally applicable to any knowledge graph available in some collection. For this ways have to be found for: • dealing with missing information in knowledge graphs, for example, in EventKG the person Louis the XVIth is not given as a participant of his own beheading. • automatic creation of hermeneutic filters for mapping the events contained in a knowledge graph to possible interpretations thereof. One approach for this could employ generic ontology design patterns [ 22 ]. • adding register to the natural language generation o23u]t.pLuetxi[cal choices can also express sentiments and ideological positioning about some entity, e.g. the diference between referring to an atom bomb as a tacvteircsauls a nuclear weapon carries some opinion with it [ 24 ].

To evaluate the output of our system a typBeleoufScore evaluation could easily be undertaken [ 25 ]. However, since it is not the intend of this work to create textual narratives for human readers, an evaluation of their structural similarity to real conflictual n9a]rwroautlidvebse[ more appropriate. For this some annotation-based metrics, e.g. Cohen’s K2a6p],paco[uld be employed to compare semantic annotations of real and generated narratives. We hope this work constitutes the beginning of a research efort that complements current eforts that focus on going from natural language expressions to formal specifications - either narratives or knowledge graphs - by looking at the reverse direction. Ultimately, the long-term research efort behind this undertaking concerns an improvement of our understanding of narrative mechanics, i.e. how they are constructed, manipulated and finally expressed as natural language utterances.

Acknowledgments

This work was funded by the by the FET-Open Project #951846 “MUHAI – Meaning and Understanding for Human-centric AI” by the EU Pathfinder and Horizon 2020 Program and by the German Research Foundation (DFG) as part of Collaborative Research Center (SFB) 1320 EASE – Everyday Activity Science and Engineering, University of Bremen in subproject P01.

References

[1] T. Fullerton , Game Design Workshop. A Playcentric Approach to Creating Innovative Games, 2008 . doi: 10 .1201/b22309.

[2] Y. N. Harari , Sapiens : a Brief History of Humankind , Harper, New York, 2015 .

[3] D. Herman , Narrative theory and the cognitive sciences , Narrative Inquiry 11 ( 2003 ) 1 - 34 . doi: 10 .1075/ni.11.1.01her.

[4] M. Beetz , D. Beßler , A. Haidu , M. Pomarlan , A. K. Bozcuoglu , G. Bartels, Knowrob 2 .0 - a 2nd generation knowledge processing framework for cognition-enabled robotic agents , in: International Conference on Robotics and Automation (ICRA) , Brisbane, Australia, 2018 .

[5] R. Porzel , V. S. Cangalovic , What Say You: An Ontological Representation of Imperative Meaning for Human-Robot Interaction , in: Proceedings of the JOWO - Ontology Workshops , Bolzano, Italy, 2020 . URhLt: tp://ceur-ws. org/ Vol- 2708 /robontics4.pdf.

[6] V. B. Carlo Meghini , D. Metilli , Representing narratives in digital libraries: The narrative ontology , Semantic Web Journal ( 2021 ).

[7] B. Krieg-Brückner , M. Codescu , M. Pomarlan , Modelling episodes with generic ontology design patterns , in: Proceedings of the Workshop on Scalable Knowledge Graph Engineering 2020 , 2020 .

[8] S. Gottschalk , E. Demidova, EventKG - the Hub of Event Knowledge on the Web - and Biographical Timeline Generation , volume 10 , IOS Press, 2019 , pp. 1039 - 1070 .

[9] L. Spillner , C. Santagiustina , T. Mildner , R. Porzel , Towards conflictual narrative mechanics , in: Proceedings of the IJCAI/ECAI Workshop on Semantic Techniques for Narrative-based Understanding , 2022 .

[10] R. Evans , J. Hernández-Orallo , J. Welbl , P. Kohli , M. Sergot , Making sense of sensory input , Artificial Intelligence 293 ( 2021 ) 103438 .

[11] H. Kroll , D. Nagel , W.-T. Balke, Modeling narrative structures in logical overlays on top of knowledge repositories , in: G. Dobbie, U. Frank, G. Kappel, S. W. Liddle , H. C. Mayr (Eds.), Concpetual Modeling , Springer, Heidelberg, 2020 , pp. 19 - 33 .

[12] D. Beßler , R. Porzel , M. Pomarlan , A. Vyas , S. Höfner , M. Beetz , R. Malaka , J. Bateman , Foundations of the Socio-physical Model of Activities (SOMA) for Autonomous Robotic Agents , CoRR ( 2020 ). URL:https://arxiv.org/abs/ 2011 .11972.

[13] A. Gangemi , P. Mika , Understanding the semantic web through descriptions and situations , in: On The Move to Meaningful Internet Systems 2003 : CoopIS, DOA, and ODBASE - OTM Confederated International Conferences , CoopIS, DOA, and ODBASE 2003, Catania , Sicily, Italy, November 3- 7 , 2003 , 2003 , pp. 689 - 706 .

[14] D. Marcheggiani , L. Perez-Beltrachini , Deep graph convolutional encoders for structured data to text generation , in: Proceedings of the 11th International Conference on Natural Language Generation , Association for Computational Linguistics, Tilburg University, The Netherlands, 2018 , pp. 1 - 9 . URLh: ttps://aclanthology.org/W18-6501. 1d0o .i: 18653 /v1/ W18 -6501.

[15] L. J. Kurisinkel , N. F. Chen , Graph to coherent text: Passage generation from knowledge graphs by exploiting edge representations in sentential contexts , in: Proceedings of the AAAI Workshops on Commonsense Knowledge Graphs (CSKGs) , Virtual, 2021 .

[16] I. Blin , Building narrative structures from knowledge graphs , in: European Semantic Web Conference , Springer, 2022 , pp. 234 - 251 .

[17] E. Reiter , R. Dale , Building Natural Language Generation Systems, Natural Language Processing , Cambridge University Press, 2000 . URhLt:tp://prp.contentdirections.com/mr/ cupress.jsp/doi=10.2277/052102451X. doiD: OI: 10 .2277/052102451X.

[18] J. A. Bateman , Enabling technology for multilingual natural language generation: the KPML development environment , Journal of Natural Language Engineering 3 ( 1997 ) 15 - 55 .

[19] J. A. Bateman , J. Hois , R. Ross , T. Tenbrink , A linguistic ontology of space for natural language processing , Artif. Intell . 174 ( 2010 ) 1027 - 1071 .

[20] C. F. Baker , C. J. Fillmore , J. B. Lowe , The Berkeley FrameNet Project, in: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1 , ACL '98/COLING '98, Association for Computational Linguistics, USA, 1998 , p. 86 - 90 . UhtRtLp:s://doi.org/10. 3115/980845.980860. doi: 10 .3115/980845.980860.

[21] C. M. Keet , An exploration into cognitive bias in ontologies , in: E. M. Sanfilippo , O. Kutz , N. Troquard , T. Hahmann , C. Masolo , R. Hoehndorf , R. Vita , M. M. Hedblom , G. Righetti , D. Sormaz , W. Terkaj , T. P. Sales , S. de Cesare, F. Gailly , G. Guizzardi, M. Lycett , C. Partridge , O. Pastor , D. Beßler , S. Borgo , M. Diab , A. Gangemi , A. O. Alarcos , M. Pomarlan , R. Porzel , L. Jansen , M. Brochhausen , D. Porello , P. Garbacz , S. Seppälä , M. Grüninger , A. Vizedom , D. M. Dooley , R. Warren , H. Küçük-McGinty , M. Lange , A. Algergawy , N. Karam , F. Klan , F. Michel , I. Rosati (Eds.), Proceedings of the Joint Ontology Workshops 2021 Episode VII: The Bolzano Summer of Knowledge co-located with the 12th International Conference on Formal Ontology in Information Systems (FOIS 2021), and the 12th International Conference on Biomedical Ontologies (ICBO 2021 ), Bolzano, Italy, September 11-18 , 2021 , volume 2969 ofCEUR Workshop Proceedings, CEUR-WS.org, 2021 . URL:http://ceur-ws. org/ Vol- 2969 /paper38-CAOS.pdf.

[22] B. Krieg-Brückner , T. Mossakowski , M. Codescu , Generic ontology design patterns: Roles and change over time , 2020 . URLh:ttps://arxiv.org/abs/ 2011 .09353. d1o0i:.48550/ARXIV. 2011 . 09353 .

[23] J. A. Bateman , C. L. Paris, Constraining the development of lexicogrammatical resources during text generation: towards a computational instantiation of register theory , in: E. Ventola (Ed.), Recent Systemic and Other Views on Language, Mouton , Amsterdam, 1991 , pp. 81 - 106 .

[24] E. H. Hovy , Pragmatics and Natural Language Generation, Artificial Intelligence 43 ( 1990 ) 153 - 197 .

[25] K. Papineni , S. Roukos , T. Ward , W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting on association for computational linguistics , Association for Computational Linguistics , 2002 , pp. 311 - 318 .

[26] J. Cohen , A Coeficient of Agreement for Nominal Scales , Educational and Psychological Measurement 20 ( 1960 ) 37 .