3 The purpose of this article is to present a new approach for the discovery and labelling of the implicit conceptual schema of texts through the application of the Thematic Progression theory. The underlying conceptual schema is the core component for the generation of summaries that are genuinely consistent with the semantics of the text. Automatic Summary Generation was first proposed in the late 1950s. Outstanding examples of this early stage are Luhn (1958), whose method is based on sentence extraction relying on its words weightings, inferred from TF-IDF metrics, or Edmundson (1969), who proposed novel sentence weighting metrics, such as the presence of words from a predefined list, the presence of the words of the title of the document or its positioning at the beginning of documents and paragraphs. These are paradigmatic examples of the first extractive summarization techniques: techniques based on the verbatim extraction of the most relevant parts of a text. The generated text summary was, thus, a collection of sentences considered relevant but, often, semantically inconsistent because of the overall weakness in coherence (the text does not make overall sense) and cohesion (the sentences are connected incorrectly). The summary generated was consequently a poorly connected text with no global meaning, presumably due to the assumption of independence of the extracted sentences (Lloret et al. 2012).
Currently, five main approaches to extractive techniques can be
distinguished: (i), statistical approaches
Although historically less addressed in the literature, abstractive
models try to address the lack of coherence and cohesion in the
summaries produced, using some source of semantic internal
representation of the text (which can be merely the output of an
extractive process) to generate the ultimate summary, composed of
sentences not necessarily included in the original text. Although this
approaches theoretically improve the consistency issue, they
introduce a new complexity layer: a natural language generator
module. Despite this greater complexity, nowadays text
summarization research is progressively shifting towards abstractive
approaches
Traditionally, abstractive summarization techniques have been
classified into structure-based, intended to populate predefined
information structures out of the relevant sentences of the texts, and
semantic-based, involving a wide variety of knowledge
representation techniques. Regarding the former, depending on the
structural schema chosen, it is possible to identify, (i), tree-based
models
The aforementioned tendency towards abstractive approaches in recent years is framed at a stage when the new Deep Learning models have proved to be particularly promising for using vector spaces as a way to address the shortcomings of discrete symbols as the input for Natural Language Processing tasks, such as tokens or lemmas, which cannot represent the underlying semantics of the concepts involved. This new paradigm has provided techniques for both extractive and abstractive summarization, such as the clustering of sentence and document embeddings, or the generation of correct sentences given a sentence embedding and a language model.
Remarkable examples are the contributions of Templeton et al. (2018), who compare different methods of sentence embeddings computing their cosine similarity, or Miller et al. (2019), who
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). proposed k-means clustering to identify sentences closest to the centroid for summary selection.
In addition to the distinction between extractive and abstractive
approaches, there is a crucial challenge in automatic summarization
which affects them both: the subjectivity of the accuracy scoring of
summaries. This implies a new difficulty in the creation of objective
gold datasets composed of correct summaries. In this context,
unsupervised summary models, such as the one proposed in this
paper, which does not require training labelled data, has become
particularly relevant. Among the unsupervised approaches we can
highlight, (i), approaches which are extensions of word embedding
techniques, such as the n-grams embeddings
This paper addresses one of the weaknesses of extractive models discussed in the previous section, i. e. the lack of coherence in the summaries produced, especially when there are insufficient linguistic datasets in a language for applying machine or Deep Learning methods. In this respect, the solution we propose identifies implicit conceptual schemas from texts using the morpho-syntactic knowledge currently provided by NL analyzers.
The paper is organized as follows: in section 2 we define the hypothesis and objectives of the research work. In section 3 we present a review of the linguistic theories on which we base our solution: the Thematic theory and Thematic Progression theory. In section 4 we present our solution: the application of both theories for the identification of the text conceptual schema. In section 5 we study the feasibility of our solution for the automatic extraction of thematic progression in Spanish, a language with few linguistic datasets for text summarization. Finally, in section 6 we draw the conclusions of this work and present our future research lines.
Our hypothesis is that applying the Thematic Theory and the Thematic Progression Theory to annotate the discourse features theme, rheme and their coreferences will allow us to extract thematic progression schemas, which represent the implicit conceptual schemas of texts.
Therefore, our aim is obtaining an internal representation of the text informational structure as a formal representation for text summarization. The advantage of this solution is that it can be applied to any language regardless of whether or not there are enough training data for the implementation of machine learning and Deep Learning techniques. In our work we will use Spanish as the language to study the feasibility of the solution. We also hope to contribute to the generation of summaries in Spanish, a task currently performed with moderate efficiency due to the limited availability of linguistic resources.
3.1.
The thematic theory is framed within the optics of linguistic analysis
corresponding to the informational layer. The uses and applications
that the authors have been giving to terms such as theme, focus, topic
and notions such as new information or emphasis have been
overwhelmingly numerous (Gutiérrez 2000). In accordance with the
Thematic theory, in descriptive and narrative texts, which are the
ones most typically to summarize, known information, or theme, is
consensually described to be positioned at the beginning of
sentences. By contrast, the phrases containing the informative
contribution of the sentence, also known as rheme, tend to be located
further to the right, ahead in the time of enunciation. This description
is consistent with how the acquisition of new knowledge is described
at the neurological level, through networking the known with the
novel or by altering pre-existing relationships
In order to clarify how we will use these concepts, we present here a series of examples adapted from Gutiérrez (2000: 18) and their corresponding answers: 1. Who joined Luis this morning?
Ludwig was joined this morning by Peter. 2. When did Pedro join Luis?
Pedro joined Luis this morning. 3. Who joined Pedro this morning?
This morning Peter joined Ludwig.
That these statements are different is a standard judgment for any native speaker. Although they share the same representative function, i. e. their syntactic and semantic relations do not differ, they show different informative functions. Therefore, in spite of transmitting the same information about the world, they do not inform in the same way. Accordingly, the underlying assumption of our proposal is that the thematic status of a phrase (like who, when and who in the examples above, respectively) is relevant in terms of the prevalence of the concept involved for the summarization of a document. Their clustering along a document, taking into account the thematic progressions patterns found, as further explained in the next section, is expected to reveal the conceptual schema of the text. 3.2.
Daneš (1974: 114) presents the thematic progression as the choice and arrangement of discourse themes, their concatenation, hierarchy and relation with the topics of texts. Accordingly, he argues that it is possible to infer the informational schema of a text from its themerheme organization. It is considered that there are three main typologies of thematic progressions: (i), linear progression, in which the rheme of one sentence is the theme of the subsequent sentence, (ii), constant progression, in which a theme is repeated over several sentences, and, (iii), derived progression, in which several topics are derived from the same inferred hypertheme. Apart from these three basic types, Daneš (1974: 120) also proposed that the combination of them can lead to thematic progressions of higher levels of abstraction, such as, (iv), the split rheme progression, which consists of the existence of a complex rheme, whose hyponyms and meronyms are themes of the subsequent sentences. Finally, he concludes (1974: 122) that the study of the thematic organization of a text could be useful for numerous practical applications, among which outstands information retrieval, given the performance achieved nowadays by the tools available for the automatic text analysis.
The usefulness of the thematic or rhematic roles of concepts along texts for automatic text summarization arises from two main facts. On the one hand, the theoretical validation of the concept of thematic progression enjoys consensus among researchers as a relevant description for the semantic structure of texts. On the other hand, although it has been traditionally examined through the optics of the Pragmatics layer, the thematic or rhematic status of a concept is actually embodied in the surface syntactic layer, which is prone to be represented in an easy-to-compute form.
Concerning the correlation between the theme of a sentence and its
syntactic structure, which is crucial for its automatic annotation,
Halliday (1985) proposed an interesting categorization based on the
concept of linguistic markedness. Thus, in SVO languages, such as
English or Spanish, for declarative sentences there are unmarked
themes, prototypically the syntactic subjects preceding principal
verbs, and marked themes, such as circumstantial attachments,
complements, or sentences with predicate construction. Examples
for the former are the first and second sentence of the examples
provided above with Ludwig and Pedro as unmarked themes
respectively, whilst the third sample sentence is an example for the
latter, with this morning as theme. Thematic equative sentences,
such as What I want is a proper cup of coffee, would be excluded
from this categorization. For interrogative sentences, the unmarked
themes are definite verbs for yes-no questions, such as did deliver in
Did the president deliver a speech, and interrogative pronouns and
similar phrases for non-polar questions, such as where in Where is
Berlin located?, whilst marked themes are circumstantial adjuncts.
Besides, a constituent which is not the theme of a sentence may
appear occasionally in a prototypical theme position. This
phenomenon has been referred to by several names, such as
focussing
In short, it is possible to locate the discourse elements theme and rheme using syntactic knowledge. To the extent that syntactic analysis is a task that can be considered well solved in NLP, it seems feasible to be able to automatically locate the theme and rheme in every sentence of a text. The next natural step in order to obtain the thematic progression schema, i. e. the conceptual schema of a text, is to connect each theme and rheme of the sentences of the text in the ultimate thematic path. 5.
Aiming to conduct a first study to verify the applicability of the Thematic Progression theory for the extraction of the underlying conceptual schema of a text, we carried out an exploratory corpus survey with Spanish descriptive texts. We analyzed the mean ratio and the ratio per text of preceding subjects, since they are the prototypically unmarked themes. The examined corpus is AnCora Surface Syntax Dependencies3 (AnCora GLiCom-UPF 1.1), published in 2014 at Pompeu Fabra University, which contains 17,376 sentences manually annotated with the lemmas, the PoS tags plus other morphological features and both dependency heads and relations for every token. The analysis was based on a symbolic rulebased grammar expressed as sub-tree extraction operations from the dependency tree of the sentences. In order to ensure the generality of the grammar, the elements obtained through rules for sampling preceding subjects were compared with the corresponding results in a second version of the corpus, resulting from its automatic annotation with the Freeling analyzer. The thematic progression analyzer scripts used to process the grammar and to generate the outputs of the corpus rendering are publicly accessible from github4. Basically, the grammar and the analysis algorithm to extract the thematic progression schema of the text were designed in three consecutive steps: (i), first, the automatic identification and labelling of themes for every sentence; (ii), second the subsequent identification of their rheme; and, (iii), finally, the identification of concepts corresponding to the same theme or rheme in a text. Each step was carried out using a symbolic syntactic-semantic rule-based grammar expressed as sub-tree extraction operations. For the grammar definition, an approach based on the transformation of dependency trees has been applied. Thus, for the simplest scenario of finding unmarked subjects in SVO languages, such as Spanish, two categories of rules were defined: (i), matching rules of child dependencies from a selected head token, consisting of the identification of a dependency relation as the name of a relation of arity two with a first argument as the key and the value expected for the selected parent and a second argument with the options for matching, being ALL if all children nodes from the head of a scope should be matched or ONE if only the immediate child should be matched. For example, the SUB(deprel:ROOT, ALL) rule would match subtrees consisting of all child nodes of the SUB children of a token tagged with a ROOT dependency relation (as shown in figure 1, obtained from the Freeling 4.1 demo5); and, (ii), matching rules of head dependencies from a selected child token, consisting also of the identificator of a dependency relation as the name of a relation of arity two, whose arguments are the same as for the child dependency rules but in the opposite order. The second type of rules can apply, for example, for sentence compression when several propositions are involved. The analyzer accepts various corpus formats as input and transforms them into the universal CoNLL-U format, where the additional theme and rheme features are added for every token in the main proposition of sentences. We found that, with our first version of the rules, theme and rheme annotation was correct in roughly half of the cases, as shown in table 1. Through a careful manual review of the data, these results have enabled us to confirm a significant correlation between the syntactic-semantic and discourse layers outlined by the Thematic theory, and, consequently, the feasibility of automating identification of, at least, half of the themes. With the syntactically annotated GLiCom-UPF 1.1 version of the AnCora corpus we seek to provide objective metrics, not prone to major annotation errors, as the corpus annotation has been manually reviewed. The qualitative analysis of this data is intended to assess whether or not the preceding subjects match the main themes of sentences in order to ultimately detect the underlying thematic progression template of texts. By contrast, with the Freeling version of the corpus we aim to assess the accuracy in applying the rule for the extraction of unmarked themes with no dependence on manually annotated data, because Freeling is the best option for syntax annotation in Spanish at this stage. According to this objective, we have generated two different files, one for each version of the corpus, with the suspected overmatched and undermatched sentences, as shown in table 2. As the figures suggest, a pervasive tendency for undermatching sentences with preceding subjects by Freeling has been confirmed through careful examination. We found that the vast majority of mismatched annotations involve some type of coordinated or juxtaposed clauses. These syntactic structures are analyzed by Freeling with a highly fluctuating dependency structure, which is quite different from the analysis in GLiCom-UPF 1.1. This high variability in the syntactic tree accounts for the vast majority of both the undermatched and overmatched sentences ratios.
A qualitative analysis of unmatched sentences has also been conducted, revealing a strong presence of thematization as the most relevant finding. This sentence pattern has been referred to by the Thematic theory as a relevant discourse feature, indicating a break in the information flow of the text, as further discussed above. A promising conclusion of the analysis of the patterns found is their suitability for being implemented in the rule formalism designed. Besides, as a synthesis of the findings obtained from the qualitative analysis of the manually annotated version, there is a strong presence of subordinate clauses in the corpus, which implies the necessity of more complex rules to select the most informative proposition in the sentence. The observed patterns have been categorized into three main categories (the most informative clauses appear in bold and the selected theme is underlined):
Sentences whose root clause is the most relevant (e. g. Since pharmacists work with a high profit margin, the business opportunity is huge).
Sentences whose root clause is not the most relevant. (e. g. The main factor is that electricity consumption during the summer is now not much lower than it used to be).
Sentences whose root clause is not the most relevant but provides a crucial modality feature for information retrieval (e. g. Investigators are convinced that someone deliberately cut that rubber).
As observed in the study conducted, this first approach to rule-based theme annotation seems to claim the theoretically hypothesized correlation between the syntactic-semantic and discourse layers required by our proposal. However, the qualitative analysis of both matched and unmatched sentences revealed the need for more complex tree rewriting rules to achieve a more accurate theme selection in order to obtain thematic progression schemas from texts. Regarding subordination, i. e. sentences with several propositions with different syntactic status, we are working on two feasible options for sentence compression: (i), the choice of the most relevant proposition for every sentence, and, (ii), the choice of the ordered subset of its n more relevant clauses. In addition, this study shows the necessity to implement an algorithm to infer the modality from the main verb. We also found that the rules should be refined in order to capture the various ways in which coordinated and juxtaposed clauses could be analyzed, given the high variability observed in the automatic syntactic annotation. Finally, the study revealed that it will be necessary to design lexicon-based rules to capture lexical semantic generalizations. With all this, in principle, it seems possible to apply this new linguistic approach for the extraction of implicit conceptual schemas from texts.