The paper considers the information technology for extraction of Ukrainian and English coherent text fragments from scientometric systems. The up-to-date methods of Data Mining, in particular, statistical models of Information Extraction and Machine Learning for identifying text fragments are analyzed. Based on intelligent data processing tools, the technology will make it possible to determine common information spaces of authors' scientific interaction due to identification of statistically significant collocations that display the topic of texts. The technology includes the distributive-statistical model and Natural Language Processing tools for identification and extraction coherent text fragments. The distributional semantic model MI is applied at the stage of probable collocation identification. Based on POS-tagging formalism, regular expressions, developed in accordance with the grammar of a particular language, allow extracting grammatically correct constructions, i.e. potential substantive, verb and adjective collocations. To identify coherence between these text fragments, dependency parsing is applied. The advantage of the developed technology is that both the grammatical structure and frequency of collocations are taken into account. The corpora of Ukrainian and English scientific texts are built on the basis of abstracts from articles indexed in Google Scholar and ScienceDirect scientometric databases. The effectiveness of the developed technology is assessed and exceeds the results of analogs. The use of the proposed technology could improve the quality of natural language processing. The solution to the problem of automatic extraction of coherent text fragments can be employed to monitor the development of scientific directions, extend research fronts, identify texts of the same domain, extract facts, etc.
are insufficient in the development of information support for libraries, electronic catalogs, computer bibliography, systems for automated import of documents, etc. Therefore, it is necessary to apply novel technologies for extraction of an implicitly expressed connection between text fragments from scientometric databases that will make it possible to provide relevant search and access to research works performed on the similar topics.
Such short text fragments, inter alia, can be collocations that represent a non-random syntactic and semantic combination of two or more lexemes and provide more specific semantic information than certain words. So, automatic extraction of collocations that display the topic of text documents can be an additional tool to identify common information spaces of authors’ scientific interaction.
Currently, documents, as online multimedia information with hyperlinks, have become a means of monitoring, influencing and communicating. One of the ways of storing such unstructured data, namely text documents, is a scientometric system that is a bibliographic and abstract database with tools for checking the citation of articles published in scientific journals. As the result, it is possible to identify the state-of-the-art directions of scientific research.
However, to form information spaces of scientific communities adequately, the level of
automation of unstructured data (text) processing needs increasing [
In general, intelligent analysis of data or Data Mining is defined as a decision support process
based on searching hidden patterns (information patterns). Data Mining technologies are one of the
most promising tools to extract valuable knowledge from massive amounts of data, discovering
structures, relations, and interconnectedness of data [
Thus, we can say that the challenging task of extraction of hidden information patterns such as collocations is highlighted at the intersection of Information Extraction and Machine Learning methods.
To identify collocations, distributive-statistical models are considered to be most applicable, the
sense of which is the statistical analysis of the probable co-occurrence of variables (lexemes) in large
data streams. Traditionally, these types of algorithms [
Statistical metrics or association measures are based on the frequencies of collocates (words) included in collocations to calculate the stability of lexical units. In total, there are more than 80 measures to assess the strength of connectivity. The most commonly used measures are MI, PMI, t-score, log-likelihood, probability coefficient, Pearson’s chi-squared test and others. However, statistical methods extract noisy data and ignore syntactic links between words.
At the same time, the analysis of Data Mining methods and statistical models of distributive semantics shows that collocation extraction requires the use of morphological and syntactic tools in addition to statistical models. In this case, identification of collocations allows us not only to take into account the probability of co-occurrence of collocates, but also to formalize grammatical dependencies between the main and dependent components of text fragments.
To solve the problem, we propose the productive combination of a statistical measure,
in particular, the model of mutual information MI that compares dependent context-related
frequencies with independent ones, and Natural Language Processing tools based on the Universal
Dependency (UD) formalism [
Forming common information spaces of scientific interaction of authors, the developed technology for extraction of Ukrainian and English coherent text fragments includes four stages (Fig. 2).
At the first stage of extraction of abstracts from articles indexed in Google Scholar (a freely accessible scientometric database) and ScienceDirect (a database of scientific publications of the Dutch publisher Elsevier), the scientific text corpora were designed. Each corpus contains 350 abstracts of research articles on Artificial Intelligence in the Ukrainian and English languages over a period of 2017-2019. The dataset is of 70 000 (Ukrainian) and 65 000 (English) words.
Since the research on collocations in Natural Language Processing focuses on two related tasks:
collocation identification and collocation extraction [
Consequently, based on the model of distributive semantics MI, we identify potential candidates for the main and dependent components of collocations at the second stage. The metric makes it possible to single out rare colocations and identify terminology and other constructions where frequency rates of collocates are very small: ( , ) =
( , ) × 2 ( ) × ( ) , (1) where n is a main word; c is a collocate; f (n, c) is the frequency of the main word n in pairs with the collocate c; f(n), f(c) are absolute (independent) frequencies of the main word n and the collocate c in the corpus; N is the total number of words in the corpus.
The result of MI computation is represented by the list of statistically significant bigrams and trigrams, probable collocations, that display the topic of processed texts identified in corpora.
The next stages are devoted to establishing the linguistic correctness of the identified candidates, namely identification of the types of potential collocations and syntactic dependency between collocates via Stanza part-of-speech (POS) tagging and dependency parsing.
Stanza [
The application of Stanza POS-tagging was conducted to extract 3 types of collocations:
substantive, verb and adjective. According to the corpus-oriented approach, these types of
collocations represent significant text fragments that are most often found in corpora [
In this way, at the third stage, we define the following POS-tags in accord with the main types of collocations: 1. Substantive collocations: • <NOUN> <NOUN> (e.g. input process, scrambling of data, “налаштування пристрою” device setting); • <ADJ> <NOUN> (e.g. user-centered approach, “програмне забезпечення” - software); 2. Verb collocations: • <VERB> <NOUN> (e.g. to meet requirements, “нести відповідальність” - to bear responsibility); • <VERB> <ADP> <NOUN> (e.g. to be caused by changes, “використовувати в описах” - to use in manuals); • <ADV> <VERB> (e.g. to process iteratively, “уважно ознайомитися” - to see thoroughly); 3. Adjective collocations: • <ADV> <ADJ> (e.g. highly accurate, “абсолютно симетричний” - absolutely symmetric).
Using Stanza DepparseProcessor at the final stage of the technology, each sentence in the output is parsed into the UD structure. As the result, syntactic dependency relations between extracted collocates are defined, representing coherence of the text fragments, i.e. collocations: • nmod: nominal modifier shows nominal dependents of another noun or a noun phrase, corresponds to a genitive complement (in Ukrainian). The nmod relation is used in Substantive collocations (<NOUN> <NOUN>); • compound: compound relation is used for noun compounds (in English) in Substantive collocations (<NOUN> <NOUN>); • amod: adjectival modifier of a noun serves to modify the noun. The relation is applied in Substantive collocations (<ADJ> <NOUN>); • obj: object of a verb denotes the entity acted upon, the object is marked by the accusative case (in Ukrainian). This core argument of a verb is represented in Verb collocations (<VERB> <NOUN>); • obl: oblique nominal relation functionally corresponds to the prepositional construction in the double object construction, as well as temporal and locational modifiers or nominal modifiers for the agent of a passive verb. It occurs in Verb collocations with prepositions (<VERB> <ADP> <NOUN>); • advmod: adverbial modifier functions like adverbs and serves to modify predicates in Verb collocations (<ADV> <VERB>) or modifier words like adjectives in Adjective collocations (<ADV> <ADJ>).
Thus, collocations are considered as coherent text fragments if grammatical characteristics of collocates, identified via MI model, satisfy POS-tagging and syntactic dependency parsing features. The collocations extracted from scientific text corpora allow not only identifying the missing combinations of words in Ukrainian and English linguistic sources, but also calculating the statistical indicators of their stability.
The developed technology is implemented as software applications to extract English (Fig. 3) and Ukrainian (Fig. 4) coherent text fragments from large data streams. Software implementation allows you to download text corpora, run POS-tagging via Stanza POSProcessor and dependency parsing of texts via Stanza DepparseProcessor, previously processed by the TokenizeProcessor, MWTProcessor, and LemmaProcessor, and display all the extracted collocations with examples of their contexts in descending order of MI coefficient values.
Running the software program, coherent text fragments are identified that display the topic of scientific texts and form a common information space (a scientific front) of researchers’ interaction. Applying the identification results subsequently makes it possible to get new abstracts of the same topic and extend scientific fronts.
To assess the effectiveness of the developed technology and carry out comparative analysis with existing technologies for extracting collocations from large data streams, the precision score was calculated.
To compare the obtained results with another existing software implementation dealing with a
similar problem, the web application Sketch Engine [
Table 2 shows the comparative analysis of the precision coefficient, determined by the ratio of the number of relevant decisions made by the system (Correctly Extracted Collocations) to the total number of collocations extracted by the system (All the Extracted Collocations). The expert evaluated the extracted collocation as relevant if its morphological and syntactic dependencies were identified correctly and false otherwise.
Based on the statistical model of distributive semantics and Natural Language Processing tools, the information technology is proposed for determining common information spaces of authors’ scientific interaction due to identification of statistically significant collocations that display the topic of texts. The effectiveness of the technology is assessed and exceeds the results of analogs.
The developed software implementation will improve the quality of collocation extraction, and can also be proposed as an advanced model for existing text processing systems. In instance, it can be applied to monitor the development of scientific directions, extend research fronts, classify texts of the same topic, extract facts, etc.
In future studies, we intend to broaden the scope of our research and focus on a more complex challenging problem of distant multi-gram collocations extraction. Additionally, our further work will extend the domain of the texts studied. In prospect, we intend to spread our dataset for free access to fulfil similar approaches.