Processing and text mining are becoming increasingly possible thanks to the development of computer technology, as well as the development of artificial intelligence (machine learning). This article describes approaches to the analysis of texts in natural language using methods of morphological, syntactic and semantic analysis. Morphological and syntactic analysis of the text is carried out using the Pullenti system, which allows not only to normalize words, but also to distinguish named entities, their characteristics, and relationships between them. As a result, a semantic network of related named entities is built, such as people, positions, geographical names, business associations, documents, education, dates, etc. The word2vec technology is used to identify semantic patterns in the text based on the joint occurrence of terms. The possibility of joint use of the described technologies is being considered.
Pullenti is a program for processing unstructured
natural language texts. Program functions: breaking down
text into words, performing morphological analysis,
determining of all possible parts of speech of words
(regardless of context), normalizing words, bringing words
to the desired case / gender / number, highlighting named
entities, multiplication of functions with numeric, nominal
and verbal groups, brackets, quotes and other useful
features [
In Pullenti, such objects as persons, organizations,
dates, geographic objects, sums of money, etc. are
distinguished. There are specialized analyzers that cover a
certain subject area. For example, identifying the structure
of a regulatory act and a contract with its details, analyzing
the title pages, literary characters, incidents, etc [
Fig. 1. Graph of selected named entities.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Pullenti ChatBot technology is designed to develop the intellectual part of chat bots. The technology is based on the SDK Pullenti (www.pullenti.ru), which contains various linguistic processing procedures, including morphological analysis. In addition, the technology offers a number of specific handlers of typical situations arising in the process of dialogue. For example, the selection of a phone number from a sequence in which the numbers are given in words, which takes place at the output of voice recognition systems ASR (automatic speech recognition), assessment of emotional state, typical situation (agreement, refusal, greetings ...), etc. The technology is aimed at developing the part of the chat bot that mimics its “brain”, that is, responsible for analyzing text fragments from the user (if it’s a voice, then after recognizing it), understanding, extracting data from the text and generating text answer.
Here is an example of using Pullenti through Python (Jupyter Notebook). The following program selects name groups from arbitrary text: «American President Donald Trump wrote on Twitter on Thursday that it was time for the US to recognize the Golan Heights as Israel in the interests of the security of Israel and the region as a whole.
A number of countries in the Middle East and Europe have already expressed regret in connection with this decision, and the Russian Foreign Ministry called it irresponsible and leading to the destabilization of the region».
The result of the program: «['AMERICAN PRESIDENT', 'PRESIDENT', 'TRUMP', 'THURSDAY', 'TWITTER', 'PORA', 'GOLANA HEIGHT', 'HEIGHT', 'INTEREST', 'SECURITY', 'REGION', 'WHOLE', 'SERIES', 'STRANA', 'NEAR EAST', 'EAST', 'EUROPE', 'Uzh', 'COMPLAINT', 'CONNECTION', 'DECISION', 'DECISION', 'MFA', 'MASTER','DESTABILIZATION', 'REGION']».
Pullenti does not include context definition functions, therefore the meaning of a word must be performed by other means, not by the program Pullenti. One of these tools is a program from Google – word2vec.
Word2vec – is a set of models for the analysis of the
semantics of natural languages, which is a technology that
is based on distributional semantics and vector
representation of words [
Before the appearance of neural networks, to analyze the proximity of words, a table of frequency of each word was compiled. That is, they made a matrix where words were horizontally and vertically laid out, and the frequency of the word in the specified line with the word specified in the column was indicated in the cells.
This paper discusses the skip-gram algorithm for
predicting a neighboring word. Further, this approach
covers several words [
At the input of the neural network, pairs of words are fed, the window size is selected, and then the moving window slides through the text over all the pairs of words in this window. If you select a window equal to one, the window will contain one word to the left of the target word and one word to the right of the target word. If the size of the window is equal to two, then to the left and right of the target word there will be two words each. Below is an example for a window equal to two (articles removed): Old abandoned house stands on the edge of the forest Training phrases: (old, abandoned), (old, house) Old abandoned house stands on the edge of the forest Training phrases: (abandoned, old), (abandoned, house), (abandoned, stands)
Old abandoned house stands on the edge of the forest Training phrases: (house, old), (house, abandoned), (house, stands), (house, on)
Old abandoned house stands on the edge of the forest Training phrases: (stands, abandoned), (stands, old), (stands, on), (stands, edge)
The neural network will learn statistics on the frequency of occurrence of each pair of words. Every word needs to be converted to digital form. One of the common ways is to present it as a column vector (one-hot encoding), for example, like this: 0 1 ⃗ = 0 . . .
[ 0 ]
Here our word, which we represent as a vector, takes second place in the dictionary. Transformations using a neural network can be represented as follows (fig. 2).
Here x is the input word (or several words) by which we want to predict, y is the word (or several words).
h (hidden layer of the neural network) is a vector obtained by multiplying the word vector x by the matrix of weight coefficients w:
ℎ⃗⃗ = ⋅ ⃗ w – is a matrix containing weights, it has the dimension: (dictionary length) * (number of attributes). The number of signs is set once before the launch of the neural network, it is selected to obtain the best result. Example: Google used 300 tags to train a neural network on a variety of Google News data. The weighting coefficients at the initial moment of time take random values, then they are adjusted in accordance with the subtracted proportion of values w, so the matrix w′ method of back error propagation.
∑ =1
After the hidden layer h, taking into account another matrix of weight coefficients w ’, the vector u is formed: ⃗⃗ = ′ ⋅ ℎ⃗⃗ = ′ ⋅ ⋅ ⃗
The dimension of the vector u coincides with the dimension of the vector ⃗⃗.
To normalize the output vector ⃗ in the range [0; 1], we use the softmax function (it is used as the activation function, see (⃗⃗ ) figure 2): where N is the number of signs.
As a result, we obtain that is the probability of observing (predicting) the i-th word (or phrase) in the dictionary with the incoming word (context) x.
The purpose of the neural network, shown in Figure 1, is to determine the weights w and w’. The criterion for convergence of calculations is the maximization of the probability y for all possible output words (phrases). As a result of mathematical transformations (taking the logarithm of the probability y, then calculating the derivative of the logarithm of the probability y using the variable w’) we get an equation for which it is impossible to find the optimum. Therefore, it is necessary to use numerical methods. One of the best numerical methods is the gradient descent method. The result is that you need to solve a recursive task:
′ ← ′ − (
⋅ (1 − ))
Thus, if the probability for the output word being searched is maximal, then the expression in parentheses is close to zero, and probability of output word is very small, then from w’ new w ' old w ' . Otherwise, when the approaches to the matrix w.
In the example below, we are processing several articles and collections of articles related to virtual reality and modeling.
The processing objects are the following documents: materials of conferences on programming, computer science, collections of articles, presentations, dissertations. The processing program is written in python.
Before processing was only 473 337 words.
As a result of processing module Pullenti formed 320 564 words.
Virtual
After the processing the above documents for 20 cycles with the word2vec module there were highlighted 8 words closest to the word 'virtual'. Here they are: scholar reality system google pubmed research analysis environment
In this work, we use only certain functions of Pullenti that have common functions for highlighting some entities. Pullenti can use different libraries for different situations. The quality of building model depends on which class the text belongs to. Classification of texts using neural networks will allow us to choose special methods of text processing and improve the quality of the resulting model.
In Pullenti for complex mining tasks, a higher level presentation of data may be required.
Pullenti denotes named entities based on the construction of a chain of adjacent words. The use of neural networks and, in particular, the genism library for additional analysis of the text, allows us to define significant verbose terms that are in the sentence quite far from each other. In this case, it will be possible to form semantic named entities and carry out their identification throughout the text based on the analysis of the word environment.
An example of the work of the program Pullenti has been analyzed, a drawback has been revealed - the lack of definition of the context of words.
An example of the work of the word2vec technology has been analyzed, and the problem of training on a small amount of data has been revealed.
During the training of the word2vec model, satisfactory results were obtained with the number of cycles equal to 20.
The use of methods based on neural networks for the analysis of texts will allow us to switch from text parsing to partially semantic modeling.
The approach outlined in this document can be used to analyze texts, compare and classify documents.
This work is supported by Russian Foundation for Basic Research, grants 18-07-01111, 18-07-00909, 19-0700857 and 16-29-09527.
We are grateful to the Russian Foundation for Basic Research for financial support of our projects.