Recently, such teaching methods as distance learning, e-learning and self-study have been actively developed. The widespread adoption of the Internet in all aspects of information technology has also affected the organization of the learning process. Now computer courses are very popular, which are electronic textbooks and provide for independent study. They are being replaced by distance and online learning. Concerning, this paper presents the current development of methods for computer processing of open tasks to conduct an effective analysis of the learning of educational material. Purpose of work was to analyze the methods and approaches used for computer processing of natural language, as well as tools to solve the problem of identifying test messages. The subject of research was mathematical models and methods of computer processing of natural language.
1.1
The world pandemic of the new coronavirus has changed the usual way of life and
approaches to education [
However, the pandemic only accelerated the process of digitalization of the
educational process, and trends in the transition of training to online have been observed for
several years [
One of the integral parts of the existence of mankind is the creation, transfer of
accumulated knowledge and skills, that is, the transfer of information. Recently, such
teaching methods as distance learning, e-learning and self-study have been actively
developed [
Now computer courses are very popular, they represent electronic textbooks and
provide independent study, but they are being replaced by distance and online learning
[
Traditional methods of remote control of knowledge use closed types of test tasks,
based on the usual calculation of the correct answers, and this does not always correctly
reflect the student’s knowledge [
The aim of the research is to analyze the methods and approaches used for computer processing of natural language, as well as tools to solve the problem of identifying test messages, and to build intelligent information technology for open tasks processing.
For tasks of this type, machine learning approach is usually used [
Learning ability is one of the main advantages of neural networks over traditional algorithms. Technically, training consists in finding the coefficients of connections between neurons. In the learning process, the neural network is able to identify complex relationships between input and output, as well as perform generalization. This means that if training is successful, the network will be able to return the correct result based on data that were not in the training sample, as well as incomplete and / or “noisy”, partially distorted data.
Artificial neural networks are used to solve various classes of problems, such as:
decision making and management, clustering, forecasting, approximation, data
compression, data analysis, optimization [
To teach word selection without pre-marked data, we first need to solve several
problems [
Take this example: The cat pushed the glass off the table. The data we need will come out as in figure 1. Each bracket denotes a single context window. The blue field indicates the input one-hot vector (target word), the red field indicates the output onehot vector (any word in the context window except for the target word, the so-called context word). Two data elements come out from one context window (there are two neighboring ones per target word).
Embedding layer stores the vectors of all words in the dictionary, where the number of
words x is the dimension of the space of the compressed vector representation of words
[
Let the vector ⃗ from real numbers be input, and at the output of the classifier, the exponent Y is calculated by the formula: = (⃗⃗⃗ · ⃗ + ⃗ ) = (∑ + ), (1) where ⃗⃗⃗ is the real vector of weights, ⃗ is regularization coefficient, which does not allow parameters to go beyond reasonable limits (through retraining), is the scalar product transformation function.
The values of the weights of the vector ⃗⃗⃗ are determined during machine learning on prepared samples. The function is usually a simple threshold function that separates one class of objects from another. In more complex cases, the function has the meaning of the probability of a solution.
The linear classification operation for two classes can be imagined as the reflection of objects in multidimensional space onto a hyperplane, in which those objects that fall on one side divides the lines, belong to the first class («yes»), and the objects on the other side - to the second class («no»).
Linear classifier is used when it is important to carry out fast calculations with high speed. It works well when the input vector x ⃗ is sparse. Linear classifiers can work well in multidimensional space, for example, to classify documents according to the word-birth matrix. In such cases, objects are considered to be well regularized. 2.3
( ⃗ ) ∶
→ ℝ ( ⃗ ) ∶ min ( )
⃗
In the case when you need to find the maximum, instead of ( ⃗ ), − ( ⃗ ) is used. The main idea of the method is to go in the direction of the steepest descent, and this direction is given by the anti-gradient - ∇ :
⃗ [ +1] = ⃗ [ ] − ⃗[ ]∇ ( ⃗ [ ]) where λ ⃗ ^ [j] sets the gradient descent speed and can be selected: − constant (in this case, the method may diverge); − descending in the process of gradient descent; − guarantees quick descent.
To find the minimum of ( ⃗ ) we get: [ ] = [ ] = To find the maximum of ( ⃗ ) we get: Gradient Descent Algorithm: ( ⃗ [ +1]) = ( ⃗ [ +1]) = ( ⃗ [ ] − ∇ ( ⃗ [ ])) ( ⃗ [ ] + ∇ ( ⃗ [ ])) 1. set the initial approximation and calculation accuracy ⃗ 0, ; 2. calculate ⃗ [ +1] according to the formula 4 and і [ ] according to the formula 5. 3. check the stop condition:
if then j=j+1going to step 2; otherwise ⃗ = ⃗ [ +1] is a solution.
| ⃗ [ +1] − ⃗ [ ]| > , | ( ⃗ [ +1]) − ( ⃗ [ ])| > (2) (3) (4) (5) (6) (7) 2.4
Softmax [
of the resulting vector is represented by a real number in the
interval [
The coordinates σ are calculated as follows: σ(z) = ∑ =1
= −
The Softmax function is used in machine learning for classification problems when the number of possible classes is more than two (for two classes a logistic function is used). The coordinates σ
of the resulting vector are interpreted as the probabilities that the object belongs to class . The column vector is calculated as follows: where is the column vector of features of an object of dimension × 1, is transposed matrix of weighting coefficients of features, has dimension × , is a column (8) (9) vector with limit values of dimension is the number of features of the objects.
Typically, Softmax is used for the last layer of deep neural networks for classification tasks. To train the neural network, cross entropy is used as an option for losses.
× 1, is the number of classes of objects, 2.5
The maximum likelihood method (also known as highest probability method) [28-29] in mathematical statistics is a method for estimating an unknown parameter by maximizing the likelihood function. It is based on the assumption that all information about the statistical sample is contained in this function. Maximum likelihood assessment is a popular statistical method that is used to create a statistical model based on data and provide an estimate of model parameters.
Let us have a sample 1, … , according to the distribution , where ∈ Θ is an unknown parameter. Let ( | ) ∶ Θ → ℝ be the likelihood function, where ∈ ℝ. Point estimate - called the maximum likelihood estimate of the parameter . ̂МП = ̂МП( 1, … , ) = arg max ( 1, … , | ) ∈Θ (10)
Thus, the maximum likelihood estimate is an estimate that maximizes the likelihood function for a fixed sample implementation. 3 3.1
TensorFlow [30] is an open machine learning software library developed by Google to solve the problems of building and training a neural network to automatically detect and classify images, achieving the quality of human perception.
The library allows developers to quickly and easily get started with deep learning in the cloud. The platform has wide industry support and has become a common research solution for deep learning and application development, especially in areas such as machine vision, natural language understanding, and speech translation.
TensorFlow comes with a complete set of visualization tools that simplify understanding, tuning and optimizing applications. Thanks to the support of various styles (from images and audio to histograms and graphs) you can quickly and easily create large deep neural networks. 3.2
While the system is learning, we need to check how well the network works in terms of accuracy. We do this with a test suite, taken from testing data, so it does not have duplication with training data.
Using validation sets based on testing data allows you to better understand how well the network can generalize what it is studying and apply it to other contexts. If we check the training data, the network may be oversaturated - in other words, finding out specific examples and remembering the answers to them does not help the network answer new questions.
After a little workout, let's take a look inside and see what answers we get from the network. In the diagrams below, we visualize the attention for each of the episodes (rows) for all sentences (columns) in our context. Darker colors pay more attention to this sentence on this episode. (fig. 4).
You should see a change in attention between at least two episodes for each question, but sometimes the attention will be answered within one and sometimes it will take all four episodes. If the attention turns out to be empty, it can be saturated and pay attention to everything at once.
The information presented in this work may form the basis for further development and improvement of the intelligent information technology of tasks processing for open tests. According to the developed models, a software product was created for further practical use. Within this research theoretical material on natural language processing was analyzed.
The following steps are used to solve the neural network training problems: data collection for training, preparation and normalization of data, experimental selection of training parameters, own training, validation of training, adjustment of parameters, final training.
The software was implemented in Python and the TensorFlow library was used. A neural network-based quality control system was created using a closed domain. 28. Mashtalir, V.P., Shlyakhov, V.V., Yakovlev, S.V.: Group structures on quotient sets in classification problems. In: Cybernetics and Systems Analysis, Vol. 50, №4, pp. 507-518 (2014). 29. Mashtalir, V.P., Yakovlev, S.V.: Point-set methods of clusterization of standard information.
In: Cybernetics and Systems Analysis, Vol. 37, №3, pp. 295-307 (2001). 30. Zeng, Z., Gong, Q., Zhang, J.: CNN Model Design of Gesture Recognition Based on Tensorflow Framework. 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China, pp. 1062-1067. (2020).