An analysis of some long text if authors are unknown or if it was written by one author is still an interesting problem and it could be done using methods of data analysis and data mining, and using structural analysis. In the paper, it is described a system of modi ed S elf-Organizing Maps working on probabilistic sequences built from a text. The sequences were built on letters and on words as n-grams, 1 n 4. The system is trained to input sequences and after the training it determines text parts with anomalies (some di erent characteristics) using a cumulative error and a complex analysis. In tested long texts the system was successful, it covered a composition of texts.
and on words). In the fourth section, we describe our new developed system to anomaly detections. The fth section contains an evaluation and illustration of results on English and Arabic texts. In the conclusion we give a resume of our work and some plan of further work in the area. 2 2.1
The applied method to an analysis of word or letter sequences is a probabilistic model. The working sequences will be presented by probabilities of their occurrences. It is possible to use individual relative frequencies of words or letters. But some of word combinations have a higher probability, some of them are used only once. It means, we will use a conditional probability of a word given by the previous words.
- a nite alphabet of symbols; j j is the number of symbols in ; in our texts, A will be Arabic and E
V - a nite vocabulary of words in a given text (the alphabet ) presented in the alphabetic order; jV j the numbers of di erent words in the vocabulary V ; d1N = hd1; : : : ; dN i; di 2 V - the text as a nite sequence of words; N - the number of words in the text d1N ; s1M = hs1s2 : : : sM i; si 2 ; - the text as a nite sequence of symbols; M { the number of symbols in the text s1M (including spaces); The probability of a complete sequence of words d1N can be represented by Pd(d1; d2; : : : ; dN ) if each word is supposed as an independent event.
The probability of a complete sequence of symbols s1N can be represented by Ps(s1; s2; : : : ; sM ) if each symbol is supposed as an independent event.
According to [Jurafsky2000] the probabilities Pd and Ps can be decomposed using conditional probabilities in the following way:
Pd(dN ) = P (d1)P (d2jd1)P (d3jd12) : : : P (dN jd1N 1) =
iN=1P (dijdi1 1); where di1 1 = hd1 : : : di 1i Ps(sN ) = P (s1)P (s2js1)P (s3js12) : : : P (sN js1M 1) = iM=1P (sijsi1 1); where si1 1 = hs1 : : : si 1i The problem to compute the probabilities is solved using some simpli cation: an approximation of the probability of the word (symbol) given by all the previous words (symbols). The probability is depending on the probability of all preceding words but we will use only small number of them, for example 2; 3; 4. It means, we work with 2; 3; 4-grams.
The n-gram model approximates the probability of a word di; 1 i N (symbol si; 1 i M ) using all the previous probabilities of words Pd(dijd1 i n+1)) by the conditional probability of preceding i n+1) (symbols Ps(sijs1 words Pd(dijdi 1) (symbols Ps(dijdi 1)). It means,
Pd(dijdi1 1)
i 1 Pd(dijdi n+1); 1
Ps(sijsi1 1)
Ps(sijsii 1n+1); 1 i
M;
i i
N; N; i 1 Pd(dijdi n+1)
C(dii 1n+1di) C(dii 1n+1) ; 1
Ps(sijsii 1n+1)
C(sii 1n+1si) C(sii 1n+1) ; 1 i
M; where C(dii 1n+1di) is the count of n-grams dii n+1di and C(dii 1n+1) is the count of all (n 1)-grams dii 1n+1: (1) (2) (3) (4)
1. The sequence Ss of symbol probabilities:
C(si) { the number of occurrences si in the text D;
The sequence Ss is prepared by the formula: 2. The sequence Ssn of n-gram of symbol probabilities: fs(si) = C(si) ; 1
M i
M: The substring sisi+1si+2 : : : si+n 1; i = 1; : : : M the length of the n-gram; Cn(ng) { the number of occurrences of n-gram ng in the text D; Cn 1((n 1)g) { the number of occurrences of (n 1)-grams (n 1)g in the text D; The sequence Ssn contains the frequencies of ng in the text D n + 1, n = 1; 2; : : : is called n-gram of symbols; n 3. The sequence Sw of word probabilities:
C(di) { the number of occurrences di in the text D; The sequence Sw is prepared using the following formula: fsn(ng) =
Cn(ng) Cn 1(n 1g)
: fw(di) = C(di) ; 1
N i
N: (5) (6) (7) (8) 4. The sequence Swn of n-gram of word probabilities:
The subsequence didi+1di+2 : : : di+n 1; i = 1; : : : N n + 1, n = 1; 2; : : : is called n-gram of words; Cnw(ng) { the number of occurrences of n-gram ng in the text D; Cnw 1((n 1)g) { the number of occurrences of (n The sequence Swn is prepared by formula: 1)-grams (n 1)g in the text D; fwn(ng) =
Cnw(ng) Cnw 1(n 1g) : 3 3.1
The Self Organizing Map belongs to the class of unsupervised and competitive learning algorithms [Kohonen2007]. This type of neural network is used to map the n-dimensional space to the lower-dimensional space, usually to two-dimensional space. The neurons are arranged usually to the two dimensional lattice, frequently called a map. This mapping is topology safe and each neuron has its own n-dimensional weights vector to an input. If input is represented by some sequence (for example, time series), when the order of values is important, then it is necessary to follow the order and do not change it.
The steps of the algorithm: 1. Initialization. The weight vectors of each node (neuron) in the lattice are initialized to a small random value from the interval h0; 1i. The weight vectors are of the same dimensions as the input vectors. 2. Winner identi cation for an input vector. Calculate the distance of the input vector to the weight vector of each node. The node with the shortest distance is the winner. If there are more than one node with the same distance, then the winning node is chosen randomly among the nodes with the shortest distance.
The winning node is called the Best Matching Unit (BMU). Let i be index of the winning node. 3. Neighbors calculation. The following equation is used: h(i ; i; t) = exp jjri(t)
ri (t)jj ; 2(t) where (t) means the radius of the neighborhood function, t is an iteration step, ri(t) and ri (t) are the coordinates of units i and i in the output array. 4. Weights adaptation. Only of the weights of the nodes within the neighborhood radius will be adapted using the equation: w~ new = w~ old + h (i ; i; t) ~x w~ old ; where w~ new is the vector of the new weights, w~ old are old weights, actual input vector. 2 (0; 1) is the learning rate, ~x is the After the algorithm make changes in the weights, it presents the next input vector from the remaining input vectors to input and continues with the step 2 and so on until there is no input vector left. 3.2
In the rst layer, it has SOMx; x 2 fwords, w2-grams, w3-grams, s3-gramsg neural networks they are trained to di erent sequences built according to the text T . The shape of the model is very similar to model developed in [Almarimi2016], but here the di erent analysed sequences are used for a training and an evaluation. The training of each SOMx is done on sequences Sw; Sw2; Sw3; Ss3. (9) (10) After the SOMx was trained it is possible to evaluate how good was the training prepared by an evaluation of errors for all input vectors (all windows in the text). We will use a quantization error Erx de ned by (11) as a measure of proximity input vector x+ to the learned winner vector wi of i -th neuron (winner for input vector x+) in the SOMx.
Erx(x+; wi ) = jjx+
wi jj; fErx(x+(t); wi (t)gtR=1; where R is the number of training vectors, t is the order of the member in input sequence. For the anomaly detections we will use thresholds developed by [Barreto2009]. Let be a signi cance level ( = 0:01 or = 0:05). We suppose the percentage of normal values of the quantization error will be 100 (1 ). Let N be the real number such that a percentage 100 (1 ) of the error values is less than or equal to N . Then The important interval is h ; +i, the values out of it could be detected as anomalies.
The quantization vectors Erx and intervals h Sx ; Sx i are computed in the panels ErrorSx ; x fwords, w2-grams, w3-grams, s3-gramsg and they are used in two the following evaluations:
then the text needs some next analysis.
CEr = 1
Erw + 2
Erw2 + 3
Erw3 + 4
Ers3; where i; i = 1; 2; 3; 4; Pi4=1 i = 1 are parameters for a contribution of Eri to the cumulative error. The values of the parameters i should be chosen after the analysis of all errors. If the cumulative error has higher value than the threshold hup given by formula (14) hup = 1 + Sw + 2
S+w2 + 3
S+w3 + 4 + Ss3 (11) (12)
2 (13) (14)
The text T will be divided into r; r > 1 disjunctive parts, T = T1T2 : : : Tr. For each text part Tk; 1 k r the following evaluation will be done: T k = T1 : : : Tk 1Tk+1 : : : Tr will be used as a training text and Tk will be a testing text. After the training using T k in our system, the testing text Tk will be evaluated using the quantization vector (12) and intervals h ; +i, the interval is built on training data. The percentage of values x; x 2 h ; +i express how the text Tk is similar to the training text. Evaluation of four sequences constructed from the texts gives more complex view to the text. 4 4.1
In the text analysis we use English recommended texts from benchmark [CorE2011] and Arabic texts from [CorA2011]. In Table I we describe some information about 3 Arabic and 3 English texts, the texts A14 and E777 were constructed as a combination of two di erent texts (important for an illustration of our analysis).
éJÊ« , é<Ë@ [allah,alyah] 747 ÈAK nal 3027 øQK , B@ [ala,tra] 115 ÕÎð , éJÊ« [alyah,wasallm] 10 (15) The used values were 1 = 0:5614; 2 = 0:0440; 3 = 0:0087; 4 = 0:3859: The in uence of the word probability in windows and 3-grams of symbols probability in the same window is higher than the others. In the text E777 there exist some windows with the higher cumulative error than the threshold hup de ned by (14), the text needs some further analysis. It should have some anomalous features.
Text part evaluation. The text E777 was divided into 4 parts with the same lengths. Evaluation of their text parts is given in the Table 2. According to the similarity percentage in the columns "2-gram of words" and "3-grams of words" each part of text is not similar to the rest of the text E777 (the values are lower than 50%).
Cumulative error. Figure 3, the rst part shows the analysis of the cumulative error of the text A14. The experiment was done with parameters 1 = 0:3339; 2 = 0:0314: 3 = 0:0157; 4 = 0:6189, computed according to the formula (15). For all types of errors, there exist error values above the thresholds and for the threshold of the cumulative error too.The text should have some anomalous features.
Text part evaluation. The text A14 was divided into 4 parts with the same lengths. Evaluation of its text parts is given in the Table 3. The the similarity percentage in the columns "2-gram of words" and "3-grams of words" except the part T4 is lower than 50%, it means the text A14 is not consistent text. The result con rms that the text is combined from two di erent texts. The main goal of the system is to nd some anomalies in the given long text. It means, our developed system have to be trained on some parts of the text and tested on the other parts. The system was evaluated for 40 Arabic texts [CorA2011] and 40 English texts [CorE2011]. In Table I we show some information about some of used texts. Arabic Texts
English Texts 5
The strategy in the evaluation - to split each text into four parts, three of them were used for training and the fourth for testing. We had four possibilities for each text. The training and testing of each text was done for di erent input sequences to the system: words, 2-gram of words, 3-gram of words, 3-gram of symbols. The results of each text were analyzed according to criteria [Almarimi2016]:
Critical text { if the the values of percentage was less than 75:000%. The critical text means that the tested part of the text gives under critical values.
Each text was evaluated 16 times, four times for each of the following methods: words, 2-gram of words, 3-gram of words, 3-gram of symbols. Results of the methods were combined into the last classi cation of each text (good or critical). In the Table 4, we show the information about 40 Arabic and 40 English texts. The research is supported by the Slovak Scienti c Grant Agency VEGA, Grant No. 1/0142/15. We thank to Bc. Peter Sedmak from Pavol Jozef Safarik University in Kosice for his help in a programming in
[Durgin2005] N.A. Durgin and P. Zhang. Pro le-based adaptive anomaly detection for network security.
SAND2005 (7293), 1-44, 2005. [CorA2011] CorpusArabic. King saud university corpus of classical Arabic. http://ksucorpus.ksu.edu.sa, 2011.