The work deals with the offered method of fuzzy analysis of texts and their rubrics actualization. The method is oriented to analyze electronic nonstructural texts of not big size in the following conditions: first, nonstationary composition and the importance of the keywords of the rubric field, second, in the absence or weak stucturization of these texts, third, if there are grammar or syntaxes inaccuracy and errors. The offered method is based on the original approach to the identification of the degree of the texts words fuzzy correspondence according to the well-founded set of syntactical characteristics with subsequent finding the degrees of text documents fuzzy correspondence to all rubrics. The method also allows to carry out monitoring of changes and actualization of rubrics according to the results of checking the formulated conditions of rubric field changes for the following typical situations: formation of the additional rubrics on the “boundary” of the existing rubrics; rubrics division, creating new rubrics, rubrics exclusion, rubrics combining. The offered method allows to raise the accuracy of analysis and the quality of texts classification at the expense of using the fuzzy approach for the accounting of analysis conditions uncertainty and nonstationarity of thesaurus of these texts as well as at the expense of operational actualization of rubrics depending on the composition and importance of the rubrics key words.
relatively small size of such texts; such texts weak structuredness or no structuredness at all (no marking and fields for computer processing); presence of grammar and syntaxes inaccuracy and errors; analysis conditions uncertainty and nonstationarity of composition and importance of rubric field key words; high degree of rubrics interdependency.
These features put considerable limitations on the usage of traditional models and methods of
morphological, syntaxes and semantic analysis of the texts. However, famous models and methods of
knowledge acquisition from the text information take the requirements of operational rubric changes
into account not sufficiently, this leads to the growth of the number of errors because of the wrong
classification of the processing texts [
Therefore, the actual problem is to make a method of fuzzy analysis of electronic nonstructural texts and actualization of rubrics taking into account the detection of the following situations requiring operational changes of the rubric field: the additional rubrics formation on the “boundary” of the already exiting rubrics, rubrics division, creating new rubrics, rubrics exclusion, combining rubrics.
The offered method of texts fuzzy analysis and rubrics actualization includes the following main stages:
Stage 1. Rubric tasks and texts presentation on the basis of the detected syntaxes characteristics.
Stage 2. Texts analysis on the basis of the degree defining of their fuzzy correspondence to the rubrics.
Stage 3. Checking the rubric field changes conditions and rubrics of field actualization according to the results of this checking.
Let us consider the problems solving on the stages of the offered method in more details.
On the basis of the preliminary texts analysis the initial rubric multitude is given:
R {Rj | j 1..J},
where for all j 1..J Rj wjm , rjm | m 1..M j , w jm – m- word in the rubric R j , rjm [
For such texts presentation the «unification» of the set of the following syntaxes characteristics,
detected, for example, by analyzer LinkGrammar is done ([
S {sn | n 1..N}, then N 5 ,
where s1 – the root word or predicate; s2 – the subject; s3 – the adverbial modifier; s4 – the subject
under action; s5 –the predicate [
The texts multitude is presented in the form of:
SD {SDk | k 1..K},
where SDk { SDn(k) | n1..N}, SDn(k) – word multitude of k- text, corresponding the syntaxes
parameter sn.
3. Texts analysis on the basis of the identification of the degree of their fuzzy correspondence to
the rubrics
First, the degrees of fuzzy correspondence jn (SDn(k) ) [
n p1 where u(jpk) – the degree of correspondence of p-word from SDn(k) , primarily given for this word from rubric R j .
To determine the degree of the text fuzzy correspondence to the rubrics let us introduce the parameter (SDk , Rj ) characterizing the degree of text SDk fuzzy correspondence to rubric Rj: (SDk , Rj ) 1 1 N
N
n1 Rj (Rjn ) Rj n (SDn(k) )2 ,
[
N 1 R n (SDn(k) )2 . n1 j
Text SDk refers, in the greatest degree, to that rubric Rl for which the degree of correspondence is maximum:
Rl : mj1a..xJ 1%(SDk , Rj ).
To check the conditions of changing of rubric field let us introduce additionally the following parameters: j J , 0±,5 (SDk , Rj ) 1 1 N
N 0,5 R n (SDn(k) )2 , n1 j j J
j J , 0%(SDk , Rj ) 1 1%(SDk , Rj ), where parameter 0±,5 (SDk , Rj ) characterizes the degree of uncertainty of text SDk referring to rubric R j and parameter 0%(SDk , Rj ) characterizes the degree of text SDk discrepancy to rubric R j .
This stage realization considers the calculation of parameters 1%(SDk , Rj ) , 0±,5 (SDk , Rj ) , 0%(SDk , Rj ) for all texts and their analysis, according to the results of the analysis on the basis of the conditions given bellow, the revision of composition and rubric field structure is performed.
Let us consider the formulated conditions of detection and revision rules of composition and rubric field structure for the following basic situations: additional rubric formation, rubric division, new rubrics creation, rubric exclusion, rubric combining.
The basis for the additional rubric formation on the «boundary» of the already existed rubrics Ri and R j is the revealing of the considerable amount of texts (equal to the rubrics or more than the number of rubrics), for every of which the following condition is valid: 1%(SDk , Ri ) 1%(SDk , Rj ) 0±,5 (SDk , Ri ) 0±,5 (SDk , Rj )
0%(SDk , Ri ) 0%(SDk , Rj )
Rl R, l i j : 1%(SDk , Rl ) 0±,5 (SDk , Rl ) 0%(SDk , Rl ) ,
where α and β – the upper and the lower boundary values (usually, α = 0.4 and β = 0.7 [
When revealing the number of texts equal to the rubrics or more than the number of rubrics, for which the above mentioned condition is performed, the conclusion about the reasonability of additional «boundary» rubric is made.
The basis for the rubric division R j is the revealing of the considerable number of texts for each of which the following condition is fulfilled:
1%(SDk , Rj ) 0±,5 (SDk , Rj ) 0%(SDk , Rj ) Rl R, l j : 1%(SDk , Rl ) 0±,5 (SDk , Rl ) 0%(SDk , Rl ) , where j – the number of the divided rubric.
The basis for the new rubric creation is the revealing of the considerable number of texts for each of which the following condition is fulfilled:
Rl R : 1%(SDk , Rl ) 0±,5 (SDk , Rl ) 0%(SDk , Rl ) .
The basis for the rubric exclusion is the revealing of the considerable number of texts for each of which the following condition is fulfilled:
1%(SDk , Rj ) 0±,5 (SDk , Rj ) 0%(SDk , Rj ) .
The basis for the rubric Ri and R j combining is the revealing of the considerable number of texts for which the following condition is fulfilled:
1%(SDk , Ri ) 1%(SDk , Rj ) 0±,5 (SDk , Ri ) 0±,5 (SDk , Rj ) 0%(SDk , Ri ) 0%(SDk , Rj ) Rl R, l i j : 1%(SDk , Rl ) 0±,5 (SDk , Rl ) 0%(SDk , Rl ) , where Ri and R j – combining rubrics.
The offered method was used in Administration of Smolensk region when automated analysis of
electronic nonstructural texts documents was performed, and it allowed to provide the operational
actualization of rubrics depending on the structure and parameters of the text documents in the
conditions of nonstationary composition of thesaurus and changes of the rubrics keywords importance.
Automated rubrication of 5062 massages received in 2016–2017 was performed through the internet
portal and by electronic mail. The analysis showed the presence of 17 different interconnected
rubrics, among them there are rubrics such as general issues of society and politics, separation of
powers and duties in Administration, social sphere, education, family, culture, housing and
communal service etc. The results of rubrication showed that rubrics dynamic accounting, when using
the probabalistic classification algorithm of text information as a basic tool of analysis [
The offered method was used in Administration of Smolensk region when automated analysis of electronic nonstructural texts documents was performed, and it allowed to provide the operational actualization of rubrics depending on the structure and parameters of the text documents in the conditions of nonstationary composition of thesaurus and changes of the rubrics keywords importance. Eventually, the number of erroneously rubricated texts was managed to be reduced to 13.3 % on average.
The work is conducted under support of the Russian Foundation for Basic Research (project 18-0100558).