s of it. The purpose of the research is to study changes of percentage of key words in the texts of an original abstract and an abstract based on statistics calculated using Zipf's first law. It is the authors' opinion that the percentage of key words defined by the user separately for each given text should increase in the target abstract as compared with the original abstract.
Most texts we encounter in real life are natural. An example of an artificial text is a text created for web crawlers to improve site’s rank on the search engine results page. Search engines fight this ‘black’ optimization actively, to the point of excluding such sites from the search index.
Zipf's laws describe regularities of frequency distribution of words in a text written in any natural language. These laws are empirical: they have no strict mathematical proof and are based on statistical distribution of words in large corpora of texts in various languages. Nevertheless, their correctness is proven statistically. 2
The Zipf’s first law [
Zipf's first law (1) states that the probability of discovery of any word multiplied by its rank is a constant (C).
C P r ,
P
where С is a constant, r is word rank, P is probability of discovery of a given word in
a text. And C coefficient is different for different languages [
Probability P is defined by (2).
f
N , where P is probability, f is frequency, and N is the total number of words. (1) (2) 3
The first published stemmer was written by Julie Beth Lovins in 1968. A later
stemmer was written by Martin Porter and was published in 1980. This algorithm is
implemented in stemmer [
An approximate heuristic process of removing suffices and inflections from a word is
usually called stemmatization [
At the first stage, the system uses the stemming algorithm to flag words and calculates numbers of their appearances.
At the second stage, the system uses the generated statistics of numbers of words in the text to calculate ideal and recommended number of appearances of every word in y
b the text according to (3). According to Zipf’s first law, the chart of words distribution in a natural text must be approximate to the graph of negative correlation. a x
, (3) where x is word rank, y is word frequency.
The least square method was used in the research to find coefficients a and b. If the ideal value is higher than the current number of appearances of a word in a text, then the recommended value will be the ideal one rounded down to the nearest whole number. If the ideal value is less, then the recommended one will be rounded up to the nearest whole number.
Fig. 1 shows a chart of word distribution in a text. The red line shows initial
rankfrequency distribution of the analyzed document. The green line shows ideal values of
word appearances in the text. According to Zipf’s first law, those are non-integral
values. The blue line shows integral values used for generation of the second abstract.
Next, two abstracts are generated: the first is based on initial numbers of appearances
of words in the text, and the second is based on recommended values of appearances
of words. The abstract generation algorithm is described in detail in [
Next, key words characteristic of the analyzed text are chosen. The number of appearances of these words in generated abstracts is calculated. Next, the statistics is displayed. An example of the statistics is shown in Fig. 2. The red column shows the number of appearances of a specific word in the original abstract. The yellow column shows the number of appearances of the word in the target abstract generated using new values of appearances of the word in the text. Fairytales were chosen as objects of research of naturalness of texts because, despite a relatively small volume, key words appear frequently in them and are different for every separate fairytale. The key words were: Ivan, Tsar, Koschei, Vasilisa, princess, etc. These key words were supposed to vividly demonstrate changes in the content of an abstract after words in the texts were approximated to natural distribution.
Dependence of the number of key words in abstracts on the size of abstracts for small texts However, in long texts, such as “Tale of Cipollino” (Fig. 4), the content of key words increases correspondingly with the increase of the abstract length. This is mainly due to increase in appearances of the first three most frequent key words.
The paper has reviewed Zipf’s first law on distribution of words in natural texts as well as an algorithm of stemming used to flag words. The research results on dependence of key words percentage in abstracts generated on the basis of recommended number of word appearances in the text were presented. This research allows to conclude that the application of Zipf’s first law for short texts does not lead to increase of key words percentage in new abstracts. It is reasonable to apply this law to longer texts. Also, the work on mitigating the influence of non-informative words on content of the abstract, such as exclusion of frequent verbs and other parts of speech, should be continued. The stemming algorithm provides an opportunity to find parts of speech by specific inflections.