This paper studies word length in the Tatar language examining data of fiction texts (the sample includes examples of both prose and poetry). Word length is a stochastic phenomenon depending on a great number of factors, including language type, text organization, its addressee, etc.; however, there are internal linguistic laws governing parameters of word length and frequencies of words, and the issue comprises universal and language specific features. We found that ration of words of different length are dissimilar in individual texts, and the most common words are those composed of 5 phonemes and 2 syllables. We evaluated word length in Tatar texts and attempted to fit a model based on Poisson distribution (in particular, a model based on one-displaced Poissonuniform distribution was used), so description of empirical data was complemented with fitting theoretical values for word frequencies. Besides, Shannon's entropy of word lengths was evaluated, and a weak correlation between the average word length and entropy was found.
Rigorously designed computational models can help making theoretical proposals by
providing clues about their limitations or internal inconsistency, so they can
contribute to improvement of theoretical approaches setting quite strict requirements for
them. Length of linguistic items is one of significant formal features of languages
which finds application in text processing, spell checking algorithms, language
teaching, etc. Word length is studied by specialists in linguistics and text analysis as well as
mathematicians and statisticians working on related issues. Basic approaches to word
length are presented in papers by P. Grzybek [
Word length can be measured by the number of phonemes, morphemes or syllables
in it, depending on research goals. A number of words of different length depends on
the language type: for example, in synthetic languages, there is a higher
morphemeto-word ratio than in analytic languages [
Abundant linguistic data on word length provide great opportunities for selecting
parameters and model fitting (Zipf ́s law, Menzerath-Altmann law, approximating by
means of different distributions, etc.). Behind a superficial simplicity of the concept
of word length, a number of surprises may hide, “so here nothing helps but incessant
testing, modeling, different viewing of data, modification of hypotheses, collecting of
data from new languages, etc. Every “new” language can falsify a beloved theory or
force us to modify it” [
There is a lack of special works devoted to word length in Turkic languages,
although some data on Turkish is presented in overviews and papers covering
multilingual data (for example, in [
The aim of this paper is to empirically evaluate word length in Tatar and to model it using Poisson distribution. Tatar fiction texts are used as empirical source; the written texts were brought into a phonologically relevant form, to allow counting the number of phonemes and syllables per word. 2
We examined a distribution of words of different length in 10 Tatar fiction texts
(prose and poetry texts were used, 5 of both kind; brief information on selected texts
is presented in Appendix). The written texts were brought into the standard form: 1
letter – 1 sound, and special rules were set to convert Tatar texts into a phonologically
relevant form. When tokenizing, co-compounds (like ata-ana ('mother' + 'father')
'parents' [
Figure 1 represents the number of words of different length in the text by A. Eniki; the word lengths are measured in phonemes. This text contains 2,169 words with length from 1 to 14 phonemes. Words with length 5 are the most frequent. Words in the text by A. Eniki contain 4,962 syllables; the words are composed of one to six syllables, with the most frequent words being those with two syllables (see Figure 2). The distribution of words depending on the number of phonemes and the number of syllables is represented in Figure 3. The most frequent are words consisting of 4 and 5 phonemes divided into 2 syllables. 0.007 0.081 0.132 0.163 0.219 0.127 0.101 0.087 0.042 0.023 0.008 0.005 0.003 0.001
0
WL in
phonemes Eniki
We computed the ratio of words consisting of a different number of phonemes in the
selected texts, then calculated the entropy [
A great variety of distributions has been tested in word length studies, and Poisson
distribution is often in focus of researchers who can modify it in different ways.
Poisson distribution is a very simple, and in many cases sufficient means for modeling.
Researchers believe that Poisson distribution, either in its usual form, or displaced to
the right, or truncated above the zero point (positive Poisson distribution) should be
used at the very beginning of any investigation [
We follow this way and rely upon the approach by V. Kromer [
Px = (λ - 1)x – 1 / (x – 1)! * e-(λ – 1) , x = 1 , 2 , 3 , ... ,
(1)
where Px is probability of textual word occurrence with length x, and λ is the
distribution parameter [
We computed the probabilities of the textual word occurrence with the given length and obtained theoretical values for occurrences of words with the given length in each text. The results are given in Table 2 (fitted values with λ0 parameter).
Then parameter λ0 was replaced by lambdas belonging to the interval: [λ0 - 0.1 * λ0, λ0 + 0.1 * λ0]; length out was 150 for best approximation, and theoretical values of word occurrences for each case were computed. Then the discrepancy between experimental and theoretical data was evaluated by the Pearson Chi-square criterion χ 2 and the best fitted values were selected (λ*). The results for 5 texts are represented in Table 2.
WL in syllables 1 2 3 4 5 6 λ χ2 Total 390 1030 567 171 23 2 Eniki 611 778 495 210 67 17 2.273 401.6 2183 We empirically evaluated word length in Tatar texts and attempted to fit a model based on Poisson distribution. Word lengths were measured in terms of phonemes and in terms of syllables.
Texts are not homogeneous because of internal rules of self-organization, so portions of words of different length are dissimilar in individual texts, and the most common words are those composed of 5 phonemes and 2 syllables.
Shannon's entropy of word lengths was evaluated, and a weak correlation between the average word length and the entropy was found with correlation coefficient equaling 0.57. According to the examined sample of texts, the average entropy of word lengths in prose is greater, and less significant in poetry, which may be the case due to the requirements of the poetic meter.
For modeling lengths of Tatar words, the one-displaced Poisson-uniform distribution was used. Although the results are generally consistent with those for other languages described in literature, nevertheless there is a significant discrepancy between the observed and fitted values, so using other modifications of Poisson distribution as well as using other distributions is needed in further research.
The results of the study of word length in Tatar can help in development of applications for style and register detection, authorship analysis and language teaching as well as they can be used in theoretical studies of language structure and complexity. 5
The work is carried out according to the Russian Government Program of Competitive Growth of Kazan Federal University.
No 2 3 4 8 9 10
Şüräle / Forest Spirit
Su anasɪ / Aquatic Woman Käcä belän sarɪk äkiyäte/ The tale of the goat and the ram
Häyät Hayat, Chapter 1
Sertotmas ürdäk / The Talkative Duck
Oçraşu / Встреча
Dürt mizgel / Four moments Novel, prose fairy tale in
verse fairy tale in
verse fairy tale in
verse
Story, prose poem poem 925 419 579 548 444 917 1,014 355 163
2,169 Number of syllables 4,962 1,917 854 1,211 1,310 1,085 2,093 2,351 815 360