-Constant developments in information and computer technologies make it possible to handle constantly increasing amount of data, thereby expanding the research possibilities. In this article, we discuss and compare distance and similarity measures used in stylometric analysis which could be applied to analyze Lithuanian texts. As corpus for the analysis, transcripts of parliamentary debates by two politicians of the Lithuanian Parliament were chosen. Furthermore, comparison of distance measures, stylometric analysis and visualization were performed. Objective of the experiment was to identify what measures would perform better when executing stylometric analysis of Lithuanian texts and explore where these differences in the performance occur. Summarizing the experiment results, the recommendations are as follow: number of Most Frequent Words used should be at least 1000, Eder's Simple Delta measure can be used in general stylometric analysis of transcriptions of parliamentary debates of Lithuanian Parliament, in a case when Most Frequent Words are limited to 2000, Binomial Index shows an increase in performance over Eder's Simple Delta and thus it is more suitable. Index Terms-stylometry; computational stylistics; parliamentary speech; R; statistical analysis; distance measure; similarity measure; data visualization
Stylometry refers to the study of linguistic style, usually to
written language. It uses variety of statistical methods to
analyze a text to determine the text’s author. Common technique
used is to calculate distances or similarities between texts
and process the output using different visualization methods.
There have already been significant number of experiments
performed by various researchers in order to figure out what
measures show better results in different cases of stylometric
analysis. F. Jannidis and S. Evert analyzed three collections
of novels in English, French and German languages and have
shown that Cosine Delta measure outperforms all other
measures on our three collections [
Copyright c 2017 held by the authors.
This paper presents an on-going experimental work on
identifying the most suitable measures used in stylometry
when analyzing Lithuanian texts. The objective of these
experiments is to compare the performance of distance and
similarity measures already used in stylometry and other fields
of research [
These experiments cover a domain of transcriptions of parliamentary debates of Lithuanian Parliament which is only a small fraction of Lithuanian language, however they represent richness and variety of language quite well, and hence, we expect, that the results could be useful in analysis of other Lithuanian texts.
Data for the analysis will be taken from corpora,
collected in ASTRA project. The data consists of transcriptions
of parliamentary debates of Lithuanian Parliament. For this
investigation we use transcripts of two politicians from the
term 2008-2012. The criteria used for selecting the right data
for comparison of measures were chosen in such a way that
measures would provide the biggest difference between texts
of different authors (or speakers, in our case). Considering this,
authors for experiments were chosen of different gender and
political standing (position and opposition) to strengthen the
dissimilarity. This is a different type of approach compared to
a similar research which was done by J. Kapocˇiu¯ te˙-Dzikiene˙
where her goal was to identify best methods and features
(MFW were not used) for authorship attribution based on
machine learning [
Automatic extraction of style applied to individual authors and groups of authors, http://dangus.vdu.lt/~jkd/?page_id=2
The selected authors were Virginija Baltraitiene˙ (Labor
Party Political Group (Darbo partijos frakcija, DPF), who
belonged to opposition at the given time) and Donatas
Jankauskas (Homeland Union - Lithuanian Christian
Democrat Political Group (Te˙vyne˙s sa˛jungos-Lietuvos krikšcˇioniu˛
demokratu˛ frakcija, TS-LKDF)). The mean word count for
both authors are 190.39 and 203.96 accordingly, which means
that their texts have to be concatenated so that one text
object would contain around 5000 words. This is because
Lithuanian language have many different words with many
different word forms, and in this experiment we are not
converting words to their base form. In addition, we will
consider these 5000 words to be sufficient for a text object,
because documents themselves contain as few as 100 words
and it was shown by M. Eder that after more than 5000 words
the quality of author attribution barely increases [
To evaluate the measures, Z-index was used, which in other words is a difference between means of standardized values. z = j s vj = 1
Xms sl ms l=1 1
Xmv vk mv k=1 ; (1) where s and v are out-group and in-group means of standardized values, ms and mv are out-group and in-group number of comparisons, sl and vk are out-group and in-group distance/similarity measurement for the corresponding comparison, and are the overall mean and standard deviation of all measurements for comparisons, excluding comparisons when compared documents are the same.
In-group data set contains comparisons of documents of speeches by the same politician, while out-group data set contains comparisons of documents of speeches by different politicians. In this experiment we assumed that the bigger difference between means of standardized values, the better performance of certain measure was.
After inspecting results of the experiment, Cluster Analysis and Multidimensional Scaling was used to visualize relation among speeches of selected politicians, using distance measure with better performance and with regards to the quantity of MFW. In addition to Z-index, a dependency on the quantity of MFW (from 100 MWF to 5000 MFW) to the results was analyzed as well.
For stylometric analysis (calculating word frequencies,
values of the distance measure, and plotting the relations
among documents) R, free software environment for statistical
computing and graphics, was used [
All distance/similarity measure computations and
stylometric analysis were performed using “stylo” [
The objective of this investigation was to identify which
measures perform better with different number of MFW in
stylometric analysis of Lithuanian texts. All the evaluated
measures are presented in Table II. Hierarchical Clustering [
Initial experiments where performed with 100, 1000 and 5000 MFW. In each case all distance measures were sorted according to the calculated Z-index. Naturally, a distance measure with the highest Z-index should have been considered the best, but in our experiments there were no clear consistency in the measures regarding their performances, e.g., the measure Number of Words Number of Different Words Mean Number of Words in One Text that showed better performance, i.e., higher Z-index with 100 MFW, would perform worse with 1000 and 5000 MFW and vice versa. To investigate further, a second experiment was executed.
To find out exactly how every distance measure behaves when increasing the number of MFW, a graph was plotted where all the measures were presented in Z-indexes, taking into consideration quantities of MFW taken for experimentation, see Fig. 1. All Z-indexes were calculated for 100, 200,. . . , 5000 MFW. In general, every distance measure was displayed as a function. A plot was generated with total of 50 points for every measure, which was enough to detect their behavior when quantity of MFW increased.
In the Fig. 1 we can see that most distance measures possess high values in Z-index throughout the plot, but four of them performed very poorly. These four distance (or similarity) measures are Alternative Gower Similarity, Horn’s modification of Morisita’s Overlap Index, Cosine Distance and Euclidean Distance (see Table II for details). In order to investigate further, we removed these distance measures from the experiment and concentrated on the remaining ones.
After removing distance measures with bad performance, we generated a new graph where remaining measures could be compared more precisely,see Fig. 2. We can see that up to around 1000 MFW, Z-index is always increasing in value and after this point it either decreases or shows similar results when MFW number is increased. Analyzing the plot further, we can observe three main groups of distance (or similarity) measures: 1) Measures which show very good performance until around 2000 MFW were reached. After that, the performance downgraded very fast. Distance measure that performed best was Binomial Index. Other distance measures which behaved similarly were: a) Argamon’s Quadratic Delta, b) Burrows’ Delta, c) Argamon’s Linear Delta, d) Gower Similarity. 2) Measures which slowly reach their performance peak only at around 2000 MWF and then their performance very slowly declined. In general, these distance measures showed a very good performance and stability. In this
Value
1 1 0 0
group, Eder’s Simple Delta had the best results. Other measures which behaved similarly were: a) Eder’s Delta, b) Mountford Index, c) Canberra Distance. 3) Measures which had a continuously improving performance together with the increasing MFW quantity, though this improvement was extremely gradual. In comparison to other groups, this group performed worse than the first and the second groups until reached 3000 MFW and after that it still performed worse than the second group, but it showed better performance over the first group. Distance measure that performed best in this group was Jaccard Index. Other measures which behaved similarly were: a) Manhattan Distance, b) Bray-Curtis Similarity, c) Kulczynski Distance.
We noticed that Eder’s Simple Delta measure performed really good: between the first group, only around 2-3% worse than the Binomial Index and between the second group, where Eder’s Simple Delta showed the highest Z-index that was 13% better than Eder’s Delta. So not only it could be used for bigger quantities of MFW, but for smaller ones as well. For this reason, in this experiment we considered Eder’s Simple Delta as the best performing distance (or similarity) measure when analyzing Lithuanian texts in general.
We used Eder’s Simple Delta measure for exemplary
stylometric analysis in order to visualize the difference in lexical
style (usage of words) between the selected members of the
Lithuanian Parliament. For this analysis 2000 MFW was used,
as it showed the best performance for the chosen distance
measure. The first method used in order to map relations
among the transcribed speeches made by before mentioned
politicians was Hierarchical Cluster Analysis (see in Fig. 4b).
In this research an agglomerative hierarchical clustering with
Ward linkage, where the criterion for choosing the pair of
clusters to merge at each step is based on the optimal value of
an objective function [
In Fig. 4a visualization and relation mapping among documents were performed using Multidimensional Scaling. The colors represent different authors of the parliamentary speeches, corresponding to the previous Figure (Fig. 4b). The division among parliamentary speeches made by different politicians was executed well, but in this case distribution of points was extended from the perspective of the vertical axis. This behavior might lead to different results if a lot more authors would be analyzed at once. In general, the results were good, which also contributed to our conclusion that the distance measure we used had an overall good performance.
In addition to the two methods we have already used for relation/position mapping of the documents as well as visualization, we tried to display the results with a Heat Map. Fig. 3 shows one of the simple ways to visualize stylistic differences among the documents without any further computations. In this example the color palette used for the plot was gradient from white to dark red, where white means a complete match. Since documents were sorted by author in the horizontal and vertical axes, the rectangle shapes formed out of light and darker colors. Light color showed that the documents were written (or speech spoken, in our case) probably by the same author and dark color – by a different author. Since the brightness of the color matched every pair of documents well enough, Heat Map could be considered to show positive results in regards to the Eder’s Simple Delta measure.
To summarize, every method for visualization and relation/position mapping for stylometric analysis that we used did not object our statement that Eder’s Simple Delta performed well in analysis of Lithuanian texts. But by no means this is the only measure that produces very good results. Other measures surely could also provide similar performance (as was seen in Fig. 2). Different corpus, parameters for text analysis, selection of Most Frequent Words - these options could still affect the performance of every distance (or similarity) measure. However, to reach solid conclusion, further research is needed.
In this experiment we focused on Lithuanian texts, when corpus was composed of parliamentary speeches from Lithuanian Parliament. Hence, the following recommendations could be applied to the experimentation with the latter corpus. 1) Distance measure can be selected according to the quantity of the Most Frequent Words used in the analysis. 2) At least 1000 of Most Frequent Words should be used.
After this point Z-index value either decreases or shows similar results. 3) If quantity of MFW does not exceed 5000 by a wide margin, Eder’s Simple Delta measure performs well. 4) If Most Frequent Words are limited to 2000, Binomial Index shows an increase in performance over Eder’s Simple Delta and thus it is more suitable in this case.
Finally, these recommendations can be applied for stylometric analysis of generic Lithuanian texts, but precautions must be taken and therefore we plan to experiment with texts belonging to different domains in the future.