Word frequency statistics and lexical diversity measures can provide insights into discourse differences between texts. The ZipfExplorer, a tool and online app for the interactive visualization and comparison of word frequencies in two texts, shows side-by-side rank-frequency profiles and interactive tables of shared lexis, enabling keyword analysis and shedding light on discourse differences. Four lexical diversity measures (type-token ratio, Gini coefficient, powerlaw alpha parameter, and Shannon entropy) are calculated for the shared word types. Word frequency information is provided for a selection of mainly literary texts, and users can upload their own files. This paper provides an overview of the visualization of word frequency distributions, describes the functionality of the ZipfExplorer tool and demonstrates some of its features, and briefly discusses the lexical diversity measures calculated by the tool.
Word frequencies are a fundamental starting point for many analytical procedures in
corpus-based linguistic, literary, or cultural analysis and for natural language
processing tasks.1 The study of word frequency distributions and their statistical properties
continues to be an active topic of research in computational linguistics [
In language and linguistics or literary or cultural studies, the comparison of word
frequencies in two texts or between a selected text and a reference corpus is a primary
method for gaining insight into differences in discourse content. The ZipfExplorer2 is
an online tool for the interactive visualization of word frequencies in texts or corpora,
1 This paper, an expanded version of [
Copyright © 2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
named after Zipf’s Law [
Among the first to systematically study lexical type frequencies was the early 20th-century American Germanist George Zipf, who noted that when the words of a text are ordered in decreasing frequency, the relationship between a the frequency and the rank for a word of rank r can be expressed as ≈ −1, where C represents a constant. A Zipfian rank-frequency profile, when plotted in double logarithmic space, is typically close to a straight line, but the shape of a frequency distribution for only those lexical types that are shared with a comparison text or reference corpus depends not only on the frequency information of the particular texts under consideration, but also on the degree of textual overlap between the two texts. Visualizations of shared lexis, in addition to highlighting discourse similarities and differences between texts through the examination of particular word types, can also give insight into the interplay between frequencies, derived lexical diversity measures, and the shape of discrete frequency distributions in general.
Following Zipf [15, pp. 45–48; 16, p. 25], word frequency distributions are typically displayed in double logarithmic space, with frequency on the y-axis and frequency rank on the x-axis, as in the top right quadrant of Figure 1, which shows four visualizations of the word frequency distribution for Charles Dickens’ 1859 novel A Tale of Two Cities. Each circle on the plot corresponds to a distinct word type. The most frequent type, at the top left of the plot, is the word type “the”, occurring 8,058 times in the text, followed by “and”, “of” and other common words. 3 https://github.com/stcoats/zipf_explorer The plot in the top left represents the same information in linear space, whereas the lower left plot is the so-called degree distribution (sometimes also referred to as the frequency spectrum): Here, the word frequency counts themselves have been binned, so that the top left circle is the proportion of all word types that occur once in the novel (the hapax legomena). Hapax comprise 47% of the word types in the novel; words that occur twice (dis legomena) 16%, and so on. While the information contained in the Zipf rank-frequency plot and the degree distribution plot is equivalent, the latter plot is more difficult to interpret in terms of discourse, as points on the plot do not correspond to individual word types. In the bottom left of Figure 1, the complementary cumulative distribution function is depicted: the cumulative proportion of types with a frequency equal to or greater than a given frequency. Thus, 100% of types in the novel occur at least once, 53% at least twice, 37% at least three times, and so on. The complimentary cumulative distribution visualization is the reflection of the Zipf double-logarithmic profile across a line extending from the bottom left to the top right of the subplot. Because the upper two plots in Figure 1 are intuitively easier to understand, the ZipfExplorer visualizes rank-frequency utilizes them, rather than the degree distribution or the complementary cumulative distribution. 3
In Figure 2 the default linear-scale view for the shared vocabulary types in Mary
Shelley’s Frankenstein and H. G. Wells’ War of the Worlds is depicted: Each subplot shows
the rank-frequency profile for the text selected via the dropdown menus to the right of
the plots. Points on the plots show word relative frequency (per 10,000 words) on the
y-axis and type rank in an ordered list of the frequencies of all words in the shared lexis
on the x-axis. Values for the lexical diversity measures type-token ratio, Gini
coefficient, alpha exponent of the best-fit power-law distribution, and Shannon entropy are
shown above the plots. Hovering over a word type will show its rank, frequency,
relative frequency, and the log-likelihood measure [
Words can be highlighted with a hover tool and selected with a box-drawing tool (in the toolset above the right-hand subplot). Selected words are highlighted in the sortable tables below the plots; clicking on a word in one of the tables highlights it in the plots. The tables show frequency rank, the word form, frequency, relative frequency, difference in relative frequency compared to the other text, and the log-likelihood value: higher log-likelihood values indicate are calculated for types with larger frequency differences.
The default texts available for comparison are selectable via a drop-down menu to
the right of the plots. In addition, users can upload their own texts for comparison with
the upload buttons. A ‘Remove most frequent words’ drop-down list removes 0, 10, 20,
50, 100, or 200 of the most frequent words in English, based on the Project Gutenberg
English Corpus from Sketch Engine [
The source texts are a selection of mainly literary texts from Project Gutenberg, a
corpus of inaugural addresses of U.S. presidents from NLTK [
The columns in the tables below the subplots can be sorted. They show original word order in the left-hand text, word form, rank in the frequency table, relative frequency, difference to the other text in relative frequency, and log-likelihood score. Sorting can show items that are much more relatively frequent in a text. In Figure 2, the personal pronouns ‘my’, ‘you’, and ‘I’ are more frequent in Frankenstein, a text with a firstperson point of view, than in the third-person War of the Worlds.
The types ‘up, ‘out’, and ‘there’ (Fig. 4) are more relatively frequent in War of the Worlds; when considered along with other prepositions, place adverbials and location names, it becomes clear that spatial organization plays a greater role as a narrative element in War of the Worlds than in Frankenstein.
Hapax can also shed light on discourse differences. Highlighting the hapax types in Frankenstein (in the left-hand rank-frequency profile of Fig. 5) shows their ranks and relative frequencies in War of the Worlds: although many are also hapax in the other text, or are found mainly in the tail of the frequency distribution for Frankenstein, the types ‘smoke’ and ‘red’ are much higher in the profile in War of the Worlds – a frequency difference that reflects the discourse content of the latter novel. Using the drop-down menu to the right of the subplots, 10, 20, 50, 100, or 200 of the most-frequent types in the Gutenberg corpus can be removed from the visualizations and tables. Because these types, which are mostly function words such as determiners, pronouns, and relativizers or common verbs, structure texts in important ways but contribute relatively little to discourse content, removing them may serve to highlight discourse differences between two texts.
In terms of the distribution shape and the derived lexical diversity statistics, the removal of common words has the effect of increasing the relative frequency of the remaining words. In effect, removing stopwords tends to change the shape of the Zipf profile in double-logarithmic space to a more curvilinear form – when function words are no longer considered, word frequencies deviate substantially from a power-law distribution. As can be expected, removal of common words tends to increase the lexical diversity of the texts for the remaining shared types, which tend to be more uniformly distributed in terms of their relative frequencies. 4
has a maximum theoretical value of log2(n), for data consisting of n unique types.
The diversity statistics calculated by the tool provide evidence for the sensitivity of
lexical diversity measures to sample size [
The ZipfExplorer displays four lexical diversity measures: the type-token ratio, the Gini
coefficient, the exponent α for the best fit of a power-law distribution, and the Shannon
entropy [
has a range in The Gini coefficient, which can be calculated with = 2 ∑ ∑ − + 1 ranges from 0 (no diversity) to 1 (maximum diversity) for n word types with relative frequencies x. 1 + 1 ( =
−1
The exponent α results from the best-fit line to the degree distribution function
(lower left plot in Fig. 1) for the frequency information for the shared lexical types,
calculated using the powerlaw package in Python [
Shannon entropy [
log2 higher type-token ratio, whereas the longer text will exhibit a smaller α exponent and a higher value. Removing frequent words will often increase values for the type-token ratio and the alpha parameter, and decrease the values for the Gini coefficient and the Shannon entropy, although this depends on the texts in question, their original frequencies, and the degree of textual overlap. For texts with a relatively large proportion of shared types, such as two novels by the same author, and with the removal of frequent function words, the lexical diversity measures may give insight into topical diversity in terms of narrative development. For texts that share relatively few types, the relationship between the measure values and the properties of the underlying original texts is less straightforward. 5
The ZipfExplorer enables the interactive exploration of word frequencies in the shared lexis of two comparison texts or corpora, potentially shedding light on discourse similarities and differences and properties of frequency distributions. The lexical diversity measures type-token ratio, Gini coefficient, alpha parameter of the power-law function, and Shannon entropy, calculated by the tool, vary according to text length and textual overlap and are also affected by the removal of common function words.
In a pedagogical context, the ZipfExplorer provides a hands-on way to make frequency information concrete. Given the increasing importance of artificial intelligence models not only in linguistics and other sciences, but ultimately in many working-life and administrative domains and in the contexts of daily life, the tool can serve as a starting point for understanding how linguistic frequency distributions underlie the large data sets used train machine learning models.
The tool may also be useful for the comparison of various discrete distributions in computational studies of language or digital humanities, and for applied analysis in literary, historical, or cultural studies in which “distant reading” approaches are employed. Planned further development of the tool is to allow upload of different file formats, enable text extraction from URLs, and enable automatic annotation of part-ofspeech tags or named entities whose frequency distributions may be of interest. It is also hoped that other researchers will use the code for the tool (or parts thereof), available at GitHub, in order to create new and exciting ways to visualize linguistic data such as word frequency information.