=Paper= {{Paper |id=Vol-2723/long35 |storemode=property |title=Weak Genres: Modeling Association Between Poetic Meter and Meaning in Russian Poetry |pdfUrl=https://ceur-ws.org/Vol-2723/long35.pdf |volume=Vol-2723 |authors=Artjoms Šeļa,Boris Orekhov,Roman Leibov |dblpUrl=https://dblp.org/rec/conf/chr/SelaOL20 }} ==Weak Genres: Modeling Association Between Poetic Meter and Meaning in Russian Poetry== https://ceur-ws.org/Vol-2723/long35.pdf

Weak Genres: Modeling Association Between Poetic
Meter and Meaning in Russian Poetry
Artjoms Šeļaa,b , Boris Orekhovc and Roman Leibovb
a
Institute of Polish Language (Polish Academy of Sciences), al. Mickiewicza 31, 31-120 Kraków
b
University of Tartu, Ülikooli 18, 50090 Tartu, Estonia
c
Higher School of Economics, ul. Miasnitskaia 20, 101000 Moscow

Abstract
This paper aims to formalize an established theory in versification studies known as ”semantic halo
of a meter” which states that different metrical forms in modern poetry accumulate and retain
distinct semantic associations. We use LDA topic modeling on a large-scale corpus of Russian
poetry (1750-1950) to represent each poem in one topic space and then proceed to represent each
meter as a distribution of aggregated topic probabilities. Using unsupervised classification and
extensive sampling we show that robust form-meaning associations are present both within and
between metrical forms: two samples of the same meter tend to appear most similar, while two
metrical forms of the same family tend to group together. This effect is present if corpus is controlled
for chronology and is not an artifact of population size. We argue that similar approach could be used
to align and compare semantic halos across languages and traditions to give meaningful general-level
answers to questions of literary history.

Keywords
poetry, semantics, meters, topic modeling, clustering

1. Introduction
The existence of a connection between a poetic form and its meaning may seem trivial. Histor-
ically, metrical differentiation was driving distinction in genres or types of poetic speech, all
the way to theorized Indo-European ”long” meter of epic and ”short” meter of lyric verse [13,
28]. We generally do not expect an introspective meditation from a limerick, while we may ex-
pect it from a sonnet. European imitations of Dactylic Hexameter or elegiac distich in modern
versification systems are thematically bound to their Classic Age sources. Does an association
between poetic form and its semantics also survive in the ”general-use” meters, in a modern
poetic tradition where the normative connections between a genre and a form quickly decayed?
The agreed answer is yes.
The ability of poetic meters to accumulate and retain distinct semantic features over time
is also known as ”the semantic halo of meter” in Russian school of quantitative metrical stud-
ies [50, 44, 53]. Initial observations were based on the usage of meters by single poets [54] or
on anecdotal evidence (notably a few scattered in time poems composed in Trochaic Pentame-
ter). Early scholars saw meter-meaning association as organic, i.e. some intrinsic features of

CHR 2020: Workshop on Computational Humanities Research, November 18–20, 2020, Amsterdam, The
Netherlands
£ artjoms.sela@ijp.pan.pl (A. Šeļa); nevmenandr@gmail.com (B. Orekhov); roman.leibov@gmail.com (R.
Leibov)
Ǳ 0000-0002-2272-2077 (A. Šeļa); 0000-0002-9099-0436 (B. Orekhov)
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)

12
rhythm shaped meter’s semantics [24, 49]. Based on the close reading of thousands of 19th
century poems Mikhail Gasparov demonstrated that the connection should be historical and
is determined by a meter’s origins in a local tradition and usage over time that accumulates a
distributed, yet distinct semantic profile [14].
Despite the attractiveness of the findings, lack of formalization makes the semantic halo a
target easy to criticize and hard to defend. Even if some specific ”halos” are not a product of
a simple sampling error, any generalizations about the mechanism itself and the structure of
relationships between metrical forms remain elusive. A few previous empirical attempts to ap-
proach meter-meaning association in Russian [35] and Bashkir [33] poetry were able to broadly
confirm lexical differences between metrical forms while relying on wordlists comparison, which
provides us an entry point to the problem.
This study tries to address the presence of the semantic halo in Russian poetry using a
set of abstracted semantic features (topics) that describe each individual poem in a uniform
way. Having all texts aligned within one model allows for performing flexible tests and using
classification algorithms to explicate and verify scholarly assumptions. We rely on hierarchical
clustering to assess the level of within-meter semantic similarities (are meters similar to them-
selves) and between-meter relationships (how metrical forms relate to each other). Following
the analysis we discuss how formalizing the semantic halo of meter could enhance our under-
standing of it as a mechanism of cultural transmission and how a similar approach could be
used to study the halo effect across various languages and traditions.

2. Corpus
Data used in this study comes from Poetry Sub-collection of the Russian National Corpus [40]
that includes texts spanning from the 18th to the late 20th century. It roughly covers the
whole history of modern Russian versification that started with the introduction of German
accentual-syllabic verse in 1730s. The corpus has a clear canonicity bias in its design: 18th and
19th century texts were included in the collection based on their availability in the 20th century
scholarly editions [25]. This leaves a lot of earlier poetic production outside of academic canon
unaccounted for and partially drives the inequality in chronological distribution of the poems:
more than 75% of texts come from the 20th century. This is also a very non-uniform pool of
texts because starting with 1917 Russian poetry split in three generally isolated traditions –
Soviet, emigrée and unoﬀicial underground. Having no automated way to separate them, we
limit the corpus by the year of 1950, which roughly excludes most of the underground works
and stops the timer before the noticeable drift towards the non-classical versification begins.
After all subsetting operations and preprocessing steps (see Section 3) we are left with 47,804
texts (2,275,233 words).
This study is mainly focused on the accentual-syllabic (AS) metrical (and usually rhymed)
poetry, which survived in the Russian versification much longer than in the Western traditions
that turned to verse libre [17]. AS systems of versification are based on strict limitations
for both the number of stresses and the number of syllables in a line, as compared to purely
accentual (only stress count matters) or purely syllabic (only syllable count matters). The AS
meters are built of recurring smaller units of rhythm – feet that organize stressed and unstressed
syllables in patterns, usually of two or three (binary or ternary feet). Since metrical scheme
is an abstraction of a poetic rhythm and is constantly altered (expected stressed positions left
unstressed and vice versa), we usually speak of strong vs. weak positions in a meter, instead

13
of ”stressed” or ”unstressed”. Table B.3 provides a summary of all the classical AS forms that
were used in this study. The exception are so-called ”dolniks”, which step away from AS by
loosening rules for syllable count, but their abundance in the 20th century cannot be ignored.
We utilize the existing corpus metadata that includes annotations for poetic form to, if
possible, label each poem with a single unambiguous metrical formula. Corpus annotation
was done institutionally under the supervision of experts in linguistics and prosody, however,
annotators’ agreement or error rate was not reported [16]. We expect the accuracy to be
very high, especially in classic AS forms that are easy distinguishable even with the minimal
training. We asked three literary scholars to verify 100 original corpus annotations for metrical
forms: on average, they marked 97.7% labels as ”true”. Mean inter-annotator agreement was
96.6% (low agreement on what to consider ”false” labels).
We were conservative in labeling texts with corpus metadata, preferring homogeneous met-
rical notations and excluding most of the complex cases of polymetry, logaeds or other het-
erogeneous forms. We also used simplified information on stanza, relying on just a general
clausula pattern. Throughout this paper we use a metrical notation derived from the Russian
school of metrics, e.g. Iamb-4-fm stands for Iambic Tetrameter with regularly alternating lines
of feminine and masculine clausula (or acatalectic and catalectic lines).
For the purposes of this study we infer three levels of metrical expression from a single
metrical formula:

1. A general family of metrical pattern (e.g., Trochee, a meter based on binary feet with
the strong position on the first syllable);
2. A meter of the poem based on the number of feet (e.g., Trochee-5, Trochaic Pentameter
composed of five trochaic feet)
3. A catalectic variant of the meter that describes the pattern of non-stressed syllables after
the last stressed one (e.g., Trochee-5-fm; f – stands for feminine (Xu), m – masculine (X),
d – dactylic (Xuu) ending of a line).

Fig. 1 captures this three-level distribution of poems relative to six most frequent metrical
families in the corpus (only two most frequent variants per meter are displayed) and provides
absolute counts of poems in a family. The dominance of Iambic meters and specifically Iamb-4
as the ”normative” meter of Russian accentual-syllabic versification is self-evident. To deal
with this extreme inequality in metrical forms further in the paper we heavily rely on random
sampling and iterative experiments.

3. Modeling semantics
We aim to model meter-meaning association through the semantic features of individual poems.
To do so, we train one Latent Dirichlet Allocation (LDA) [5] topic model on the whole corpus,
without any aggregation of poems, writing metrical labels and other metadata in document
names.
Topic models is a collective name for a large family of information extraction algorithms that
look for groups of co-occurring elements in a collection of documents. These groups are labelled
topics (the original goal was text mining), but models are transferable to, e.g. molecules [55],
music [27] and genes [4], or any task that requires to abstract groups of similar behaviour from
large number of features (words, chords, genes, chemical elements, etc.). Topic modeling is
now widely used for text mining and classification in humanities and social sciences [20, 56, 8,

14
Figure 1: Relative distribution of metrical forms in a family. For each meter that had at least 200 poems
two of its most frequent variants are represented. Absolute poem counts include everything.

42]; it was also shown multiple times that LDA is applicable to the corpora of smaller poetic
texts [1, 31, 19, 36].
Topic models were promisingly used for modeling general questions of cultural history: rate
of change in popular music [27], modes of scientific exploration of information [30], or innovation
and retention in historical political discourse [3]. In these cases topical representation of entities
served as a mere approximation of ”contents”. We aim for a similar abstracted representation
of poetic language, very vaguely mimicking scholars who operated high-order semantic labels
to describe meanings specific to meters like Night, Road or Death (themes that, according to
Gasparov, collectively express some of the main semantic directions of Trochaic Pentameter in
Russian poetry [14]).
LDA is a generative probabilistic model that is based on a few very important assumptions:
1) each text in the collection is assumed to be generated from k number of topics; 2) each
topic is a probability distribution over all available features (where most of the features are very
unlikely). LDA represents each document as probability distribution over all k topics, so that
all documents could essentially be described by the equal-sized vectors in one ”topic space”. In
other words, LDA tries to infer a specific number of groups of co-occuring words from a corpus
automatically; as a consequence, each document becomes represented as combination of these
groups. We consider the use of topic models crucial to our goals, because 1) LDA allows to
do uniform semantic abstraction on the level of single poems; 2) it expresses each document
with potentially low number of interpretable dimensions; 3) topic probabilities of poems allow
for a straightforward follow-up analysis; 4) topic models make our approach independent from
language and specific domain expertise.

15
We followed several corpus preprocessing steps before training a model:

1. All texts were lemmatized using mystem 3.1 [43];
2. General-purpose stop-word list was applied to the corpus (removing conjunctions, parti-
cles, prepositions, pronouns and numerals);
3. We wanted to reduce lexical variance of the corpus, taking only 5,000 most frequent words
to build a model. LDA output usually improves with less sparse data, so removing rare
words is a common procedure. However, since our goal was semantic simplification of
poetic language we used a separate word embedding model trained on the same corpus to
replace words outside of this ”core” 5,000 (word2vec implementation via gensim Python
library, vector size=300). A word was replaced if it had a semantic neighbour among
its 10 contextually most similar words (measured in cosine similarity of corresponding
vectors) that was also a member of the top-1000 words. This procedure allowed us to
replace words with their hyponyms, more frequent synonyms or grammatical variants
(e.g. replacing diminutive forms) and, in some cases, to explicate traditional metonymy
of poetic language (e.g. replacing ”Pontus” with ”ocean”). The procedure was not perfect
and introduced some noise, which however did not have noticeable effect on the model.
We also note here that our results do not alter radically if this contextual replacement
does not happen or if we use another limit on most frequent words (e.g. 1,000). Despite
insignificant effects, we still report results for data with contextual replacements, since
we believe that the chosen direction towards the semantic abstraction is important and
should be improved in the future. We provide all main results for the unaltered corpus
in the Appendix (Table B.5).
4. The corpus was also limited by text size to introduce the LDA with at least comparable
range of word distributions in a document. We removed extra-small (less than 4 lines)
and extra-large (more than 100 lines) poems which left us with approximately 95% of
total texts. We further trimmed the corpus based on word counts, leaving out the texts
between .10 and .90 percentile of size distribution (between 20 & 102 words, which
approx. corresponds to 12 & 50 lines poems when accounted for stop-words removal).
These limitations mean that our model primarily is focused on short lyrical poetry (a
dominant form in Russian tradition that experienced rapid shrinkage of mean poem
length [45]). We believe however, that whatever results we have should also apply to
long narrative poetry, where semantic traditions of metrical usage were predicted to be
much more pronounced [14]. Final text count after all operations is 47,804 (of which
39,220 texts have a single label for their form, derived from the corpus annotations).

There is no universally recognized way to determine an optimal number of topics for the
model [41]: in this paper we report results for LDA trained with 80 topics, which was a midpoint
model in a trade-off between topic coherence (log-likelihood) and perplexity (”surprise” of the
model when predicting unseen data). Main study procedures were also applied to a range of
LDA models with variable number of topics (from 10 to 200) that showed a robust performance
overall (Table B.5 in the Appendix). We set LDA priors to alpha=0.1 (we do not expect many
topics generating single text, since we do not want to swamp distribution) and beta=0.3 (we
do not expect too many words contributing to a topic, but some).
To perform a quick sanity check of the final model we can look at the distinctive topics in a
few meters that were described before qualitatively (Table B.4 in the Appendix). While some
topics could be seen as compatible with assumed semantic halo of meters, there is, of course, no

16
direct relationship. Topics do not correspond to the abstracted metrical ”themes” (Gasparov
also did not use them systematically across his descriptions of different meters) but still they
appear interpretable and could be used for our purposes of the distributed representation of a
meter’s ”content”.

4. Tracing the halo
4.1. Within-meter similarities
The theory of ”semantic halo of meter” assumes that meaning is non-randomly distributed
across metrical forms, that each meter historically builds a unique semantic valency. The
theory also (somewhat implicitly) considers the halo effect cumulative: we would not be able to
reconstruct a meter’s semantics looking at an isolated poem, but a distinct pattern will emerge
from much larger sample of meter’s usage in a tradition. We can rephrase these premises to
say that meter-meaning association assumes some kind of self-similarity within a poetic form.
If the halo effect exists, then two independent pools of poems coming from the same meter
should appear semantically closer to each other than to other samples coming from different
meters.
Let us say we are an observer of the whole tradition, looking at the metrical halos from the
year of 1950 (our corpus upper chronological boundary). To test if the meaning-meter asso-
ciation is noticeable on a general level we perform unsupervised classification on two random
samples (without replacement) of 200 poems for each meter that has at least 500 poems. Per
each sample we calculate mean topic probabilities to represent aggregated topic distribution
within a meter. Since we are dealing with probability distributions, we proceed to calculate
Jensen-Shannon divergence (symmetrical Kullback-Leibler divergence [26]) between all sam-
ples and build hierarchical clusters from the resulting distances. Resampling and recalculation
then continues for 100 times. This results in 100 dendrograms with clustering information that
is used to build a ”majority-rule” consensus tree [11]: it draws branches that correspond to
50% agreement across all dendrograms, so that two branches will not be connected if they did
not cluster together at least in half of the trees (Fig. 2a).
The same procedure could be applied at the level of metrical variants. Because of data
sparsity and noise in metrical annotation, we use only variants of Iamb-4 that have at least
200 poems, while removing the most frequent variant for its diffused semantics (Iamb-4-fm).
This leaves us with only four forms of Iamb-4 (Fig. 2b).
Without addressing further complications of this approach, it is clear that within-meter
semantic similarities are present in the corpus. It is also apparent that semantic difference
could be traced in specific metrical variations, although this level of detail will require much
better annotations and stanza information. Topic information alone is enough to consistently
group two arguably large samples coming from the same meter together (if the median size
of a poem in our corpus is 50 words, then meter-meaning association is quite pronounced in
samples of 10,000 words). We can check the ”cumulative” effect of metrical halo by looking
at how the performance of hierarchical clustering change with the sample size.
To evaluate unsupervised classification we use two metrics: simple Cluster Purity (CP) [7]
(sum of cluster matches divided by number of individual samples) and Adjusted Rand Index
(ARI) [23] that are designed to compare two classifications with the latter also accounting for
classification by chance (returning values around 0). We use the same threshold of 500 available
poems per meter and the same procedure on the increasing sample sizes (up to 250 poems),

17
Figure 2: a, majority-rule consensus tree showing clustering agreement (Jensen-Shannon divergence,
complete linkage) in 100 iterations, 2 random samples per 9 meters, 200 poems per sample. b, consensus
tree showing agreement between clustering of iambic variants, 100 poems per sample. c, Performance of
hierarchical clustering in CP and ARI against ”ground truth” (metrical lables) and against randomly assigned
clusters. Run on the same set of meters (at least 500 poems per each), 100 iterations per sample size.

calculating CP and ARI for each case of clustering. As expected, clustering accuracy grows
with the increasing number of poems in a sample, up to the median ARI=0.73 and CP=0.90
(Fig. 2c). However, it is important that non-random clustering can be noticed early and some
semantic patterns in meters are recognized at 20-40 poems per sample.

4.2. Between-meter similarities
Consensus tree on Fig. 2a also hints at the overarching semantic relationships between par-
ticular metrical forms. Some iambic meters tend to cluster together, the same happens with
ternary forms that clearly make one group (with low distinction between Amphibrach forms).
We do not really have a ”ground truth” for how poetic forms should relate to each other seman-

18
Figure 3: a, The strength of meter association by their corresponding family, clusters are calculated in
equal number of meters per family (k=6, n=12, sample size = 300). 100 iterations per each 20 metrical
sets. b, Consensus tree demonstrating stable semantic associations between meters (k=6, n=19, sample
size = 300, trees = 100).

tically, except for some observations of their historical usage, similar origins, etc. However, we
may expect that the transmission of meaning is at least partly bounded by metrical families
(i.e. Iambic or Trochaic), since they cast broadly similar rhythmic and grammatical bound-
aries on language [12]. In some cases meters of one family also share historical connections.
For example, most Iambic meters were initially used in high-prestige genres: Iamb-4 in ode,
Iamb-5 in drama, Iamb-6 in elegy. At the same time, Trochaic forms were often perceived as
a counterpart to elite iambic verse; some of their rhythmical features overlapped with an oral
tradition which defined their use as folklore imitations and set up corresponding association
range.
We design a fairly conservative experiment to address the assumed ”family influence”: since
the number of available meters per family is not equal, we consider only those families that
have at least two populated meters (> 400 poems). We take two meters per family 20 times at
random; in each set of meters every meter is represented by 300 random poems; clusters are
calculated the same way as described in Section 4.1 , except that now we verify clustering of
forms against their family ”ground truth” (Iamb, Trochee, etc.). The process is repeated 100
times for each of 20 sets of meters. We report the distribution of mean ARI and CP values
per instance of sampled meters as compared to randomly assigned clusters (Fig. 3a).
While the resulting clustering evaluation may not seem high (median ARI is around 0.44,
CP – 0.76), these values are enough to confirm that, at least to some extent, between-meter
relationships are driven by the metrical families. To better demonstrate this effect, we calcu-
lated a consensus tree out of 100 clustering results without any restriction on number of meters
per metrical family (Fig. 3b. Iambs, Dactyls, Anapests and some Trochees tend to consistently
cluster together; semantics of ternary forms remain somewhat diffused, but they still form one
cluster with each other.

19
Cases of ”wrong attribution” are potentially informative and align with the scholarly work
on the subject. The similarity of Iamb-3 and Trochee-4 is well-known [14]: they both originate
in the 18th century’s anacreontics and share ”song” semantics across many variants. Iamb-43
with regular alternation of lines of different feet count also grew from a lyrical song and a
ballad and was associated with lyro-epic poetry.

4.3. Semantic halo in time
It is time to abandon the position of an observer who is ”looking back” at the whole tradition.
Fig. 2 and Fig. 3 make it clear that clustering is to some extent enhanced by the difference in
meters’ distribution in time. Since LDA algorithm exploits patterns of word co-occurrence in
documents, it naturally ends up with groups of words coming from different times (e.g. ”folk”
topic, ”Soviet” topic or naturalistic war topic). In turn, this drives the divergence calculations:
Soviet topic has close to zero probability in 18-19th century texts. We can see this from Dolnik
and Trochee-5 consistently appearing in one cluster – two very popular 20th century forms,
both rare earlier.
”Free” Iamb could be a good example here. This form was composed of unregulated al-
ternation of iambic lines of varying feet length and almost exclusively used for specific set of
genres: poetic epistolary, fables and epigrams, It was also entirely abandoned after 1850s. In
all 1,200 poems of free Iamb there are only two topics that account together for 20% proba-
bility (48,animals & 46,communication). Despite its name, this meter is frozen in time with
a combination of genres imprinted on it, two samples of which is easy to cluster together. In
short, our semantic abstraction is not abstract enough to disregard chronological differences.
The time, however, does not invalidate the general presence of metrical halo; after all,
asynchronous development and fashion fluctuations of metrical forms shape their perceived
differences and pin their associations to distinct periods (Iamb-4-fm to the ”Golden Age”
of 1820-1830s, Dactyl-3 to civic and political sentiment of 1850-1880s, Dolnik to modernism
poetry). Controlling the corpus for chronology is useful not only for testing the halo effect
on a smaller scale, it also creates opportunities to examine the meter-meaning association
mechanism behavior in time. We only briefly address it here, since it is a separate problem
that deserves its own experimental design and comparative data.
First, we want to see if within-meter semantic similarities are present if all poems samples
are coming from the same time frame. We divide our corpus to 30-year bins (excluding the 18th
century because of low variety in popular meters). For each time frame we take its six most
frequent meters and run 100 iterations of clustering, taking two random samples of poems per
meter and report mean ARI values (Table 1). These values are not directly comparable since
we use different pools of meters and sample sizes (half of the minimum meter count per period,
floored), but they are enough to point out that halo effect could be traced within limited time
frames and, in fact, in samples much smaller than expected (see Fig. 2) for some periods.
Second, it is possible to utilize chronological information to ask questions about halo behavior
and semantic accumulation. If semantic halo of a meter is taken as a historical phenomenon and
not as an organic one, we expect a non-uniform, decreasing strength of association between
meter and meaning across time. Specifically, we expect to find a discrepancy in clustering
evaluation between the early 19th century poetry and the second half of the century. This would
confirm a historical diffusion of metrical semantics and unchaining the connection between
form & genre imposed on poetry by normative aesthetics. Since our approach to classification
depends on sample size, we simply split 19th century data in two halves and observe distribution

20
Table 1
Meter-meaning association in different chronological frames, 100 iterations per period
Period Median ARI Poems per sample Meters
1800-1829 0.51 30 I4 If T4 I6 I5 I3
1830-1859 0.47 50 I4 T4 I6 I5 If Amph3
1860-1889 0.23 30 I4 T4 I6 I5 An3 An43
1890-1919 0.77 270 I4 I5 T4 I6 An3 T5
1920-1949 0.58 250 I4 I5 T4 T5 Dk3 An3

Table 2
First and second half of the 19th century, aligned clustering. 1,000 sampling iterations for each period.
Difference in ARIs distributions is significant (t-test, t=19.6, p < 0.0001)
Period Median ARI Poems per sample Meters
1800-1849 0.48 40 I4 T4 If I6 I5 Amph4
1850-1899 0.33

of ARI values when clustering is performed on the strictly the same set of meters and sample
sizes for each of the two chronological groups.
The difference between two periods turns out to be significant (Table 2). Samples of 40
poems – the maximum possible sample size to use for two groups – were better distinguishing
meters in the first half of the 19th century, signaling a focused metrical usage. If we increase
sample size for the second half of the 19th century, then mean clustering accuracy predictably
goes up (sample size=100, ARI=0.43), which points to the process of semantic accumulation
in meters – they become more diffused semantically, but not unrecognizable.

4.4. Topic expression and frequency of a meter
From the start of this paper there was one daunting question: what if the whole halo effect
arises from the simple fact that all metrical forms differ in popularity? It is easy to draw any
kind of topic configuration from Iamb-4, less so from Iamb-3, while options for Trochee-3-dm
are almost non-existent. The semantic halo thus may be born from a combination of sampling
error, chronological differences and a confirmation bias in scholars reading same-meter poems
similarly.
It becomes more complicated, since scholars tell us to expect that the distinctiveness of
a halo is naturally reducing with the increase in a frequency of a meter. There is just no
space for semantic variation in rare forms. Thus we assume that 1) the relationship between
the strength of expression of a meter and its frequency should be linear; 2) if semantics are
randomly assigned to meters a trend should be different.
To measure how ”distinct” is semantics within a single meter we can utilize the curves of
topic probabilities distribution and look at them from the perspective of ”inequality”. In less
populated meters we expect to find fewer topics that dominate the distribution than in highly
frequent forms. Fig. 4a shows an example: topic probabilities are aggregated from all poems for
each of the two metrical variants and arranged according to their overall contribution to meter
semantics. For each of the resulting curves a Gini coeﬀicient – originally designed for measuring
national inequality in wealth distribution – could be calculated. Gini takes a value of 1 when a

21
Figure 4: a, The difference in topic inequality (Gini coeﬀicient) between a general-use Iamb-4-fm (0.12)
and a specific rare form An-43-mf (0.23). b, Semantic inequality decay based on meter’s popularity (black)
vs. randomly redistributed poems (grey), 20 independent redistributions. The slopes of two linear models
are different (-0.28, -0.39), and the model describes more variation in redistributed data (R2 = 0.96) than
in empirical (R2 = 0.81).

distribution is demonically unequal (a single topic is 100% probable) and 0 when it is perfectly
equal (each of the 80 topics is 1.25% probable). It is apparent that this coeﬀicient is enough
to capture how focused a meter’s semantics is at least relatively, while Gini’s absolute values
would be influenced by LDA priors (0.1 alpha assumes more inequality in topic probabilities
of a poem than 0.5 alpha).
To verify the diffusion of a halo and its (non-)random nature we first calculate Gini coef-
ficients for each metrical variant which was used in corpus at least 10 times. Then the same
calculation is done when semantics are redistributed – per each frequency n of a real metrical
form we randomly sample (without replacement) n poems from the corpus. In the end all po-
ems end up being randomly reassigned to empty ”meter bins” and this redistribution is done
20 separate times. If the halo could not be simply sampled at random, then some divergence in
how inequality correlates with sample size between two groups of points should be noticeable.
Fig. 4b shows differences between inequality distributions in randomly aggregated poems
and actual metrical forms on log-log scales. A decrease of semantic inequality based on a
meter’s frequency happens as expected with very telling outliers that signal concentrated halo
in frequent metrical forms (most notably free Iamb). On the other hand, inequality in randomly
redistributed data decays quicker, leaving lots of actual meters above the line. This suggests
that even if comparable levels of topical inequality in the very infrequent forms might occur
randomly, there is no reason to expect it for semantic halo in general. It is highly improbable

22
to randomly sample semantic curve even of Iamb-4-fm which was always considered a neutral
and a general-use form.

5. Discussion
We were able to show that topical information alone could be used to recognize a poetic form.
Poems originating in one meter retain semantic similarity to themselves. Separate meters
demonstrate stable relationships between them which in part are driven by their origins in
metrical family. Historical differences in classification accuracy also suggested semantic accu-
mulation in metrical forms and a diffusion of meter’s ”meaning” over time without swamping
it beyond recognition. These findings, we believe, confirm the theory of semantic halo and its
main assumptions at least on a general level.
In the future the effect of metrical halo could be better understood in cultural evolution
framework [29, 48], which provides plenty of options to explicate the way of how we think
about the workings of historical processes and cultural transmission. Cultural evolution is an
emergent field that studies variation, retention and diffusion of cultural information (which is
usually defined as any information acquired via social learning). This framework encompasses
the research across various disciplines and domains: it was used to reconstruct cultural phyloge-
nies of archaeological artifacts [32], folktales [47] and medieval manuscripts [2, 59]; understand
how people learn and contribute to cumulative forces of culture [52]; address innovation rates
in popular music [27]; study macro patterns in language evolution [15, 6], or the effects of
population size on the diffusion and retention of cultural information [22, 37].
It would not be too far-fetched to say that all new poems originate from previous ones.
Most often they are products of imitation: it is extremely rare for a poet to completely escape
any exposure to a tradition or single-handedly create a whole versification system. Should
it happen, chances are high that these individual efforts will not survive for too long, simply
because there will not be enough followers. Poetic forms are persistent and conservative: things
like Iambic Pentameter, or rhyming, or a sonnet pattern could survive for centuries. This means
new poems share plenty of formal features – like meter – with their predecessors, effectively
replicating the previously used form. Arguably nothing should stop a poet in a modern highly
individualized tradition of lyrical self-expression from using a metrical form absolutely free,
independent from its semantic inertia. We see this is not the case.
Meters and poetic forms could be seen as behaving similarly to ”transmission isolating mech-
anisms” [10, 51] of culture. These mechanisms are certain conditions (marriage traditions,
household organization, etc.) that maintain some level of ”vertical” (parents to offspring)
transmission of information in culture which is usually seen as the domain of extensive ”hori-
zontal” connections (peers to peers). Meters, similarly, are limiting the semantic possibilities
of poems and pushing meaning creation into fuzzy, yet distinct pathways. This ensures that
meters during the modern poetic histories act as ”weak genres”, replicating an expanding set
of features in poems that also share similar formal origins.
Why this formal isolation should happen in poetry at all? An obvious answer lies in meter’s
capacity as an effective mnemonic device that pushes language into a high-order pattern and
enhances memorability [17]. Poetic forms originated in oral traditions, which, in turn, relied
on many formal mechanisms (meter, rhyme, formulaic language, formulaic plots, etc.) that
limited options for how a text could be made and retold to facilitate its memorization and
transmission [39]. Apparently the mnemonic strength of a meter would matter in written

23
traditions too. It is not just a form that is remembered and reproduced; by simply turning its
wheels a meter carries its ever-expanding baggage further. The individual histories of metrical
forms in Russian poetry had a lot of twists and turns and consciously guided revolutions in
metrical expectations (because these expectations existed). However, it seems that no one
could truly escape the mnemonic tyranny of the accentual-syllabic verse.
We expect the semantic halo to be present (to larger or smaller extent) in any poetic tradi-
tion based on any versification system that allows for distinct and stable poetic forms across
time. Topic models could give us a set of abstract derived measures (classification accuracy,
inequality, etc.) that could be compared across languages and traditions. This, in turn, pro-
vides an access to general questions of literary history: how poetic genres ”fall” in time, to
what extent meters continue to be recognizable (if at all), how the invention of new forms
happen or what is the role of individual poets and poems in shaping semantic halo.

Acknowledgments
AŠ was funded by “Large-Scale Text Analysis and Methodological Foundations of Computa-
tional Stylistics”(NCN 2017/26/E/ HS2/01019) project supported by Polish National Science
Centre. We would like to thank two anonymous reviewers for their attentive reading that
allowed us to make this paper better. We thank Joanna Byszuk, Maciej Eder, Antonina
Martynenko, Vera Polilova and Oleg Sobchuk for all of their contribution, help and support.

References
[1] E. Asgari, M. Ghassemi, and M. A. Finlayson. “Confirming the themes and interpretive
unity of Ghazal poetry using topic models”. In: Neural Information Processing Systems
(NIPS) Workshop for Topic Models (2013).
[2] A. C. Barbrook et al. “The phylogeny of The Canterbury Tales”. In: Nature 394.6696
(Aug. 1998), pp. 839–839. issn: 1476-4687. doi: 10.1038/29667. (Visited on 09/17/2020).
[3] A. T. J. Barron et al. “Individuals, institutions, and innovation in the debates of the
French Revolution”. In: Proceedings of the National Academy of Sciences 115.18 (May
2018), pp. 4607–4612. issn: 0027-8424, 1091-6490. doi: 10.1073/pnas.1717729115. (Vis-
ited on 07/26/2020).
[4] M. Bicego et al. “Investigating Topic Models’ Capabilities in Expression Microarray Data
Classification”. In: IEEE/ACM Transactions on Computational Biology and Bioinformat-
ics 9.6 (2012), pp. 1831–1836.
[5] D. M. Blei. “Latent Dirichlet Allocation”. In: Journal of Machine Learning Research 3
(2003), pp. 993–1022.
[6] R. Bouckaert et al. “Mapping the Origins and Expansion of the Indo-European Language
Family”. In: Science 337.6097 (Aug. 2012), pp. 957–960. issn: 0036-8075, 1095-9203. doi:
10.1126/science.1219669. (Visited on 09/17/2020).
[7] F. Cafiero and J.-B. Camps. “Why Molière most likely did write his plays”. In: Science
Advances 5.11 (Nov. 2019). issn: 2375-2548. doi: 10.1126/sciadv.aax5489. (Visited on
12/16/2019).

24
[8] P. DiMaggio, M. Nag, and D. Blei. “Exploiting aﬀinities between topic modeling and
the sociological perspective on culture: Application to newspaper coverage of U.S. gov-
ernment arts funding”. In: Poetics 41.6 (Dec. 2013), pp. 570–606. issn: 0304422X. doi:
10.1016/j.poetic.2013.08.004. (Visited on 03/10/2019).
[9] H.-G. Drost. “Philentropy: Information Theory and Distance Quantification with R”. In:
Journal of Open Source Software 3.26 (June 2018). issn: 2475-9066. doi: 10.21105/joss
.00765. (Visited on 09/13/2020).
[10] W. H. Durham. “Advances in Evolutionary Culture Theory”. In: Annual Review of An-
thropology 19.1 (1990), pp. 187–210. doi: 10.1146/annurev.an.19.100190.001155.
[11] J. Felsenstein. “Confidence Limits on Phylogenies: An Approach Using the Bootstrap”.
In: Evolution 39.4 (1985), pp. 783–791. issn: 1558-5646. doi: 10.1111/j.1558-5646.1985
.tb00420.x. (Visited on 07/26/2020).
[12] M. L. Gasparov and M. Tarlinskaja. “The Linguistics of Verse”. In: The Slavic and East
European Journal 52.2 (2008), pp. 198–207. issn: 0037-6752. doi: 10. 2307 / 20459662.
(Visited on 07/26/2020).
[13] M. Gasparov. A History of European Versification. Oxford, New York: Oxford University
Press, July 1996.
[14] M. Gasparov. Metr i smysl: ob odnom iz mekhanizmov kulturnoi pamiati. Moscow: Izda-
telskii tsentr RGGU, 1999. (Visited on 07/26/2020).
[15] R. D. Gray and F. M. Jordan. “Language trees support the express-train sequence of
Austronesian expansion”. In: Nature 405.6790 (June 2000), pp. 1052–1055. issn: 1476-
4687. doi: 10.1038/35016575. (Visited on 09/17/2020).
[16] E. Grishina et al. “Poeticheskii korpus v ramkah NKRIA: obschaia struktura i perspek-
tivy ispolzovania”. In: Natsionalnii korpus russkogo iazyka: 2006-2008. Novye rezultaty i
perspektivy. St. Petersburg: Nestor-Istoria, 2009, pp. 71–113.
[17] M. Gronas. Cognitive Poetics and Cultural Memory: Russian Literary Mnemonics. 1
edition. New York: Routledge, 2010.
[18] B. Grün and K. Hornik. “topicmodels: An R Package for Fitting Topic Models”. In:
Journal of Statistical Software 40.13 (2011), pp. 1–30. doi: 10.18637/jss.v040.i13.
[19] T. N. Haider. “Diachronic Topics in New High German Poetry”. In: Proceedings of the
International Digital Humantities Conference. Utrecht, 2019, pp. 1–7. url: https://dev
.clariah.nl/files/dh2019/boa/1031.html (visited on 09/17/2020).
[20] D. Hall, D. Jurafsky, and C. D. Manning. “Studying the History of Ideas Using Topic
Models”. In: Proceedings of the Conference on Empirical Methods in Natural Language
Processing. EMNLP ’08. Stroudsburg, PA, USA: Association for Computational Linguis-
tics, 2008, pp. 363–371. (Visited on 12/11/2019).
[21] E. Henderson, D. Desrosiers, and M. Chirico. ghibli: Studio Ghibli Colour Palettes. Apr.
2020. url: https://CRAN.R-project.org/package=ghibli (visited on 09/13/2020).
[22] J. Henrich. “Demography and Cultural Evolution: How Adaptive Cultural Processes can
Produce Maladaptive Losses: The Tasmanian Case”. In: American Antiquity 69.2 (2004),
pp. 197–214. issn: 0002-7316. doi: 10.2307/4128416. (Visited on 09/17/2020).

25
[23] L. J. Hubert and P. Arabie. “Comparing partitions”. In: Journal of Classification 2.2-3
(1985), pp. 193–218. issn: 0176-4268(Print). doi: 10.1007/BF01908075.
[24] R. Jakobson. “Toward a Description of Mácha’s Verse”. In: R. Jakobson. Selected Writ-
ings. Vol. 5. The Hague, Paris, NY, 1979, pp. 433–485.
[25] K. Korchagin. “Poezija XX veka v poeticheskom podkorpuse Natsional’nogo korpusa
russkogo iazyka: problema reprezentativnosti”. In: Trudy instituta im. V.V. Vinogradova
6 (2015), pp. 235–256. (Visited on 07/26/2020).
[26] S. Kullback and R. A. Leibler. “On Information and Suﬀiciency”. In: The Annals of
Mathematical Statistics 22.1 (1951), pp. 79–86. issn: 0003-4851. (Visited on 07/26/2020).
[27] M. Mauch et al. “The evolution of popular music: USA 1960–2010”. In: Royal Soci-
ety Open Science 2 (2015). url: http://dx.doi.org/10.1098/rsos.150081 (visited on
07/26/2020).
[28] A. Meillet. Les origines indo-européennes des mètres grecs. Paris, Les Presses univer-
sitaires de France, 1923. url: http : / / archive . org / details / lesoriginesindoe00meiluoft
(visited on 07/26/2020).
[29] A. Mesoudi. Cultural Evolution: How Darwinian Theory Can Explain Human Culture
and Synthesize the Social Sciences. University of Chicago Press, 2011.
[30] J. Murdock, C. Allen, and S. DeDeo. “Exploration and Exploitation of Victorian Sci-
ence in Darwin’s Reading Notebooks”. In: Cognition 159 (Feb. 2017), pp. 117–126. issn:
00100277. doi: 10.1016/j.cognition.2016.11.012. (Visited on 07/26/2020).
[31] B. Navarro-Colorado. “On Poetic Topic Modeling: Extracting Themes and Motifs From
a Corpus of Spanish Poetry”. In: Frontiers in Digital Humanities 5 (2018), p. 15. issn:
2297-2668. doi: 10.3389/fdigh.2018.00015.
[32] M. J. O’Brien and R. L. Lyman. “Evolutionary archeology: Current status and future
prospects”. In: Evolutionary Anthropology: Issues, News, and Reviews 11.1 (2002), pp. 26–
36. issn: 1520-6505. doi: 10.1002/evan.10007. (Visited on 09/17/2020).
[33] B. Orekhov. Bashkirskii stikh XX veka. Korpusnoe issledovanie. St. Petersburg: Aletejia,
2019. (Visited on 07/26/2020).
[34] T. L. Pedersen. patchwork: The Composer of Plots. June 2020. url: https://CRAN.R-p
roject.org/package=patchwork (visited on 09/13/2020).
[35] A. Piperski. “Semantic halo of a meter: a keyword-based approach”. In: Kompiuternaia
lingvistka i intellectualnyie tekhnoloigii. Vol. 2. Kompiuternaia lingvistika: lingvistich-
eskiie issledovaniia. Moscow: RGGU, 2017, pp. 342–354.
[36] P. Plechac and T. N. Haider. “Mapping Topic Evolution Across Poetic Traditions”. In:
arXiv:2006.15732 [cs, stat] (Aug. 2020), pp. 1–6. url: http://arxiv.org/abs/2006.15732
(visited on 09/28/2020).
[37] A. Powell, S. Shennan, and M. G. Thomas. “Late Pleistocene Demography and the
Appearance of Modern Human Behavior”. In: Science 324.5932 (June 2009), pp. 1298–
1301. issn: 0036-8075, 1095-9203. doi: 10.1126/science.1170165. (Visited on 09/17/2020).
[38] R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Aus-
tria: R Foundation for Statistical Computing, 2019. url: https://www.R-project.org/.

26
[39] D. C. Rubin. Memory in Oral Traditions: The Cognitive Psychology of Epic, Ballads,
and Counting-Out Rhymes. Oxford University Press USA, 1995.
[40] Russian National Corpus. 2003. url: https://ruscorpora.ru/new/en/index.html (visited
on 09/13/2020).
[41] S. Sbalchiero and M. Eder. “Topic modeling, long texts and the best number of topics.
Some Problems and solutions”. In: Quality & Quantity 54.4 (Aug. 2020), pp. 1095–1108.
issn: 1573-7845. doi: 10.1007/s11135-020-00976-w. url: https://doi.org/10.1007/s1113
5-020-00976-w (visited on 09/17/2020).
[42] C. Schöch. “Topic Modeling Genre: An Exploration of French Classical and Enlighten-
ment Drama”. In: Digital Humanities Quarterly 011.2 (May 2017). issn: 1938-4122.
[43] I. Segalovich. “A Fast Morphological Algorithm with Unknown Word Guessing Induced
by a Dictionary for a Web Search Engine”. In: MLMTA. 2003, pp. 273–280.
[44] M. Shapir. “”Semanticheskii oreol metra”: termin i poniatie”. In: Shapir, M. Universum
versus: iazyk, stikh, smysl v russkoi poezii XVIII-XX vekov, volume 2. Moskva: Iazyki
slavianskoi kul�tury, 2015, pp. 395–404.
[45] A. Shelya and O. Sobchuk. “The shortest species: how the length of Russian poetry
changed (1750–1921)”. In: Studia Metrica et Poetica 4.1 (Aug. 2017), pp. 66–84. issn:
2346-691X. doi: 10.12697/smp.2017.4.1.03. (Visited on 07/26/2020).
[46] J. Silge and D. Robinson. Text Mining with R: A tidy approach. 2020. url: https://ww
w.tidytextmining.com/ (visited on 07/27/2020).
[47] S. G. da Silva and J. J. Tehrani. “Comparative phylogenetic analyses uncover the ancient
roots of Indo-European folktales”. In: Royal Society Open Science 3.1 (2016), p. 150645.
doi: 10.1098/rsos.150645. (Visited on 09/17/2020).
[48] O. Sobchuk. “Charting artistic evolution: an essay in theory”. PhD thesis. Tartu: Tartu
University, 2018. (Visited on 07/26/2020).
[49] K. Taranovskii. “O vzaimootnosheniah stikhotvornogo metra i tematiki”. In: American
Contributions to the Fifth International Congress of Slavists 1 (1963), pp. 287–332.
[50] M. Tarlinskaja and N. Oganesova. “Meter and Meaning: The Semantic Halo of Verse
Form in English Romantic Lyrical Poems (Iambic and Trochaic Tetrameter)”. In: The
American Journal of Semiotics 4.3/4 (1986), pp. 85–106. doi: https://doi.org/10.5840
/ajs198643/422. (Visited on 07/26/2020).
[51] J. Tehrani and M. Collard. “Do Transmission Isolating Mechanisms (TRIMS) influence
cultural evolution? Evidence from patterns of textile diversity within and between Ira-
nian tribal groups”. In: Understanding Cultural Transmission in Anthropology: A Critical
Synthesis (eds. Ellen, R., Lycett, S. & Johns, S. Berghahn). Vol. 26. 2013, pp. 148–164.
[52] C. Tennie, J. Call, and M. Tomasello. “Ratcheting up the ratchet: on the evolution of
cumulative culture”. In: Philosophical Transactions of the Royal Society B: Biological
Sciences 364.1528 (Aug. 2009), pp. 2405–2415. issn: 0962-8436. doi: 10.1098/rstb.2009
.0052. (Visited on 09/17/2020).
[53] M. Trunin. “Towards the concept of semantic halo”. In: Studia Metrica et Poetica 4.2
(Dec. 2017), pp. 41–66. issn: 2346-691X. doi: 10.12697/smp.2017.4.2.03. (Visited on
07/26/2020).

27
[54] G. Vinokur. “Vol’nye iamby Pushkina”. In: Pushkin i ego sovremenniki: Materialy i
issledovania. Vol. 38/39. Leningrad, 1930, pp. 23–26.
[55] M. Woźniak et al. “Linguistic measures of chemical diversity and the “keywords” of
molecular collections”. In: Scientific Reports 8.1 (May 2018), p. 7598. issn: 2045-2322.
doi: 10.1038/s41598-018-25440-6. (Visited on 07/26/2020).
[56] T.-I. Yang, A. Torget, and R. Mihalcea. “Topic Modeling on Historical Newspapers”.
In: Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural
Heritage, Social Sciences, and Humanities. Portland OR, 2011, pp. 96–104.
[57] G. Yu et al. “Two Methods for Mapping and Visualizing Associated Data on Phylogeny
Using Ggtree”. In: Molecular Biology and Evolution 35.12 (2018), pp. 3041–3043. issn:
0737-4038. doi: 10.1093/molbev/msy194.
[58] A. Zeileis and C. Kleiber. ineq: Measuring Inequality, Concentration, and Poverty. July
2014. url: https://CRAN.R-project.org/package=ineq (visited on 09/13/2020).
[59] J. van Zundert. “Computational methods and tools”. In: Handbook of Stemmatology. Ed.
by P. Roelli. De Gruyter Reference. De Gruyter, 2020, pp. 292–356.

28
A. Code and data availability
Document-Term Matrix, final preprocessing steps, final models & full analysis are openly
available: https://github.com/perechen/semantic_halo_rus We have used R 4.0.2. [38] for
the analysis, LDA implementation in topicmodels package [18], relied on tidytext wrappings
for the model’s output [46], phylentropy [9] and ineq [58] for calculations, ggtree for drawing
trees [57], patchwork for assembling plots [34]. ghibli package [21] provided the color palette
”MononokeMedium” for the plots.

B. Appendix

29
Table B.4
Distinctive topics (words translated) in three meters compared to Gasparov’s descriptions [14]. Top 10 topics
with most deviation from the mean are listed. Topics that could be relevant for halo are highlighted
Meter Halo (Gasparov) Topic Top words
Trochee-5-fm Night, Landscape, Love, Death, Road 69 to know, to live, to be, to die, nothing
41 war, to go, soldier, battle, bullet
61 goodbye, last, to go (away), hand, parting
25 wind, steppe, sand, grass, desert
66 garden, green, leaf, branch, linden
45 train, wheel, smoke, to fly, wind
38 window, house, wall, room, table
39 water, river, shore, to swim, lake
31 to go, path, road, to cross, leg
Trochee-3-fm Song, Road, Nature, Yearning, Love, Death 76 to sing, song, nightingale, voice
77 matter, take, give, comrade, most
23 red, to go, oi, white, ”ka” (folksong love topic)
43 snow, white, ice, winter, snowy
51 door, house, enter, window, wait
51 woods, pine, green, tree
10 wind, leaf, autumn, rain, autumn
21 dream, to dream, night, to wake, morning
22 night, darkness, murk, dark
31 to go, path, road, to cross, leg
Iamb-4-dm dangerous movement through space / love 62 city, tower, wall, stone
59 horror, death, evil, blood
1 star, world, sky, earth, abyss
6 shade, dream, ghost, pale
4 soul, dream, beauty, world, power
68 hour, wait, to come, soon, or
12 god, temple, tsar, before, world
61 goodbye, last, to go (away), hand, parting
80 city, road, house, light, to go
50 poem, write, poet, book, word

30
Table B.5
Median Adjusted Rand Index values for within-meter clustering (Fig. 2c) in different samples (ARI @ n
poems per sample), using LDA models with varying k number of topics . ARI family column holds ARI
values for between-meter similarity tests (Fig. 3a). The robust performance of clustering in various models
shows that number of topics has little influence on the experimental results, and, thus, surprisingly, is not
really an influential variable. LDA with 20 topics shows one of the highest performances at 250 poems
per sample, which further highlights the effectiveness of the reduction of semantic information. There is,
however, a slight noticeable trade-off between local (within-meter) and global (between-meter) recognition
(models with 80 and 100 topics seem to provide the most balanced performance). Additional tests were
made for LDA models trained on the original Document-Term Matrix (without replacements of less frequent
words with their more frequent semantic neighbours).
DTM k ARI@10 ARI@60 ARI@100 ARI@150 ARI@250 ARI family
w/ replacement 10 0.07 0.28 0.47 0.55 0.62 0.33
20 0.09 0.28 0.45 0.57 0.76 0.40
40 0.08 0.31 0.44 0.60 0.71 0.36
60 0.09 0.30 0.41 0.55 0.70 0.43
80 0.09 0.29 0.44 0.56 0.73 0.43
100 0.09 0.31 0.45 0.60 0.76 0.39
120 0.04 0.30 0.45 0.54 0.70 0.48
150 0.08 0.28 0.42 0.57 0.70 0.41
200 0.04 0.28 0.42 0.54 0.70 0.42
w/o replacement 20 0.08 0.34 0.46 0.55 0.70 0.32
80 0.09 0.31 0.41 0.56 0.76 0.32
150 0.08 0.27 0.45 0.60 0.73 0.45