Introduction

A good deal of the 'catchiness' of popular music can be attributed to the frequent recurrence of patterns or schemas in multiple musical domains like form, rhythm, melody, and harmony [40,22]. Despite the simplicity of the term as used in academic, educational, and commercial contexts, 'popular music' covers a vast and diverse range of mostly 20th-century musical styles and genres that share some but by far not all patterns and schemas [2]. Understanding patterns in popular music and how they change over time is thus crucial for understanding the music itself. The repetitive harmonic structure of popular music in particular is thought to ease information compression and thus to facilitate both retention and recall, for which there is also some empirical support [24,39]. A particularly telling example of harmonic patterns in popular music are so-called 'four-chord songs' that consist-entirely or in large parts-of a single repeated sequence of four harmonic units [30]. 1 Empirical research on patterns in popular music research can draw on a number of data sources for analysis. Some genres have produced songbooks that contain essential melodic and harmonic information to be used in performance. Commonly, these songbooks contain melodies annotated with chord symbols that experts have transcribed from recordings of the original pieces of music, for which authoritative versions usually do not exist: popular music is recorded rather than written down. However, the annotated chords in songbooks have sometimes gained a status of authority, the most famous example probably being the volumes of the Real Book for the case of jazz [7].

Another source for harmonic analysis of popular music are methods from music information retrieval (MIR) applied to audio recordings of music, either through manual transcription or algorithmic inference [37,23,26,60,35]. The harmonic information thus obtained can then further serve as a basis for corpus studies that aim at understanding characteristics of a particular musical style, or to trace historical developments throughout the lifetime of a genre [51].

With the rise of corpus studies in musicology in the last decades, harmonic analysis in popular music has become a frequent use case. Examples include the study of chord idioms in the Beatles and Real Book jazz standards [32,55,56,27], or US-pop charts [6,54]. Apart from the stylometry of particular genres or music groups, researchers have also widely engaged in genre classification [46,34,17], and addressed the question of historical changes in popular music idioms [33,18,52,43]. More recently, researchers have turned to more methodological meta-questions and addressed concerns regarding biases and issues of representation in the selection of corpora [31,57], and have pointed out that content descriptors as well as the reliance on curated chart lists are insufÏcient to fully characterize the complexities of popular music genres [8].

While the aforementioned studies have laid the ground for large-scale stylistic analysis of several styles in popular music, they remain largely descriptive. There are still relatively few studies on popular music dealing explicitly with models of diachronic information transmission. The growing field of cultural evolution [50,65] provides a promising methodological framework to extend the existing studies on the cultural evolution of music [62,63,64,44,52,45,25,19] in the domain of popular music.

In the present study, we explicitly address the research question of how harmonic patterns in popular music change over time, and how their diachronic transmission can be modeled formally. We build on a recent model for selection and mutation probabilities in Japanese pop music [44] and apply it to different popular musical styles with a more fine-grained measure of harmonic similarity.

Data

To study the fluctuating transmission probabilities of harmonies in different popular music genres, we assemble several datasets by drawing on existing scholarship in music corpus studies. Specifically, we analyze datasets of chord symbols from three different cultural contexts: Japanese Pop songs, US-Pop songs, and Brazilian Choro pieces. Summary statistics are shown in Table 1, and Figure 1 displays the absolute numbers of pieces per year for the three corpora. Japanese pop songs This dataset was collected by the author of [44] and is comprised of Japanese popular songs composed in years from 1927 to 2019. In this period, the musical style changed drastically under a strong influence from Western popular culture. The songs were taken from the top ranked songs in the yearly charts in Japan and also from a collection of songbooks published in Japan. The chord progressions were extracted from published scores.

US-Pop songs

We analyze US popular music by drawing on the Billboard Hot 100 charts between 1958 and 1991 [5,4] that are frequently employed in music information retrieval (MIR) research. The dataset is taken to be representative of the most popular songs.

Brazilian Choro

This mostly instrumental music genre emerged in Rio de Janeiro in the late-19th century. While the genre has had an immense influence on the development of other Brazilian genres like Samba, there are only a few data-driven approaches to analyze its style to date [43,53]. The data is taken from the Choro Songbook Corpus [41].

Methods

Modeling chords

Assembling data from different sources usually means that they do not adhere to the same encoding standards, a big challenge in many comparative studies [16]. In our case, we were confronted with various encodings for harmony, each of which containing different selections of harmonic features. Another issue is that the labels were generated in very different processes with diverging-sometimes genre-dependent-conventions that may or may not be explicitly stated. For instance, a G:13 chord in jazz is usually assumed to be a short-hand notation for G: (7,9,11,13), that is, all thirds below the stacking implied. These labels can be found in lead sheets that are used in jazz performance. On the other hand, labels in the pop corpora are analytical in the sense that someone (a person or a machine) listened to the songs and estimated the best-fitting chord symbol for a given segment of music. Moreover, the labels in the corpora used are analytical (describing what was played in a certain recording), but those are not fixed as some genres involve improvisation and performance variation. Another difÏculty relates to the functional interpretation. In pop music, sometimes it is not clear what the global or local key is,2 hence chords can not be interpreted in function to the tonic, rendering the harmony ambiguous [48,12]. This can, of course, be an intentional artistic device. For these reasons, finding an appropriate data representation for chords is challenging. In music information retrieval (MIR) research, one of the most commonly used standards for chords is the syntax proposed by Christopher Harte [21,20]. In this notational scheme, chords are encoded as interval structures above a given root in the form <root>:<intervals>. For example, C:maj represents a C-major chord and Ab:sus4(7,9) represents a dominant-seventh chord on the root A♭ with a suspended fourth.

Note that this syntax represents chords in an absolute manner that does not assign any functional roles to the harmonies, which avoids the problem of functional ambiguity. While we do not assume functional roles in encoding the chords, using a common syntax ensures that chord sequences are comparable. We follow the common practice to transpose all pieces in a corpus to the same key (C major) in order to ensure that relative chord distances are comparable, while still maintaining a functionless interpretation. Naturally, the keys of popular music pieces can be as ambiguous as are the functional roles of chords, but these ambiguities often concern the primacy of mediants, e.g. whether a piece is in C major or A minor.

Modeling chord distances

As we have seen, musical chords are complex objects that do not imply a canonical distance measure. In fact, music theorists have proposed a variety of different distance metrics between harmonies, each of which emphasizes a particular aspect of chords. Among the proposed metrics, one can distinguish between theoretical or mathematical [13,15,59,14,9,61,36] and perceptual [29,58,49] notions of harmonic distance, as well as their mutual relation [28,38].

The simplest distance metric is to assess whether two chords are identical or different (binary). This, however, obfuscates more subtle differences, e.g. the fact that, in tonal harmony, one would generally consider a G chord to be more similar to a G7 or a C chord than to a E♭ chord.

For chord parsing we use The Harte Library,3 a convenient framework to work with chord labels in Harte notation [11] that builds upon the music21 [10] data structure for chords. The encoding of chords 𝑐 with the Harte syntax easily translates to the representation as root-interval pairs 𝑐 = (𝑟, 𝑒), because short-hand codes such as maj or min can be expressed as sets of intervals above the root 𝑒 = (3,5) or 𝑒 = (3, ♭5), respectively.We use three types of distances suitable for chords in a symbolic representation. For chords 𝑐 1 = (𝑟 1 , 𝑒 1 ) and 𝑐 2 = (𝑟 2 , 𝑒 2 ):

Chord-type distance The coarsest distance compares whether two chords 𝑐 1 and 𝑐 2 are of the same type, i.e. whether they share all features except their roots.

𝑑 type (𝑐 1 , 𝑐 2 ) = { 0 if 𝑒 1 = 𝑒 2 1 if 𝑒 1 ≠ 𝑒 2(1)

Line-of-fifths distance Focusing on chord roots, we measure their distance as the difference of their positions on the line of fifths (LoF) [42],

𝑑 LoF (𝑐 1 , 𝑐 2 ) = |𝑙(𝑟 2 ) − 𝑙(𝑟 1 )|,(2)

where the positions of chord roots on the line of fifths are given by

𝑙 ∶ {… , B♭, F, C, G, D, …} ↦ {… , −2, −1, 0, 1, 2, …} ≡ ℤ.(3)

Pitch-class set distance

To measure the distance between two chords represented as pitchclass sets, we employ the Jaccard distance [47] 𝑑

Jaccard (𝑐 1 , 𝑐 2 ) = |𝑐 1 ∩ 𝑐 2 | |𝑐 1 ∪ 𝑐 2 | .(4)

Modeling chord substitutions

We denote the (infinite) vocabulary of chord symbols in Harte syntax with Ω. For each piece in each of the corpora, we extract 𝐿-grams (chord sequences of length 𝐿), were repetitions are retained. Many chordal patterns correlate with the formal structure of songs that frequently consists of multiples of four bars. Thus, we only examine chord sequences of length 𝐿 = 4, and denote the collection of 4-grams of all pieces released in year 𝑡 by

𝑆 𝑡 = {w (𝑖) | 𝑡 𝑖 = 𝑡},(5)

where chord 4-grams are given by

w (𝑖) = (𝑤 (𝑖) 1 , 𝑤 (𝑖) 2 , 𝑤 (𝑖) 3 , 𝑤 (𝑖) 4 ) , 𝑤 (𝑖) 𝑙 ∈ Ω,(6)

for some year-specific index 𝑖 of distinct 4-grams. The set of 4-grams in earlier years is defined as 𝑆 <𝑡 = ⋃ 𝑡−1 𝑗=1 𝑆 𝑗 . We assume that 4-grams w ∈ 𝑆 𝑡 are generated randomly by mutating an existing reference chord sequence from a past song, w ′ ∈ 𝑆 <𝑡 . First, a reference chord sequence w ′ is sampled from 𝑆 <𝑡 according to its selection probability, 𝑃 𝑠𝑒𝑙 (w ′ | 𝑆 <𝑡 ). The actual chord sequence w is then generated by a pure replication or by replacing one or several elements of w according to the substitution probabilities

𝑃 𝑠𝑢𝑏 (w | w ′ ) = 𝐿 ∏ 𝑙=1 𝜋 𝑠𝑢𝑏 (𝑤 𝑙 | 𝑤 ′ 𝑙 ).(7)

Note that, here, substitution probabilities for chords within one 4-gram are considered independent. While it is possible to employ more elaborate probabilistic music sequence models, these would introduce further parameters and would thus necessitate a larger amount of data for reliable inference. The substitution probabilities between all chords in a corpus can be expressed as a 𝑁 × 𝑁 transition matrix, where 𝑁 is the number of chord types in the corpus, that is not necessarily symmetric because, in general, 𝜋 𝑠𝑢𝑏 (𝑤 𝑙 | 𝑤 ′ 𝑙 ) ≠ 𝜋 𝑠𝑢𝑏 (𝑤 ′ 𝑙 | 𝑤 𝑙 ). In the selection process, we suppose that a reference 4-gram is selected by first randomly choosing a past song and then choosing a 4-gram in that song. Since the selection probability of a 4-gram is then proportional to its frequency in the past data, 𝑃 𝑠𝑒𝑙 (w; 𝑆 <𝑡 ) is proportional to the relative frequency #w 𝑗 of a chord 4-gram w in year 𝑗. In order to make the selection process more realistic, we introduce a bias that affects the selection probabilities. Specifically, we assume that more recent songs exert a stronger influence on present songs than songs further in the past, corresponding to a recency bias, which we model by a weighting factor 𝑒 −𝛽(𝑡−𝑗) for a 4-gram observed in year 𝑗, where 𝛽 is the parameter controlling the strength of the recency bias. The selection probability is thus given by

𝑃 𝑠𝑒𝑙 (w; 𝑆 <𝑡 ) ∝ 𝑡−1 ∑ 𝑗=1 𝑒 −𝛽(𝑡−𝑗) ⋅ #w 𝑗 .(8)

Finally, the full specification of our generative model is given by the following equation:

𝑃(w; 𝑆 𝑡 ) = ∑ w ′ ∈𝑆 <𝑡 𝑃 𝑠𝑢𝑏 (w | w ′ ) ⋅ 𝑃 𝑠𝑒𝑙 (w ′ ; 𝑆 <𝑡 ).(9)

Inference

The model parameters 𝛽 and 𝜋 𝑠𝑢𝑏 (w | w ′ ) are inferred using the expectation-maximization (EM) algorithm [3] with grid search and likelihood optimization, where the reference chord 4-grams w ′ are considered to be latent variables for each observed chord 4-grams w in the datasets used.

After estimating the mutation probabilities, we analyze the temporal evolution of substitution probabilities from the posterior probabilities of the referential 4-grams for each year. For some years, there are only a small number of songs, or even no songs, in our datasets, and we apply additional smoothing techniques to reliably estimate the substitution probabilities for each year. Specifically, we use the Kalman smoothing method [1] in which we assume that the temporal continuity of each substitution probability can be modeled by a Gaussian Markov process.

Results and discussion

We analyze the three corpora under three perspectives: (1) evolution of the chord vocabulary, (2) historical changes in the substitution probabilities, and (3) relations with the chord distances defined above. As the chord vocabularies have different encodings and sizes, it is difÏcult to compare them directly in a quantitative manner. For this reason, we focus on the 25 most frequent chords or substitutions, analyze each corpus separately, and interpret the results qualitatively against the background of music theory.

Evolution of chord-symbol frequencies

We first analyze the changing frequencies of occurrence of chords symbols. For J-Pop (Figure 2(a)), one can observe a trend towards a more uniform distribution of harmonies used. Whereas a few chords dominate the earlier phases (notably diatonic triads and applied dominants), chord frequency entropy increases rendering the harmonic vocabulary in use more diverse. The strongest changes occur roughly between the 1970's and the mid-90's. It is difÏcult to assert whether this trend is due to a change in the harmonic language or due to changes in data, as this phase lies between the two peaks visible in Figure 1. However, after this phase, the entropy of the chord vocabulary is distinctively higher than in the earlier years.

In US-Pop (Figure 3(a)), the chord vocabulary remains relatively stable throughout, although here, too, one can observe a slight increase in entropy towards the end of the timeline (chord frequencies become even more uniform). Choro (Figure 4(a)), in contrast, shows a more mixed behavior, with some local fluctuation (as opposed to US-Pop), but no clear trend (as opposed to J-Pop).

Probability FM9 Gsus4 CM7 Bm7(b5) Bb Fm A D Em7 FM7 C7 B7 Am7 D7 A7 Dm7 E Em G F Dm G7 E7 C Am (a) Chord frequencies.G -> Gsus4 Am7 -> FM7 Dm7 -> FM7 FM7 -> Dm7 G -> Em7 FM7 -> F F -> FM7 Em -> Em7 Em7 -> C G -> Dm7 G -> E Am7 -> Am Dm7 -> F Dm7 -> Dm Em -> G F -> Dm7 G -> G7 E7 -> Em E7 -> E C -> Am C -> Em F -> Dm Am -> F Dm -> F G7 -> G (b) Substitution probabilities.

Evolution of substitution probabilities

The evolution of substitution probabilities is shown in Figures 2(b as the left and right plots in Figures Figures 2-4 are clearly correlated and one can see a relatively high entropy of chord substitution probabilities towards the ends of the respective timelines. This is particularly strong in the case of J-Pop and, to some extend, also in Choro.

Chord-symbol distances

Finally, we relate the substitution probabilities to the three chord distances defined above. For each of the three corpora, we correlate the mutation probability curves, and qualitatively interpret the resulting hierarchical clustering.

The results are clearest for the case of J-Pop (Figure 5). Here, two distinct clusters emerge, revealing two types of chord mutation dynamics. Their meaning, however, is less clear, as the clusters do directly correspond toUS-Pop again maintains a relatively stable scenario. one of the three chord distances introduced above.

For US-Pop (Figure 6), the clustering is less pronounced, although here, too, two clusters appear. In contrast to J-Pop, their meaning is better interpretable: the smaller cluster contains mostly (but not exclusively) mutations between chords that are relative to one another (e.g., G:min and Bb:maj), (applied) dominants being replaced by their tonics (e.g. A:maj and D:maj), or, interestingly, stepwise downward mutations (e.g., D:min and C:maj). The second cluster contains more subdominant-to-tonic patterns than the fist (e.g., G:maj and D:maj), but also a number of mutations between relative chords. Moreover, a number of stepwise upward replacements can be found (e.g., G:maj and A:min).

In the case of Choro (Figure 7), there are three large clusters, and the picture is more mixed. The first cluster in the upper left constists of chord inversions (e.g., D:min/b3 and D:min). In the second (middle) cluster, we see chord inversions as well, but also dominant replacements and chromatic alterations (D:min and D:7), while the third cluster in the bottom right exclusively contains chord mutations that maintain the same root (leading to a minimal line-of-fifths distance).

Summary and conclusion

In this study, we have employed a statistical evolutionary model to understand stylistic changes in chord usage in popular music from different cultural contexts. Utilizing a common representation for harmonic units in all corpora, we have observed historical changes in chord frequencies as well as the evolution of substitution probabilities in chord 4-grams.

Our results indicate stylistic differences in how patterns change over time, but their relation to chord distances is less pronounced than anticipated. Further research needs to investigate to what extend this needs to be attributed to the different data sources, and to what extend these are musical factors. Moreover, future work needs to expand in several directions, specifically testing whether our findings can be replicated in different repertoires, and looking into how cross-cultural influences can be incorporated in the analyses.