In this paper, we present MedleyAssistant, a system to assist in the creation of music medleys from segments of existing music. Our goal is to make medley creation more accessible to novices, while still allowing them to express their own creative style. Our system addresses two key challenges in medley creation - determining which segments of music sound natural when transitioning to which other segments of music, and determining specific transition points between two given segments of music. This constrains the problem so that medleys created with our system tend to sound natural while allowing the user to be creative with music selection. We also provide a music visualization that helps users understand the musical principles of medley creation.
Author Keywords music medley; creative MIR; personalized music creation. INTRODUCTION AND MOTIVATION A music medley is a piece of music that is composed or arranged from a series of songs or musical segments. In a high quality medley, each segment tends to naturally flow into the next segment, and the transitions typically sound seamless. Medleys provide a way to create new variations of music starting from existing music. They can be used for music playback by itself or as backing tracks for media such as videos and video games. They provide a way to customize a piece of music so that different sections of the media correspond to different segments of music.
Manual creation of high quality medleys can be a challenging task and typically requires a background in music and audio editing. They are often created by musicians and DJs. The typical sequence of steps to create a medley is as follows: 1. Select a number of candidate musical segments from various pieces of music. 2. Determine a musically natural sequence of segments for the proposed medley from the above candidate set of segments. ©2018. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes.
MILC ’18, March 11, 2018, Tokyo, Japan
RELATED WORK
Recent advances in Music Information Retrieval (MIR)
techniques have given rise to intelligent musical interfaces [
SYSTEM OVERVIEW In this section, we describe the workflow and interface of MedleyAssistant. A user creates a medley using our system as follows: 1. The user provides a number of candidate songs based on their preference in music. 2. Our system provides a visualization of these songs as shown in Figure 1. Specifically, it visualizes the chord structure and tempo. This visualization could help guide the user in the subsequent steps. 3. The user chooses the first segment of the medley based on the music that they would like to be the introduction. This forms the foundation of the medley because it in turn dictates subsequent segments. The user can choose this first segment based on listening as well as using the visualization as a guide. The beginning and end of the selection snaps to the nearest beat. 4. Based on the first segment, the system assists in choosing the second segment. It estimates multiple potential candidates for the second segment (over all songs) and highlights them, as shown in Figure 2. The opacity of each gray box indicates a confidence level of how good the segment will sound. The system attempts to choose candidate segments that will musically flow well from the first segment based on chord structure, tempo, and timbre. The chosen segments will be 16 beats or less. We use a relatively small size so that the user can adjust the length to whatever they desire in the next step. The user can choose to use one of these segments, and the system will then perform the low level edits to concatenate the two segments. However, the user can alternatively choose any part of any song (even if it is not highlighted) and use that as the second segment. 5. After choosing a segment, the user can drag the segment ending boundary to adjust the length of the segment. 6. The user chooses the next segment as outlined in the previous two steps and continues this process until the medley is complete.
The interface in our system is realized as a web application using wavesurfer1, an API built on top of Web Audio API2 and HTML5 Canvas.
Visualization As shown in Figure 1, our system visualizes both the chord structure as well as the tempo for each song. This visualization provides a tool for users to better understand the segments highlighted by the system, helps users choose new segments that are not highlighted, and can serve as an educational tool to better understand the music theoretical principals of medley creation.
We visualize the chord structure of each song by overlaying
the waveform on colored blocks where each color corresponds
to a different chord. We use only Major and Minor chords,
1https://wavesurfer-js.org/
2https://www.w3.org/TR/webaudio/
so we have a total of 24 chords. The color scheme we used
in visualizing each chord is inspired by Alexander Scriabin’s
“Clavier à lumières" (“keyboard with lights") [
We visualize tempo below the waveform, with a bar representing rhythmic density of the music as a function of time. For the rhythmic feature, we first extract the dynamic tempos of the song, and then group the dynamic tempos into different sections. We use color interpolation to indicate the tempo density (i.e: deeper color for a faster tempo).
Our system demonstrates the visualization of two features (chord structure and tempo), but this can be extended to various other features. We think that different kinds of musical features could be used to assist in different forms of medley creation styles.
ALGORITHMS In this section, we describe the algorithms that we use for visualization and automatic segment selection (as described in steps 2 and 4 of the workflow in the previous section). We start with a pre-processing step in which we extract features at each beat of each song. These features are used both in the visualization and computation of the cost function described below.
Given a specific segment of the medley, which we refer to as the query segment, the goal of our algorithm is to estimate potential candidates for the next segment. We select the next candidate segment based on specific criteria as follows: 1. Acoustically smooth and seamless transitions between consecutive segments. We compute this smoothness based on a cost function between the query segment and every other segment in every song based on a sliding window. 2. Consecutive segments should conform to a harmonic progression specified by music theory rules. Based on the acoustically smooth candidates (as determined by the previous step), we select a subset of candidate segments based on a harmonic progression factor.
Feature Extraction
We first estimate beat locations in each song by applying
beat tracking on the onset envelope of the audio signal. At
every beat of every song, we compute timbral features
(Melfrequency cepstrum coefficients), signal energy level (root
mean square), and harmonic features (chroma vectors). We
also compute tempo over a window around each beat. We
compute these features using librosa [
Dt (S1; S2) st sm sr Acoustic Smoothness Given the query segment, S1 of length N, and the potential candidate segment S2 of length M = 16, our goal is to compute the optimal transition point from S1 to S2. We do this by computing a cost function between four-beat windows sliding over both S1 and S2. We define the optimal transition point by the location of the sliding windows with the lowest cost, which we refer to as the highest acoustic smoothness. We consider four beat sliding windows starting from beat 14 N to beat N 3 of S1, and beat 1 to beat 34 M of S2. We do not consider the beginning of S1 and the ending of S2 in order to ensure that the resulting combination of S1 and S2 retain at least a part of each segment.
Our cost function over a four-beat sliding window of S1 to S2 is:
3 C(S1i; S2 j) = a å Dc(CS1 [i + k];CS2 [ j + k]) k=0
3 åk=0 Dm(MS1 [i + k]; MS2 [ j + k]) Dr(RS1 [i 4 : i 1]; RS2 [ j : j + 3]) (1) Host. These estimated chords are used in the visualization under the waveform.
Where Dc denotes the cosine distance of the chroma features, Dm denotes the Euclidean distance of the timbre features, Dr denotes the difference of root mean square energy, and Dt denotes the tempo difference. a, b , g, and d are tuning factors, whereas sm, sr, and st represents the standard deviations. Figure 3 illustrates the computation of the cost function. We compute our tuning factors apriori as follows. The goal is to define the tuning factors that help indicate acoustic smoothness. By definition, the transitions between consecutive segments of a given song are maximally smooth. Therefore we compute the cost function between all consecutive segments of a number of songs using a number of different combinations of tuning factors (each tuning factor can vary from 0 to 1:0). We choose the combination of tuning factors that on average yields the lowest cost.
When we compute the cost function for segments S1 and S2, we choose the optimal transition point between S1 and S2 based on the i and j that yield the lowest cost. Given the optimal i and j, the optimal transition between the segments is to go from beat i 1 of S1 to beat j of S2.
The query segment S1 has a cost with respect to each segment S2 (associated with the optimal transition point) of each song. We choose all of the segments S2 that have a cost under a threshold as candidate segments for the next step.
Harmonic Progression Factor After obtaining a set of candidate segments with acoustically smooth transition points from S1, we determine which of these candidates yield a music theoretically valid harmonic progression when transitioning from S1.
Given candidate beat b1 in S1 that connects to beat b2 in S2, we analyze the four-beat harmonic progression from S1[b1 1 : b1] to S2[b2 : b2 + 1], namely the last two beats of the transition point in S1 and the first two beats in transition point in S2, as well as the two-beat harmonic progression from S1[b1] to S2[b2]. We assign a score Qb1;b2 as: (2) + Ppop(S1[b1
1 : b1]; S2[b2 : b2 + 1])
Where P5th is the chord transition probability from the last
beat of the transition point in S1 to the first beat of the transition
point in S2 based on a circle of fifth, and Ppop is the four-beat
harmonic transition probability from the last two beats of the
transition point in S1 to the first two beats of the transition
point in S2, trained from the SALAMI dataset [
CONCLUSION We present medleyAssistant, an interactive music medley creation system that enables users to create personalized music medleys. Our informal pilot study showed that our interface makes medley creation significantly easier for novices. We believe that it could be a useful tool for experts as well as it could help them create medleys more quickly.