EarSketch is an online learning environment that teaches coding and music concepts through the computational manipulation of sounds selected from a large sound library. It features sound recommendations based on acoustic similarity and co-usage with a user's current sound selection in order to encourage exploration of the library. However, students have reported that the recommended sounds do not complement their current projects in terms of two areas: musical key and rhythm. We aim to improve the relevance of these recommendations through the inclusion of these two musically related features. This paper describes the addition of key signature and beat extraction to the EarSketch sound recommendation model in order to improve the musical compatibility of the recommendations with the sounds in a user's project. Additionally, we present an analysis of the efects of these new recommendation strategies on user exploration and usage of the recommended sounds. The results of this analysis suggest that the addition of explicitly musically-relevant attributes increases the coverage of the sound library among sound recommendations as well as the sounds selected by users. It reflects the importance of including multiple musical attributes when building recommendation systems for creative and open-ended musical systems.
iflter by artist, genre, instrument, or key signature, and mark them as favorites for future use (see Fig 1) and can EarSketch [1] is a computational music remixing environ- preview or copy these sounds into their code as constants. ment designed to teach music and computing concepts through the process of writing code to creatively manipulate audio loops. It is a web application that contains a code editor for students to write Python or JavaScript code using a custom API, and a Digital Audio Workstation for them to view and listen to the musical output produced by their code.
Previous analysis of EarSketch users revealed that a sense of creative ownership and expression for their work has been linked to intentions to persist in computer science education [2]. To this end, EarSketch was designed with the goal of being authentic to industry tools in terms of music production, musical content, and computing languages. It achieves this with the design of its interface and API as well as with the inclusion of a large sound library for students to explore and find sounds that are personally expressive and meaningful to them.
EarSketch contains a library of over 4,500 sounds produced by professional artists such as sound designer Richard Devine, hip-hop producer/DJ Young Guru, and additional stems from popular musicians such as Alicia Keys, Ciara, Common, Dakota Bear, Irizzary y Caraballo, Jayli Wolf, Khalid, Milk + Sizz, Pharrell Williams, and Samian. Users are able to search for sounds by name,
scripts. It was hypothesized that this was due to dificulty a unique project than when matching sounds to others in navigating the sound browser, as users reported that out of the context of EarSketch [6]. it was hard to discover groups of sounds relevant to their While the initial recommendation system, a hybrid current work. In order to address this under-utilization of model using collaborative filtering and content-based the sound library and to promote compositional diversity similarity metrics, improved the number of sounds examong its users’ projects, a recommendation system was plored by users, users have reported a lack of musical coadded to EarSketch [3]. hesion between sounds after they have already included
Diversity and coverage, measures of how diferent a contrasting elements in a project, as well as a lack of set of recommendations are from each other and how sound suggestions that facilitated specific compositional much of a set of available options is being recommended, ideas such as creating a new section of a song. This work are common design goal of recommendation systems aims to improve the recommendation system’s impact on [4]. Recommendation systems to present diverse com- sound exploration and usage by adding two additional positional material are prevalent in music production musical features as inputs: key signature and beat simplatforms with which EarSketch aligns its design goals. ilarity. These features are musically motivated in that, The EarSketch sound recommendation system was de- unlike the existing system’s use of Short-time Frequency signed to assist in the process of navigating the sound Transform, they use explicit human-understandable lalibrary by presenting relevant, novel sounds for users bels grounded in music theory. Although EarSketch does to include in their code. By giving users more easily- not include western music notation by design, each tonal accessible sound options that match the content of their sound was originally composed with a major or minor in-progress compositions, the system aims to improve key signature in mind. As such, by adding explicit key the variety of sounds that users preview and copy into labels [7] to sounds, the overall key of a user’s current their scripts. It uses collaborative filtering [ 5] and acous- project can be estimated and sounds with that key can tic similarity metrics to minimize or maximize co-usage have their recommendation scores increased. In addition and similarity scores in various combinations to generate to tonal similarity, the system can prioritize recommenrecommendation scores, which can be used for diferent dations that are rhythmically consistent with a user’s recommendations such as “Songs that Fit Your Script” project [8]. Beat detection is performed by generating a or “Others Like You Used These”. Combining multiple numerical vector representing the rhythm of each sound recommendation strategies allowed for increased user in the sound library, then computing the distance beexploration and sound usage and that users preferred tween two sounds’ vectors and factoring it into their diferent types of recommendations when freely creating pairwise recommendation scores.
By adding the above features, we aim to answer the compute the key signatures for the dataset where key following question: signatures were appropriate.1 • How does the addition of salient musical features Beats were extracted using librosa’s [12] beat track in the EarSketch sound recommendation system prediction method. The method takes an audio signal impact the diversity of sounds recommended and and predicts its tempo and beat track. Details of the used in student projects? method can be found in Daniel Ellis’ paper [13], which is the implementation used by librosa. The beat track
The contributions of this work include the augmenta- prediction provided by librosa is a series of timestamps tion of a hybrid recommendation system, combining col- indicating where a beat might be. We take these time laborative filtering with multiple aspects of feature-based stamps and construct an audio signal that is a click at audio similarity, and the evaluation of sound recommen- those time stamps. For the sake of computational and dations in a creative, open-ended task. The rest of this space eficiency when computing scores, we downsampaper will detail the process of adding the musically moti- pled the signal from 44100 Hz to 100 Hz. vated features of key signature and beat similarity to the In the paper detailing the implementation of the beat EarSketch recommendation system (the dataset, the key tracker, it is shown that the dynamic programming apsignature and beat similarity extraction, and how they proach is capable of achieving 93.4% accuracy on the were incorporated into the recommender), followed by MIREX beat tracking dataset [13]. Since we did not have the methodology and analysis of an evaluation of these ground truth annotations, we manually verified the beat recommendations on aggregate statistics of users on the predictions internally by using generated click tracks on EarSketch website. a random subset of the sound library with an informal subjective evaluation. Using 5 sets of 16 sounds at a time, 2. Implementation testers from the EarSketch development team rated the implementation as appropriately matching their perception for each example.
The EarSketch web client continuously monitors the sounds included in a user’s project as they edit their code. Once a change is detected, the recommendation system generates a set of recommendations using the newly stored list of sounds as input [6]. The output is presented to users as a list of recommendations in the sound browser (Figure 1). This section will discuss the implementations of key signature estimation and beat similarity calculation, as well as their addition to the existing EarSketch recommendation algorithm at the time of generation.
In order to extract key signatures for the clips in the sound library, we used Essentia [9], a popular software package for music information retrieval. It implements several key-profiles to estimate the key signature for a given sound such as “edmm” [10], a profile that is generally suited to estimating key signatures from electronic music and “braw” [11], a more general key signature estimation profile. In addition to the key signature, Essentia’s key signature estimator also produced a strength score indicating how strong the presence of an annotated key signature is in the sample.
Identifying the best key signature profile in Essentia was done using the annotated subset of the library described in the section above. For each profile, we compared the predicted key signatures for the subset against the ground truth annotations. The “edmm” profile stood out as the best profile since it predicted the largest number of correct annotations. Therefore, it was used to
The previous algorithm to recommend sounds to users was described in [3]. In short, sounds in the library were assigned a score
− 1 − 1 = + + where and are acoustic feature distances between a given sound and every sound in the library and is the co-usage score, i.e, a score indicating how often two sounds were used together.
In order to add key signatures into our algorithm, we compute the key signature of the project proj as the most frequent key signature label across all sound clips in a project. For a given sound clip S and its corresponding key signature S we compute it’s key signature score as = 0
otherwise {︃1, if S ∈ {(proj | relative(proj)} (1) (2) where is set to 1 if the clip’s key signature matches the project or has a relative major/minor key relationship with the project’s key signature.
To add a beat similarity score, we compute the Hamming distance [14] between two given beat tracks. This − 1 is denoted as ℎ. We assume that users might select 1We excluded purely percussive sounds and short, single shot examples - for e.g, snare samples a set of samples with varying attributes, for example, genre or instrumentation, but happen to have a consistent rhythmic structure. Hamming distances have been shown by Toussiant et al[? ] as a good measure of rhythmic similarity. Given that EarSketch time-stretches samples to match a specified tempo, we wanted to choose a similarity measure that was tempo invariant and primarily focused on the diference in how the rhythms are performed directly.
Adding key and beat information to the system was done as an addition to the score described in Equation 1: − 1 − 1 − 1 = + + + + hamm Like the co-usage and acoustic similarity scores in the initial version of the recommendation system [6], the key signature estimation and beat extraction processes are performed ofline for the whole sound library. Their results are deployed to the EarSketch web client to be retrieved for individual sound-sound pairs and used in realtime recommendations. This is done to allow for faster recommendations without the requirement for heavy audio processing while users are editing a project.
As with a previous evaluation of the recommendation sys- Figure 4: The distribution of how recommended sounds tem [6], the impact of this recommender was measured were added to projects before and after the addition of key through statistical analysis of the sounds recommended signature and beat similarity to the recommendation system. and added to projects by EarSketch users. The key signa- The figure is scaled by the number of unique user sessions in ture and beat similarity recommendation changes were the two time periods. added to the EarSketch website in October 2022. Using an analytics engine, the actions of 103,828 users before the update were recorded between July and September recommended. The higher average and lower skew of 2022, and the actions of 133,349 users after the update the distribution after the addition of key signature and were recorded between October and December 2022. beat similarity indicates that more sounds from the EarS
During each session for a given user, each unique rec- ketch library are more likely to be recommended. In the ommendation made is stored as a separate event rec- period prior to the update, a given sound would be recommendation. A separate event, recommendationUsed, ommended an average of 1790 times (1.72% of sessions). is stored when the student uses a recommended sound Comparatively, following the update a given sound would in their project by copying it directly from the sound be recommended 3240 times (2.42% of sessions). Using browser interface or by writing the name of the sound a two-sample t-test, we note that the diference in recinto their code. The density of sounds being recom- ommendation frequency across the entire library was mended is a function of recommendationUsed / recom- statistically significant ( < 0.05). mendation for each individual sound constant. We deter- When comparing the usage of recommendations across mine coverage of the sound library by the distribution both periods, we measured the frequency distribution of of unique recommendations made, as well as the rate recommendationUsed events, or the unique instances of at which these recommendations are used in student previously recommended sounds being used in projects projects. (see Fig 4). A two sample t-test shows that there is a sta
The analysis of this data suggests that the inclusion tistically significant increase in recommendation usage of musically driven features further improves the diver- frequency following the update ( < 0.05). On aversity of sounds suggested by our hybrid recommenda- age, a recommended sound was used 0.94% of times it tion system. Fig 3 depicts the frequency density of each was recommended following the update compared to the sound based on the number of times that sound was average of 0.73% prior to the update.
We also observed that the inclusion of rhythmic features coincides with a noticeable uptick in the usage of percussive loops with a larger proportion of used sounds from a recommendation being percussive. We investigated the top 10 sounds being recommended and used during both periods. We found that 6 of the 10 most frequently used recommendations after the update are categorized as purely percussive sounds. In the period prior to the update, there was no majority among the instruments in the most used recommendations. 4. Discussion and Future Work how recommendation behavior influences the song creation process for students. By rating the understanding of and satisfaction with recommendations by users with and without the musical features in a controlled setting, we can determine how efective these features are and what visual design changes are necessary to enhance the efectiveness of musically-informed recommendations.
In conclusion, this analysis of the impact that salient musical features have on EarSketch users reveals multiple insights for the design of recommendation systems and other creative systems. The use of recommendation density to compare groups shows how artifact analysis can represent trends in user interaction with a creative musical assistant, even at its most simple form. The significant change in the density of unique sound recommendations shows the efectiveness of multimodal domain knowledge on recommendation generation. As the EarSketch recommendation system either minimizes or maximizes co-usage scores as well as acoustic similarity [3], the addition of features to multiple types of recommendations shows the importance of understanding task specifications when discussing recommendations for a creative system.
EarSketch sound recommendation system, to improve the diversity and coverage of sounds that are being recommended to users and to make more musically relevant suggestions for a student’s project. We analyzed two periods of data above to identify trends in usage before and after the addition of these two musical features.
In our results, we were able to successfully demonstrate that the inclusion of these features improves the diversity and coverage of recommended sounds. By comparing the distributions of unique recommendations per sound before and after the change, we found that the 5. Acknowledgments number of recommended sounds was more evenly distributed across the sound library after the change. This This material is based upon work supported by the Namay be because the algorithm is able to pick up on more tional Science Foundation Award No. 1814083. Any opinsounds that are pertinent to a given user’s project more ions, findings, and conclusions or recommendations exfrequently. Additionally, there was a statistically sig- pressed in this material are those of the author(s) and nificant increase in how often students elected to use a do not necessarily reflect the views of the National Scirecommendation. This could be attributed to the promi- ence Foundation. EarSketch is available online at https: nence of beat similarity in the recommendation algorithm //earsketch.gatech.edu providing sounds that stylistically match a user’s current sounds and as such present more viable options to try in a given project. References
We noticed that there has been a shift in the types of recommended sounds that are more frequently used [1] B. Magerko, J. Freeman, T. Mcklin, M. Reilly, E. Livacross both periods. Following the introduction of our ingston, S. Mccoid, A. Crews-Brown, Earsketch: A updated algorithm, we found that a majority of the most steam-based approach for underrepresented popuused recommendations were percussive or primarily rhyth- lations in high school computer science education mic. We believe that this is an artifact of how the key sig- 16 (2016) 1–25. doi:10.1145/2886418. natures and rhythmic similarities of sounds are weighted [2] T. McKlin, B. Magerko, T. Lee, D. Wanzer, D. Edin the recommendation process. We speculate that stu- wards, J. Freeman, Authenticity and personal credents are largely seeking rhythmic sounds at the begin- ativity: How EarSketch afects student persistence, ning stages of the song-creating process. Given that the in: Proceedings of the 49th ACM Technical Symweighting for pitched sounds necessitates the existence posium on Computer Science Education, 2018, pp. of a key signature, the recommendation algorithm skews 987–992. doi:10.1145/3159450.3159523. heavily towards rhythmic sounds at the start of a new [3] J. Smith, D. Weeks, M. Jacob, J. Freeman, project. Additionally, users with developed projects may B. Magerko, Towards a Hybrid Recommendation prefer recommendations that do not clash with their cur- System for a Sound Library, in: Joint Proceedings rent selections, such as the percussion samples without of the ACM IUI 2019 Workshops, CEUR-WS, 2019. a key signature. In order to understand this behavior [4] C. C. Aggarwal, Recommender Systems, Springer better, we need a more in-depth user study to understand International Publishing, 2016. URL: http://link.