Modern music platforms like Spotify support users to interact with music through diferent interactive tools; from creating a playlist to liking or skipping a song. A prominent feature of such platforms is interacting with users by allowing them to react to music via likes and skips. Some video sharing interfaces like Niconico and Bilibili allow users to view and add overlaid commentary on videos in a synchronized fashion creating a sense of shared watching experience. However, integration of emoticons with real-time music has been rare. In this work, we propose additional channels for interacting with music through emoticons. Emoticons have been widely accepted and integrated as a medium of communication and expression, especially in text. It conveys more information than its matching text and is space optimal. We aim to integrate emoticons into an interactive music-listening interface. We believe that emoticon representation of music allows for a finer granularity in representing emotions and provides users with additional options to interact with music. We propose to build an interface which presents emoticon representation of music with basic music player functionalities.
1. Introduction chine learning to sentiment analysis [1, 2, 3], and music interaction [4, 5]. Despite the progress, we are yet to see an interface that combines music and emoticons. Emoticons have played a major part in the sentiment analysis domain, especially in understanding emotions from text or tweets. Emoticons also have been widely integrated into text messages and lend a meaningful value in determining context in text and natural language processing applications [6]. In our work, we build an emoji-informed interface, through which the emotion of the music is displayed in real time and users can also input their emoticons associated with a music piece. First, we developed a back-end machine learning system that decodes emoticons from audio by using lyrics as a proxy. Secondly, we design an interface which simultaneously displays audio properties with their corresponding emoticons. This interface is built on top of the back-end ML system as it uses the output emoticons from the system to display with music. Finally, we extend the interface to enable the user to interact with the music they are listening via emoticons in real-time. This additional feedback from the user interaction is then used to retrain our model and improve the performance of our machine learning system. CEUR htp:/ceur-ws.org ISN1613-073 © 2023 Copyright for this paper by its authors. Use permitted under Creative
CEUR
Workshop Proceedings (CEUR-WS.org)
Our design is diferent from SmartVideoRanking [ 7] and MusicCommentrator [8], both of which estimate ments by the users rather than the metadata from the audio itself. 2.
Methodology Our system is designed for interactive demonstration of musical emotions through machine learning and emoticons, which contains three parts. First, preparing a fresh dataset with matching emoticon labels for a song by leveraging the underlying time-annotated lyric representation. Second, training a machine learning model via supervised learning by representing audio signals as corresponding spectrograms and using the generated emoticon symbols as target labels. Finally, we integrate this system into our interactive display which utilizes the music-emoticon pair as a starting point and follows up by enabling users to interact with music by selecting their preferred emoticons. We dive into some of the details of each of these sections below and present an outline of the system in
We have used the DALI dataset [
We first transform audio signals to spectrograms via short-time fourier transforms. It has been studied before that spectrograms ofer a rich representation of audio [11]. Then, we utilize the spectrogram-emoticon pair to train our model. Since we’ve represented audio signals as spectrograms, we leverage transfer learning to extract information.
Transfer Learning in audio has been mainly focussed on pretraining a model on a large corpus of audio datasets. We follow an approach similar to [12] where we leverage the power of transfer learning shifting focus from audio datasets to image datasets. We train DenseNet [13] and ResNet [14] which are convolutional neural network (CNN) architectures trained on the ImageNet [15] dataset.
EmoMusic Player Song Level Emoji
Pop-Up
Reactions User Reaction
Paragraph Level Emoji Both DenseNet and ResNet are fine-tuned to predict 62 emoticons. To accomplish this, we add a fully connected layer followed by a sigmoid layer to obtain class proba- Figure 2: Interface with real-time emoticon representation of bilities. This subprocess is represented in Figure 1 titled audio alongside user enabled reactions. Training. 3. Interface Design and with real-time audio playback. The emoticon icon opens a pop up for users to input their selection of emotiIn our proposed integrated music player design, we aim cons. The pop up icon on the audio waveform section to build a web based music player that includes basic fea- inputs user interaction for real-time feedback while the tures like Play/Pause, Next/Previous, Playlist etc. When pop-up icon embedded in the player(horizontally to the a song is playing, we intend to display the corresponding song title) provides a song level emoticon feedback. From audio waveform, current emoticon representation (de- these user interactions, we intend to improve the model’s rived from the paragraph based annotation), and song performance by re-training our model. A visualization level emoticon representation. In addition, we enable the of the music player is shown in Figure 2. user to interact with the music by selecting emoticons.
We propose to build a two-level interaction; with a song