Traditional approaches for music recommender systems face the known challenges of providing new recommendations that users perceive as novel and serendipitous discoveries. Even with all the music content available on the web and commercial music streaming services, discovering new music remains a time consuming and taxing activity for the average user. The goal for our proposed system is to provide novel music recommendations based on contextual sensor information. For example, contextual place information can be inferred with intelligent use of techniques such as geo-fencing and using lightweight sensors like accelerometers and compass to monitor location. The inspiration behind our system is that music is not in the past, neither in the future, but rather enjoyed in the present. For this reason, the system does not rely on learning the user's listening history. Raw sensor data is fused with information from the web, passed through a cascade of Fuzzy Logic models to infer the user's context, which is then used to recommend music from an online music streaming service (SoundCloud) after ltering out songs based on genre preferences that the user dislikes. This paper motivates and describes the design for a mobile application along with a description of tests that will be carried out for validation.
H.3.3 [Information Storage and Retrieval]: Information Search, Retrieval and Filtering
Music plays a central role in the daily lives of many people. Today, music streams are readily available through services such as YouTube, Spotify and Apple Music. However, the process of discovering music remains a tedious and unpleasant experience. With a general trend towards streaming music as opposed to downloading music, many music services aim to learn user music tastes and listening behaviors to provide personalized music recommendations. The assumption of the learning approach, namely, that past behavior is a good predictor of future behavior, is certainly not ill founded. Such services will certainly satisfy users looking for a highly predictable music experience. However, users interested in expanding their music horizons will not be satis ed by algorithms that rely on previous listening history or preferences (artist/genre), since they do not support new music discovery. Such algorithms fail to provide the serendipity that is extremely important for users to discover music that is new, but is also not completely alien to them. Instead, to design the system presented here, we make a new assumption. We consider music listening to be independent of the past (history) or the future (prediction) and instead consider it as a function of the present (current context). We use the term context to refer to the sum of a user's experience at a given moment, including place, surroundings, activities that the user is currently pursuing and atmospheric e ects on the user's mood. We assume that listeners have similar expectations of which music ts a particular context. We rely on the idea that this collective conception of `music that ts a moment' will provide users with a sense that the recommendations of our system t their current needs, and at the same time allow them to discover music that they would not have otherwise found themselves. 2.
There is a large volume of prior research in the eld of
context-aware music recommender systems (e.g., [
In [
Our main design concept is|as the title states|from sensors to songs. We want to recommend novel music to users by inferring their context from sensory data. To achieve the desired surprise and delight factor, the system should not have to learn the user's music tastes and listening behavior. We believe this non-learning characteristic of the system to be, currently, a quite radical approach to music recommendation. It allows users to discover new music continually without any impediments, such as the need to interact frequently with the system. Through inference of user preferences based on collection-wide user experiences of context, we think the system will achieve a level of personalization that is ideal for music recommender systems|without the need to learn everything about the user's listening history.
We are aware that user music preferences are also highly personal. However, instead of making the assumption that music recommendation is \all about personalization", our system strives to integrate \minimum necessary personalization". We do this in two ways. First, we rely on the idea of the context as mentioned above. The situations in which users nd themselves can be expected to re ect their lifestyles and overall music preferences for places and activities. A system like ours that relies on collective music preferences of users for speci c contexts, is actually providing a level of personalization, albeit indirectly. Second, we allow users a minimum degree of control, e.g., in excluding songs from genres that the user dislikes.
Inspired by the design concept, our system focuses on providing novel music recommendations with an emphasis on incorporating contextual user information. Our design methodology aims to inform the possibilities for a sensorbased music recommender, with a user centered approach. The goal of sensors embedded within any device is to `sense' the environment for information such as temperature, acceleration etc. This inherent capability of sensors makes them an ideal choice for use in interpreting user context, especially since most users carry `smart' devices such as smartphones close to them at all times. This allows the system to respond to major context changes implicitly without requiring any user action. It is also important for our system to be lightweight and run e ciently and not drain the device's battery during normal usage.
Given that the context inference might not be perfect due
to `noisy' sensory data, we want to give the user a choice of
playlists. As discussed in our design concept, to exploit the
communal behaviors of music listening across di erent
contexts, we will generate contextual tags to retrieve music from
SoundCloud1. Knees and Schedl [
We conducted an intensive focus group study with 6 Master's students from di erent faculties at the Delft University of Technology and the feedback gave us key insights for our design process. All of them described music discovery as a tedious and challenging activity even with all the music available on the web. They described their ideal music recommender system would know which song to play for any given situation and not just based on their past history.
One of their main complaints about current music recommender systems was that most systems tend to repeat the same type of songs unless the user has explicitly made a di erent selection. They were also of the opinion that even though such a system might provide `bad' recommendations at times, they would simply move on to the next song and continue listening. This insight suggested that our system does not have to infer the user context perfectly and that we could hedge our predictions by providing the user a choice of playlists for the most likely contexts. The group also mentioned that they all had di erent music tastes and each had their own music preferences for di erent contexts|this led us to include a genre preferences block as shown in Figure 1 so that in addition to knowing what the user enjoyed listening to of late, more importantly, the system \knows" the kind of music the user really does not enjoy hearing. 5.
The proposed system architecture as shown in Figure 1 is the materialization of our design concept, methodology, and the focus group feedback. The system is divided into three main components: context inference, music retrieval/analysis and music recommendation. 5.1
Context inference as shown in Figure 1 is done by fusing sensor data and passing it through fuzzy logic models. 5.1.1
1https://developers.soundcloud.com/docs/api/reference
Using fuzzy logic for context inference makes the system extremely exible and easy-to-understand, and allows it to process imprecise sensor data with ease. Motivated by our non-learning design concept, fuzzy logic makes it possible to translate user-supplied human language rules into mathematical values that can be used for making decisions, thus making the system logic easily understandable. Given the computational challenges of fusing multi-modal sensor data, fuzzy logic provides an extremely light-weight and e cient technique. The Fuzzy Logic Context Modeling block comprises two main internal models as shown in Table 2.
The Atmospheric-based model generates values for
valence and arousal based on prior psychology research on the
impacts of di erent weather factors on people's mood (e.g.,
[
For the situation-based context model, the focus group results informed us of the most common situations in which participants listen to music and we chose to pick the top 7 for our system: waking up, commuting, working/studying, exercising, relaxing, housework and sleeping. To determine the situation, we use fuzzy rules such as the following:
IF Activity IS Stationary AND DayOfWeek IS Weekday AND TimeOfDay IS Afternoon AND Indoor/Outdoor IS Indoor AND Place IS O ce THEN Context IS Working or Studying
The activity states that our system identi es are
stationary, walking, running and driving. These activity states are
provided by the iOS platform. To accurately distinguish
between the stationary and driving state, we utilize GPS to get
the user's speed and make a decision accordingly. The
indoor/outdoor sensor inputs to this model determine whether
the user is indoors or outdoors using sensors such as light
and compass and is adapted from Zhou et. al's proposed
system in [
Our system is currently able to identify ve general place categories for users|home, o ce, library, gym and other. These areas are recognized without the user having to explicitly enter information. The system monitors signi cant location updates and marks any visited locations as possible candidates for any of the above ve places in a two-step process.
First, using the Foursquare Venues API we reverse geocode the location's coordinates to the library or gym place categories. If no results are returned, the visit information is then passed through an internal fuzzy model to determine the home and o ce place categories based on fuzzy rules. Once a place has been annotated with a category (not always), the system sets up a geofence around it for a speci ed radius. From this point on, any time the user enters or leaves this place, a place context change event is triggered and the user's context is recomputed by processing all the other sensory inputs as shown in the Situation-based Context Model in Table 2. If a change in user context is detected, a new contextual song query is formulated to request a new set of songs from SoundCloud. The proposed technique of monitoring places ensures that we do not drain the smartphone's battery by only using the GPS when needed. 5.2
Once the user's context has been analyzed, the next step
is to retrieve songs from SoundCloud based on this
information. The system performs query expansion using Last.fm
APIs3 to translate the fuzzy model's output into query tags.
For example, for the `Exercising' tag, Last.FM returns a set
of similar tags such as ` tness', `workout' and `motivation'.
These results are aggregated and the top-ten tags are used
by the system for this context. For the weather-to-mood tag
generation process, social mood tags from [
Post song retrieval, the system then performs music content analysis using the EchoNest APIs4 for each of the retrieved songs by looking at parameters such as energy and valence. However, since each song is analyzed at runtime by uploading the tracks from SoundCloud to EchoNest, this process takes some time and increases as the song duration increases. For this reason, this analysis is done in the background while the user is listening to music. After the songs have been retrieved from SoundCloud and/or EchoNest, the system then generates situational-based and atmosphericbased playlists. It is important at this step that the user identi es the subtle di erences between the recommended playlists and we plan on using visual imagery and colors to convey the contextual di erences to the user.
The system is currently under development as a mobile application on the iOS platform on iPhone models 5S and newer running iOS 85. The evaluation plan is to have about 20-25 participants use the application for a week and answer the following research questions: \What information shown 3http://www.last.fm/api 4http://developer.echonest.com/docs/v4 5https://developer.apple.com/library/ios/navigation/ on the user interface would make the music recommendations transparent for users?", \Which modality (mood-basedcontext or activity-based-context) in uences the user more?", \Is the content-based re-ranking for song relevancy necessary for recommendations?".
Future work in this topic includes a number of challenges such as removing the hard-coding of contextual tags and making the tag generation process dynamic. Other alternatives would be to include playlist titles and tracks within the recommendations for playlists. Our design concept and motivations for this system however remain the same|to expand the musical horizons of users while making the music discovery process less tedious and more serendipitous. 7.
The contribution of the second author was funded in part by CrowdRec (EC FP7 Project 610594). 8.