=Paper=
{{Paper
|id=Vol-1405/paper-07
|storemode=property
|title=From Sensors to Songs: A Learning-Free Novel Music Recommendation System using Contextual Sensor Data
|pdfUrl=https://ceur-ws.org/Vol-1405/paper-07.pdf
|volume=Vol-1405
|dblpUrl=https://dblp.org/rec/conf/recsys/SenL15
}}
==From Sensors to Songs: A Learning-Free Novel Music Recommendation System using Contextual Sensor Data==
https://ceur-ws.org/Vol-1405/paper-07.pdf
From Sensors to Songs: A learning-free novel music
recommendation system using contextual sensor data
Abhishek Sen, Martha Larson
Intelligent Systems
Delft University of Technology, Netherlands
a.sen@student.tudelft.nl, m.a.larson@tudelft.nl
ABSTRACT process of discovering music remains a tedious and unpleas-
Traditional approaches for music recommender systems face ant experience. With a general trend towards streaming mu-
the known challenges of providing new recommendations sic as opposed to downloading music, many music services
that users perceive as novel and serendipitous discoveries. aim to learn user music tastes and listening behaviors to
Even with all the music content available on the web and provide personalized music recommendations. The assump-
commercial music streaming services, discovering new music tion of the learning approach, namely, that past behavior
remains a time consuming and taxing activity for the average is a good predictor of future behavior, is certainly not ill
user. The goal for our proposed system is to provide novel founded. Such services will certainly satisfy users looking for
music recommendations based on contextual sensor infor- a highly predictable music experience. However, users inter-
mation. For example, contextual place information can be ested in expanding their music horizons will not be satisfied
inferred with intelligent use of techniques such as geo-fencing by algorithms that rely on previous listening history or pref-
and using lightweight sensors like accelerometers and com- erences (artist/genre), since they do not support new music
pass to monitor location. The inspiration behind our system discovery. Such algorithms fail to provide the serendipity
is that music is not in the past, neither in the future, but that is extremely important for users to discover music that
rather enjoyed in the present. For this reason, the system is new, but is also not completely alien to them. Instead, to
does not rely on learning the user’s listening history. Raw design the system presented here, we make a new assump-
sensor data is fused with information from the web, passed tion. We consider music listening to be independent of the
through a cascade of Fuzzy Logic models to infer the user’s past (history) or the future (prediction) and instead consider
context, which is then used to recommend music from an it as a function of the present (current context). We use the
online music streaming service (SoundCloud) after filtering term context to refer to the sum of a user’s experience at a
out songs based on genre preferences that the user dislikes. given moment, including place, surroundings, activities that
This paper motivates and describes the design for a mobile the user is currently pursuing and atmospheric effects on the
application along with a description of tests that will be user’s mood. We assume that listeners have similar expec-
carried out for validation. tations of which music fits a particular context. We rely on
the idea that this collective conception of ‘music that fits
a moment’ will provide users with a sense that the recom-
Categories and Subject Descriptors mendations of our system fit their current needs, and at the
H.3.3 [Information Storage and Retrieval]: Information same time allow them to discover music that they would not
Search, Retrieval and Filtering have otherwise found themselves.
Keywords 2. RELATED WORK
Context-aware, Music recommender systems, Fuzzy logic, There is a large volume of prior research in the field of
Sensor data fusion context-aware music recommender systems (e.g., [2], [?],
[10], [7]). Bonnin and Jannach present a comprehensive lit-
1. INTRODUCTION erature survey on automated playlist generation and cate-
Music plays a central role in the daily lives of many people. gorize existing approaches in [1]. They mention the impor-
Today, music streams are readily available through services tance of context in automatic playlist generation and also
such as YouTube, Spotify and Apple Music. However, the how similarity-based algorithms are an obvious approach
when the system’s goal is to maximize the homogeneity of
the playlist. As a downside, serendipity and diversity are
negatively affected since most songs recommended will be of
a similar type, i.e., the same with respect to artist or genre.
One of their core recommendations for future research is to
assess multiple criteria at the same time and explore the
trade-offs between homogeneity and diversity of playlists.
Our system, explained further in Section 5, addresses these
Copyright held by the author(s).
recommendations by balancing diversity and homogeneity
LocalRec’15, September 19, 2015, Vienna, Austria. and does not rely on learning the user’s past listening be-
haviors. user action. It is also important for our system to be light-
In [10], Wang et. al propose a system that is context- weight and run efficiently and not drain the device’s battery
aware, probabilistic and learns the user’s listening habits during normal usage.
over time for better recommendations. Their system uti- Given that the context inference might not be perfect due
lizes contextual sensor data and integrates this information to ‘noisy’ sensory data, we want to give the user a choice of
with music content analysis to provide relevant music rec- playlists. As discussed in our design concept, to exploit the
ommendations per context. However, the study requires the communal behaviors of music listening across different con-
musical signal of the songs to be pre-analyzed by music anal- texts, we will generate contextual tags to retrieve music from
ysis and was also evaluated with offline music. In the version SoundCloud1 . Knees and Schedl [5] discuss tags as a form
presented in this paper, our system instead focuses on mu- of text-based approach given their community-based char-
sic metadata that is directly available and does not rely on acteristics. SoundCloud has a music database of over 100
learning the user’s listening behaviors. Okada et. al present million songs, which are richly annotated with tags. Tags of
a system in [8] that focuses on the user interface aspects a track that are related to the context provide us with evi-
of context-aware music recommender systems, an area of- dence that listeners generally associate the track with that
ten ignored by researchers. One of their core objectives is context. Contextual tag-based queries then allow us to re-
to explore how context plays a key role in a user’s listening trieve songs from SoundCloud that both, fit contexts and
behavior and how this information can be conveyed to the allow users to discover new music.
user. In the next sections, we will see how this prior work We conducted an intensive focus group study with 6 Mas-
inspired key design choices in our system. ter’s students from different faculties at the Delft University
of Technology and the feedback gave us key insights for our
design process. All of them described music discovery as
3. DESIGN CONCEPT a tedious and challenging activity even with all the music
Our main design concept is—as the title states—from sen- available on the web. They described their ideal music rec-
sors to songs. We want to recommend novel music to users ommender system would know which song to play for any
by inferring their context from sensory data. To achieve the given situation and not just based on their past history.
desired surprise and delight factor, the system should not One of their main complaints about current music recom-
have to learn the user’s music tastes and listening behavior. mender systems was that most systems tend to repeat the
We believe this non-learning characteristic of the system to same type of songs unless the user has explicitly made a
be, currently, a quite radical approach to music recommen- different selection. They were also of the opinion that even
dation. It allows users to discover new music continually though such a system might provide ‘bad’ recommendations
without any impediments, such as the need to interact fre- at times, they would simply move on to the next song and
quently with the system. Through inference of user prefer- continue listening. This insight suggested that our system
ences based on collection-wide user experiences of context, does not have to infer the user context perfectly and that we
we think the system will achieve a level of personalization could hedge our predictions by providing the user a choice of
that is ideal for music recommender systems—without the playlists for the most likely contexts. The group also men-
need to learn everything about the user’s listening history. tioned that they all had different music tastes and each had
We are aware that user music preferences are also highly their own music preferences for different contexts—this led
personal. However, instead of making the assumption that us to include a genre preferences block as shown in Figure
music recommendation is “all about personalization”, our 1 so that in addition to knowing what the user enjoyed lis-
system strives to integrate “minimum necessary personal- tening to of late, more importantly, the system “knows” the
ization”. We do this in two ways. First, we rely on the kind of music the user really does not enjoy hearing.
idea of the context as mentioned above. The situations in
which users find themselves can be expected to reflect their
lifestyles and overall music preferences for places and ac- 5. PROPOSED SYSTEM
tivities. A system like ours that relies on collective music The proposed system architecture as shown in Figure 1 is
preferences of users for specific contexts, is actually provid- the materialization of our design concept, methodology, and
ing a level of personalization, albeit indirectly. Second, we the focus group feedback. The system is divided into three
allow users a minimum degree of control, e.g., in excluding main components: context inference, music retrieval/analysis
songs from genres that the user dislikes. and music recommendation.
5.1 Context Inference
4. DESIGN METHODOLOGY Context inference as shown in Figure 1 is done by fusing
Inspired by the design concept, our system focuses on sensor data and passing it through fuzzy logic models.
providing novel music recommendations with an emphasis
on incorporating contextual user information. Our design 5.1.1 Sensors
methodology aims to inform the possibilities for a sensor- Table 1 shows a list of contextual information categories
based music recommender, with a user centered approach. and the sensors used for their inference. All the sensors used
The goal of sensors embedded within any device is to ‘sense’ in the system are embedded inside most smartphones and
the environment for information such as temperature, accel- this trend is likely to continue with future ‘smart’ devices
eration etc. This inherent capability of sensors makes them such as smartwatches and other wearables. The system is
an ideal choice for use in interpreting user context, especially scalable and additional sensors can be easily integrated to
since most users carry ‘smart’ devices such as smartphones further improve the context inference process.
close to them at all times. This allows the system to respond
1
to major context changes implicitly without requiring any https://developers.soundcloud.com/docs/api/reference
Figure 1: Proposed playlist generation system architecture: Sensor-based novel music recommender
impacts of different weather factors on people’s mood (e.g.,
Table 1: Contextual Information [3], [4]). Weather information is integrated into the system
Category Sensors
using Yahoo API2 . Mood is a very difficult characteristic to
Location Wifi, GPS, Accelerometer, Com-
judge on a personal level—especially since everyone’s mood
pass, Cellular
could be influenced by a multitude of factors. For this rea-
Indoor/Outdoor Compass, Light
son, we decided to use the most important weather condition
Activity Accelerometer, Gyroscope factors that are thought to most universally affect people in
Date/Time System Clock a certain geographic area to get a rough estimate of which
Weather Temperature, Humidity, Pressure, quadrant of Russell’s widely accepted circumplex model of
Sunshine affect the user might be in [9]. The objective here is not to
accurately determine the user’s mood but to get a general
5.1.2 Fuzzy Logic Context Modeling idea depending on the impacts of weather on their mood.
Using fuzzy logic for context inference makes the system
extremely flexible and easy-to-understand, and allows it to 5.1.4 Situation-Based Context Model
process imprecise sensor data with ease. Motivated by our For the situation-based context model, the focus group
non-learning design concept, fuzzy logic makes it possible results informed us of the most common situations in which
to translate user-supplied human language rules into math- participants listen to music and we chose to pick the top 7
ematical values that can be used for making decisions, thus for our system: waking up, commuting, working/studying,
making the system logic easily understandable. Given the exercising, relaxing, housework and sleeping. To determine
computational challenges of fusing multi-modal sensor data, the situation, we use fuzzy rules such as the following:
fuzzy logic provides an extremely light-weight and efficient IF Activity IS Stationary AND DayOfWeek IS Weekday
technique. The Fuzzy Logic Context Modeling block com- AND TimeOfDay IS Afternoon AND Indoor/Outdoor IS
prises two main internal models as shown in Table 2. Indoor AND Place IS Office THEN Context IS Working or
Studying
The activity states that our system identifies are station-
Table 2: Fuzzy Logic Context Models ary, walking, running and driving. These activity states are
Category Inputs provided by the iOS platform. To accurately distinguish be-
Atmospheric-Based Temperature, Humidity, Pressure, tween the stationary and driving state, we utilize GPS to get
Sunshine the user’s speed and make a decision accordingly. The in-
Situation-Based Activity, Day of week, Time of day, door/outdoor sensor inputs to this model determine whether
Indoor/Outdoor, Place the user is indoors or outdoors using sensors such as light
and compass and is adapted from Zhou et. al’s proposed
system in [11]—we do not use cellular signal strength in our
5.1.3 Atmospheric-Based Context Model system due to lack of development support on iOS.
The Atmospheric-based model generates values for va-
2
lence and arousal based on prior psychology research on the https://developer.yahoo.com/weather/
Our system is currently able to identify five general place on the user interface would make the music recommenda-
categories for users—home, office, library, gym and other. tions transparent for users?”, “Which modality (mood-based-
These areas are recognized without the user having to ex- context or activity-based-context) influences the user more?”,
plicitly enter information. The system monitors significant “Is the content-based re-ranking for song relevancy necessary
location updates and marks any visited locations as possi- for recommendations?”.
ble candidates for any of the above five places in a two-step Future work in this topic includes a number of challenges
process. such as removing the hard-coding of contextual tags and
First, using the Foursquare Venues API we reverse geocode making the tag generation process dynamic. Other alter-
the location’s coordinates to the library or gym place cat- natives would be to include playlist titles and tracks within
egories. If no results are returned, the visit information is the recommendations for playlists. Our design concept and
then passed through an internal fuzzy model to determine motivations for this system however remain the same—to
the home and office place categories based on fuzzy rules. expand the musical horizons of users while making the mu-
Once a place has been annotated with a category (not al- sic discovery process less tedious and more serendipitous.
ways), the system sets up a geofence around it for a specified
radius. From this point on, any time the user enters or leaves 7. ACKNOWLEDGEMENTS
this place, a place context change event is triggered and the
The contribution of the second author was funded in part
user’s context is recomputed by processing all the other sen-
by CrowdRec (EC FP7 Project 610594).
sory inputs as shown in the Situation-based Context Model
in Table 2. If a change in user context is detected, a new
contextual song query is formulated to request a new set of 8. REFERENCES
songs from SoundCloud. The proposed technique of moni- [1] G. Bonnin and D. Jannach. Automated generation of
toring places ensures that we do not drain the smartphone’s music playlists: Survey and experiments. ACM
battery by only using the GPS when needed. Comput. Surv., 47(2):26:1–26:35, 2014.
[2] D. Griffiths, S. Cunningham, and J. Weinel. A
5.2 Music Retrieval & Recommendation discussion of musical features for automatic music
Once the user’s context has been analyzed, the next step playlist generation using affective technologies. In
is to retrieve songs from SoundCloud based on this informa- Audio Mostly Conference, AM ’13, 2013.
tion. The system performs query expansion using Last.fm [3] A. Hannak, E. Anderson, L. F. Barrett, S. Lehmann,
APIs3 to translate the fuzzy model’s output into query tags. A. Mislove, and M. Riedewald. Tweetin’ in the rain:
For example, for the ‘Exercising’ tag, Last.FM returns a set Exploring societal-scale effects of weather on mood. In
of similar tags such as ‘fitness’, ‘workout’ and ‘motivation’. International AAAI Conference on Weblogs and Social
These results are aggregated and the top-ten tags are used Media, ICWSM ‘12, 2012.
by the system for this context. For the weather-to-mood tag [4] E. Howarth and M. S. Hoffman. A multidimensional
generation process, social mood tags from [6] were used as approach to the relationship between mood and
tag seeds for the query expansion. Through this process of weather. British Journal of Psychology, 75(1):15–23,
query expansion and the ever-evolving music community on 1984.
SoundCloud, the chances of retrieving novel music is very [5] P. Knees and M. Schedl. A survey of music similarity
high. and recommendation from music context data. ACM
Post song retrieval, the system then performs music con- Trans. Multimedia Comput. Commun. Appl.,
tent analysis using the EchoNest APIs4 for each of the re- 10(1):2:1–2:21, 2013.
trieved songs by looking at parameters such as energy and
[6] C. Laurier. Automatic Classification of musical mood
valence. However, since each song is analyzed at runtime
by content-based analysis. PhD thesis, Universitat
by uploading the tracks from SoundCloud to EchoNest, this
Pompeu Fabra, 2011.
process takes some time and increases as the song duration
[7] S. Nirjon, R. F. Dickerson, Q. Li, P. Asare, J. A.
increases. For this reason, this analysis is done in the back-
Stankovic, D. Hong, B. Zhang, X. Jiang, G. Shen, and
ground while the user is listening to music. After the songs
F. Zhao. Musicalheart: A hearty way of listening to
have been retrieved from SoundCloud and/or EchoNest, the
music. ACM SynSys ‘12, 2012.
system then generates situational-based and atmospheric-
based playlists. It is important at this step that the user [8] K. Okada, B. F. Karlsson, L. Sardinha, and T. Noleto.
identifies the subtle differences between the recommended Contextplayer: Learning contextual music preferences
playlists and we plan on using visual imagery and colors to for situational recommendations. In ACM SIGGRAPH
convey the contextual differences to the user. ‘13, 2013.
[9] J. Russell. A circumplex model of affect. Journal of
personality and social psychology, 39(6):1161–1178,
6. OUTLOOK & CONCLUSIONS 1980.
The system is currently under development as a mobile [10] X. Wang, D. Rosenblum, and Y. Wang. Context-aware
application on the iOS platform on iPhone models 5S and mobile music recommendation for daily activities. In
newer running iOS 85 . The evaluation plan is to have about ACM Multimedia ‘12, 2012.
20-25 participants use the application for a week and answer [11] P. Zhou, Y. Zheng, Z. Li, M. Li, and G. Shen.
the following research questions: “What information shown Iodetector: A generic service for indoor outdoor
3 detection. In ACM SenSys ‘12, 2012.
http://www.last.fm/api
4
http://developer.echonest.com/docs/v4
5
https://developer.apple.com/library/ios/navigation/