We present two novel music interaction systems developed for casual exploratory search. In casual search scenarios, users have an ill-de ned information need and it is not clear how to determine relevance. We apply Bayesian inference using evidence of listening intent in these cases, allowing for a belief over a music collection to be inferred. The rst system using this approach allows users to retrieve music by subjectively tapping a song's rhythm. The second system enables users to browse their music collection using a radio-like interaction that spans from casual mood-setting through to explicit music selection. These systems embrace the uncertainty of the information need to infer the user's intended music selection in casual music interactions.
When interacting with a music system, listeners are faced
with selecting songs from increasingly large music
collections. With services like Spotify, these libraries can include
many songs the user has never heard of. This retrieval is
often a hedonic activity and may not serve a particular
information need. Users do not always have a song in mind
and are often just interested in setting a mood or nding
something `good enough' [
By asking `What would this user do?', we can develop a
likelihood model of user input within an interaction. With
Bayes theorem, this allows for an uncertain belief over a
music space to be inferred. Users can provide evidence of
their listening intent as part of a casual music interaction,
not needing to be fully engaged in the music retrieval. This is
an explicitly user-centered approach, focusing on how a user
will interact with the system. Both the systems discussed
here have been iteratively developed by comparing real user
behaviour against that predicted by the user input models.
We present two novel music retrieval systems which explore
two challenges with this approach: i) how to correctly
interpret evidence which may be subjective and ii) how to allow
users to set their current level of engagement:
i) `Query by Tapping' is a music retrieval technique where
users tap the rhythm of a song in order to retrieve it [
User 2
In this section we describe our e orts to model the
subjectivity of rhythmic queries, yielding a query by tapping system
for casual music retrieval which can be trained to users to
account for their subjective querying style. After training
the system, a user can tap a rhythm to re-order their music
collection by rhythmic similarity to their query. The top 20
highly ranked results are listed on-screen as a music playlist,
from which the user could also then select a speci c song.
Query by tapping provides an example of a casual music
interaction which su ers from subjective queries. In
mobile music-listening contexts, it can often be inconvenient
for users to remove their mobile device from their pocket
or bag and engage with it to select music. This tapping of
music as a querying technique for music is depicted in gure
2. Tapping a rhythm is already a common act and rhythm
is a universal aspect of music [
In order to assign a belief to the songs in the music
collection given a rhythmic query, we compare the query to
those predicted by the user input model. This comparison is
done using the edit distance from string comparison
methods, scaling the mismatch penalty to the time di erences
between the rhythmic sequences [
`Query by Tapping' has received some consideration in the
Music Information Retrieval community. The term was
introduced in [
In previous implementations of QBT, each IOI is de ned
relative to the preceding one [
t querymodel
Gen. Model Baseline Dial position
Dial position
Dial position
(a)
(b)
(c)
The approach to rhythmic interaction in [
The most important metric for the system to be usable was
whether a rhythmic input produced an on-screen (top 20)
result. We asked eight participants to provide queries for
songs selected from a corpus of 300 songs which we had
complete note onset data for. Participants listened to the
songs rst to ensure familiarity and were asked to provide
training queries for each song. These training queries were
used to train the generative model using leave-one-out
crossvalidation. We use a state-of-the-art onset detection
algorithm (based on measuring spectral ux [
We consider casual search interactions as spanning a range
of levels of engagement. How much a user is willing to
engage with a system and provide evidence of their listening
intent will undoubtedly vary with listening context. An
interaction which is xedly casual would be as problematic
as one which requires a user's full attention, with users
unable to take control when they wish to. An example of this
would be old analogue radios { whilst they o er a simple
music interaction, users have limited control over what they
hear. Previous work by Hopmann et al. sought to bring the
bene ts of interaction with vintage analog radio to modern
digital music collections [
We explore how the inference of listening intent can be
conditioned upon the user's level of engagement, with the music
interaction spanning from casual mood-setting through to
speci c song selection. While it would be desirable to bring
the simplicity of radio-like interaction to modern music
collections, mapping a modern music collection to a dial such as
in gure 5 would require prolonged scrolling. An alternative
would be to instead support scrolling through an overview of
the music space however this removes granularity of control
from the user, leaving them unable to select speci c items.
We developed a radio-like system called FineTuner that
allows users to navigate their music, which is arranged along
a mood axis. Users can `tune in' to a mood to hear
recommended songs based on their listening history. FineTuner
allows the user to assert control over the music
recommendation by applying pressure to a sensor. This enables users to
seamlessly transition from a casual style of interaction akin
to a radio to controlling styles such as specifying a particular
sub-area of interest in a music space, or even selecting
individual songs. FineTuner provides a single interaction which
supports casual search through to fully engaged retrieval.
Our system enables both casual and engaged forms of
interaction, giving users varying degrees of control over the
selection of music. In casual interactions where users apply less
pressure, the system can become more autonomous { making
inferences from prior evidence about what the user intended.
This handover of control was termed the `H-metaphor' by
Flemisch et al. where it was likened to riding a horse { as
the rider asserts less control the horse behaves more
autonomously [
As the user begins an interaction, they have not applied pressure and therefore are not asserting control over the system. The inferred selection is thus broad, covering an entire region of their collection and is biased towards popular tracks ( g. 4a). The music in the inferred selection is visualised by randomly sampling tracks from it and drawing beams from the dial position to the album art. The user may press in the knob to accept the selection and the sampled track is played. At low levels of assertion it is likely that most tracks played would be highly popular tracks. This behaviour is a design assumption, users may want the system to use other prior evidence. When the user applies pressure, the system interprets this as an assertion of control. The inferred selection is smaller and the spread of beams becomes narrower, the album art visualisation zooms in to show the smaller selection ( g. 4b). This selection is a combination of evidence from the dial position with prior evidence i.e. their last.fm music history. When users fully assert control (max. pressure), they navigate the collection album by album ( g. 4c) and can make exact selections. By varying the pressure, users seamlessly move through this continuous range of control. The smooth change in engagement is achieved using a simple model of user input. We assume that in an engaged interaction, users will point precisely at the song of interest (as in g. 4c). For more casual selection, we assume that users will point in the general area (mood) of the music they want, modelled using a normal distribution as in ( g. 4b). As less pressure is applied the distribution is widened, leading to less precise selection and a greater role for a prior belief over the music collection such as listening history.
The scenarios explored here involve casual music retrieval, where users have an ill-de ned information need and browse for hedonic purposes or to satis ce a music selection. In these cases, considering what input a user would provide for target songs and inferring selections is an intuitive approach which avoids the issue of de ning relevance. We show two music interactions which support the uncertain selection of music, inferred from casual user input such as tapping a rhythm or turning a radio dial.
We have shown that modelling user input for inferring music selection can address issues of subjectivity by taking a user-centered approach to model development. The model can be iterated by comparing its predictions against actual user behaviour. Accounting for this subjectivity can yield signi cant improvements in retrieval performance as well as creating a more personalised search experience. A key feature of the second system, FineTuner, is its ability to span seamlessly from casual search scenarios, such as satis cing, through to more explicit selections of music. By conditioning the inference upon the user's level of engagement, we are able to interpret the same input space (in this case the dial) according to the current context.
Our approach to casual music interaction empowers the user to enjoy their music while expending as much or as little e ort in the retrieval as they wish, providing queries in their own subjective style. Instead of focusing solely on optimising the retrieval process, we consider it equally important to design retrieval systems which suit how the user currently wants to interact. By considering how users might provide casual evidence for their listening intent, we achieve music interactions as simple as tapping a beat or tuning a radio. 6.
We are grateful for support from Bang & Olufsen and the Danish Council for Strategic Research.