<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Music Discovery with Social Networks</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Cédric S. Mesnage</string-name>
          <email>cedric.mesnage@usi.ch</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Asma Rafiq</string-name>
          <email>a.rafiq@qmul.ac.uk</email>
          <email>q@qmul.ac.uk</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Romain P. Brixtel</string-name>
          <email>rbrixtel@info.unicaen.fr</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Simon Dixon</string-name>
          <email>simon.dixon@eecs.qmul.ac.uk</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Centre for Digital Music, Queen Mary University of</institution>
          ,
          <addr-line>London, London</addr-line>
          ,
          <country country="UK">UK</country>
          <addr-line>E1 4NS</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Faculty of Informatics, University of Lugano</institution>
          ,
          <addr-line>Lugano</addr-line>
          ,
          <country country="CH">Switzerland</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>University of Caen</institution>
          ,
          <country country="FR">France</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Current music recommender systems rely on techniques like collaborative ltering on user-provided information in order to generate relevant recommendations based upon users' music collections or listening habits. In this paper, we examine whether better recommendations can be obtained by taking into account the music preferences of the user's social contacts. We assume that music is naturally di used through the social network of its listeners, and that we can propagate automatic recommendations in the same way through the network. In order to test this statement, we developed a music recommender application called Starnet on a Social Networking Service. It generated recommendations based either on positive ratings of friends (social recommendations), positive ratings of others in the network (nonsocial recommendations), or not based on ratings (random recommendations). The user responses to each type of recommendation indicate that social recommendations are better than non-social recommendations, which are in turn better than random recommendations. Likewise, the discovery of novel and relevant music is more likely via social recommendations than non-social. Social shu e recommendations enable people to discover music through a serendipitous process powered by human relationships and tastes, exploiting the user's social network to share cultural experiences.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Categories and Subject Descriptors</title>
      <p>WOMRAD 2011 2nd Workshop on Music Recommendation and Discovery,
colocated with ACM RecSys 2011 (Chicago, US)
Copyright c . This is an open-access article distributed under the terms
of the Creative Commons Attribution License 3.0 Unported, which permits
unrestricted use, distribution, and reproduction in any medium, provided
the original author and source are credited.</p>
    </sec>
    <sec id="sec-2">
      <title>1. INTRODUCTION</title>
      <p>
        An interesting mechanism of music discovery occurs when
family and friends recommend each other music that they
discover. The emergence of Social Networking Services (SNS)
and Web Music Communities (WMC) provide us the
opportunity to develop music recommender applications to
support this mechanism. SNS are becoming increasingly
popular means for people to socialise online. Music is playing
a similar role on these platforms as in real life social
networks. It is shared, discussed, recommended and discovered
with social contacts. It is noteworthy that music
information seeking behaviour has been indicated as `highly social'
[
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. This social aspect of music should be incorporated in
current music recommender systems. WMCs like Last.fm1,
Pandora2 and Ping3 are playing a vital role in helping music
listeners to build relationships with similar music-listeners
and get recommendations based on their current music
collections. WMCs are very popular among music fans.
Music discoveries often result from passive behaviour [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
In [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] authors also indicated that music discovery was
seldom a conscious activity until the project participants were
given the task of writing diaries when they encountered new
music. Therefore, it is quite likely that many people are
interested in discovering music but not actively seeking for
it on WMCs. Possibly due to the region-speci c content
access restrictions, since sites such as Pandora can only be
used within U.S. territories and Last.fm radio is not available
without paid subscription to countries other than UK, US
and Germany. On the other hand, SNS such as Facebook4
allow social interaction with family and friends around the
globe.
      </p>
      <p>In this work, we model that music discoveries take place
1http://www.last.fm
2http://www.pandora.com
3http://www.apple.com/itunes/ping
4http://www.facebook.com
via natural di usion of music through social networks or
randomly. An experiment was conducted to reproduce this
process so that we could analyse how people respond to the
recommendations. The recommendations arising from such
processes are either randomly picked from the pool of tracks
of the data set or collaboratively from the tracks
recommended by other people on the SNS. A successful music
discovery occurs when the user of the application likes a track
that s/he has never heard before.</p>
      <p>This paper has been divided into 7 sections; section 1 is the
introduction, section 2 elaborates the rationale behind this
research experiment. In section 3, we discuss the
methodology. In section 4, the results are presented. In sections
5, 6 and 7, limitations are discussed, the research work is
concluded, and future work is proposed respectively.</p>
    </sec>
    <sec id="sec-3">
      <title>BACKGROUND AND MOTIVATION</title>
      <p>
        Research on nding new music shows that music discovery
often occurs with the personal acquaintances playing music
to the respondents, and that social networks continue to
play a vital role in music discovery in the digital age [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ].
Our social contacts may in uence our music preferences [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
Social context plays a signi cant role in improving music
recommendation algorithms [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Music can both re ect and
de ne social identity and membership in a given subculture
[
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. Information retrieval from social media aids the
collection, storage and review of music of the users. It presents
opportunities for improved music recommender systems
incorporating music preferences of the user's social contacts.
An online survey conducted by Entertainment Media
Research Company (EMRC) and Wiggin (2009) with 1,608
participants from the United Kingdom, indicated that social
networking sites are frequently used for music streaming [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
Although Last.fm uses collaborative recommendation
algorithms, it does not explicitly provide an option to restrict
recommendations to the user's social contacts rather than
the whole WMC. Interestingly, Pandora has recently
attempted to add the feature of \Music Feed" on Pandora One,
which shows the activities of friends such as likes, tracks they
are listening to and comments. It is a similar concept to Ping
but is only available to the paid subscribers and is currently
in testing phase [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. The number of active users on
Facebook5 outnumbers any of the WMCs mentioned above by a
signi cant margin. Therefore, it is more likely to nd real
life friends on Facebook as compared to the WMCs, forming
another motivation to conduct our experiment on Facebook.
In [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] authors show that collaborative ltering based on
social relationships and tags outperforms standard
information retrieval techniques by running simulations on users'
listening history. Other research has shown that music
listeners sometimes enjoy randomly ordered recommendations
[
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. We approach the problem with a di erent methodology
by conducting a live experiment in which users listen and
rate the recommendations.
      </p>
    </sec>
    <sec id="sec-4">
      <title>METHODOLOGY</title>
      <p>The aim of the experiment is to test that discoveries are
di used through the social network. The problem is treated
as a recommendation problem de ned as follows: given a
pool of items, select an item that the subject has not heard
and that is relevant to her/him. A collaborative
recommendation makes use of items rated previously by other subjects
to choose the item to be recommended. If collaborative
recommendations based on ratings by people from the social
network of the subject lead to more successful
recommendations than ratings from people not in the subject's social
network, then there is an indication that social
recommendations are more appropriate for collaborative
recommendations.
3.1</p>
    </sec>
    <sec id="sec-5">
      <title>Experimental setting</title>
      <p>
        The idea of the social shu e is to recommend tracks (see
Section 3.2 for more details) and di use discoveries through
the social network. Figure 1 shows an example of this
process when a recommendation is posted by a user in her/his
SNS. In this case, Joe gets a random recommendation, gives
it a 4 star rating, the track is then di used to his social
network. If a friend of Joe gives a high rating to the same track,
it will be di used to her network as well (in this case, the
example of Alice). If Joe's friend does not enjoy the track
and gives it 2 or less rating then the social di usion stops
and here Aleks' friends will not get this recommendation.
Each time a track is recommended, the subject of the
experiment rates the track on a 0 to 5 Likert scale; a psychometric
scale introduced by Rensis Likert [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] visually represented
as stars, where selection of 0 indicates dislike and 5 indicates
favourite tracks.
3.1.1
      </p>
      <sec id="sec-5-1">
        <title>Dataset</title>
        <p>In order to have a representative pool of available tracks
for random selection, the dataset was built from Last.fm
containing a million tracks fetched through their API. We
explored a subset of the Last.fm of about 300,000 people and
their friends. The tag pro le of each user was fetched, giving
us the list of tags they used to organize tracks, yielding a
set of more than 300,000 unique tags. For each tag the API
allows to fetch the top 50 most popular tracks tagged with
this tag. Only the set of tracks is relevant for this study.
On Facebook, we started by fetching the network of the rst
author, his friends and friends of friends interested in
participating in this experiment. Many people share music using
Youtube6 by posting links on Facebook. The advantage of
using music videos from Youtube is that the full track can
be played (if the video is available), which is not the case for
public users of Last.fm for instance which limits the playback
to 30 seconds. The limitation pertaining to Youtube videos
is that sometimes these are blocked in various countries and
5http://www.facebook.com/press/info.php?statistics
6http://www.youtube.com
removed for copyright violations. The application detects
when such errors occur and does not recommend that video
afterwards. On the Youtube API, videos were searched for
each track with the artist name and track title. We selected
the most popular video for each track and restricted to fetch
the videos tagged as \Music" only. Using this process about
a quarter of the tracks from Last.fm had a video on Youtube,
totaling around 252,000 tracks.
3.1.2</p>
      </sec>
      <sec id="sec-5-2">
        <title>Facebook application</title>
        <p>Starnet is a Facebook application fed by ratings on
random selections. A positive rating spawns di usion through
the user's social network, i.e. the shu e recommendation
becomes social. The interface (Figure 2) consists of the
current track description (title and artist name), the
music video associated, a tag cloud of the user's pro le and a
rating form consisting of 5 stars. Also, it has \next" button
to play the next track and a \bail" button which sets the
stars to 0 and plays the next track. The user is asked to
indicate whether the track is already known, in order to learn
whether it is a music discovery. We assume it is a reasonable
estimate for true unknown tracks.</p>
        <p>The subjects are the people who used the Starnet
application. A total of 68 subjects participated in this
experiment and allowed access to their social pro les. Each
subject acted as his/her own control group by getting and
rating random recommendations. In about 4 months (from the
29th June 2010 to the 18th October 2010), 31 subjects made
4966 ratings using the Starnet Application. Participation of
other 37 subjects was insigni cant in terms of ratings.
3.2</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>Recommendation Strategy</title>
      <p>The recommender system produced the following types of
recommendations:</p>
      <p>Random recommendations: The random selection is a
query which selects tracks that have not been rated by
the subject and orders them randomly.</p>
      <p>Collaborative recommendations: Selects a track
randomly from the set of tracks that have been rated with
a rating above average (i.e. greater than 2 stars) and
not yet rated by the subject. The collaborative
recommendation can be social or non-social.</p>
      <p>Social recommendations: The social recommender
selects a track randomly, from the tracks that have been
rated by friends of the subject, with a rating superior
to 2 stars.</p>
      <p>Non-social recommendations: The non-social
recommender selects a track randomly, from the tracks that
have been rated by people who are not friends of the
subject, with a rating superior to 2 stars.</p>
      <p>The likelihood for selecting random recommendation, social
recommendation and non-social recommendation is 0.5, 0.25
and 0.25, respectively.
4.</p>
    </sec>
    <sec id="sec-7">
      <title>RESULTS</title>
      <p>This section presents the results from the analysis of the
ratings made by the subjects of the experiment. The success
of a recommendation model to discover new and relevant
tracks for a subject can be evaluated without further user
input.</p>
      <sec id="sec-7-1">
        <title>Discoveries.</title>
        <p>A measurement of the success of a recommendation model
to discover new and relevant tracks for a subject is the ratio
of already discovered tracks and all rated tracks. Figure 3
represents how these sets relate for a particular user.</p>
        <sec id="sec-7-1-1">
          <title>Already discovered To be discovered</title>
          <p>New
Rated</p>
        </sec>
        <sec id="sec-7-1-2">
          <title>Poor recommendations Profile</title>
          <p>Relevant</p>
          <p>Histograms are used to compare distributions of ratings.
The x axis represents number of stars/rating and the
corresponding percentage of ratings on the y axis. Heat maps
are used to show how the ratings of a subject relates to
ratings on the same tracks from friends and non-friends. The
ratings for social and non-social recommendations of known
and unknown tracks are compared, showing that social
recommendations give better ratings than non-social ones.</p>
          <p>Figure 4 shows the ratings for the tracks known to the
subject and for those speci ed as unknown, respectively
represented as black and white bars. This shows a clear
distinction between known and unknown tracks, most unknown
tracks are disliked whereas most known tracks are liked. In
this gure we looked indi erently at all the recommenders,
in the next gure we di erentiate between collaborative and
random recommendations.</p>
          <p>Figure 5 represents the proportions of ratings for
collaborative and random recommendations. Collaborative
recommendation outperforms random recommendation as 45% of
unknown
known</p>
          <p>Col aRbaonradtoivme
2</p>
          <p>3</p>
          <p>Ratings
0
1
4
5
0
1
2</p>
          <p>3
Ratings
4
5
its recommendation get 3 star or above ratings. The
percentage drops to around 17% in the case of random
recommendations. In this visualization, an increased number of
ratings above the threshold of 3 stars is evident for
collaborative recommendations. This is due to the way
collaborative recommendations are made: only tracks which get a
rating superior to 2 stars from other users are used as
collaborative recommendations. This shows that there is some
consistency in users' ratings. We now look only at the
collaborative ratings dividing them between social and non-social
recommendations.</p>
          <p>Figure 6 represents the ratings from two types of
collaborative recommendations. The social and non-social ratings
have di erent distributions indicating that subjects react
di erently to recommendations coming from their friends
than from people they do not know. The subjects are not
aware of the source of the recommendation. More than 47%
of social recommendations get 3 or more stars as compared
to the non-social ones, (33%). In this visualization, we see
that social recommendations tend to lead to better ratings,
but we mix known and unknown tracks.</p>
          <p>Figure 7 represents the ratings of collaborative
recommendations on known and unknown tracks. 94% of the ratings
state that the track is unknown to the subject, which is why
the left histogram on Figure 7 is similar to the histogram
of Figure 6. In general, the social recommendations lead
to more unknown good recommendations (46% at least 3
stars against 34%) than the non-social ones and less bad
recommendations (40% at most 2 stars against 60%). The
histogram on the right shows ratings where the subject
speci ed she knew the track. Although, the overall rating
distributions are very di erent, the same trend of higher ratings
for social recommendations is evident.</p>
          <p>Figure 8 shows two heat maps. Each square represents the
proportion of ratings made by all users of the application on
the same tracks for each rating. The left hand heat map
represents the relation between ratings of people who are
not friends and the right one the relation between ratings of
people who are friends.The contrasted regions on both heat
map shows that people agree on ratings on the same tracks.
Interestingly, the social heat map is more contrasted,
showing that friends agree more with their ratings than people
who are not friends.
5.</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-8">
      <title>LIMITATIONS</title>
      <p>Our results show that people tend to prefer the music that
their friends prefer. One of the limitations of this work is
that it is based on the assumption that the relationships in
SNS or WMC are with real life friends and family which
might not be true for some subjects. Therefore, they might
not be very good source for music recommendation or
discovery in that case, as they might not share user's music
taste. Results suggest otherwise.</p>
      <p>
        Studies on music behaviour also suggest that sometimes
youngsters try to distance themselves from the previous
generation [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] in which case, social recommendations from older
members of the social contacts might not work well. So,
there should be an option for tuning the users' social
network for music recommendation, when such a feature is
integrated in a music recommender system.
      </p>
    </sec>
    <sec id="sec-9">
      <title>CONCLUSION</title>
      <p>The analysis of the results of the experiment on social
di usion. lead us to the following conclusions:
50
40
30
20
10
0
50
40
30
20
10
0</p>
      <p>These conclusions support the view that social di usion is a
good mechanism for music recommendation and discovery.
It is anticipated that it shall form the foundation for the
framework of better music recommender systems combined
with social media.</p>
    </sec>
    <sec id="sec-10">
      <title>7. FUTURE WORK</title>
      <p>An improved recommendation system could make use of
the genres (or tags) of the tracks previously rated by the
subject. We have been working on a di erent input for
music videos. We are now fetching the Youtube videos posted
on Facebook by registered people to the Starnet application
and their friends. This changes the settings of the
application and requires us to focus on interaction mechanisms
for people to explore a social network. The new application
shall allow users to add and remove social contacts to their
network, to tune their personalised social radio.
Another application of interest is a music recommender
system that recommends music based on the type of event and
music preferences of the people attending it. The initial
prototype of the system is available as a Facebook
Application7. People post events (such as party, wedding, etc.) on
SNS. Analysis of music taste of the attendees of the events
can enable the event organiser to make better decisions on
what type of music shall be enjoyable for most of the
attendees. However, event categorisation and determination of
suitability of music within that particular category can be a
challenge.
7http://apps.facebook.com/music_valley/</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Baker</surname>
          </string-name>
          .
          <article-title>Social Networking and Music: the impact of myspace as a Digital Platform for Discovery and Distribution</article-title>
          .
          <source>Bachelor's thesis</source>
          , Cardi School of Journalism, Media and
          <string-name>
            <given-names>Cultural</given-names>
            <surname>Studies</surname>
          </string-name>
          , Cardi University,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>S.</given-names>
            <surname>Cunningham</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Bainbridge</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>McKay</surname>
          </string-name>
          .
          <article-title>Finding new music: A diary study of everyday encounters with novel songs</article-title>
          . pages
          <volume>83</volume>
          {
          <fpage>88</fpage>
          . Austrian Computer Society, International Society for Music Information Retrieval,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>T.</given-names>
            <surname>DeNora</surname>
          </string-name>
          . Music in Everyday Life, pages
          <fpage>4</fpage>
          <lpage>{</lpage>
          8. Cambridge University Press,
          <year>2000</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>I.</given-names>
            <surname>Konstas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Stathopoulos</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Jose</surname>
          </string-name>
          .
          <article-title>On social networks and collaborative recommendation</article-title>
          .
          <source>In SIGIR '09: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval</source>
          , pages
          <volume>195</volume>
          {
          <fpage>202</fpage>
          , New York, NY, USA,
          <year>2009</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>A.</given-names>
            <surname>Laplante</surname>
          </string-name>
          .
          <article-title>Everyday life music information-seeking behaviour of young adults: an exploratory study</article-title>
          .
          <source>PhD thesis</source>
          , McGill University, Montreal, Canada,
          <year>2008</year>
          . pages
          <fpage>4</fpage>
          -
          <lpage>65</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>T.</given-names>
            <surname>Leong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Vetere</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S.</given-names>
            <surname>Howard</surname>
          </string-name>
          .
          <article-title>The serendipity shu e</article-title>
          .
          <source>In OZCHI '05</source>
          , pages
          <fpage>1</fpage>
          <lpage>{</lpage>
          4.
          <string-name>
            <surname>Computer-Human Interaction</surname>
          </string-name>
          Special Interest Group,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>D.</given-names>
            <surname>McEnnis</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Cunningham</surname>
          </string-name>
          .
          <article-title>Sociology and music recommendation systems</article-title>
          . pages
          <fpage>185</fpage>
          {
          <fpage>186</fpage>
          . Austrian Computer Society, 8th International Conference on Music Information Retrieval,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>A.</given-names>
            <surname>Minks</surname>
          </string-name>
          .
          <article-title>Growing and grooving to a steady beat: Pop music in fth-graders' social lives</article-title>
          .
          <source>Yearbook for traditional music</source>
          ,
          <volume>31</volume>
          :
          <fpage>77</fpage>
          {
          <fpage>101</fpage>
          ,
          <year>1999</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>M.</given-names>
            <surname>Siegler</surname>
          </string-name>
          .
          <article-title>Dancing to the drumbeat of HTML5, New Pandora is brilliant, beautiful</article-title>
          .
          <source>Website</source>
          ,
          <year>2011</year>
          . http://techcrunch.com/
          <year>2011</year>
          /07/12/new-pandora/.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>S.</given-names>
            <surname>Tepper</surname>
          </string-name>
          and
          <string-name>
            <given-names>E.</given-names>
            <surname>Hargittai</surname>
          </string-name>
          .
          <article-title>Pathways to music exploration in a digital age</article-title>
          .
          <source>Poetics</source>
          ,
          <volume>37</volume>
          (
          <issue>3</issue>
          ):
          <volume>227</volume>
          {
          <fpage>249</fpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>J.</given-names>
            <surname>Uebersax</surname>
          </string-name>
          .
          <article-title>Likert scales: dispelling the confusion</article-title>
          .
          <source>Website</source>
          ,
          <year>2006</year>
          . http://john-uebersax.com/stat/likert.htm.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>