<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Testing a Recommender System for Self-Actualization</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Daricia Wilkinson</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Saadhika Sivakumar</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Pratitee Sinha</string-name>
          <email>psinha@g.clemson.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Bart P. Knijnenburg</string-name>
          <email>bartk@clemson.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Clemson University Clemson</institution>
          ,
          <country country="US">USA</country>
        </aff>
      </contrib-group>
      <fpage>3</fpage>
      <lpage>9</lpage>
      <abstract>
        <p>Traditionally, recommender systems were built with the goal of aiding users' decision-making process by extrapolating what they like and what they have done to predict what they want next. However, in attempting to personalize the suggestions to users' preferences, these systems create an isolated universe of information for each user, which may limit their perspectives and promote complacency. In this paper, we describe our research plan to test a novel approach to recommender systems that goes beyond \good recommendations" that supports user aspirations and exploration.</p>
      </abstract>
      <kwd-group>
        <kwd>Recommender Systems</kwd>
        <kwd>Filter Bubble</kwd>
        <kwd>Choice Overload</kwd>
        <kwd>Self-Actualization</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Recommender systems have become ubiquitous in daily user interactions across
many e-commerce websites, social networking sites and streaming services. The
main purpose of these systems is to provide users with relevant information, and
as such, much of the content consumed online is personally tailored [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
      </p>
      <p>
        Although personalized content has numerous bene ts, presenting items that
are only based on some of users' expressed preferences could hinder the e
ectiveness of the recommender, and trap users in a \ lter bubble" [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] that limits their
perspectives, discourages exploration and prevents genuine taste development.
      </p>
      <p>
        Recently, scholars have acknowledged the importance of other user-centered
factors beyond accuracy that contribute to the e ectiveness of recommender
systems [
        <xref ref-type="bibr" rid="ref12 ref6">6, 12</xref>
        ]. This shift \beyond the accuracy" has spawned investigations into
solutions that improve all aspects of the user interaction experience. For instance,
to increase understandability, researchers have suggested providing explanations
of the recommendations [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. However, these explanations could increase the
already high conformity, as users simply trust the system's explanation rather
than engaging in true understanding and exploration [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>
        This could have long-term societal consequences as the persuasive nature of
recommendations could replace human creativity and understanding [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ],
turning humans into \input" for systems rather than acknowledging the
opportunities for taste development. This paper outlines our research plan to build upon
our recent proposal for recommender systems for self-actualization- systems that
helps users in understanding their unique tastes through development and
exploration [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Algorithmic Features</title>
      <p>
        Previous research on critiquing [
        <xref ref-type="bibr" rid="ref15 ref16 ref2 ref3">2, 3, 15, 16</xref>
        ] and diversifying recommendations [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]
has already investigated better options for the Top-N suggestions, yet the focus of
these alternative methods is to provide \good recommendations". However, our
approach is fundamentally di erent, since it carefully considers the psychology
of consumer choice processes, and supports (rather than replaces) these processes
by featuring new recommendation lists. In addition to displaying a Top-N list,
our system also di ers from previous studies by simultaneously displaying the
following lists that promote exploration and taste development (for more details
see [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]): Our alternative lists will address the following issues:
Incorrect negative predictions. In conventional recommenders, items that a
system predicts that you may dislike are never shown. While they are mostly correct,
it is possible for the system to be mistaken on some. These mistakes are hard
to correct, because items with low-valued predictions are never recommended.
Presenting users with a list of things we think you'll hate will allow users to
correct or con rm low-valued predictions.
      </p>
      <p>Our list of things we think you'll hate contains items that have a low predicted
rating for this user, compared to the average predicted rating. To populate this
list, compute the di erence between the total average rating of the item and the
user predicted rating, with the following formula:
items = max(average predicted rating
user predicted rating)</p>
      <p>This allows users to correct mistakes quickly.</p>
      <p>Unknown preferences. It is di cult for recommenders to predict items for which
there is insu cient information about whether the user will like them or not.
As a result, recommender usually tailor to users' known preferences only. Rather
than only catering to known preferences, we propose to display a list of things
we have no clue about. This list consists of items with a user predicted rating for
which the system has the lowest con dence. Current recommender algorithms
do not provide con dence intervals, so for our study we estimate the system's
con dence by computing the di erence in user-predicted ratings for di erent
algorithms, e.g. matrix factorization (mf) and k-nearest neighbors (knn):
items = max(predicted ratingmf</p>
      <p>predicted ratingknn)2</p>
      <p>This allows the system to learn information about all of a user's preferences,
rather than just a subset of their preferences.</p>
      <p>
        Novel items. New items are an enduring complication in recommender systems.
Since users have yet to try them, they rarely show up among the
recommendations. Most recommender systems solve this cold start problem through content
based techniques to approximate predicted ratings. However, this solution
ignores the fact that user may at time actually be excited to try new things, even
if it does not always t their preferences [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. We propose to resolve the cold
start problem by simply presenting items with limited rating data to users who
are excited to try them. These \hipster" users are likely to appreciate things
you'll be among the rst to try, and their feedback on these items will help to
populate the available information and hence, improve the system. We detect
these \hipster" users by detecting their high percentage of top-rated items with
very few ratings, and then show them more of such items.
      </p>
      <p>users = max(% top rated items with (#ratings &lt; threshold))</p>
      <p>items = min(#ratings)
Controversial items. Recommenders usually identify a set of users that are
similar to the current user, and then calculate recommendations based on the
preferences of these nearest neighbors. This often leads to recommendations that the
neighbors unanimously like. These \safe" recommendations do not challenge a
user's tastes beyond what is generally agreed upon as \good" among like-minded
users. However, these neighbors may not always be an unvaried group, and there
may be certain polarizing items that some of them really like, but others hate.
Our fourth feature will detect these polarizing items. This list of things that are
controversial can help users to develop their unique tastes.</p>
      <p>Among the four proposed features, identifying polarizing items is arguably
the most challenging from an algorithmic perspective. The simplest approach to
identify polarizing items is to select items that have the highest rating variability
or range (rather than average) among the neighbors:</p>
      <p>items = max(var(neighbors0 predicted ratings))</p>
      <p>A more sophisticated approach would be to cluster the identi ed neighbors
based on their ratings, and then select items that best discriminate between
clusters.</p>
      <p>The proposed features could improve recommenders? ability to support
people in life-altering decisions (e.g. choosing an education, a job, an insurance plan,
or a retirement fund) where it is important that they develop a strong sense of
determination about the chosen path. The features would also improve
recommenders? ability help people make lifestyle choices (e.g., about music, movies, or
fashion) based on carefully developed personal tastes. Our proposed plan to test
and evaluate these features in a movie recommender system are outlined below.
The goal of our research is to develop, test and evaluate a recommender
system that supports rather replaces the decision-making process for users. Our
proposed features that were mentioned in the previous section will be tested
alongside a traditional Top-N recommender. The system will have the
capability to display a Top-N recommendation list, as well as the lists of the four
new features. We will train the system using the MovieLens dataset. An online
experiment will be conducted to test the RSSA features.
The experiment will be conducted on Amazon Mechanical Turk (MTurk) with at
least 300 participants. Including the traditional \Top-N only" recommendations,
the experiment will also test the four RSSA features in combination with a Top-N
list (see Fig. 1).</p>
      <p>In our study, participants will see two lists of 10 items: one list will be the
traditional Top-10 (\Things you might like"), while the other list will be
manipulated between-subjects with the following ve conditions:
{ \More things you might like"; i.e. the next 10 recommendation (Top-11-20)
{ \Things we think you will hate"
{ \Things we are not sure about"
{ \Things you'll be among the rst to try"
{ \Things that are controversial"</p>
      <p>
        After being randomly assigned to one of the ve experimental conditions,
participants will be asked rate 15 movies that they have seen before, to use as
a base for their recommendations. Next, we will show them the
recommendations the Top-10 list of \things you might like" on the left, while the list on the
right will feature 10 items that are based on the randomly selected experimental
condition. At this point, we will ask participants to rate the movies from the
two lists. After this nal round of rating we will update the two lists, and ask
participants to select one movie that they would watch right now. Finally,
participants will be asked to complete a questionnaire to evaluate their experience
with using the system. The behavioral and objective aspects to be evaluated
are outlined in Table 1. We will adopt highly validated questionnaire items from
previous studies [
        <xref ref-type="bibr" rid="ref1 ref10 ref11 ref19 ref7 ref8">1, 7, 8, 10, 11, 19</xref>
        ] and develop additional scales along the lines of
the Knijnenburg et al. user experience framework for recommender systems [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]
which will contribute to the theory of recommender systems evaluation.
      </p>
      <p>
        Existing scales [
        <xref ref-type="bibr" rid="ref1 ref10 ref11 ref19 ref7 ref8">1, 7, 8, 10, 11, 19</xref>
        ]
      </p>
    </sec>
    <sec id="sec-3">
      <title>Conclusion</title>
      <p>In this paper, we describe a new direction for recommender systems that move
towards supporting our aspirational selves rather than pushing content to users
based on their history. We outline our research plan for developing and evaluating
the new interface and algorithmic features. Aside from this, we are also working
with several companies and organizations to build these new features into real-life
recommenders. We believe that our Recommender Systems for Self-Actualization
acknowledges the multidimensionality and evolving nature of human beings, and
can fundamentally change the way recommender systems are used.</p>
    </sec>
    <sec id="sec-4">
      <title>Acknowledgments</title>
      <p>This research was supported in part by the NSF award IIS 1565809.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>D.</given-names>
            <surname>Bollen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. C.</given-names>
            <surname>Willemsen</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Graus</surname>
          </string-name>
          .
          <article-title>Understanding choice overload in recommender systems</article-title>
          .
          <source>In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys '10</source>
          , pages
          <fpage>63</fpage>
          {
          <fpage>70</fpage>
          , New York, NY, USA,
          <year>2010</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>L.</given-names>
            <surname>Chen</surname>
          </string-name>
          and
          <string-name>
            <given-names>P.</given-names>
            <surname>Pu</surname>
          </string-name>
          .
          <article-title>Interaction design guidelines on critiquing-based recommender systems</article-title>
          .
          <source>User Modeling</source>
          and
          <string-name>
            <surname>User-Adapted</surname>
            <given-names>Interaction</given-names>
          </string-name>
          ,
          <volume>19</volume>
          (
          <issue>3</issue>
          ):
          <volume>167</volume>
          {
          <fpage>206</fpage>
          ,
          <string-name>
            <surname>Aug</surname>
          </string-name>
          .
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <given-names>L.</given-names>
            <surname>Chen</surname>
          </string-name>
          and
          <string-name>
            <given-names>P.</given-names>
            <surname>Pu</surname>
          </string-name>
          .
          <article-title>Critiquing-based recommenders: Survey and emerging trends. User Modeling</article-title>
          and
          <string-name>
            <surname>User-Adapted</surname>
            <given-names>Interaction</given-names>
          </string-name>
          ,
          <volume>22</volume>
          (
          <issue>1-2</issue>
          ):
          <volume>125</volume>
          {
          <fpage>150</fpage>
          ,
          <string-name>
            <surname>Apr</surname>
          </string-name>
          .
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>G.</given-names>
            <surname>Friedrich</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Zanker</surname>
          </string-name>
          .
          <article-title>A taxonomy for generating explanations in recommender systems</article-title>
          .
          <source>AI Magazine</source>
          ,
          <volume>32</volume>
          (
          <issue>3</issue>
          ):
          <volume>90</volume>
          {
          <fpage>98</fpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>F.</given-names>
            <surname>Gedikli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Jannach</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Ge</surname>
          </string-name>
          .
          <article-title>How should i explain? a comparison of different explanation types for recommender systems</article-title>
          .
          <source>Int. J. Hum.-Comput</source>
          . Stud.,
          <volume>72</volume>
          (
          <issue>4</issue>
          ):
          <volume>367</volume>
          {
          <fpage>382</fpage>
          ,
          <string-name>
            <surname>Apr</surname>
          </string-name>
          .
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <given-names>C.</given-names>
            <surname>He</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Parra</surname>
          </string-name>
          , and
          <string-name>
            <given-names>K.</given-names>
            <surname>Verbert</surname>
          </string-name>
          .
          <article-title>Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities</article-title>
          .
          <source>Expert Syst. Appl.</source>
          ,
          <volume>56</volume>
          :9{
          <fpage>27</fpage>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bostandjiev</surname>
          </string-name>
          ,
          <string-name>
            <surname>J.</surname>
          </string-name>
          <article-title>O'Donovan, and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Kobsa</surname>
          </string-name>
          .
          <article-title>Inspectability and control in social recommenders</article-title>
          .
          <source>In Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys '12</source>
          , pages
          <fpage>43</fpage>
          {
          <fpage>50</fpage>
          , New York, NY, USA,
          <year>2012</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. J.</given-names>
            <surname>Reijmer</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M. C.</given-names>
            <surname>Willemsen</surname>
          </string-name>
          .
          <article-title>Each to his own: How di erent users call for di erent interaction methods in recommender systems</article-title>
          .
          <source>In Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys '11</source>
          , pages
          <fpage>141</fpage>
          {
          <fpage>148</fpage>
          , New York, NY, USA,
          <year>2011</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Sivakumar</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Wilkinson</surname>
          </string-name>
          .
          <article-title>Recommender systems for self-actualization</article-title>
          .
          <source>In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys '16</source>
          , pages
          <fpage>11</fpage>
          {
          <fpage>14</fpage>
          , New York, NY, USA,
          <year>2016</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. C.</given-names>
            <surname>Willemsen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Gantner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Soncu</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Newell</surname>
          </string-name>
          .
          <article-title>Explaining the user experience of recommender systems</article-title>
          .
          <source>User Modeling</source>
          and
          <string-name>
            <surname>User-Adapted</surname>
            <given-names>Interaction</given-names>
          </string-name>
          ,
          <volume>22</volume>
          (
          <issue>4-5</issue>
          ):
          <volume>441</volume>
          {
          <fpage>504</fpage>
          ,
          <string-name>
            <surname>Oct</surname>
          </string-name>
          .
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <given-names>A.</given-names>
            <surname>Kobsa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Knijnenburg</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Livshits</surname>
          </string-name>
          .
          <article-title>Let's do it at my place instead?: Attitudinal and behavioral study of privacy in client-side personalization</article-title>
          .
          <source>In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14</source>
          , pages
          <fpage>81</fpage>
          {
          <fpage>90</fpage>
          , New York, NY, USA,
          <year>2014</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Konstan</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.</given-names>
            <surname>Riedl</surname>
          </string-name>
          .
          <article-title>Recommender systems: From algorithms to user experience. User Modeling</article-title>
          and
          <string-name>
            <surname>User-Adapted</surname>
            <given-names>Interaction</given-names>
          </string-name>
          ,
          <volume>22</volume>
          (
          <issue>1-2</issue>
          ):
          <volume>101</volume>
          {
          <fpage>123</fpage>
          ,
          <string-name>
            <surname>Apr</surname>
          </string-name>
          .
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <given-names>J.</given-names>
            <surname>Lanier</surname>
          </string-name>
          .
          <article-title>You Are Not a Gadget: A Manifesto</article-title>
          . Thorndike Press,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>E. Pariser.</surname>
          </string-name>
          <article-title>The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think</article-title>
          . Penguin Books, New York, NY, USA,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <given-names>P.</given-names>
            <surname>Pu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Faltings</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhang</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Viappiani</surname>
          </string-name>
          .
          <article-title>Usability guidelines for product recommenders based on example critiquing research</article-title>
          . In F. Ricci,
          <string-name>
            <given-names>L.</given-names>
            <surname>Rokach</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Shapira</surname>
          </string-name>
          , and P. B. Kantor, editors,
          <source>Recommender Systems Handbook</source>
          , pages
          <volume>511</volume>
          {
          <fpage>545</fpage>
          . Springer,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <given-names>P.</given-names>
            <surname>Resnick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. K.</given-names>
            <surname>Garrett</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Kriplean</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. A.</given-names>
            <surname>Munson</surname>
          </string-name>
          , and
          <string-name>
            <given-names>N. J.</given-names>
            <surname>Stroud</surname>
          </string-name>
          .
          <article-title>Bursting your ( lter) bubble: Strategies for promoting diverse exposure</article-title>
          .
          <source>In Proceedings of the 2013 Conference on Computer Supported Cooperative Work Companion, CSCW '13</source>
          , pages
          <fpage>95</fpage>
          {
          <fpage>100</fpage>
          , New York, NY, USA,
          <year>2013</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <given-names>P.</given-names>
            <surname>Resnick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Iacovou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Suchak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Bergstrom</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Riedl</surname>
          </string-name>
          . Grouplens:
          <article-title>An open architecture for collaborative ltering of netnews</article-title>
          .
          <source>In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, CSCW '94</source>
          , pages
          <fpage>175</fpage>
          {
          <fpage>186</fpage>
          , New York, NY, USA,
          <year>1994</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <given-names>A. I.</given-names>
            <surname>Schein</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Popescul</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. H.</given-names>
            <surname>Ungar</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D. M.</given-names>
            <surname>Pennock</surname>
          </string-name>
          .
          <article-title>Methods and metrics for cold-start recommendations</article-title>
          .
          <source>In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '02</source>
          , pages
          <fpage>253</fpage>
          {
          <fpage>260</fpage>
          , New York, NY, USA,
          <year>2002</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>M. C. Willemsen</surname>
            ,
            <given-names>M. P.</given-names>
          </string-name>
          <string-name>
            <surname>Graus</surname>
            , and
            <given-names>B. P.</given-names>
          </string-name>
          <string-name>
            <surname>Knijnenburg</surname>
          </string-name>
          .
          <article-title>Understanding the role of latent feature diversi cation on choice di culty and satisfaction</article-title>
          .
          <source>User Modeling</source>
          and
          <string-name>
            <surname>User-Adapted</surname>
            <given-names>Interaction</given-names>
          </string-name>
          ,
          <volume>26</volume>
          (
          <issue>4</issue>
          ):
          <volume>347</volume>
          {
          <fpage>389</fpage>
          ,
          <string-name>
            <surname>Oct</surname>
          </string-name>
          .
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>C.-N. Ziegler</surname>
            ,
            <given-names>S. M.</given-names>
          </string-name>
          <string-name>
            <surname>McNee</surname>
            ,
            <given-names>J. A.</given-names>
          </string-name>
          <string-name>
            <surname>Konstan</surname>
            , and
            <given-names>G.</given-names>
          </string-name>
          <string-name>
            <surname>Lausen</surname>
          </string-name>
          .
          <article-title>Improving recommendation lists through topic diversi cation</article-title>
          .
          <source>In Proceedings of the 14th International Conference on World Wide Web, WWW '05</source>
          , pages
          <fpage>22</fpage>
          {
          <fpage>32</fpage>
          , New York, NY, USA,
          <year>2005</year>
          . ACM.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>