<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Personalizing an Online Parenting Library: Parenting-Style Surveys Outperform Behavioral Reading-Based Models</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Mark P. Graus</string-name>
          <email>m.p.graus@tue.nl</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Martijn C. Willemsen</string-name>
          <email>m.c.willemsen@tue.nl</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Chris C. P. Snijders</string-name>
          <email>c.c.p.snijders@tue.nl</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="editor">
          <string-name>Author Keywords
Personalization; Parenting; User Experience; Cold Start;
Psychological Traits; Psychological Models; User Models</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Eindhoven University of</institution>
          ,
          <addr-line>Technology, IPO 0.17, 5600 MB Eindhoven, the</addr-line>
          ,
          <country country="NL">Netherlands</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Eindhoven University of</institution>
          ,
          <addr-line>Technology, IPO 0.20, 5600 MB Eindhoven, the</addr-line>
          ,
          <country country="NL">Netherlands</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Eindhoven University of</institution>
          ,
          <addr-line>Technology, IPO 1.[20], 5600 MB Eindhoven, the</addr-line>
          ,
          <country country="NL">Netherlands</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>The present study set out to personalize a digital library aimed at new parents by reordering articles to match users' inferred interests. The interests were inferred from reading behavior as well as parenting styles measured through surveys. As prior research has shown that parenting styles are related to how parents take care of their children, these styles are likely to be related to what content a parent is interested in. The present study compared personalization based on parenting styles against other types of personalization. We conducted a user study with 106 participants, in which we compared the effects of four different approaches of personalization to our users' reading behavior and user experience: a non-personalized baseline, personalization based on reading behavior, personalization based on parenting styles measured through surveys, and a hybrid personalization based on both reading behavior and parenting styles. We found that while the reading behavior was not significantly influenced by different types of personalization, participants had a better user experience with our survey-based approach. They indicated they perceived a higher level of personalization and satisfaction with the system, even though in terms of objective metrics this approach performed worse.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>INTRODUCTION
Becoming a parent is for many a big challenge in life. New
parents have to get used to a new set of responsibilities and
©2018. Copyright for the individual papers remains with the authors.
Copying permitted for private and academic purposes.</p>
      <p>HUMANIZE ’18, March 11, 2018, Tokyo, Japan
have to learn a whole new set of care-taking skills, ranging
from practical (such as changing diapers) to more emotional
(such as recognizing and reacting to a child’s emotions). There
are numerous ways to acquire these skills: parents can get
advice from relatives, or alternatively rely on vast amounts of
books, websites, videos, and other types of media.
Parents have different styles of parenting and as such some
topics may be very relevant to a parent, while others are
completely irrelevant. In this sense, helping parents find their way
in content related to the parenting domain is similar to
personalization areas such as movie or book recommendations. A
challenge in personalizing content on parenting is that
firsttime parents have to find their own way in a domain that is
completely new to them. Parents may not have a clear view
yet on the range of alternative ways of taking care of a child
that match their styles. It might not be easy for them to judge
what content is relevant and they might read content that is not
in line with their parenting styles or interests. As such, there
might be a discrepancy between what content new parents
read and what is actually relevant to them. As a result
personalization based on reading behavior (as is common) might not
provide the desired results.</p>
      <p>An additional challenge is that parenting is an activity people
are very committed to and about which they hold strong beliefs.
As a result, new parents might find certain types of content
extremely irrelevant, to the point of being offended. A mother
who struggled and eventually gave up breastfeeding might be
hurt by receiving unwanted breastfeeding advice. Being wrong
in personalization in this domain thus has a bigger impact than
in other domains.</p>
      <p>We aim to help parents in finding relevant content by
personalizing a digital library of information articles on parenting.
Because the content is aimed at new parents, we think a
discrepancy can exist between reading behavior and reading
interests and parenting styles measured through surveys might
provide more reliable information for predicting reading
interests. To investigate this, we personalize a library using both
behavior data and survey data.</p>
      <p>
        Research Question and Hypotheses
The current paper aims to investigate how a library comprising
articles on parenting can be improved by personalizing the
order in which the articles are presented1. A screenshot of
what the library interface looked like can be found in Figure. 1.
In addition the paper investigates if and how parenting styles
can contribute to this personalization. The main research
question thus is “How does personalization based on parenting
styles compare to personalization based on reading behavior
in terms of user behavior and user experience?”
We try to answer this research question by investigating the
effects of personalization based on survey responses
measuring parenting styles (explained in Section 1.4) and more
conventional ways of personalization that rely on behavior
data. Specifically we compare the effects of survey-based
personalization with personalization based on reading behavior,
personalization based on both reading behavior and survey
responses and a non-personalized baseline. We are interested
in the effects of this personalization both in terms of
influenced behavior (e.g. does personalization based on surveys
increase the number of articles users read?) and in terms of
user experience (e.g. does personalization based on surveys
result in a higher satisfaction with the digital library?). To
investigate the effects on the user experience we adopted the
User-Centric Evaluation Framework for personalized systems
by Knijnenburg and Willemsen [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. We designed a UX
survey with items aimed to measure different aspects of the user
experience.
      </p>
      <p>With the survey we aimed to measure three aspects of the user
experience specifically and formulated survey items to do so:
the perceived level of personalization (“The library shows
articles I find interesting”), the system satisfaction (“It was easy
to find relevant/interesting articles”), and reading satisfaction
(“I enjoyed reading the items I read”). We hypothesize that the
different ways of personalization influence the perceived level
of personalization. A higher level of personalization should
lead to a higher system satisfaction, which should lead to a
higher reading satisfaction. The higher system satisfaction is
also expected to increase the amount of reading by the user.
In terms of improving user satisfaction and increasing reading
behavior we hypothesize the following order in the different
personalizations, from worst to best:</p>
    </sec>
    <sec id="sec-2">
      <title>The non-personalized library</title>
      <p>The library personalized based on just reading behavior
The library personalized based on just survey responses
The library personalized based on both reading behavior
and survey responses
The remainder of this section will introduce the theoretical
background on which this study is based.</p>
      <p>
        Personalization
Personalization is the process of altering a system to fit to
the needs and/or preferences of an individual [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. Examples
1The content and the design of the library were taken from Philips’
uGrow app. Available for iOS https://itunes.apple.com/app/
ugrow-healthy-baby-development/id1063224663 and Android
https://play.google.com/store/apps/details?id=com.philips.
cl.uGrowDigitalParentingPlatform
of personalization can be found on numerous websites, for
example in the form of recommendations on Amazon, or as
filters on social media feeds such as Twitter and Facebook. In
general the goal is to alter a system in a way that it caters to
the individual needs of a user to influence user behavior or
user experience. A typical goal of influencing behavior is to
make users consume more content in a media browsing system
or purchase more items in a webshop, while a typical goal of
influencing user experience is to make it easier for users to
reach their goals.
      </p>
      <p>
        Personalization can be implemented in many different
ways [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ], but the most widely adopted methods rely on
historical data describing how users interact with a system, and
combine these data across users to make predictions on what
content a user will find relevant. The system is subsequently
altered so that the user is exposed to more of the content he/she
is likely to find relevant.
      </p>
      <p>
        A standard problem related to this approach to personalization
is the cold start problem [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. More specifically, three cold
start problems exist: the system cold start, the user cold start,
and the item cold start. The system cold start occurs when
not enough data are available within the system as a whole to
make predictions. The user and item cold start occur when
there are not enough interaction data available corresponding
to respectively the user or the item, so that no predictions can
be made for respectively the user or the item.
      </p>
      <p>In the context of parenting an additional challenge occurs.
Apart from being new to a system, (some) parents are also
new to being parents and they might find it hard to identify
what content is relevant to them. This can result in a mismatch
between the content they read and the content that they are
actually interested in. In systems in which user evaluations of
content are not being tracked explicitly, assuming that content
is appreciated because it was read may well lead to inaccurate
predictions about user preferences. Because of this, a library
aimed at parents might benefit from relying on other types of
data for personalization.</p>
      <p>
        Personalization and Psychological Traits
Many psychological traits have been incorporated in
personalization applications. Hauser et al. [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] personalized an online
tool to compare contracts for mobile phones based on
cognitive styles (i.e. the way in which individuals prefer to process
information) and showed that providing users information in
a way that matches their cognitive style (e.g. textual versus
visual information) increases buying propensity. Germanakos
and Belk [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] found that adapting an online learning
environment to the working memory capacity of its students resulted
in higher test scores.
      </p>
      <p>
        Similarly, Fernandez-Tobias et al. [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] showed that
incorporating personality in collaborative filtering algorithms allowed
them to better predict recommendations across domains (e.g.
recommending movies based on someone’s music listening
behavior). They did this by extending the SVD++ algorithm [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ],
an algorithm used to predict ratings that users will assign to
items. Fernandez-Tobias et al. used a part of the
myPersonality dataset2 comprising 160k users and in total just over 5
million likes over 16k items (consisting of books, movies or
music artists). The personality traits (the five factor model
with the traits openness to experience, conscientiousness,
extraversion, agreeableness and neuroticism [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]) were available
for all users and were used to predict likes. Their results
showed that incorporating the personality information
substantially improved the extent to which likes on Facebook could
successfully be predicted.
      </p>
      <p>
        These studies demonstrate that personalization can benefit
from considering and incorporating personal characteristics
(such as personality traits or cognitive styles). In the case
of parenting, parenting styles are psychological traits that are
likely to play a role in what content parents find relevant. In the
present study we measure parenting styles and subsequently
use them for personalizing the online library.
2Available from http://mypersonality.org/wiki/doku.php?id=
download_databases
Parenting Styles
Zhao [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] performed a literature review on research on
parenting with the goal of understanding how scholars operationalize
and measure parenting styles. Zhao was in particular interested
in how parenting styles relate to the actual care-taking
behavior and as such, the review was primarily focused on research
that comprised both questionnaires and a behavioral aspect.
She found that parenting as a whole is a combination of
cognitive factors, the physical task of taking care of a baby, and
the interplay between the two (cf. [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]). Zhao in addition found
that researchers conceptualize parenting styles as individual
differences along two cognitive dimensions: structure (i.e.
how important parents think structure is for their children) and
attunement (i.e. how much parents value reacting to a child’s
needs and how able they are at reading those needs) [
        <xref ref-type="bibr" rid="ref1 ref16 ref3">1, 3, 16</xref>
        ].
Prototypical parenting styles are the resulting combinations
of scores along these two dimensions (high attunement/high
structure, high attunement/low structure, low attunement/high
structure and low attunement/low structure). Other cognitive
factors that have been identified in literature to play a role are
parental distress, perceived self-efficacy, and the perceived
difficulty of the child.
      </p>
      <p>
        The cognitive factors allegedly have an interplay with how
parents actually take care of their children. To validate these
parenting styles and investigate how they relate to care-taking
behavior, Zhao [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] conducted a survey study in which she
measured parenting styles and asked respondents to self-report
on how they take care of their children. The analysis of the
survey data showed support for the conceptualization of
parenting styles along the previously mentioned dimensions of
structure and attunement. In addition it showed that parenting
styles are related to the actual care-taking behavior of parents.
For example, parents scoring low on attunement are less likely
to engage in breast-feeding and more likely to opt for
bottlefeeding. As parenting styles are related to how parents take
care of their children, they are likely to be useful predictors
for what type of content parents are interested in. For example,
parents that find structure important put their children to bed
at a fixed bedtime instead of waiting for the kid to become
sleepy. As a result they might be more conscious of the fact
that their child does not fall asleep easily and will thus be more
interested in content on how to get a baby to sleep well than
people that value flexibility over structure and wait for their
child to get sleepy.
      </p>
      <p>STUDY DESIGN
To investigate our research question we designed a user study
that consisted of two main parts, with a first part aimed at
collecting initial data to be used for personalizing the “My
Articles” page and a second part aimed at investigating the
effects of the different ways of personalization on the reading
behavior and user experience.</p>
      <p>During the first part, participants were asked to complete a
survey to measure their parenting styles, after which they
were invited to browse the non-personalized library (i.e. a
library with a fixed order of articles). The responses to the
surveys were stored for personalization later. The information
regarding what articles participants read during the browsing
phase was used for personalization based on reading behavior.
The order of articles for the second part of the study was
calculated in one out of four ways (described in more detail in
section 3). For each participant we selected at random which
set of predictions was used to personalize the library.
In the second part of the study the participants were re-invited
to interact with their now personalized digital library.
Subsequently, participants evaluated their experience with the
system through our UX survey. We first report on the initial
phases of the study.</p>
      <p>Initial Data Collection: Survey and Reading Behavior
We implemented the online library on a website that was
accessible through browsers on computers and mobile phones. We
recruited participants through posts in online forums dedicated
to parenting and through Facebook ads targeting parents in
the United Kingdom and United States with children younger
than two years old. In total 234 parents clicked on the link to
participate in the study. All participants that completed the
entire study were compensated with $4.50 or £3.50 of shopping
credit for amazon.co.uk or amazon.com. The ad campaign and
data collection took place in May and June 2017.</p>
      <p>The first part of the study consisted of two steps. In the first
part people were asked to complete the survey to measure
parenting styles. After completing the survey, the participants
were presented with the digital library and invited to browse
through it and read the articles that they were interested in.
The participants were invited to read as many articles as they
wanted for as long as they wanted and to click a link labeled
“I’ve finished reading” once they felt like they read enough.
After clicking this link participants were asked to submit their
email address for the second step of the study.</p>
      <p>
        In total 181 participants completed the survey (15 men/166
women, 99 first time parents, with an average age of the baby
11.39 (SD: 7.96) months). On average the whole session
lasted just over 6 minutes (378 seconds, SD: 279:80 seconds).
The survey consisted of 15 items of the original survey of
Zhao [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. For the five cognitive (structure, attunement,
maternal self-efficacy, parental disstress and perceived difficulty
of the child) we selected per factor the two items with most
extreme factor loadings. We added items concerning the
demographics of the parent (gender, level of education, whether
they were first time parents) and child (gender, age) that had
had large effects on the self-reported behavior in the original
analysis. The factor scores for our participants were calculated
by using the factor loadings from the original survey and are
displayed in Figure 2. These scores show similar distributions
and correlations as the factors in the original survey.
The interface of our library was made to have the look and
feel of the original library (see Fig. 1) as much as possible.
As in the original interface, the articles are subdivided in
categories that are displayed in rows. Within the category rows
the articles are displayed horizontally. The user is able to scroll
up and down to different categories and left and right within
categories to the different articles. As in the original interface,
the order of articles and categories was fixed: every participant
had exactly the same order of categories and articles.
      </p>
      <p>The initial part of the data collection was concluded with
offering the participants to freely browse the online library.
Participants opened on average 2:23 articles (SD: 3:37 articles)
from 1.25 categories (SD: 1.51 categories). These data and the
survey responses were used to calculate relevance predictions
for the individual participants.</p>
      <p>CALCULATING RELEVANCE PREDICTIONS
Based on the data collected in the first step of the study we
calculated per participant four different relevance rank predictions
for all articles. As a baseline we used the non-personalized
General Top-N. The three other ways of predicting differed
in what data from the first step were used. A survey-based
ordering was based on the data from the survey responses
of the participants and on reading behavior at the aggregated
level. A reading-based ordering used only data regarding the
articles that people had read in the first step. Finally, a hybrid
ordering used both the survey responses and the individual
reading behavior. The way these orderings were calculated is
described in the following sections.</p>
      <p>
        Survey-Based Predictions
We used the survey responses collected in the first step to
predict relevance of the different articles for the participants
in our study. To do this, the participants were subdivided
in segments, by performing median splits on the 2 cognitive
factors: attunement and structure. The user segment was then
defined to be the combination of these two scores, resulting
in four segments. We considered incorporating the three other
factors measured in the study (self-efficacy, parental distress
and perceived difficulty of the child), but given the number
of users in our dataset adding additional factors resulted in
segments that became too small.
As the participants read on average just over 2 articles, there
was not enough data to show differences on the level of
individual articles (i.e. articles were not read often enough to allow
for enough variance), but participants from different segments
did prefer different categories, as can be seen in Figure 3.
When investigating these predictions, the popularity order for
these categories seems to make sense intuitively. For example,
the breastfeeding category is predicted to be more popular for
segments with high attunement, which is congruent with the
relationship with breast-feeding and high attunement in the
original survey [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ].
      </p>
      <p>As a result we decided to sort the categories based on the
attunement-structure segment and sort the articles within each
category based on general popularity. That is, the survey-based
predictions only personalized the order of the categories, not
the articles within each category. We tried basing segments on
other factors than attunement and structure, but the resulting
predictions were not as easily interpretable as the predictions
based on these segments.</p>
      <p>
        Reading-Based Predictions
For the conditions based on reading behavior alone, we used
the Bayesian Personalized Ranking Matrix Factorization (or
BPRMF) algorithm implemented in MyMediaLite [
        <xref ref-type="bibr" rid="ref17 ref6">6, 17</xref>
        ] to
predict relevance. BPRMF is an extension to classic matrix
factorization [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] that allows it to calculate recommendations
from positive only feedback instead of rating data.
Conventional matrix factorization attempts to complete the
matrix R with dimensionality of U (number of users) and I
(number of items). In this matrix the cells represent ratings
the user has given to the corresponding item. This matrix is
decomposed into two k-dimensional sub-matrices P and Q in
which the rows of P and Q represent respectively users and
items in a k-dimensional latent feature space. These matrices
are constructed so that the predicted rating rˆui is calculated by
taking the inner product pu qi (see Equation 1).
      </p>
      <p>
        rˆui = qi pu
(1)
Rendle et al. [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] extended this matrix factorization into
BPRMF to allow using positive only feedback to calculate
per user a ranking of the articles from highest predicted
relevance to lowest predicted relevance. In the current study,
the positive only data describe whether or not a user read an
article in the first step, and the predictions would indicate what
items a user is most likely to read. In order to translate this
positive-only, binary feedback into a ranking, pairs of items
are semi-randomly selected per user, where each pair consists
of an item that the user has interacted with and one with which
the user has not interacted. The assumption is that the first
is preferred over the second. Sampling a large number of
pairs per user, results in a ranking that can be used in matrix
factorization and the resulting model then calculates a relative
relevance score instead of a rating.
      </p>
      <p>
        Hybrid Predictions
The BPRMF algorithm was extended to combine reading
behavior and the individual parents’ user attributes inferred
from the survey for the calculation of hybrid predictions. The
BPRMF algorithm was extended similarly to how
FernandezTobias et al. [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] extended the SVD++ [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] algorithm to
incorporate personality in predictions.
      </p>
      <p>Where the original BPRMF algorithm uses two matrices P and
Q to calculate predictions, our user attribute aware BPRMF
algorithm uses a third matrix Y . Y describes the user attributes
on the same k latent features the users and articles are
expressed in. In our case we used high and low scores for the
five cognitive factors from our parenting style survey as user
attributes. We decided again to use the median splits per factors
to assign each user a high or low score for each factor in order
to prevent overfitting. Every user has thus 5 user attributes and
the relevance predictions are similar to the original BPRMF
algorithm with an additional matrix in which user attributes
are represented. The predicted relevance is then calculated
according to equation 2.</p>
      <p>rˆui = qi
pu +</p>
      <p>
        å ya
a2A(u)
!
(2)
This model is fit using stochastic gradient descent. Each
iteration consists of two steps. In the first step the P and Q matrices
are fit, while leaving the Y matrix constant. In the second step
the Y matrix is fit, while leaving the P and Q matrix
constant. We implemented this algorithm in the MyMediaLite
library [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>
        Calculated Relevance Predictions
In total the dataset contained 221 users and 508 reads3. For
each user predictions using the four methods described above
were calculated. The predictions were then sorted in two steps.
First the 7 categories were ordered based on the article with
the highest predicted relevance (a strategy called min-rank
that has been shown to work well in similar circumstances [
        <xref ref-type="bibr" rid="ref21 ref4">4,
21</xref>
        ]). Within the categories the articles were ordered based on
predicted relevance.
      </p>
      <p>The algorithms for the reading-based and hybrid predictions
required the tuning of a set of regularization
hyperparameters, which we carried out using Bayesian Optimization. The
3We included reading data from a pilot study to ensure we had enough
data to calculate predictions
algorithm
baseline
survey
reading
hybrid
0:840</p>
      <p>0:832
0:769
0:083</p>
      <p>0:079
0:080
0:065</p>
      <p>0:061
0:059
0:424</p>
      <p>0:411
0:404
0:706
0:650
0:767
0:807
0:146
0:060
0:176
0:214
0:104
0:062
0:114
0:126
0:477
0:353
0:522
0:561
Bayesian Optimization was performed using 5-fold cross
validation, using AUC as the target measure. Once optimal values
for the hyperparameters were established, the predictive
models were constructed and the predictive performance (i.e. the
reading-based and hybrid recommendations) was investigated
through 5-fold cross validation also. Table 1 shows these
performance metrics of the three algorithms under the column
‘5-fold Cross Validation’. The performance metrics appeared
to be adequate4. However, the baseline, reading-based, and
hybrid predictions are calculated on the level of the individual
articles, they cannot be easily compared to the survey-based
predictions that are calculated first on the category level and
then within the categories on an individual article level. In
order to make a fair comparison, we performed a post-hoc
analysis by recalculating the performance metrics for the sets
of recommendations to correspond to the survey-based
predictions. We did this by calculating the lists of recommendations
and sorting all lists first by category based on the minimum
predicted rank of the article within that category and
subsequently sorting the articles within their categories based on the
predicted relevance for the individual articles. We then
calculated performance metrics by using the actual reading behavior
as ground truth. The outcome of these recalculations can be
found under the columns ‘Post-hoc Comparison’ in Table 1.
These numbers indicate the most accurate predictions for the
hybrid predictions, followed by the reading-based predictions,
the survey-based predictions and finally the non-personalized
baseline. Based on these metrics we would expect the hybrid
predictions to be most in line with what participants will read,
and the survey-based least. This order is different from the
order in the k-fold cross validation metrics because no k-fold
cross validation was applied to be able to compare with the
survey-based recommendations (i.e. the train and test set were
identical).</p>
      <p>
        RE-ENGAGING WITH THE NOW PERSONALIZED SYSTEM
The second part of the study was used to investigate our
research question and test our hypotheses. To this end
participants were re-invited to interact with the website, where they
were now shown the library personalized in one out of four
ways (selected at random). The invitations were sent out after
all predictions were calculated, which means that the time
between finishing the first part and starting the second part
differed between participants (median 42.6 days. SD: 15.4
days). In this step the interface was personalized by
reordering both the categories and the articles within the categories.
The categories were ranked based on the minimum predicted
4An overview of the different metrics and how to interpret them can
be found in [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
relevance rank (or highest predicted relevance) within the
category, so that the category with the article with the highest
predicted relevance was shown on top. This way of sorting
categories has been shown to be one of the best strategies in
terms of reducing browsing time [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Within categories the
articles were ordered by predicted relevance rank, with the
article with the lowest predicted relevance rank to the left of
the list.
      </p>
      <p>Participants were allowed to browse the library freely during
which we measured what articles the participants opened.
Participants were shown a link labeled “I have finished reading”
that would take them to the survey as soon as they felt they
read enough. The survey contained 11 items aimed at
measuring Perceived Level of Personalization, System Satisfaction,
and Reading Satisfaction.</p>
      <p>Participants
All 181 users from the first part were invited to join the second
part of the study via email. Of the 181 users we sent
invitations to, 150 visited the second part and 121 completed the
study. A number of cases were removed, for either trying to
complete the study with multiple email addresses (3 users),
having missing data in the survey (1 user), or finishing the
second part of the study in less than 50 seconds (11 users). For
our final data analysis we ended up with 106 users (9 men/97
women, 50 first time parents, mean (SD) age of the baby 10.63
(8.45) months)
These users were distributed roughly equally over conditions
(baseline: 29, survey: 29, reading-based: 22, hybrid: 26). In
addition, there appeared to be no bias in response rate for the
different parenting style segments of the participants, with
response rates of .56 for the low structure/high attunement
segment, .65 for the high structure/low attunement segment,
.73 for the high structure/high attunement segment and .60
for the low structure/low attunement segment(c2(3) = 3:239,
p = 0:356).</p>
      <p>Results
To gain insight in how the different methods of predicting
relevance influenced the final recommendations participants
received, we calculated the difference of the recommendations
with the general Top-N in terms of Spearman Rank
Correlation. The (Spearman) correlation coefficient r indicates to
what extent lists are similar, with a value of 1 if the order is
identical and -1 if they are in reverse order. The results are
shown in Figure 4 and they reveal that the available reading
data does not allow personalization that differs a lot from the
baseline condition (as the correlation between reading-based
and baseline is 0.91 on average). Personalization based on the
survey-based predictions is quite different from the baseline
predictions, with an average correlation of 0.37. The hybrid
predictions fall somewhere in between the reading-based and
survey-based predictions with a correlation of 0.74. These
numbers indicate that the additional data of parenting styles
allows for personalization that deviates more from the baseline
than personalization based on reading behavior alone.
One possible explanation of the reading-based personalization
not differing much from the baseline is insufficient data. As
there are a limited number of users (221 users, see Section 2.1),
that read a limited number of articles (2.44 articles on average)
from a library with a limited number of articles (102) that was
presented in a fixed order. As such the dataset might not
contain enough variance between users’ reading behavior to fully
benefit from collaborative filtering. What argues against this
is the fact that the reading-based and hybrid recommendations
appear to outperform the survey-based predictions in terms of
prediction accuracy (see Table 1).</p>
      <p>
        Reading Behavior
Participants read on average 2.72 articles (SD: 4.28 articles),
but 42 participants (39.6%) did not read any articles. The
descriptives for the number of article reads per condition are
shown in Table 2. The different conditions had no significant
influence on the number of articles people read, as negative
binomial regressions with the condition as independent
variable and the number of reads as dependent variable showed
no significant difference across conditions. This implies that
no support is found for the hypotheses regarding the effect of
our experimental manipulations on how participants interact
with their personalized libraries.
User Experience
As per the user-centric evaluation framework by Knijnenburg
and Willemsen [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] all survey items were submitted to a
structural equation model (SEM). The responses to the individual
items can be seen in Figure 5, with items belonging to
Perceived Level of Personalization (pers1-pers4), System
Satisfaction (syssat1-syssat4), and Reading Satisfaction
(readsat1readsat3). The three items for reading satisfaction show very
low variance among each other, which lead to these three items
not fitting in the model. This might have been caused by the
fact that the reading behavior did not differ across conditions
as we manipulated only the order in which the articles were
presented, and not the actual content in the library. Therefore,
people were actually able to read the same articles regardless
of experimental condition and thus the reading satisfaction
might be similar. Apart from the items on Reading
Satisfaction, two of the remaining items (pers1 and syssat2) explained
little variance and were also removed from the analysis.
Despite the fact that participants did not read a large amount
of articles, the interface did allow participants to get a general
idea of the library by looking at the categories and the article
titles. Nevertheless, we do feel that the participants who
actually read articles are better able to evaluate the library. To
account for this we introduced an additional (dummy)
variable labeled ‘Read’ indicating whether or not people read any
articles.
      </p>
      <p>A SEM was constructed using the remaining six survey items
measuring two latent constructs (Perceived Personalization
and System Satisfaction), the experimental conditions, and the
variable describing whether or not people read as exogenous
variables. The two latent factors had high correlation, but the
model showed good fit (with c 2(36) = 44:447, p = :158, CFI
= :984, TLI = :974, RMSEA = :047, 90% CI: [0:000; :088]).
For each participant we used this model to calculate the scores
on these latent factors to be used for the remainder of the
analysis.</p>
      <p>As the final model consists of only two latent constructs
(Perceived Personalization and Systems Satisfaction) that are
highly correlated, there is no clear underlying structural model
to test anymore. For the analysis we could either combine both
factors into one overall latent factor, or analyze both factors
separately. We chose to do the latter as both factors might still
capture different nuances of the user experience, despite their
high correlation.</p>
      <p>We analyzed the effect of our manipulation on the factor scores
of both constructs through linear regressions, with the factor
scores as dependent variables and the experimental condition
as independent variable. As additional moderator we included
the dummy variable representing whether or not people read
articles.</p>
      <p>The average factor scores per condition for the two measured
constructs can be found in Figure 6. The image shows an
increase in both Perceived Personalization and System
Satisfaction for the survey-based condition. The effects are higher
for the participants that did not read (represented in the red
bars) and lower for the participants that did (represented in the
green bars).</p>
      <p>The regression models in Table 3 show these effects as well.
Regression model (1) shows a positive and significant effect on
Perceived Personalization for participants in the survey-based
condition, indicating that these participants had the feeling the
library catered more to their interests5. An additional, albeit
not statistically significant, effect the table shows is an effect
with a significance level of p &lt; 0.1 for the increased perceived
level of personalization in the condition with hybrid
personalization. Although caution is needed when interpreting this
effect, it describes a trend towards participants experiencing a
higher level of personalization with the hybrid personalization.
In terms of System Satisfaction the patterns are slightly
different. Participants that received the survey-based personalization
were more satisfied with the system, as can be seen in model
5Because the factor scores are calculated through a Structural
Equation Model they are normally distributed with a mean of 0 and SD
of 1. Participants in the condition with survey-based personalization
thus had a perceived level of personalization of 0.563 SD higher than
participants in the baseline.
(2) in Table 3. Model (3) reveals how this effect holds up
for participants that read versus participants that did not. It
shows a negative interaction effect for the participants that
received survey-based personalization and read at least one
article, which suggests that only the people that do not read
any articles actually perceive a higher system satisfaction; for
those who do read at least one article the effect is strongly
reduced.</p>
      <p>In conclusion support is found for the hypothesis that
surveybased personalization outperforms the non-personalized
baseline, while no evidence was found that the reading-based and
hybrid personalization did so. The lack of effect in terms
of reader experience are in line with the comparison of the
different predicted rankings in terms of Spearman’s Rank
Correlation, that showed a high similarity between the
readingbased and non-personalized baseline. This comparison further
showed that the survey-based personalization was most
different from the baseline, which is also reflected in the user
experience (albeit stronger for the people that did not read than
the people that read). The hybrid conditions falls in between
the survey-based and reading-based and similarly the effects
on user experience appear to fall in between the effects of the
survey-based and reading-based recommendations.
CONCLUSION AND DISCUSSION
This study set out to compare personalization based on
psychological traits measured through a survey to
personalization based on reading data. Through a user study we
compared different methods against a non-personalized baseline
and showed that personalization based on survey information
about parenting styles resulted in a significantly higher
experienced user satisfaction and perceived level of personalization
despite a lower objective performance, whereas using only
historical reading behavior or the combination of historical
reading behavior and measured parenting styles did not. Our
findings speak to the potential usefulness of including data
regarding characteristics of users (collected through an initial
survey or otherwise) in personalization to alleviate the cold
start problem. While the actual reading behavior for users was
not influenced, an improved user experience may increase the
probability for users to return to the library later on.
The fact that using the survey data for personalization also
outperformed the condition where recommendations were based
on both survey data and reading behavior is likely caused by
the fact that the hybrid recommender - given how we had
implemented it - came up with suggestions that were relatively
close to the baseline condition. Hybrid predictions that would
have assigned more weight to the survey data might have
faired better. In any case, we do see that personalization based
on surveys captures the interests better, or at least increase
the reported user satisfaction, and that they lead to a more
different order in which articles are presented than based on
the reading behavior alone.</p>
      <p>From a system owner point of view it is worth noticing that the
survey-based predictions were very straightforward to
calculate and implement compared to the reading-based and hybrid
predictions. In addition, after completing the short survey the
user can immediately benefit from personalization. Both the
Perceived Personalization</p>
    </sec>
    <sec id="sec-3">
      <title>System Satisfaction</title>
      <p>reading-based and (to a lesser extent) the hybrid predictions
require reading behavior from the user before they can be
calculated. Admittedly providing explicit feedback in the form
of a survey demands more effort than the implicit feedback
provided through the natural interaction of reading. However,
the higher user experience suggests there might be a trade-off
between the costs of user effort and the benefits of accurate
personalization.</p>
      <p>Another interesting finding is that the effects of personalization
on user experience disappeared as soon as participants started
reading articles. A possible explanation for this observation
can be the number of articles people see in the second part
that they have already read in the first part. Seeing articles one
has already read may contribute to a higher perceived level of
personalization and satisfaction with the library as a whole,
while reading these articles might actually be detrimental for
the user experience. In other words, what looks good might
not necessarily be what helps the user and as such it might be
worthwhile to investigate the factors that influence user
satisfaction of a personalized system before and after consumption
and to see if and how these are different from each other. From
a more general perspective this raises the question whether
and how personalization needs to anticipate possible changes
and differences in the perception of recommendations as the
user progresses. Alternatively it might indicate that the
process of evaluating personalization is different and depends on
whether the user is evaluating through observing or through
experiencing.</p>
      <p>Shortcomings and Future Work
While the findings of this study indicate that using surveys as
a basis for personalization can improve personalized systems,
the specific application in which we tested our hypotheses
might limit the extent to which this finding can be generalized.
Dependent variable:
b
0:673
0:406
0:273
(2)
(SE)
(0:243)
(0:261)
(0:249)
0:097</p>
      <p>(0:171)
Participants in our study interacted with the system twice. One
time for an initial data collection and a second time for the
evaluation. This difference might have lead to a discrepancy,
as in the first session people were exploring the system and
possibly paying attention to other aspects than in the second
session. For example, in the first session people were getting
used to the way of navigation in the library and getting
acquainted with the system and its usability may have been an
issue. In the second session, participants are more likely to
have evolved past this stage, and they can now focus more on
what it is that they want to read. This would imply that data in
the first session is describing behavior of participants who are
getting to know a system, and as a result models trained on
this data will generate recommendations based on what an
exploring user will typically read, which may not be appropriate
to personalize a library for a participant who already knows
and is actively using a system.</p>
      <p>
        As mentioned in the results section, it is unsure how our
findings hold up in a setting with a bigger library and more
interaction data (both in terms of number of users and in terms of
interactions per user). With only 102 articles in a fixed order,
behavior for participants in the initial data collection may not
have differed enough from each other (yet) to allow the
personalization based on reading behavior to produce predictions
that are personalized sufficiently. The fact that these
personalizations stayed relative close to the non-personalized baseline
can be interpreted this way. The survey-based
recommendations on the other hand combined data from users with similar
parenting styles and as a result were able to differentiate
themselves more from the non-personalized baseline. Having more
articles and perhaps also a somewhat longer initial period will
allow for behavior with more differences between users,
allowing to more effectively leverage the predictive power and
complexity of reading-based personalization, which in turn
will provide more insight into the conditions that play a role
in how personalization based on behavior compares to
personalization based on psychological traits. However, our results
show that in this situation with limited reading data a short
survey delivers good data for initial personalization.
In line with the previous argument, it is important to realize
that in terms of data per user, our participants only interacted
with the system once and read 2:23 articles on average. They
might still have been in their cold start phase and there may
not have been enough information about the users’ reading
behavior to provide useful recommendations. What argues
against this is that both the hybrid and reading-based models
had higher prediction accuracy than the survey-based
recommendations. Given these observations it would be worthwhile
to perform a study that controls for the amount of feedback
collected from the participants. Having more feedback per
participant allows to investigate how the number of interactions
per user affects the performance of the different
personalization approaches, similar to how Kluver and Konstan [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]
investigated the effects of number of interactions on predictive
accuracy.
      </p>
      <p>Apart from the amount of data per user, the amount of data
available within the system as a whole may be another factor
that plays a role which method of personalization works best.
Evaluating how survey-based and reading-based
personalization compare over time, as more data enter the system as a
whole or per user, would provide valuable insight in which
approach works best when. One could imagine a system that
starts out from personalization based on measured
psychological traits that transitions into a system based more on behavior
or a hybrid system. Investigating this effect would require a
more longitudinal study, where users are invited to a
personalized library at multiple moments, to see whether and how the
different approaches are affected by the cold start.
Apart from the drawback of a low number of participants for
calculating relevance predictions, the low number also limited
the statistical power of our statistical analysis of the effects
of personalization. While young parents are active on the
internet, they are hard to approach. In the current study we did
not manage to detect effects of personalization on reading
behavior and only differences between some of the experimental
conditions. The effects caused by the personalization might
have been smaller than the statistical power of our analysis
allows us to detect. Conducting a study with more participants
would allow us to detect these possibly smaller effects.
In conclusion, the current paper demonstrates that measuring
psychological traits for the sake of personalization is
worthwhile and might well lead to increased user satisfaction, but
additional work is needed to establish under which conditions
this approach is valuable.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>B</given-names>
            <surname>Arnott and Amy Brown</surname>
          </string-name>
          .
          <year>2013</year>
          .
          <article-title>An Exploration of Parenting Behaviours and Attitudes During Early Infancy: Association with Maternal and Infant Characteristics</article-title>
          .
          <source>Infant and Child Development</source>
          <volume>22</volume>
          (
          <year>2013</year>
          ),
          <fpage>349</fpage>
          -
          <lpage>361</lpage>
          . DOI: http://dx.doi.org/10.1002/icd.1794
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>Diana</given-names>
            <surname>Baumrind</surname>
          </string-name>
          .
          <year>1966</year>
          .
          <article-title>Effects of Authoritative Parental Control on Child Behavior</article-title>
          .
          <source>Child Development</source>
          <volume>37</volume>
          ,
          <issue>4</issue>
          (dec
          <year>1966</year>
          ),
          <fpage>887</fpage>
          . DOI:http://dx.doi.org/10.2307/1126611
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <given-names>Jay</given-names>
            <surname>Belsky</surname>
          </string-name>
          and
          <string-name>
            <surname>Sara R. Jaffee</surname>
          </string-name>
          .
          <year>2015</year>
          .
          <article-title>The Multiple Determinants of Parenting</article-title>
          . In Developmental Psychopathology.
          <source>Number April</source>
          . John Wiley &amp; Sons, Inc., Hoboken, NJ, USA,
          <fpage>38</fpage>
          -
          <lpage>85</lpage>
          . DOI: http://dx.doi.org/10.1002/9780470939406.ch2
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>Gianluca</given-names>
            <surname>Demartini</surname>
          </string-name>
          ,
          <string-name>
            <surname>Paul-Alexandru</surname>
            <given-names>Chirita</given-names>
          </string-name>
          , Ingo Brunkhorst, and
          <string-name>
            <given-names>Wolfgang</given-names>
            <surname>Nejdl</surname>
          </string-name>
          .
          <year>2008</year>
          .
          <article-title>Ranking Categories for Web Search</article-title>
          .
          <source>In Advances in Information Retrieval</source>
          . Springer Berlin Heidelberg, Berlin, Heidelberg,
          <fpage>564</fpage>
          -
          <lpage>569</lpage>
          . DOI: http://dx.doi.org/10.1007/978-3-
          <fpage>540</fpage>
          -78646-7_
          <fpage>56</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>Ignacio</given-names>
            <surname>Fernández-Tobías</surname>
          </string-name>
          , Matthias Braunhofer, Mehdi Elahi, Francesco Ricci, and
          <string-name>
            <given-names>Iván</given-names>
            <surname>Cantador</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Alleviating the new user problem in collaborative filtering by exploiting personality information</article-title>
          .
          <source>User Modeling and User-Adapted Interaction 26</source>
          ,
          <fpage>2</fpage>
          -
          <lpage>3</lpage>
          (jun
          <year>2016</year>
          ),
          <fpage>221</fpage>
          -
          <lpage>255</lpage>
          . DOI:http://dx.doi.org/10.1007/s11257-016-9172-z
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Zeno</surname>
            <given-names>Gantner</given-names>
          </string-name>
          , Steffen Rendle, Christoph Freudenthaler, and
          <string-name>
            <surname>Lars</surname>
          </string-name>
          Schmidt-Thieme.
          <year>2011</year>
          .
          <article-title>MyMediaLite: A Free Recommender System Library</article-title>
          .
          <source>In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys</source>
          <year>2011</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>Panagiotis</given-names>
            <surname>Germanakos</surname>
          </string-name>
          and
          <string-name>
            <given-names>Marios</given-names>
            <surname>Belk</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Human-Centred Web Adaptation</article-title>
          and Personalization. Springer International Publishing, Cham.
          <volume>336</volume>
          pages. DOI: http://dx.doi.org/10.1007/978-3-
          <fpage>319</fpage>
          -28050-9
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <given-names>Asela</given-names>
            <surname>Gunawardana</surname>
          </string-name>
          and
          <string-name>
            <given-names>Guy</given-names>
            <surname>Shani</surname>
          </string-name>
          .
          <year>2015</year>
          .
          <article-title>Evaluating Recommender Systems</article-title>
          . Springer US, Boston, MA,
          <fpage>265</fpage>
          -
          <lpage>308</lpage>
          . DOI: http://dx.doi.org/10.1007/978-1-
          <fpage>4899</fpage>
          -7637-
          <issue>6</issue>
          _
          <fpage>8</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <given-names>J. R.</given-names>
            <surname>Hauser</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. L.</given-names>
            <surname>Urban</surname>
          </string-name>
          , G. Liberali, and
          <string-name>
            <given-names>M.</given-names>
            <surname>Braun</surname>
          </string-name>
          .
          <year>2009</year>
          .
          <string-name>
            <given-names>Website</given-names>
            <surname>Morphing</surname>
          </string-name>
          .
          <source>Marketing Science</source>
          <volume>28</volume>
          ,
          <issue>2</issue>
          (mar
          <year>2009</year>
          ),
          <fpage>202</fpage>
          -
          <lpage>223</lpage>
          . DOI: http://dx.doi.org/10.1287/mksc.1080.0459
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <given-names>Daniel</given-names>
            <surname>Kluver</surname>
          </string-name>
          .
          <year>2012</year>
          .
          <source>How Many Bits Per Rating ? Proceedings of the 6th ACM conference on Recommender systems - RecSys '12</source>
          (
          <year>2012</year>
          ),
          <fpage>99</fpage>
          -
          <lpage>106</lpage>
          . DOI: http://dx.doi.org/10.1145/2365952.2365974
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Bart</surname>
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Knijnenburg</surname>
            and
            <given-names>Martijn C.</given-names>
          </string-name>
          <string-name>
            <surname>Willemsen</surname>
          </string-name>
          .
          <year>2015</year>
          .
          <article-title>Evaluating Recommender Systems with User Experiments</article-title>
          .
          <source>In Recommender Systems Handbook</source>
          . Springer US, Boston, MA,
          <fpage>309</fpage>
          -
          <lpage>352</lpage>
          . DOI: http://dx.doi.org/10.1007/978-1-
          <fpage>4899</fpage>
          -7637-
          <issue>6</issue>
          _
          <fpage>9</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <given-names>Yehuda</given-names>
            <surname>Koren</surname>
          </string-name>
          .
          <article-title>Factorization Meets the Neighborhood : a Multifaceted Collaborative Filtering Model</article-title>
          . (????). DOI: http://dx.doi.org/978-1-
          <fpage>60558</fpage>
          -193-4/08/08
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Yehuda</surname>
            <given-names>Koren</given-names>
          </string-name>
          , Robert Bell, and
          <string-name>
            <given-names>Chris</given-names>
            <surname>Volinsky</surname>
          </string-name>
          .
          <year>2009</year>
          .
          <article-title>Matrix Factorization Techniques for Recommender Systems</article-title>
          . IEEE Computer (
          <year>2009</year>
          ),
          <fpage>42</fpage>
          -
          <lpage>49</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Robert R. McCrae</surname>
          </string-name>
          , Jr. Paul T. Costa, and
          <string-name>
            <surname>Thomas</surname>
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Martin</surname>
          </string-name>
          .
          <year>2005</year>
          .
          <article-title>The NEOâA˘ S¸ PIâA˘ S¸ 3</article-title>
          :
          <string-name>
            <given-names>A</given-names>
            <surname>More Readable Revised NEO Personality</surname>
          </string-name>
          <article-title>Inventory</article-title>
          .
          <source>Journal of Personality Assessment</source>
          <volume>84</volume>
          ,
          <issue>3</issue>
          (
          <year>2005</year>
          ),
          <fpage>261</fpage>
          -
          <lpage>270</lpage>
          . DOI: http://dx.doi.org/10.1207/s15327752jpa8403_05 PMID:
          <fpage>15907162</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <given-names>Bamshad</given-names>
            <surname>Mobasher</surname>
          </string-name>
          .
          <year>2007</year>
          .
          <article-title>Data mining for web personalization</article-title>
          .
          <source>The adaptive web</source>
          (
          <year>2007</year>
          ),
          <fpage>90</fpage>
          -
          <lpage>135</lpage>
          . http: //link.springer.com/chapter/10.1007/978-3-
          <fpage>540</fpage>
          -72079-9
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Stephanie L Prady</surname>
            , Kathleen Kiernan, Lesley Fairley, Sarah Wilson,
            <given-names>and John</given-names>
          </string-name>
          <string-name>
            <surname>Wright</surname>
          </string-name>
          .
          <year>2014</year>
          .
          <article-title>Self-reported maternal parenting style and confidence and infant temperament in a multi-ethnic community: Results from the Born in Bradford cohort</article-title>
          .
          <source>Journal of Child Health Care 18</source>
          ,
          <issue>1</issue>
          (
          <year>2014</year>
          ),
          <fpage>31</fpage>
          -
          <lpage>46</lpage>
          . DOI: http://dx.doi.org/10.1177/1367493512473855
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Steffen</surname>
            <given-names>Rendle</given-names>
          </string-name>
          , Wolf Huijsen, and
          <string-name>
            <surname>Karen</surname>
          </string-name>
          Tso-Sutter.
          <year>2008</year>
          .
          <article-title>State-of-the-art Recommender Algorithms</article-title>
          .
          <source>Technical Report</source>
          . www.mymediaproject.org
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Andrew</surname>
            <given-names>I Schein</given-names>
          </string-name>
          , Alexandrin Popescul,
          <string-name>
            <surname>Lyle H Ungar</surname>
          </string-name>
          , and David M Pennock.
          <year>2002</year>
          .
          <article-title>Methods and metrics for cold-start recommendations</article-title>
          .
          <source>Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval SIGIR 02 46</source>
          ,
          <string-name>
            <surname>Sigir</surname>
          </string-name>
          (
          <year>2002</year>
          ),
          <fpage>253</fpage>
          -
          <lpage>260</lpage>
          . DOI: http://dx.doi.org/10.1145/564376.564421
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>K. R. Venugopal</surname>
            ,
            <given-names>K. G.</given-names>
          </string-name>
          <string-name>
            <surname>Srinivasa</surname>
            , and
            <given-names>L. M.</given-names>
          </string-name>
          <string-name>
            <surname>Patnaik</surname>
          </string-name>
          .
          <year>2009</year>
          .
          <article-title>Algorithms for Web Personalization</article-title>
          . Springer Berlin Heidelberg, Berlin, Heidelberg,
          <fpage>217</fpage>
          -
          <lpage>230</lpage>
          . DOI: http://dx.doi.org/10.1007/978-3-
          <fpage>642</fpage>
          -00193-2_
          <fpage>10</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <given-names>Tiange</given-names>
            <surname>Zhao</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Investigating the relationship between parenting beliefs and parenting practice for in-app personalization</article-title>
          .
          <source>Master thesis</source>
          . Eindhoven University of Technology. https://pure.tue.nl/ws/files/46944250/855031-
          <fpage>1</fpage>
          .pdf
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <given-names>Zheng</given-names>
            <surname>Zhu</surname>
          </string-name>
          .
          <year>2011</year>
          .
          <article-title>Improving Search Engines via Classification</article-title>
          .
          <source>Ph.D. Dissertation.</source>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>