=Paper=
{{Paper
|id=Vol-2225/paper7
|storemode=property
|title=Online Daters’ Willingness to Use Recommender Technology for Mate Selection Decisions
|pdfUrl=https://ceur-ws.org/Vol-2225/paper7.pdf
|volume=Vol-2225
|authors=Stephanie Tom Tong,Elena F. Corriero,Robert G. Matheny,Jeffrey T. Hancock
|dblpUrl=https://dblp.org/rec/conf/recsys/TongCMH18
}}
==Online Daters’ Willingness to Use Recommender Technology for Mate Selection Decisions==
https://ceur-ws.org/Vol-2225/paper7.pdf
Online Daters’ Willingness to Use Recommender Technology
for Mate Selection Decisions
Stephanie Tom Tong Elena F. Corriero Robert G. Matheny Jeffrey T. Hancock
Wayne State University Wayne State University Wayne State University Stanford University
906 W. Warren Ave. 906 W. Warren Ave. 906 W. Warren Ave. Building 120, Room 110
569 Manoogian Hall 508 Manoogian Hall 508 Manoogian Hall 450 Serra Mall
Detroit MI 48201 Detroit, MI 48201 Detroit, MI 48201 Stanford, CA 94305
USA USA USA USA
stephanie.tong@wayne.edu elena.corriero@gmail.com rmatheny@wayne.edu jeff.hancock@stanford.edu
ABSTRACT criminals to arrest [14]. Another place that recommender
algorithms have inserted themselves is into the very “human”
Online and mobile dating services often offer recommendation context of romantic love. Dating websites and mobile
systems to facilitate decision making for their users. These applications (apps) that claim to provide users with a
recommendation systems take the form of decision aids that technological advantage over decisions of the heart have now
narrow down the size of the dating pool, or delegated agents that become a mainstay of the romantic landscape. Even Facebook
select optimal matches on behalf of users. An experiment recently entered the online dating industry by launching a new
examined three kinds of factors that influence daters to rely on service they call Dating that “will use a unique algorithm to
such recommenders when selecting dates: (a) selection task match you with potential dates, based on ‘dating preferences,
factors (e.g., number of available daters in the pool and number things in common, and mutual friends.’” [28]
of information attributes on a profile), (b) daters' personality As 15% of the American adult population reports using some
(e.g., need for cognition), and (c) daters' pre-existing trust in online or mobile dating platform [35], the ubiquity of technology
recommender technology. The results reveal that daters were in romance suggests that people should be accepting of
willing to use a decision aid under all circumstances, but that algorithmic recommenders. But an important question still
their intent to use a delegated agent was dependent on the size remains: Under what circumstances do people accept aid from a
of the choice set and levels of technological trust. This effect was recommender when searching for potential romantic partners?
further moderated by personality, such that daters who had a In current information systems research, there is a lack of
higher need for cognition displayed greater willingness to use a investigation into the “human side” of recommender systems—
recommender system when facing larger choice sets, compared that is, users’ perceptions and motivations for adoption of
to those low in need for cognition. This study provides insight recommender technology. This is despite the fact that many
into when users are willing to rely on different kinds of researchers have noted that a better understanding of human
recommendation systems for decisions in online dating contexts. decision making behavior and user experience would strengthen
the design of recommender systems [5]. With this concern in
CCS CONCEPTS mind, we examine how (1) properties of the selection task, (2)
• Human-centered computing→ Human computer interaction users’ personality and (3) users’ attitudes towards algorithmic
→ HCI design and evaluation methods→ User studies; technology affect their intentions to use recommender systems
Laboratory experiments in the context of online dating.
KEYWORDS 2 BACKGROUND
Social recommender systems, human-computer interaction, user The recent interest in recommender systems and online
interfaces, decision aids dating has been driven by the recognition that while
recommender technology may not change users’ romantic
ACM Reference format:
preferences, it does fundamentally change the process of mate
Stephanie Tom Tong, Elena F. Corriero, Robert G. Matheny, and Jeffrey
selection and relationship initiation [9]. Dating systems vary in
T. Hancock. 2018. Online Daters’ Willingness to use Recommender
Technology for Mate Selection Decisions. In Proceedings of Joint
kinds of recommendation technology they offer users: Websites
Workshop on Interfaces and Human Decision Making (INTRS ’18). like the industry standard Match.com offer decision aids that help
Vancouver, Canada, October 2018, 8 pages. users whittle down the immense dating pool into a more
manageable number of profiles. While decision aids act as a
filter, delegated agent recommenders found in sites like
1 INTRODUCTION eHarmony and mobile apps like Only and the European-based
Recommender algorithms help humans make decisions in a Once app provide daters a single optimal match once a day [see
variety of contexts, including how to maximize productivity in 17, 19].
the workplace, which stocks to buy on Wall Street, even which
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
Decision aids and delegated agent recommenders reflect defined as the ability to devote attention to the daters who more
varying amounts of technological involvement in users’ decision closely fit their previously-indicated mate preferences. The
making: Users who opt for a delegated agent recommender are authors reasoned that increasing options within the choice set
granting more control over mate selection to technology, should make mate selection more difficult by leading to less
compared to those who opt for a decision aid. Uncovering the mindful information processing and reduced ability to weed out
specific factors that determine when daters are likely to use a “poor” matches. Results revealed a linear trend across choice sets
decision aid versus a delegated agent recommender system to of 30, 60, or 90 profiles, with participants making decisions in the
facilitate their romantic decision making is the purpose of this large choice set conditions reporting lower goodness of match
work. scores and lower selectivity when compared to either the
moderate or small choice conditions [40]. From these results, it
2.1 Selection Task Difficulty was concluded that increased assortment size led daters to
One of the oft-touted advantages of online dating is the deviate more from the preferences they declared before they
increased number of potential partners available to users. began looking for dates. Although one could argue that such
However, this increased variety can complicate the choice task, deviations actually indicate better decisions (i.e., daters changed
leading to choice overload “in which the complexity of the their preferences to match available daters in the pool),
decision problem faced by an individual exceeds the individual’s increasing choice sets still produced an observable effect on
cognitive resources” [4]. Dating is a series of choices that are decision making behavior.
made within a social context, and these choices are still subject This review demonstrates that increased profile choice set
to many of the same issues as other decisions. Thus we rely on size affects perceptions of the difficulty of the selection task in
previous work on choice overload to predict that as choice task online dating (e.g., choice overload), and that increasing
difficulty increases, decisions also become more difficult, which selection task difficulty also produces effects such as decreased
increases the likelihood of a dater relying on a recommender tool decision making satisfaction and greater deviation from initial
to aid in mate selection. In this study we examine two factors preferences. Thus, an important question is whether increasing
that increase choice task difficulty by creating overload: choice choice set size will also increase daters’ intent to use a
set size and information attributes. recommender algorithm to help with mate selection as the task
2.1.1 Choice Set Size. Several experiments in the decision becomes more difficult.
making literature consistently indicate that increasing the 2.1.2 Profile Information Attributes. Another factor that may
assortment size of the choice set during a decision task increases increase the perceived difficulty of the choice task is the number
the effort required to make a choice, such that the benefit of information attributes along which a dater is described (e.g.,
provided by greater variety may not offset the energy that their demographics, hobbies, etc.). The more attributes displayed
individuals must expend to evaluate each additional option [e.g., in a dating profile increase the complexity of the choice since
1, 4, 15, 26, 30]. One study of recommender systems and overload comparisons among daters require contrasting the options across
in the context of movie selection found that increasing the more attributes, thus increasing cognitive effort. The presence of
number of recommended options in a choice set did increase multiple dimensions on which to compare potential partners
perceived variation for people, but did not necessarily boost their may also render selection more difficult as the options become
overall decision making satisfaction [1]. This result clearly more similar [26]. The number of information attributes
illustrates the aforementioned tradeoff underlying increasing set contained in profiles also represents a key design difference
size—though we may often enjoy the variety of a larger across popular dating platforms—for example, Tinder’s relatively
assortment, more options increase the difficulty of the choice. sparse amount of attributes per profile versus OkCupid’s more
One of the earliest studies [26] that examined varying choice extensive set of attributes per profile. We hypothesize that more
set size in online dating found that daters preferred choice sets information attributes displayed on a profile complicates the
containing roughly 20-50 profiles and anticipated feeling mate selection task and should lead to an increased willingness
overloaded and more frustrated by larger choice sets. In a more to rely on recommender algorithms in online dating.
recent experiment, D’Angelo and Toma [6] instructed
participants to choose a potential date from a set of either 4 or 24 2.2 User Personality: Adaptive Decision Making &
profiles, and then asked one week later if they were satisfied Need for Cognition
with their initial choice or if they would like to change their The literature also suggests that daters’ personal
selections. Daters choosing from the larger profile choice set characteristics might influence their perceptions of
were less satisfied with their choices and more likely to change recommender systems. Specifically, we predict that people’s
their choices, suggesting that choice set size affected people’s adaptive decision-making behavior—which refers to one’s ability
decision making response. to employ multiple strategies to make an optimal decision based
Elsewhere, Chiou and colleagues [24, 39, 40] predicted that on the environment [31]—may affect recommender adoption.
the larger the pool of potential dating partners, the worse the Because adaptive decision making depends on individuals’
final decision along two dimensions: (1) the “goodness of match,” overall “cognitive development, experience, and more formal
defined as the difference between the attributes of a preferred training and education”, we focus on users’ need for cognition
partner and attributes of the chosen partner, and (2) “selectivity,” (NFC) [2, 3] as a personality variable shown to be positively
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
correlated with strategic adaptive decision making behavior. In contrast, a priori trust suggests that people often approach
Indeed users’ personality characteristics have been shown to recommender technology with some general level of trust. That
affect their use of different recommender interfaces [21]. is, they do not enter into interactions with recommendation
How might NFC be associated with adaptive decision making algorithms as “blank slates”—instead, users often have some pre-
in the current context? Research in decision making suggests existing attitude toward technology that may influence their
that online daters with higher NFC should be more likely than future intentions to use it. Those few theorists who define users’
their low NFC counterparts to use a diversity of strategies, which a priori trust in machines have noted it consists of mostly
should include recommender systems. Using a recommender to positive impressions of technology as being very authoritative,
reduce options allows daters to engage in more explicit objective, highly accurate, and extremely credible [7, 36]. What
comparison of remaining profiles. Thus literature suggests that we call a priori trust has also been termed dispositional trust [29],
as the selection task gets more difficult, higher NFC individuals trust expectancy [25], and trusting propensity [21]; all of these
should exhibit a greater willingness to adopt recommender terms share a conceptual reference to those attitudes,
technology, reflecting their tendency toward adaptive behavior impressions, or expectations that users bring with them into a
[27, 31]. In contrast, those with low NFC are less likely to exhibit first encounter, prior to any experience with the recommender.
adaptive decision making because they strive to expend as little Clearly, these two classifications of human-machine trust
effort as possible. develop in very different ways. Users rely on direct observation
However, other findings point to an alternative relationship with an algorithm to form post facto trust, while a priori trust is
between daters’ NFC and their adoption of recommender “ultimately an affective response” [25]. However, most existing
systems in online dating. As reviewed above, Lee and Chiou [25] experiments often instruct participants to interact with a
found that high NFC daters were willing to search through more recommender system and then look at the effects of this
profiles than low NFC daters. Unlike the decision making interaction on user attitudes or behavior. This has led to a focus
research, these results suggest that high NFC daters may prefer on users’ post facto trust, and a failure to consider how a priori
the challenge of profile comparisons making them less likely to trust affects users’ willingness or resistance to adopt
use an algorithm compared to low NFC daters who would prefer recommender systems at the outset.
the recommender to help them select a date. To better understand how a priori human-machine trust
In this way, high NFC daters function as decision functions, the current study examines how two dimensions of a
“maximizers” who will spend the effort to find the best possible priori trust in recommender algorithms influence subsequent
option, as opposed to low NFC “satisficers” who simply find an intentions to use them: (1) a priori cognitive trust defined as users
option that fulfills basic criteria [34]. Interestingly, a study [21] “rational expectations” of the system’s integrity, ability, and
examining if differences in cognitive decision style affected reliability; (2) a priori emotional trust defined as “the extent to
peoples’ satisfaction with various recommender tools: which one feels secure or comfortable” about relying on the
Maximizers and satisficers did not display many differences in system [16]. This study tests whether users’ a priori cognitive
their satisfaction with different kinds of recommender interfaces and emotional trust in recommenders affects their desire to use
(e.g., Top-N, sort, etc.). In this study, however, choice sets were them in the context of online dating.
held constant at 80 options—it remains to be seen whether
increasing the difficulty of the choice interacts with cognitive 3. PRESENT STUDY
decision style to affect reactions to recommenders.
The present study tested how increasing choice set size and
Given this ambiguity across different literatures, the present
varying profile information attributes affected people’s intentions
study examines how daters’ NFC is related to their intent to use
to use recommender systems before actually interacting with
recommender systems for mate selection as the number of
them. After learning how many daters they would be choosing
options and attributes in each profile both increase.
from and how much information they could obtain about each
dater through their profile (see design, below), participants were
2.3 Attitude toward Technology: Trust in
asked how willing they would be to use a decision aid or a
Recommenders
delegated agent.
The final factor predicted to affect people’s use of While both recommenders were described as applying daters’
recommenders for online dating is their trust in recommenders. pre-specified criteria to the dating pool, the decision aid was
Theoretical models of human-to-machine trust often divide the described to participants as a system that assisted mate selection
trust construct into two categories: post facto and a priori trust. by reducing the number of available profiles in half. The
Users develop post facto trust after interacting with or delegated agent was described as a system that optimized mate
otherwise observing the recommender system. Post facto trust is selection by selecting the single, most compatible partner from
therefore founded upon people’s exchanges with the algorithm, the entire pool. This was a strong manipulation (reducing by half
akin to what [29] calls history-based trust. Notably, most existing vs. reducing to one), but it is reasonable in the context of online
information systems and recommender system studies have dating systems given that some major platforms reduce the
focused almost exclusively on users’ post facto attitudes and how choice pool to a single match, at least for a given time period
they affect user experience or subsequent adoption of (e.g., eHarmony, Once).
recommender tools [e.g., 7,19, 23].
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
3.1 Experimental Design 4 METHOD
Our study followed a 4 x 2 between subjects design. For our
4.1 Sample
manipulation of choice set size, we relied on previous work that
indicated that the average number of profiles a dater would A sample of 129 participants (Mage = 20.35, SD = 2.13; 76%
review in a single sitting was 170 [39]. Therefore, we created female) was recruited from a Midwestern university and
four choice set conditions with considerable range of options compensated with class credit. Beyond convenience, college
that were both below and above average: 4, 64, 204, 804. students are an appropriate population from which to sample, as
For the profile information attributes factor, the high 18-24 year olds represent the largest demographic group of
information condition (modeled on Match.com profiles) American online daters, with usage up to 27% in 2016, from just
contained 13 attributes of the dater (e.g., photo, screenname, age, 10% in 2013 [35]. Participants indicated their experience with
gender, location, height, about me section); the low information online dating using a scale of 1 = strongly disagree to 7 =
condition, modeled on Tinder profiles, only five attributes were strongly agree: “I feel totally comfortable with online dating,” “I
displayed. Figure 1 displays samples of experimental stimuli. am very experienced with online dating,” “I am familiar with
how online dating works” (α = .71, M = 4.01, SD = 1.32).
3.2 Hypotheses and Research Questions
4.2 Profile Stimuli
Our team content analyzed 150 publicly available online
dating profiles from a variety of websites and apps (PlentyofFish,
Match.com, OkCupid, Tinder, etc.) and used this information to
create the experimental stimuli. While stimuli profiles varied in
content, care was taken to ensure consistency across self-
description word count (e.g., 70-80 words), age (range 19-25), and
profile photos (e.g., no full body shots). Stimuli profile
attractiveness was controlled by carefully regulating the way in
which profiles were displayed in the ClassMate website. First,
Figure 1. Example stimulus profiles featuring main photos were judged for overall attractiveness by a group of
experimental manipulations. outside raters on a scale of 1 = “not at all attractive” to 10 “very
We first predict that elements that increase the perceived attractive”. A group of seven male judges rated female stimuli
difficulty of the mate selection task are related to intentions to photos, M = 6.32, SD = 1.03. A group of 10 female judges rated
use both types of recommender systems: male stimuli photos, M = 4.44, SD = 1.00. Using these ratings, a
H1: Increasing assortment in choice set size increases daters’ script was created so that a profile of average attractiveness
intentions to use (a) decision aid and (b) delegated agent would be displayed, followed by alternating profiles at one
recommender systems for online dating. standard deviation above and below the average. Thus
H2: Increasing information attributes in the profile increases attractiveness was balanced across choice set conditions.
daters’ intentions to use (a) decision aid and (b) delegated
agent recommender systems for online dating. 4.3 Procedure
Participants were told that the purpose of the study was to
As competing predictions exist regarding the effect of need test a new website called ClassMate.com that was being
for cognition on the relationship between selection task and developed specifically for college-aged singles. They were told
recommender system adoption, we ask a research question: that ClassMate was being tested at universities across the
RQ1: Does individual need for cognition moderate the country and that their campus was selected as a test site; in
relationship between choice task difficulty and daters’ actuality, this was a cover story. Procedures were approved by
intentions to use recommender systems for online dating? the university’s institutional review board.
Participants began by creating a profile and were told that it
Lastly, we advanced hypotheses regarding a priori trust: would be shown to others enrolled in the study. They completed
H3: Users’ a priori (a) emotional trust and (b) cognitive trust a pretest that asked for their desired preferences in a dating
are positively related to intentions to use recommender partner using 14 specific traits [37], and the need for cognition
systems for online dating. measure, M = 4.46, SD = 0.65, α = .83 [3]. This stage was done at
H4: Users’ a priori trust in recommendation algorithms home so that they could spend as much time as they desired on
moderates the relationship between choice task difficulty on their profiles.
willingness to use recommender systems, such that as mate Participants came to the lab for the next stage and were told
selection increases in difficulty, (a) greater emotional trust that their main task was to give their opinions about the
and (b) cognitive trust combine to produce greater reliance on “Selective Tracking and Relationship Test,” or START tool, that
online dating recommenders. was being developed for use in ClassMate. Participants watched
a short video that explained the two versions of the START
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
Table 1. Correlation among study variables
Variable 1 2 3 4 5 6 7 8 9
1. Profile Set
2. Information Attributes -.05
3. Perception of Overload .59** .05
4. Need for Cognition .05 .08 -.03
5. Emotional Trust -.007 -.03 .01 -.04
6. Cognitive Trust -.13 .02 -.07 -.08 .55**
7. Intent to Use Decision Aid -.06 .02 -.01 -.05 .51** .47**
8. Intent to Use Delegated Agent .18* -.06 .27** -.01 .41** .28** .32**
9. Online Dating Experience -.07 -.07 .007 -.15 .39** .33** .28** .25**
10. Sex -.19* .04 -.016 .04 -.10 -.13 -.16 -.06 0.09
Notes. N = 129. * p< .05. **p < .01.
recommender that were being tested. As noted above, the Participants were debriefed and asked whether they
decision aid recommender was described as a tool that applied suspected the purpose of the study or any of the manipulations.
users’ mate selection preferences to reduce the dating pool by Two who said they were suspicious of the experimental
eliminating 50% of the options, leaving a more narrowed choice procedures were excluded from inclusion in the final sample.
set from which daters could choose. The second variation was
the delegated agent recommender, which would make the 5. RESULTS
decision on behalf of the user by selecting the single, most
An ANOVA was used to examine whether our manipulations
compatible person from all available daters within the ClassMate
affected perceived selection task difficulty (e.g., perceptions of
network by applying their preferences.
overload). As expected, the choice set size variable produced
After receiving explanations about the different variations of
increasing perceptions of overload, F (3, 121) = 23.68, p < .001
recommender tools, participants were randomly assigned to
(Table 2). The interaction between the factors was not
experimental conditions described above and given up to 5
significant, F (3, 121) = 1.41, p = .24, nor was the main effect for
minutes to familiarize themselves with the profiles and website
information attributes on perceptions of overload, F (1, 121) =
interface. They were told they could use the ClassMate site more
1.60, p = .21. In sum, only the choice set size manipulation
extensively after they completed the posttest questionnaire.
contributed to participants’ selection task experience.
The posttest began with checks on the manipulations of
selection task difficulty, which we operationalized as perceptions 5.1 Effects of Selection Task Elements on
of choice overload. We used two items adapted from [15], “I think
Intentions to Use Recommender Systems
the number of daters in the ClassMate network is…” 1 = far too
little, 7 = far too many; “I wish the dating pool contained ___ Did the manipulations of choice set size affect willingness to
people” 1 = many fewer, 7 = many more, M = 2.70, SD = 0.79. use recommender systems? A linear contrast analysis [32] was
Because participants did not directly engage with either form used to test the predicted relationship between increasing choice
of recommender, their measures of trust were based solely on set size and willingness to use recommender systems in H1.
their initial impressions of the algorithm. To assess a priori Regarding willingness to use a decision aid, the result was not
emotional trust, we adapted 3 items [19] that used a 1 = strongly significant, F (1, 125) = 0.42, p = .52; however, the contrast was
disagree to 7 = strongly agree scale (e.g., “I feel comfortable significant for willingness to use a delegated agent, F (1, 125) =
relying on START for my romantic matching decisions”), M = 3.96, p = .049 (see Table 2). In summary H1a was not supported;
4.41, SD = 1.26, α = .92. For a priori cognitive trust, we adapted 5 data were consistent with H1b.
items [19] (e.g., “START seems to have good knowledge about There was no effect of information attributes on willingness
the daters”), M = 5.15, SD = 0.74, α = .93. to use a decision aid, F (1, 127) = 0.47, p = .50, or for on intentions
Participants indicated their intentions to adopt the decision aid to use a delegated agent, F (1, 127) = .06, p = .80. These data failed
on a 1 = strongly disagree to 7 = strongly agree scale (e.g., “I am to support H2a and H2b.
willing to use START as a tool that suggests to me a number of These initial results helped us to refine our analytic strategy:
potential partners from which I can choose”), M= 5.42, SD = 1.02, Given the results of the manipulation check and lack of support
α = .91, and intentions to adopt the delegated agent (“I intend to let for H2, the information attributes variable was dropped from
START choose my best romantic match on my behalf”), M = 3.82, subsequent analyses, as it did not affect participants’ perception
of the online dating selection task. Furthermore, the results of
SD = 1.52, α = .93. Behavioral intent is an important factor
H1 clearly indicated that choice set size produced an effect on
measured in both the Technology Acceptance Model [7] and the
intentions to use a delegated agent, but not on intent to use a
Theory of Planned Behavior [10], the latter of which was
decision aid (nor were there any interaction effects). Therefore,
adapted for the current study, following [19]. To avoid order
in examining the moderating effects of need for cognition (RQ1)
effects, the presentation of these two behavioral intention scales
and effects of a priori trust (H3, H4), we included users’ intent to
was randomized in the posttest.
use a delegated agent, and excluded intent to use a decision aid.
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
Table 2. Means and Standard Deviations for Profile Set Size (3.83) and one standard deviation above (5.11). As seen in Figure
Conditions on Perceptions of Overload and Intent to Use 2, we find that among those at or below the mean in NFC, choice
Recommenders set size did not seem to influence intentions to use the delegated
agent. Increasing choice set size only influenced those who were
Choice Set Perceptions Intent to Use Intent to Use
Condition of Overload Decision Aid Delegated high in NFC to use the recommender for mate selection tasks,
2
(M, SD) (M, SD) Agent (M, SD) ΔR = .09, F (3, 119) = 4.15, p = .01, suggesting that those higher
4 profiles 2.15 (0.71) 5.46 (0.98) 3.45 (1.61) in NFC may use the algorithm as a form of adaptive behavior.
64 profiles 2.50 (0.77) 5.54 (0.76) 3.54 (1.37)
5.3 A Priori Trust in Algorithms
204 profiles 2.79 (0.58) 5.42 (0.90) 3.84 (1.37)
804 profiles 3.22 (0.57) 5.34 (1.16) 4.13 (1.48) To examine predictions regarding a priori trust, we again
used the SPSS PROCESS macro. We began by testing the
5.2 Users’ Need for Cognition relationship between choice sets and intention to use a delegated
agent recommender, as a function of emotional trust (H4a).
Conditional process modeling [12] was used to test the Examination of the moderating effect began with a formal test of
moderation detailed in RQ2. Analyses were performed in SPSS 2
significance for ∆R , which was not significant F (3, 119) = 1.39, p
version 25 using the PROCESS macro; this allowed us to analyze = .25. However there was a significant association between a
the effect of choice set size (X) on participants’ willingness to priori emotional trust and intent to use the delegated agent that
use the SMART algorithm (Y) as a function of their NFC (M), supported H3a, b = 0.35, SE = 0.15, t = 2.36, p = .02.
using 10,000 resamples and controlling for age and online dating 2
Our test of H4b revealed that the formal test of ∆R was not
experience as covariates. Because our independent variable significant, F (3, 119) = 0.62, p = .60, suggesting no interaction of
contained four categories, PROCESS created three dummy cognitive trust and choice set size on intent to use a delegated
variables (D1, D2, D3). We selected indicator coding based on the agent recommender. A significant positive association between
pattern of means uncovered in H1. Specifically, we used the 4- users’ cognitive trust and willingness to use the delegated agent
profile condition as the reference group, which resulted in (D1 = obtained, b = 0.78, SD = 0.11, t = 2.58, p = .01, supporting H3b.
D2 = D3 = 0); D1 coded the 64-profile condition (D1 = 1, D2 = D3 =
0); D2 coded the 204-profile condition (D1 = D3 = 0, D2 = 1); and
6. DISCUSSION
D3 coded the 804-profile condition (D1 = D2 = 0, D3 = 1).
Following the analytic procedure outlined by [13], we first
tested the unconstrained model, which was significant, F (5, 123)
2
= 2.96, p = .015. The subsequent test of the increase in R when
the moderation effect of NFC was added to the model was also
2
significant, ΔR = .06, F (3, 119) = 2.77, p = .04. Thus, with respect
to RQ1 the effect of the number of profiles on participants’
willingness to use the algorithm depends on their NFC (Table 3).
To probe this effect, we used the simple slope procedure at
values of NFC’s average (4.47), one standard deviation below
Table 3. Regression of profile set condition on intent to use
the delegated agent for mate selection when need for
cognition is the moderator
Figure 2. Visualization of the interaction between profile set
b SE t p LLCI ULCI conditions and need for cognition.
Constant 4.26 1.58 2.70 .01 1.14 7.38 The current study examined how choice set size and profile
Sex -0.22 0.33 - .49 -0.87 0.42
attributes affected daters’ intent to use two types of
0.69
recommender systems. We find that within an online dating
Online Dating 0.33 0.11 3.04 .001 0.11 0.54
Experience context, and before ever interacting with a system, most users
Need for -0.42 0.33 -1.25 .21 -1.08 0.24 are comfortable ceding some control to a decision aid
Cognition recommender to streamline their mate selection. The surprising
D1 -2.27 4.89 -0.47 .64 -11.95 7.40 lack of effect of choice set size or information attributes for
D2 1.18 2.65 0.45 .66 -4.06 6.43 decision aids suggests that people have faith in using a
D3 -4.29 2.04 -2.10 .04 -8.34 -0.25 recommender tool to filter out the bottom half of their pool,
D1xNFC 0.47 1.10 0.42 .67 -1.72 2.65 perhaps because they perceive that they still maintain some
D2xNFC -0.24 0.59 -0.40 .69 -.141 0.94 control over their final choice.
D3xNFC 1.09 0.45 2.43 .02 0.20 1.98 Daters are more discerning, however, when it comes to
Note. D = dummy variables using indicator coding, b = unstandardized
coefficients, SE = standard error, LLCI and ULCI = 95% confidence intervals. delegated agent recommenders, and intended to use them only
when they perceived a difficult selection task with a large
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
amount of options. This is perhaps not surprising given how different kinds of systems that provide more variation in
much control participants had to give to the delegated agent, recommendations (e.g., a Top-N versus sort/filter interface).
essentially reducing their dating pool to one person. This We also note that the profile attributes used in the current
suggests there is a threshold for when decision making becomes study were modeled on the profiles of popular websites
so difficult that a delegated agent becomes attractive. (Match.com) and apps (Tinder) that feature these specific
Although our explanation regarding differential nature of attributes. Although the attribute manipulation did not
control across decision aid and delegated agent conditions are contribute to perceptions of choice task difficulty, the limited
speculative, they parallel previous work in both online dating range of these specific attributes is a limitation, and future work
partner selection [38] and recommender systems, more generally could examine other combinations or types of attributes to see if
[1, 21, 22]. The concept of control is an important factor to they affect decision-making in this context.
consider when studying user adoption; and as dating decisions As our focus was on people’s responses to recommenders
can be construed as more “high stakes” in comparison to those in prior to actual use, we did not examine daters’ experiences with
previous studies (e.g., movies or music), it might be an either decision aids or delegated agents. In essence, this study
interesting domain in which to conduct future testing of system focused more on users’ process as opposed to outcomes—“being
features like variable user control or system transparency. satisfied with the system itself and the outcomes of using it are
Notably, the choice set manipulation indicated that daters two separate concerns that may at times even be in conflict with
were willing to deal with the difficulty of selecting from 200 one another” [21, p. 147]. Future work should compare daters’
profiles. Although choice set size significantly influenced daters’ expectations for online dating recommender technology to their
perceptions of selection task difficulty (e.g., overload), even the actual user experience and examine how consistent they are.
largest condition only produced a sense of overload equivalent to That we examined trust regarding a fictional dating service of
a mean of 3.22 on the 7-point scale. This suggests that daters are ClassMate.com is also a limitation; however it would also be
willing to process much larger dating pools (e.g., 804) than interesting to see how users’ a priori impressions about an
research indicated approximately a decade ago. For example, already existing company affect their desires to use new
[40] included 90 daters in their “large” condition compared to services. For example, general trust in Facebook has likely
our 804. A question for future work is to understand why declined given recent events surrounding the 2016 US
increased tolerance has been observed. Presidential election (e.g., data mining by Cambridge Analytica).
One possibility is that people have adapted to higher Would users’ feelings toward Facebook as a company affect their
information density in their online experiences, including online likelihood of using the new Dating feature? Our results suggest
dating. The Flynn Effect, for example, describes the increase in that a priori trust affects users’ adoption of a new system, but
intelligence test scores, with each generation scoring higher than investigations of how trust in a familiar company may affect
previous generations, which mandates that scores be re- people’s willingness to use new services offered by that
standardized every few years [11]. It is possible the current company would be an exciting direction for future work.
generation of online daters have cognitively adapted to larger Finally, we note the limitations associated with a college
dating pools than previous generations were accustomed to. student sample—though the participants in our study are
Clearly, additional work is required to examine this change in representative of the largest group of mobile and online dating
willingness to select from large sets of dating profiles, but our users (those aged 18-24), future studies could examine
results suggest that need for cognition is an important factor: recommender use among more diverse groups of daters who
Daters high in NFC are willing to sift through larger profile sets may approach dating (and recommender technology) differently.
making them less likely to want to use the delegated agent for
smaller choice sets; it is not until they are facing 800 profiles that 6. CONCLUSION
they recognize that the task has become too difficult and adapt
The results of our experiment renew earlier calls to refine the
their behavior. In contrast, daters lower in NFC were more likely
design of recommender tools and move beyond the “one-size-fits
to use the recommender when faced with any profile set size.
all” approach [1, 17, 21, 22] and instead consider how users’
Although trust did not interact with choice conditions, the
personal characteristics affect their responses to technology.
more trust in recommender agents people reported, the more
This appears to also be important in the online dating context, in
willing they were to use the system. This is consistent with other
which choice overload is a key complaint. Our results suggest
work examining user trust [21, 22] and suggests that people’s
that daters differ in their desire for recommenders; developers
initial feelings about recommenders can affect potential use.
may consider offering customizable features that reflect
individual differences in user personality and cognitive decision
6.1 Limitations and Future Research
style will allow people to trust the recommender and facilitate
Our decision to instantiate the delegated agent condition as alignment of the system with users’ relational goals. Focusing on
selecting a single dater was a deliberate effort to reflect the the users’ psychological makeup might facilitate more effective,
functionality of websites like eHarmony and apps like Only. intelligible design of recommender systems embedded in social
While this enhanced ecological validity and increased domains, such as online dating.
experimental variance across the two recommender system
conditions, future research could examine people’s response to
IntRS Workshop, October 2018, Vancouver, Canada Tong et al.
[21] Bart P. Knijnenburg, Niels J. M. Reijmer, and Martijn C. Willemsen. 2011.
ACKNOWLEDGMENTS Each to his own: How different users call for interaction methods in
th
This study was supported by NSF #1520723; thanks to R. Slatcher recommender systems. In Proceedings of the 5 ACM Conference on
Recommender Systems. (RecSys ’11) ACM, New York, NY: 141-148. doi:
for his feedback; W. Cooper, A. Elam R. Prince, B. Jefferson, A.M. 10.1145/2043932.2043960
P Rochadiat & K. Wibowo for their help with data collection. [22] Knijnenburg B.P., Willemsen M.C. 2015. Evaluating Recommender Systems
with User Experiments. In: Ricci F., Rokach L., Shapira B. (eds) Recommender
Systems Handbook. Springer, Boston, MA: 309-352. doi: 10.1007/978-1-4899-
REFERENCES 7637-6_9
[1] Dirk Bollen, Bart P. Knijnenburg, Martijn C. Willemsen, and Mark Graus. [23] Pigi Kouki, James Schaffer, Jay Pujara, John O'Donovan, and Lise Getoor.
2010. Understanding choice overload in recommender systems. In User Preferences for Hybrid Explanations. 2017. In Proceedings of the 11th
Proceedings of the 4th ACM conference on Recommender Systems (RecSys ACM Conference on Recommender Systems. (RecSys ’17) ACM, New York,
’10) ACM, New York, NY: 63-70. ACM, doi: 10.1145/1864708.1864724 NY: 84-88. DOI: 10.1145/3109859.3109915
[2] John T. Cacioppo and Richard E. Petty. 1982. The need for cognition. [24] Chun-Chia Lee and Wen-Bin Chiou. 2016. More eagerness, more suffering
Journal of Personality and Social Psychology. 42, 1 (Jan. 1982), 116-131. from search bias: Accuracy incentives and need for cognition exacerbate the
DOI:10.1037/0022-3514.42.1.116 detrimental effects of excessive searching in finding romantic partners
[3] John T. Cacioppo, Richard E. Petty, and Chuan F. Kao. 1984. The efficient online. Journal of Behavioral Decision Making. 29, 1 (Jan. 2016), 3-11. DOI:
assessment of need for cognition. Journal of Personality Assessment. 48, 3 10.1002/bdm.1852
(May 1984), 306-307. DOI: 10.1207/s15327752jpa4803_13 [25] John D. Lee and Katrina A. See. 2004. Trust in automation: Designing for
[4] Alexander Chernev, Ulf Böckenholt, and Joseph Goodman. 2015. Choice appropriate reliance. Human Factors. 46, 1 (March 2004), 50-80. DOI:
overload: A conceptual review and meta-analysis. Journal of Consumer 10.1518/hfes.46.1.50_30392
Psychology. 26, 2 (Aug. 2015), 333-358. DOI: 10.1016/j.jcps.2014.08.002 [26] Alison P. Lenton, Barbara Fasolo, and Peter M. Todd. 2008. “Shopping” for a
[5] Li Chen, Marco De Gemmis, Alexander Felfernig, Pasquale Lops, Francesco mate: Expected versus experienced preferences in online mate choice. IEEE
Ricci, and Giovanni Semeraro. 2013. Human decision making and Transactions on Professional Communication. 51, 2 (Jun. 2008), 169-182. DOI:
recommender systems. ACM Transactions on Interactive Intelligent Systems. 3, 10.1109/TPC.2008.2000342
3 (Oct. 2013), Article 17. DOI: 10.1145/2533670.2533675 [27] Irwin P. Levin, Mary E. Huneke, and J. D. Jasper. 2000. Information
[6] Jonathan D. D’Angelo and Catalina L. Toma. 2016. There are plenty of fish in processing at successive stages of decision making: Need for cognition and
the sea: The effects of choice overload and reversibility on online daters’ inclusion-exclusion effects. Organizational Behavior and Human Decision
satisfaction with selected partners. Media Psychology. 20, 1 (Feb. 2016), 1-27. Processes. 82, 2 (Jul. 2000), 171-193. DOI: 10/1006/obhd.2000.2881
DOI: 10.1080/15213269.2015.1121827 [28] Louise Matsakis. 2018. Facebook’s New “Dating” Feature Could Crush Apps
[7] Fred D. Davis (1993). User acceptance of information technology: system Like Tinder. (May 2018). Retrieved from:
characteristics, user perceptions and behavioral impacts. International journal https://www.wired.com/story/facebook-dating/
of man-machine studies. 38, 3 (March 1993), 475-487. doi: [29] Stephanie M. Merrit and Daniel R. Ilgen. 2008. Not all trust is created equal:
10.1006/imms.1993.1022 Dispositional and history-based trust in human-automation interactions.
[8] Berkely J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm Human Interactions. 50, 2 (Apr. 2008), 194-210. DOI:
aversion: People erroneously avoid algorithms after seeing them err. Journal 10.1518/001872008X288574
of Experimental Psychology: General. 144, 1 (Feb. 2015), 114-126. DOI: [30] Harmen Oppewal and Kitty Koelemeijer. 2005. More choice is better:
10.1037/xge0000033 Effects of assortment size and composition on assortment evaluation.
[9] Eli J. Finkel, Paul W. Eastwick, Benjamin R. Karney, Harry T. Reis, and Susan International Journal of Research in Marketing. 22, 1 (Jan. 2005), 45-60. DOI:
Sprecher. 2012. Online dating: A critical analysis from the perspective of 10:1016/j.ijresmar.2004.03.002
psychological science. Psychological Science in the Public Interest. 13, 1 (Mar. [31] John W. Payne, James R. Bettman, Eric J. Johnson. 1988. Adaptive strategy
2012), 3-66. DOI: 10.1177/1529100612436522 selection in decision making. Journal of Experimental Psychology. 14, 3 (Jul.
[10] Martin Fishbein, and Icek Ajzen. 2010. Predicting and changing behavior: The 1988), 534-552. DOI: 10.1037/0278-7393.14.3.434
reasoned action approach. New York: Psychology Press. [32] Rosenthal, R., Robert, R., & Rosnow, R. L. (1985). Contrast analysis: Focused
[11] James R. Flynn. 1987. Massive IQ gains in 14 nations: What IQ tests really comparisons in the analysis of variance. CUP Archive.
measure. Psychological Bulletin. 101 (March 1987), 171-191. DOI: [33] Benjamin Scheibehenne, Rainer Greifeneder, and Peter M. Todd. 2010. Can
10.1037/0033-2909.101.2.171 there ever be too many options? A meta-analytic review of choice overload.
[12] Andrew F. Hayes. 2017. Introduction to Mediation, Moderation, and Journal of Consumer Research. 37, 3 (Oct. 2010), 409-425. DOI:
Conditional Process Analysis: A Regression-Based Approach (2nd. ed.). 10/1086/651235
Guilford Press, New York, NY. [34] Barry Schwartz, Andrew Ward, John Monterosso, Sonja Lyubomirsky,
[13] Andrew F. Hayes and Amanda K. Montoya 2017. A tutorial on testing, Katherine White, and Darrin R. Lehman. 2002. Maximizing versus
visualizing, and probing an interaction involving a multicategorical variable satisficing: Happiness is a matter of choice. Journal of Personality and Social
in linear regression analysis. Communication Methods and Measures, 11:1, 1- Psychology. 83, 5, (Nov 2002), 1178-1197. doi: 10.1037//0022-3514.83.5.1178
30, DOI: 10.1080/19312458.2016.1271116 [35] Aaron Smith and Monica Anderson. 2016. “5 facts about online dating.”
[14] Ellora Israni. 2017. When an Algorithm Helps Send You to Prison. (October (February 2016). Retrieved from http://www.pewresearch.org/fact-
2017). Retrieved from: www.nytimes.com/2017/10/26/opinion/algorithm- tank/2016/02/29/5-facts-about-online-dating/
compas-sentencing-bias.html [36] S.Shyam Sundar. 2008. The MAIN Model: A heuristic approach to
[15] Sheena S. Iyengar and Mark R. Lepper. 2000. When choice is demotivating: understanding technology effects on credibility. In Miriam J. Metzger and
Can one desire too much of a good thing? Journal of Personality and Social Andrew J. Flanagin, eds. Digital Media, Youth, and Credibility. The John D.
Psychology. 79, 6 (Dec. 2000), 995-1006. DOI: 10.1037/0022-3514.79.6.995 and Catherine T. MacArthur Foundation Series on Digital Media and
[16] Anthony Jameson. Adaptive interfaces and agents. Human- computer Learning. Cambridge, MA: The MIT Press, 73–100. DOI:
interaction handbook. J. A. Jacko and A. Sears, eds. Laurence Earlbaum 10.1162/dmal.9780262562324.073
Associates, New York, NY. 305-330. [37] Natasha D. Tidwell, Paul W. Eastwick, and Eli J. Finkel. 2013. Perceived, not
[17] Anthony Jameson, Martijn C. Willemsen, Alexander Felfernig, Marco de actual, similarity predicts initial attraction in a live romantic context:
Gemmis, Pasquale Lops, Giovanni Semeraro, and Li Chen. Human decision Evidence from the speed-dating paradigm. Personal Relationships. 20, 2 (May
making and recommender systems. In Francesco Ricci, Lior Rokach, and 2012), 199-215. DOI: 10.1111/j.1475-6811.2012.01405.x
Bracha Shapira, eds. Recommender Systems Handbook. Boston, MA: Springer, [38] Stephanie Tom Tong, Jeffrey T. Hancock, and Richard B. Slatcher. 2016.
611-648. doi: 10.1007/978-1-4899-7637-6_18 Online dating system design and relational decision-making: Choice,
[18] Sherrie X. Komiak and Izak Benbasat. 2004. Understanding customer trust in algorithms, and control. Personal Relationships. 23, (Oct 2016) 645–662. doi:
agent-mediated electronic commerce, web-mediated electronic commerce, 0.1111/pere.12158
and traditional commerce. Information Technology and Management. 5, 1-2 [39] Pai-Lu Wu and Wen-Bin Chiou. 2009. More options lead to more searching
(Jan. 2004), 181-207. DOI: 10.1023/B:ITEM.0000008081.55563.d4 and worse choices in finding partners for romantic relationships online: An
[19] Sherrie X. Komiak and Izak Benbasat. 2006. The effects of personalization experimental study. Cyberpsychology, Behavior & Social Networking. 12, 3
and familiarity on trust and adoption of recommendation agents. MIS (Jun. 5), 315-318. DOI: 10.1089/cpb.2008.0182
Quarterly. 30, 4 (Dec. 2006), 941-960. DOI: 10:2307/25148760 [40] Mu-Li Yang and Wen-Bin Chiou. 2010. Looking online for the best romantic
[20] Bart P. Knijnenburg, Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. partner reduces decision quality: The moderating role of choice-making
2012. Inspectability and control in social recommenders. In Proceedings of strategies. Cyberpsychology, Behavior & Social Networking. 13, 2 (Apr. 2010),
the sixth ACM conference on Recommender systems (RecSys '12). ACM, New 207-210. DOI: 10.1089=cyber.2009.0208
York, NY, USA, 43-50. doi:10.1145/2365952.2365966