<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>September</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>A Multistakeholder Recommender Systems Algorithm for Allocating Sponsored Recommendations∗</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Edward C. Malthouse</string-name>
          <email>ecm@northwestern.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Khadija Ali Vakeel∗</string-name>
          <email>khadija@northwestern.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Yasaman Kamyab Hessary†</string-name>
          <email>yasaman.kamyab@northwestern.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Robin Burke</string-name>
          <email>robin.burke@colorado.edu</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Morana Fudurić</string-name>
          <email>mfuduric@efzg.hr</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Northwestern University</institution>
          ,
          <addr-line>Evanston, Illinois</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>University of Colorado</institution>
          ,
          <addr-line>Boulder, Colorado</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>University of Zagreb</institution>
          ,
          <addr-line>Zagreb</addr-line>
          ,
          <country country="HR">Croatia</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <volume>1</volume>
      <fpage>6</fpage>
      <lpage>20</lpage>
      <abstract>
        <p>Retailing and social media platforms recommend two types of items to their users: sponsored items that generate ad revenue and nonsponsored ones that do not. The platform selects sponsored items to maximize ad revenue, often through some form of programmatic auction, and non-sponsored items to maximize user utility with a recommender system (RS). We develop a multiobjective binary integer programming model to allocate sponsored recommendations considering a dual objective of maximizing ad revenue and user utility. We propose an algorithm to solve it in a computationally eficient way. Our method can be applied as a form of post processing to an existing RS, making it widely applicable. We apply the model to data from an online grocery retailer and show that user utility for the recommended items can be improved while reducing ad revenue by a small amount. This multiobjective approach, which unifies programmatic advertising and RS, opens a new frontier for advertising and RS research and we therefore provide an extended discussion of future research topics.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>CCS CONCEPTS</title>
      <p>• Information systems → Data mining; Collaborative
filtering; Computational advertising.</p>
    </sec>
    <sec id="sec-2">
      <title>INTRODUCTION</title>
      <p>A two-sided market is an intermediary platform between two
distinct stakeholder groups that provide each other with network
benefits. The media have long been held up as an exemplar. Media
companies attract an audience of consumers with, for example,
news or entertainment content (e.g., a TV program or magazine),
and then sell advertisers access to the audience. The two
stakeholder groups are consumers and advertisers. In recent years, a
wide variety of diferent types of two-sided markets have been
∗Copyright 2019 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).</p>
      <p>
        Presented at the RMSE workshop held in conjunction with the 13th ACM Conference
on Recommender Systems (RecSys), 2019, in Copenhagen, Denmark.
†Both authors contributed equally to this research.
introduced, often under the name platform rather than two-sided
market [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. Here are some examples of successful platforms:
• Shopping platforms. Taobao in China and companies like
Shoprunner and Amazon Marketplace in the US are
platforms that match consumers with retailers and merchants.
      </p>
      <p>Related shopping examples include Ebay and Etsy.
• Online travel agencies (OTAs) such as Orbitz, Expedia and
Google match consumers with travel-related services (e.g.,
hotels, airlines, car rentals). AirBnB, VRBO and Uber are
other successful platforms in the travel category.
• Social media such as Facebook and LinkedIn match
consumers with user-generated content and advertisers.
• Online retailers often act as platforms, matching brands
(advertisers) with customers.</p>
      <p>This article studies recommendations on platforms with two
stakeholder groups, users and providers (e.g., advertisers). The
platform attracts users and matches them with providers through
recommendations. We assume two types of provider recommendations,
sponsored and non-sponsored, where sponsored providers (i.e.,
advertisers) pay the platform for access to the user and non-sponsored
providers do not. For example, online retailing shopping platforms
might show sponsored items from certain manufacturers, as well
as items from manufacturers or vendors that the platform thinks
the user would like.</p>
      <p>
        Sponsored recommendations are often sold on programmatic
auctions where the platform can adopt various protocols to sell slots
[
        <xref ref-type="bibr" rid="ref16 ref20">16, 20</xref>
        ] to the highest bidder, e.g., second-price auctions where the
winner pays the second-highest bid amount. Since non-sponsored
recommendations, by definition, do not generate ad revenue, we
assume that platforms select non-sponsored items to be relevant to
the user. Later we discuss other factors that the platform may take
into consideration such as one non-sponsored item could have a
larger margin than another. Core assumptions of this paper are that
sponsored recommendations are sold to maximize ad revenue to
the platform, while non-sponsored recommendations are selected
to provide utility to the user.
      </p>
      <p>The focus of this article is on deciding which sponsored items, if
any, to recommend to each user. Platforms must balance between
recommending sponsored items with high ad revenue and high
user utility. We express the two objectives in a single equation:
Q = γ (user utility) + (1 − γ )(ad revenue),
(1)
where γ is a parameter that determines how much weight to give
to each of the two objectives. A platform that allocates sponsored
recommendations with an auction has γ = 0 and is maximizing ad
revenue (depending on how much the advertiser knows about the
active user and how good its predictive models are, ad bids will also
reflect user utility). RS that consider only the user’s utility such as
those used to allocate non-sponsored recommendations have γ = 1.
Rather than focusing on one or the other, this equation allows for
a continuum between the two, competing objectives.</p>
      <p>An idealized example clarifies this trade-of, which will be
empirically investigated later in the paper. Suppose that advertiser A
places the highest bid for a sponsored recommendation to some
user. Advertiser B places a bid that is only slightly less than A’s,
but would provide the user with substantially more utility than
A. It would seem reasonable for the platform to give the slot to
B even though it would generate less ad revenue. There could be
ifnancial justifications for this decision as well, such as B may lead
to higher lifetime value, where the user buys B, and ultimately is
satisfied with it, and buys again from the platform into the future.
If the retailer had shown A then either the user would not buy it,
or would not be satisfied, and would be less likely to return in the
future. Of course, an alternative scenario is also conceivable, where
the ad revenue from B is substantially lower and/or B provides
no advantage over A in user utility. In this second scenario the
platform should display A.</p>
      <p>
        This is an example of a multiobjective, multistakeholder
recommendation system (MRS) [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. The system (platform) must
recommend one set of stakeholders (providers) to another group (users),
while balancing the two objectives of generating ad revenue and
creating utility for users. We develop the MRS problem specifically
for sponsored recommendations, where a platform must select a
personalized list of providers to recommend to each user.
      </p>
      <p>
        We formulate a post-processing binary integer programming
model to maximize the dual-objective function in equation (1) over
all users, subject to constraints that the advertiser has budget to pay
for the recommendation and the number of sponsored
recommendations to any user has not exceeded a threshold. We show how
to solve the optimization problem optimally in a computationally
eficient way. We test it using real data from an online grocery
store. With this approach, we make the following contributions.
First, we integrate the fields of computational advertising with RS.
Second, we show how to handle a situation where a platform is
interspersing non-sponsored recommendations with sponsored ones.
While sponsored recommendations guarantee the platform a
constant stream of revenue, by balancing user utility and ad revenue,
managers can generate higher customer lifetime value [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ].
      </p>
    </sec>
    <sec id="sec-3">
      <title>LITERATURE REVIEW 2 2.1</title>
    </sec>
    <sec id="sec-4">
      <title>Programmatic Auctions Maximizing Ad</title>
    </sec>
    <sec id="sec-5">
      <title>Revenue to Platforms</title>
      <p>
        Search and display advertising is often sold using programmatic
auctions. An auction is held for an impression and diferent
advertisers place bids. Auctions are typically second-price, where the
winner pays the second-highest bid amount. Second-price auctions
have an optimal strategy for an advertiser of bidding the expected
value of the impression [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. There is a rich literature on bidding
strategies. For example, [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] presented a bid-optimization approach
that combines several supervised learning algorithms, as well as
second-price auction theory, to determine the correct price and
ensure that the right message is delivered to the right person, at
the right time. [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] proposed a way to leverage machine learning to
predict the winning amount from the bidding history. It is
important to note that in the above strategies, the platform is accepting
the best bid and is therefore maximizing only its ad revenue. The
platform does not consider the user’s utility when it awards the
exposure to an advertiser.
      </p>
      <p>
        Our formulation of the MRS problem for sponsored
recommendations is related to the AdWords problem [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], which considers
“online” situations where user queries come in over time to some
system and assumes (1) a set of bids from advertisers for a query; (2)
known (or predicted) click-through rates (CTR) for each
advertiserquery pair; (3) a budget for each advertiser; and (4) an upper bound
for the number of ads to be displayed with each query. Algorithms
solving the problem return search results that do not exceed the
upper bound, and the advertisers must both bid on the search term
and have budget to pay if the ad is clicked on.
2.2
      </p>
    </sec>
    <sec id="sec-6">
      <title>Multistakeholder RS</title>
      <p>
        As we mentioned earlier, traditional RS have an exclusive focus on
one stakeholder, the user. In other words, traditional RS do not
accommodate more complicated systems with multiple objectives and
stakeholders. Multistakeholder recommender system (MRS) refers
to a designs that incorporate the interests of other parties [
        <xref ref-type="bibr" rid="ref1 ref6">1, 6</xref>
        ].
While [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] conceptualized the MRS problem, there is no known
general solution and diferent authors have started to address MRS
subproblems. [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] use an experimental study to show that RS that
consider profits can have a positive efect on purchasing without
diminishing trust. [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] also demonstrate that RS can increase
revenue without a significant drop in user satisfaction. One of the
ifrst attempts to propose a model for the MRS problem for MSPs is
[
        <xref ref-type="bibr" rid="ref22">22</xref>
        ], which focused specifically on promoting retailer retention by
guaranteeing a minimum level of exposures to each.
      </p>
      <p>
        Following [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ], we also propose a post-processing optimization
problem. Our contribution is to formulate an optimization model
specifically for the sponsored recommendation ad problem. Rather
than constraining the number of recommendations that each
retailer (provider) receives, our model includes two terms in the
objective function for the diferent stakeholders and provides a
tuning parameter to determine how much weight to give to each.
Our model allows for advertisers to have specified budgets and also
allows for the system to restrict the number of sponsored
recommendations that any user receives to not exceed some threshold.
3
      </p>
    </sec>
    <sec id="sec-7">
      <title>THE MULTISTAKEHOLDER SPONSORED</title>
    </sec>
    <sec id="sec-8">
      <title>RECOMMENDATION ALGORITHM</title>
      <p>We assume that a platform has item set I = {1, . . . , n}, some of
which are sponsored. Our task is to identify top-k lists for each
of m users in the set U = {1, . . . , m}. Let R be the m × n rating
matrix, where rui is the rating of item i by user u and brui is the
predicted rating from some traditional RS. Among the k items, at
most smax ≤ k can be sponsored, which allows the platform to set
an upper limit on the number of ads to show any single user. For
now, we assume the ofline assignment problem , where the m users
are known in advance. This situation arises when, for example, a
system plans to send email promotions to m known users.1 Let
aui be the ad revenue (or expected revenue, as discussed below)
associated with displaying item i to user u. If item i is not sponsored
then aui ≡ 0. Let βi be the total ad budget associated with item i,
where non-sponsored items have βi ≡ 0.</p>
      <p>We model the recommendation output via decision variable xui
where xui = 1 if item i is included in the top-k list for user u and 0
if not. The m × n matrix of xui variables, denoted in matrix form
as X, determines which recommendations will be delivered in the
email promotion. We propose the following optimization model:</p>
      <p>User utility

γ zÕ Õ}|

 u ∈U i ∈I


max</p>
      <p>X
subject to
Õ
xui = k</p>
      <p>(∀u ∈ U)
i ∈I
xui ∈ {0, 1} (∀i ∈ I, u ∈ U)
Õ</p>
      <p>1(aui xui &gt; 0) ≤ smax (∀u ∈ U)
i ∈I
Õ
aui xui ≤ βi (∀i ∈ I)</p>
      <p>{
brui xui +(1 − γ )
u ∈U
We now discuss each part of the model, starting with the
objective function in (2). The optimization problem is to select over all
possible X values, i.e., the top-k lists for the m users, the values of
X to maximize a two-part objective. The first part, labeled “user
utility,” measures the total utility of the items recommended to all
users. Product brui xui equals 0 for all items not selected, and the
estimated utility for those selected; terms are summed over all users
and items. The second part of the objective function, labeled “Ad
revenue,” measures the total revenue from recommended sponsored
items. Product aui xui equals 0 for non-recommended items and
aui for recommended ones. The analyst-specified tuning
parameter γ (0 &lt; γ ≤ 1) determines how much weight to assign to the
two parts. When γ = 1 the objective gives all weight to customer
utility and produces the same solution as a traditional RS. When
γ ≈ 0 the algorithm places nearly all the weight2 on ad revenue and
mostly ignores user utility, which is how programmatic advertising
auctions operate. The constraints are interpreted as follows:
1The “online” situation where we do not know which customers will require
recommendations will be discussed in the future research section. This online situation arises
when, for example, customers visit the system’s website, but the system does not know
in advance which ones will show up on a given day.
2We do not allow γ = 0 because then the algorithm would have no way of allocating
non-sponsored items when, for example, smax &lt; k. When γ is very small, e.g., γ =
10−8, then up to smax sponsored items are selected based on the ad revenue, and the
remaining non-sponsored items, which have ad revenue equal to 0 causing the ad
revenue term in (2) to equal 0, are selected based on user utility. Constraining γ &gt; 0
also addresses other degenerate cases, such as not having enough sponsored items to
ifll nsmax slots.
• (3): the number of recommended items is k for each user u.
• (4): xui takes only values 0 or 1.
• (5): the number3 of sponsored recommendations shown to
user u must not exceed analyst-specified smax. Note that 1(·)
is the indicator function, which equals 1 when the Boolean
condition in the argument is satisfied and 0 otherwise.
• (6): the total amount of ad revenue from item i does not
exceed the budget for that item, βi .</p>
      <p>We now discuss the two parts of the objective function (2), total
user utility and total ad revenue. Figure 1 shows some possible
curves for the two parts as a function of γ . First, suppose that
γ ≈ 0, indicating that the algorithm should focus on optimizing ad
revenue. The amount of user utility and ad revenue are indicated
by the black dots on the left side (γ ≈ 0) of the respective plots.
Ad revenue is at its maximum value, while utility is at its smallest
value, although the ads should produce positive utility. If we start
to consider user utility by increasing the value of γ two things will
happen: (1) user utility will either increase or stay the same, and
(2) ad revenue will either decrease or stay the same. An important
question is, how? The best scenario is when both the the user-utility
3Sometimes it is more convenient to express this constraint as an upper bound on the
fraction of sponsored items rather than the number. In such cases, one can compute
smax by multiplying the upper-bound fraction by k and using the algorithm as stated.
and ad revenue curves are concave, shown by black, solid lines. In
this case a small increase of γ from 0 produces a large increase in
utility with very little loss in ad revenue. The worst case is when
both curves are convex, shown by red, dashed lines, where a small
increase of γ from 0 results in a large decrease in ad revenue with
little improvement to utility.</p>
      <p>We start at the right sides of the plots at the red triangles (γ = 1),
where the algorithm should focus exclusively on user utility, as
with a traditional RS. User utility is maximized, while ad revenue
is at its lowest point. The same conclusions hold that the concave
functions are the best scenario and convex the worst. If we decrease
γ , allocating some weight to ad revenue, and the curves are concave,
then utility decreases by a small amount while ad revenue increases
substantially. The curvature will be investigated in our empirical
examples for a real company.</p>
      <p>The curves will be concave when the platform has flexibility.
When there are many sponsored and non-sponsored items with
roughly the same utility, the platform can substitute one item for
another in a top-k list without sacrificing much utility. Likewise,
when there are many sponsored items with roughly the same ad
revenue, substitutions can be made to improve utility.</p>
      <p>To solve the optimization problem, rewrite (2):</p>
      <p>Q
=
=
where optimization criterion cui = γ brui + (1 − γ )aui . This
expression is maximized by selecting the largest cui values, subject to the
constraints, leading to Algorithm 1 below. The most
computationally expensive part of the algorithm is sorting the mn elements of
cui in descending order. Sorting has super-linear computational
complexity, requiring on the order of mn log(mn) operations, so this
algorithm will be computationally feasible for very large problems.
Sorting is indicated by an index function that returns a user-item
pair, e.g., no other c value is larger than cu(1),i(1); also cu(h),i(h)
gives the hth largest c value. The variable su counts the number of
sponsored recommendations shown to user u, bi totals the amount
of ad budget from item i spent, and ku counts the number of items
recommended to user u. The algorithm walks through cui one
element at a time, starting from largest, until each of the m users have
received k recommendations.</p>
      <p>We discuss some potentially challenging situations. In line 11
of the algorithm there could be situations where a sponsored item
(ai &gt; 0) has the highest utility (i.e., higher than remaining
nonsponsored items), but either the ad budget (βi ) for the item has
been exhausted or the maximum number of sponsored item slots
(smax) has been filled. In these cases, the algorithm should display
the item as non-sponsored, since it has the highest utility. In order
to have transparency, there should not be a sponsored designation
displayed next to the item and the platform should not charge for the
recommendation. Note that for γ &lt; 1 the sort order of cui changes
if ad revenue is set to 0. An item has more value to the platform</p>
      <p>Algorithm 1 Multistakeholder advertising recommendation
Require: Analyst sets the weight for user utility (γ ), and the max
number sponsored recommendations shown to a user (smax).</p>
      <p>Advertiser sets budget (βi ).
1: cui ← γ brui + (1 − γ )aui , ∀u, i
2: Sort cui in descending order with index function mapping h ∈
{1, . . . , mn} to user-item pairs [u(h), i(h)] such that cu(1),i(1) ≥
cu(2),i(2) ≥ · · · ≥ cu(mn),i(mn).
3: su ← 0, ∀u ▷ Initialize counter of sponsored ads to user u
4: bi ← 0, ∀i ▷ Initialize spent ad i budget item i
5: xui ← 0, ∀u, i ▷ Initialize item to be not selected
6: ku ← 0, ∀u ▷ Initialize number recommended to user u
7: for h ← 1 until all users are assigned exactly k items do
8: if ku(h) &lt; k then
9:
10:
11:
xu(h),i(h) ← 1 ▷ Add item to top-k list
ku(h) ← ku(h) + 1 ▷ Increment recommendation count
if au(h),i(h) &gt; 0 and bi(h) + au(h),i(h) ≤ βi(h) and
su(h) &lt; smax then</p>
      <p>Label item sponsored
12:
13: su(h) ← su(h) + 1
14: bi(h) ← bi(h) + au(h),i(h)
15: end if
16: end if
17: end for
▷ Increment sponsored count
▷ Tally ad spend
when it is sponsored than when it is not: γ brui + (1 − γ )aui &gt; γ brui .
A simple way to handle this is to include all sponsored items twice
in the list, once as sponsored and once as non-sponsored (aui = 0),
and then do the sort. An additional if-then will be necessary to
guarantee that if an item has already been selected to be shown
as sponsored, it cannot also be selected as non-sponsored. Second,
when allocating the last (kth) slot there could be multiple items
with the same value of criterion cui . Since the criterion values are
tied it does not matter which one is selected, and the algorithm
could select one at random or some with tie-breaking rule.
3.1</p>
    </sec>
    <sec id="sec-9">
      <title>Commensurate units</title>
      <p>The analyst should be aware that the two parts of the objective
function have diferent units: aui is likely measured in some
currency such as dollars or euros, while the units for brui depend on
how utility is operationalized. For example, it could be the log
number of previous purchases plus one, the probability of purchase, or
the estimated rating on a five-point scale. One way to avoid this
problem is to use “unitless,” standardized versions of aui and brui in
the objective function.</p>
      <p>
        A more desirable common unit would be to express brui in the
same monetary unit as aui , e.g., dollars. Assigning a monetary
interpretation, however, is a dificult task requiring substantial modeling.
In addition to quantifying the probability of a current purchase,
one would also have to consider the long-term implications. If
a recommended item produces a sale, the retailer receives some
short-term profit from selling the item, but also has a customer with
better recency-frequency-monetary (RFM) status: recency is now
zero days, frequency is one larger and monetary value is greater
[
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. All three variables tend to be associated with higher purchase
probabilities in the future [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Failing to account for the long-term
efects produces evaluations of the value of the recommendation
that are biased downward. As noted in the future research section,
if such common units are available, one could optimize over γ as a
decision variable rather than specifying it exogenously.
3.2
      </p>
    </sec>
    <sec id="sec-10">
      <title>Ad revenue</title>
      <p>Algorithm 1 requires the analyst to provide values of aui , the ad
revenue associated with item i for user u. Notice that there are
two subscripts, allowing diferent revenue amounts for diferent
users and items. This subsection discusses the source of the aui
values. We intend for our algorithm to be general and work for
many current RS as well as systems that have not yet been devised.
We assume that advertisers can set an ad budget and there are
auctions for exposures. The advertiser can select items on which
to place bids, and may have the ability to modify the bid amount
based on additional information about the user, e.g., the advertiser
could increase the bid for users in a certain demographic segment.</p>
      <p>There are currently diferent revenue models for computational
advertising. For example, advertisers can pay for a view, click, lead,
conversion or engagement, such as signing up for an email list
or newsletter. These models have corresponding metrics such as
cost-per-view (CPV, which is often expressed in the cost for 1000
impressions, abbreviated CPM), cost-per-click (CPC), etc. In the
simplest case where there is a fixed cost per exposure and the value
aui is a constant. In more complicated cases, aui is an expected
amount, which may be computed from a predictive model or various
auction strategies. For example, there could be a model estimating
the chance of a conversion and probability of winning the auction
based on a certain bid. Or there could be a model where platform
chooses from auction strategies like a first-price auction,
secondprice auction, etc. However, we focus on the algorithm and take ad
revenue as a constant.
4</p>
    </sec>
    <sec id="sec-11">
      <title>EMPIRICAL EVALUATION</title>
      <p>We return to the example in the introduction between advertisers
A and B, where A has bid more than B and won the auction. We
investigate the sensitivity of the ad revenue and utility curves as a
function of γ using data from an online grocery retailer.
4.1</p>
    </sec>
    <sec id="sec-12">
      <title>Data and Preparation</title>
      <p>The data is from an online grocery retailer in the United States,
operating in 12 states and Washington D.C., with approximately
25,550 unique stock-keeping units (SKUs), 99,556 unique users,
and 2,286,831 orders. The first step in building a RS is to define
what an “item” is by aggregating the SKUs, since each size, flavor,
etc. has its own SKU. For example, in the subcategory “soda,” the
brand “Diet Coke” is separate from “Coke Classic.” The retailer
provided a taxonomy of 136 subcategories, which, when crossed
with brand names, gives 3,966 combinations. We clean the data
further by combining diferent brand lines of the same product
into a single brand. For example, within diapers “Pampers” is a
single brand, while Huggies has many sub-brands such as “Little
movers,” “Overnites,” “Pull ups,” etc., which are collapsed into one
Huggies brand. We count the number of orders that contain each
subcategory-brand combination. With this approach, the brand
“Dean’s” in the Ice Cream subcategory is a distinct brand from
Dean’s milk, Dean’s juice and Dean’s cottage cheese, but all flavors
and sizes of Dean’s ice cream are grouped together. Many
brandsubcategory combinations have small order counts. We minimize
rare items by relabeling any subcategory-brand combination with
fewer than 2500 orders as having brand “other,” which gives a total
of n=1,021 combinations, defining an “item” for the algorithm.</p>
      <p>We only examine regular users who have placed 15 or more
orders and have purchased within the past six months (i.e., recency
≤ 0.5 years), giving a universe of 19,360 users. We draw a simple
random sample of size n=2,000 users, and build our R matrix with
roughly two million entries (2,000 users × 1,021 items). We use
implicit ratings by taking the log number of past purchases, which
provides gradation between products that are never purchased,
purchased occasionally, and purchased often by a user.</p>
      <p>In our data items were not sponsored. In order to study
sponsored items, we assume that some fraction of the branded items
is sponsored. Private-label items are not allowed to be sponsored
because they are owned by the retailer. Likewise, we exclude “other”
subcategories, e.g., “other cottage cheese” because they would be
composed of multiple small brands. We assign a $1000 budget to
each branded item for sponsored recommendations and assume
ad revenue (ai ) drawn randomly from the Gaussian distribution
N (µ = 1, σ = 0.2). The algorithm is then executed, and the results
are compared for diferent values of γ . We conduct robustness check
by varying the fraction of sponsored items.
4.2</p>
    </sec>
    <sec id="sec-13">
      <title>Estimating User Utility</title>
      <p>
        Our algorithm post-processes ratings from a traditional RS to
address the MRS problem. Hence, an RS must first predict ratings for
items each user has not rated yet. We compare item- and user-based
collaborative filtering (IBCF and UBCF). For UBCF, we use cosine
similarity and vary the number of nearest neighbors (NNs): 5, 10, 20,
30, 50, 70, 90, 100 and 110. The performance of both traditional RS
is measured by the root mean square error (RMSE). We use five-fold
cross validation to evaluate traditional RS algorithms and follow
the established model-building process outlined in section 3.2 of
[
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. Table 1 summarizes the results. UBCF performs better than
IBCF. There is little diference in RMSE between diferent numbers
of nearest neighbors, and we use 30.
4.3
      </p>
    </sec>
    <sec id="sec-14">
      <title>Results from Proposed Algorithm</title>
      <p>After estimating the ratings with UBCF, we test our algorithm. The
goal is to recommend the top 20 items (ku = 20) to each user,
among which, at most three (smax ≤ 3) can be sponsored. To put
both ratings (rˆui ) and ad revenue (aui ) values on the same scale, we
use their Z -scores to calculate cui . Figure 2 displays the results of
the algorithm for user utility and ad revenue as a function of γ with
diferent percentages (10%, 20%, 30%, 40%, 50%) of sponsored items.
In all five cases, we see that as γ (the weight given to user utility)
increases from zero to one, user utility increases and ad revenue
decreases monotonically. The ad revenue plot (bottom) is mostly
concave as we had hoped. The user utility curve (top), however, is
convex for small values of γ , but mostly concave for large values.</p>
      <p>When γ ≈ 0, all weight is given to optimizing ad revenue. This
can be considered the status quo for sponsored ads. As shown
in ad revenue plot (Figure2, bottom), when the number of
sponsored items increase, there are more options to choose from and
ad revenue grows. In the user utility plot, having fewer options in
sponsored items would lead the algorithm to choose items with
higher user utility, generating higher overall user utility in the case
of 10%, compared to 50% of sponsored items.</p>
      <p>After γ reaches approximately 0.5, the ad revenue plots show a
faster decrease in the cases with a larger number of sponsored items.
With more sponsored items in the inventory, as γ increases, cui
puts more weight on user utility and the algorithm has more ways
to select sponsored items that have lower ad revenue but higher
ratings. At the end, the sum of ad revenue has a faster decline in
the cases with more sponsored items as to when the number of
sponsored items is limited, since those limited items might have
had higher ad revenue with higher ratings. In all cases, the ideal
scenario with both curves concave is partially realized.</p>
      <p>
        We compare our proposed algorithm for MRS with traditional RS,
i.e., UBCF. While evaluating MRS, previous studies have shown that
a slight decrease in one objective improves over the previous system
if there is a substantial increase in competing objectives [
        <xref ref-type="bibr" rid="ref19 ref6">6, 19</xref>
        ]. To
compare the performance of our algorithm with a traditional RS,
we show the change in one objective as we vary the weight given to
the competing objective. Recall when γ = 1, the algorithm chooses
the top 20 items solely based on user’s utility. At 20% sponsored
items, we observe that as γ changes from 1 to 0.75, user utility
decreases by only 0.17%, while ad revenue increases substantially
by more than 40%.
      </p>
      <p>
        We can also evaluate our algorithm by plotting total ad revenue
against total user utility for diferent values of γ , as shown in
Figure 3. For comparison, the axes have been scaled so that 0 indicates
the minimum value of ad revenue or utility, and 100 indicates the
maximum value. The values of the points on the curve correspond
to diferent γ values. This resembles a receiver operator curve (ROC)
[
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], where, in this application, the area under the curve (AUC)
measures how much flexibility the platform has. In an ideal situation,
the platform would be able to achieve 100% of both the ad revenue
and user utility, giving a point in the upper right corner of the
graph. In our example, we can see that reducing the ad revenue
by a fairly small amount increases user utility substantially. For
example, changing γ from 0 to 0.50 achieves roughly a 28 point
increase in user utility while ad revenue decreases by only 2 points.
      </p>
    </sec>
    <sec id="sec-15">
      <title>FUTURE RESEARCH AGENDA</title>
      <p>Although our optimization model and algorithm to solve it make
important contributions to the fields of computational advertising
and RS, we believe this to be only the beginning of a larger area
for future research. We therefore provide an extended discussion
of open research questions and extensions. The algorithm should
be evaluated with field tests, on data sets from diferent industries,
and with more traditional RS algorithms.</p>
      <p>
        Understand the relationship between customer utility and
customer lifetime value (CLV). When a platform recommends items
with lower utility than others, how does this afect CLV? If this
relationship is understood, then the trade-of can be between ad
revenue and CLV, which are both measured in the same units (e.g,
dollars). Then γ could potentially become a variable that is selected
by the optimization algorithm instead of being set by the manager.
Game theory [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ] could also provide relevant insights as it is often
used to model the stability of a system. It is a trade-of where the
gain of one player is a loss of another. As our study also shows, the
system manager makes a trade-of of a small portion of ad revenue
to generate more consumer-centric recommendations. The
stability of the proposed model then lies in the expertise of a system
manager on how well she can tune γ .
      </p>
      <p>
        Understand how to handle situations where a user has high utility
for a sponsored item. For example, if the item with the highest utility
for user u is sponsored, should the system show a “sponsored” label?
On the one hand, showing such a label can decrease the probability
of purchase [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], and the user might select a non-sponsored item
providing less utility than the sponsored one. On the other hand, if
the system receives ad revenue for showing an item then the item
is sponsored, and the system should be transparent, or risk losing
the trust of its users.
      </p>
      <p>Model components of ad revenue (aui ). Ad auctions tend to be
second-price, meaning that the highest bidder A pays the
secondhighest bid. This is from a game-theoretic result that keeps the
auction stable, but it makes modeling revenue more complex. The
second-price issue could be addressed by multiplying the highest bid
by a constant that adjusts to the second-highest bid. In the proposed
model, the rank is determined by user utility as well as the bid
order. The ranking function is Bid × P (click|Ad) = expected revenue.
Example: A is a spammer P (click|A) = 0.001; B is a real company
P (click|B) = 0.1. A has to bid 100 × B’s bid to win the auction, not
just B’s bid plus some ϵ, and would have to pay an amount that
provides expected revenue = B for each click. The second-price
criterion is very dificult to model under these circumstances.</p>
      <p>
        Develop algorithms for when items have diferent margins. A
grocery retailer might make more profit from selling a private-label
product, and recommend it as a non-sponsored item over other
items [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. We would need a separate set of constants for margin.
Let µ (i, r ) be the expected margin on item i for someone with
utility r (which accounts for diferent click probabilities). Then the
objective becomes:
 User utility Revenue 
 zÕ }| { zÕ Õ }| { 
Q2 = γ Õ brui xui +(1 − γ ) [µ (i, brui ) + aui ]xui  .
      </p>
      <p> u ∈U i ∈I u ∈U i ∈I 
Optimize over smax. Setting smax to a small value restricts ad
inventory, which will likely increase auction bids and thus the
ad revenue per slot, whereas larger values increase the supply of
inventory and reduce the bids. This raises the possibility of reducing
sponsored ad inventory without reducing ad revenue. One would
need to know the elasticities of the supply and demand curves,
which could be an important study on its own.</p>
      <p>
        Develop algorithms to handle online situation, where users arrive
over time and the platform does not know which ones will visit.
A simple example will illustrate the problem. Suppose that we are
to recommend k = 1 item and that there are n = 2 items A and B,
each of which is sponsored and with enough budget for exactly
one exposure today. There are m = 2 users, 1 and 2. The criterion
values are c1A = 3, c1B = 2, c2A = 5 and c2B = 2. Suppose that user
1 visits the platform first, and so showing item A is optimal since
c1A = 3 &gt; 2 = c1B . Next user 2 visits, but since A has exhausted its
ad budget, item B is shown giving c2B = 2 and a total criteria of
3 + 2 = 5. The optimal solution to the of-line problem, however,
is to show user 1 item B and user 2 item A, giving a total criteria
of 5 + 2 = 7 &gt; 5. Search advertising faces the same issue [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], and
methods such as the Balance algorithm [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] have been proposed.
Perhaps the Balance algorithm could be used in the sponsored
recommendation situation after substituting cui for click-through
rates, but the details should be carefully examined.
      </p>
      <p>Improve the eficiency of Algorithm 1. The example in the previous
paragraph also illustrates why the sorting in Algorithm 1 must
be done over all mn items to find an optimal solution. Sorting
within a user, as in the example, or within items may lead to a
sub-optimal solution. While sorting has super-linear computational
complexity and is therefore possible for large data sets, there might
be ways improve the eficiency. For example, the slowest part of
the algorithm is the sort, which can be sped up by reducing the
number of cui values. If the inventory is, say, 10% of mn, one could
drop all values less than, say, the 85th percentile, reducing the sort
cost substantially. Knowing percentiles, however, requires sorting,
unless the shape of the distribution is known. The cui values for
the grocery data tend to be moderately right-skewed. The mean,
variance and other moments could be found with a single pass,
e.g., while computing cui . This would give information about the
quantiles without sorting, which could then be used to drop
useritem pairs with small cui values.
6</p>
    </sec>
    <sec id="sec-16">
      <title>DISCUSSION</title>
      <p>This study exemplifies how platform managers can consider both
ad revenue and user utility while recommending items to the users.
This is a paradigm shift from extant computational advertising
approaches that focus on maximizing ad revenue and only consider
user utility through click probabilities. Our approach treats the
problem of MRS in the platform settings as a post-processing
optimization step to be applied after user utilities have been estimated
by an existing recommendation algorithm. Thus, our approach is
not dependent on any particular RS algorithm, and can be widely
applied. Our approach allows platform managers to trade-of ad
revenue for higher user utility. Our empirical simulation suggests
that shifting some weight to user utility will increase utility
substantially while decreasing ad revenue only slightly, although this
should be confirmed with field tests. Thus, we make computational
advertising more user-centric by explicitly trading-of ad revenue
to improve user utility while recommending sponsored items.</p>
      <p>Advertising and RS have developed independently of one another
and the two have starkly diferent underlying philosophies: the goal
of advertising has been to persuade while the goal of RS has been to
help users find items of interest to them. The ultimate goal—selling
a product or, more generally, increasing CLV—may be the same,
but the approaches are completely diferent. This study attempts to
unify the two by balancing ad revenue and user utility in the specific
context of sponsored recommendations. A more general question
is whether this hybrid approach could be applied to other areas
of advertising: can media selection and targeting work better by
explicitly considering user utility? We believe this is a profoundly
important question, as users find new ways to avoid ads.</p>
    </sec>
    <sec id="sec-17">
      <title>ACKNOWLEDGMENTS</title>
      <p>We thank Özge Sürer for helpful discussions and Northwestern
University’s Spiegel Research Center for supporting the second and
third authors.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>Himan</given-names>
            <surname>Abdollahpouri</surname>
          </string-name>
          , Robin Burke, and
          <string-name>
            <given-names>Bamshad</given-names>
            <surname>Mobasher</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>Recommender Systems as Multistakeholder Environments</article-title>
          .
          <source>In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP'17)</source>
          . ACM, New York, NY, USA,
          <fpage>347</fpage>
          -
          <lpage>348</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>Amos</given-names>
            <surname>Azaria</surname>
          </string-name>
          , Avinatan Hassidim, Sarit Kraus, Adi Eshkol, Ofer Weintraub, and
          <string-name>
            <given-names>Irit</given-names>
            <surname>Netanely</surname>
          </string-name>
          .
          <year>2013</year>
          .
          <article-title>Movie Recommender System for Profit Maximization</article-title>
          .
          <source>In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys '13)</source>
          . ACM, New York, NY, USA,
          <fpage>121</fpage>
          -
          <lpage>128</lpage>
          . https://doi.org/10.1145/2507157.2507162
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>Joeran</given-names>
            <surname>Beel</surname>
          </string-name>
          , Stefan Langer, and
          <string-name>
            <given-names>Marcel</given-names>
            <surname>Genzmehr</surname>
          </string-name>
          .
          <year>2013</year>
          .
          <article-title>Sponsored vs. organic (research paper) recommendations and the impact of labeling</article-title>
          .
          <source>In International Conference on Theory and Practice of Digital Libraries</source>
          . Springer,
          <fpage>391</fpage>
          -
          <lpage>395</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Robert</surname>
            <given-names>C Blattberg</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Edward C Malthouse</surname>
          </string-name>
          , and Scott A Neslin.
          <year>2009</year>
          .
          <article-title>Customer lifetime value: Empirical generalizations and some conceptual questions</article-title>
          .
          <source>Journal of Interactive Marketing</source>
          <volume>23</volume>
          ,
          <issue>2</issue>
          (
          <year>2009</year>
          ),
          <fpage>157</fpage>
          -
          <lpage>168</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>Robin</given-names>
            <surname>Burke</surname>
          </string-name>
          and
          <string-name>
            <given-names>Himan</given-names>
            <surname>Abdollahpouri</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>Patterns of Multistakeholder Recommendation</article-title>
          .
          <source>In VAMS Workshop</source>
          <year>2017</year>
          . arxiv, http://arxiv.org/abs/1707.09258,
          <issue>Article 1</issue>
          , 5 pages.
          <source>arXiv:1707.09258</source>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Robin</surname>
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Burke</surname>
            , Himan Abdollahpouri, Bamshad Mobasher, and
            <given-names>Trinadh</given-names>
          </string-name>
          <string-name>
            <surname>Gupta</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>Towards Multi-stakeholder Utility Evaluation of Recommender Systems</article-title>
          .
          <source>In UMAP Extended Proceedings</source>
          , Vol.
          <volume>1618</volume>
          . CEUR Workshop Proceedings, Aachen University.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>Benjamin</given-names>
            <surname>Edelman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Sonia</given-names>
            <surname>Jafe</surname>
          </string-name>
          , and Scott Duke Kominers.
          <year>2011</year>
          .
          <article-title>To groupon or not to groupon: The profitability of deep discounts</article-title>
          .
          <source>Marketing Letters</source>
          (
          <year>2011</year>
          ),
          <fpage>1</fpage>
          -
          <lpage>15</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Michael</surname>
            <given-names>D</given-names>
          </string-name>
          <string-name>
            <surname>Ekstrand</surname>
          </string-name>
          , John T Riedl,
          <article-title>Joseph A Konstan</article-title>
          , et al.
          <year>2011</year>
          .
          <article-title>Collaborative ifltering recommender systems</article-title>
          .
          <source>Foundations and Trends® in Human-Computer Interaction 4</source>
          ,
          <issue>2</issue>
          (
          <year>2011</year>
          ),
          <fpage>81</fpage>
          -
          <lpage>173</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>Tom</given-names>
            <surname>Fawcett</surname>
          </string-name>
          .
          <year>2006</year>
          .
          <article-title>An introduction to ROC analysis</article-title>
          .
          <source>Pattern recognition letters 27</source>
          ,
          <issue>8</issue>
          (
          <year>2006</year>
          ),
          <fpage>861</fpage>
          -
          <lpage>874</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>Andrei</given-names>
            <surname>Hagiu</surname>
          </string-name>
          and
          <string-name>
            <given-names>Julian</given-names>
            <surname>Wright</surname>
          </string-name>
          .
          <year>2015</year>
          .
          <article-title>Multi-sided platforms</article-title>
          .
          <source>International Journal of Industrial Organization</source>
          <volume>43</volume>
          (
          <year>2015</year>
          ),
          <fpage>162</fpage>
          -
          <lpage>174</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>Dietmar</given-names>
            <surname>Jannach</surname>
          </string-name>
          and
          <string-name>
            <given-names>Gediminas</given-names>
            <surname>Adomavicius</surname>
          </string-name>
          .
          <year>2017</year>
          .
          <article-title>Price and profit awareness in recommender systems</article-title>
          .
          <source>arXiv preprint arXiv:1707.08029</source>
          (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Bernard J Jansen and Tracy Mullen</surname>
          </string-name>
          .
          <year>2008</year>
          .
          <article-title>Sponsored search: an overview of the concept, history, and technology</article-title>
          .
          <source>International Journal of Electronic Business</source>
          <volume>6</volume>
          ,
          <issue>2</issue>
          (
          <year>2008</year>
          ),
          <fpage>114</fpage>
          -
          <lpage>131</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>Bala</given-names>
            <surname>Kalyanasundaram and Kirk R Pruhs</surname>
          </string-name>
          .
          <year>2000</year>
          .
          <article-title>An optimal deterministic algorithm for online b-matching</article-title>
          .
          <source>Theoretical Computer Science</source>
          <volume>233</volume>
          ,
          <fpage>1</fpage>
          -
          <lpage>2</lpage>
          (
          <year>2000</year>
          ),
          <fpage>319</fpage>
          -
          <lpage>325</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Jure</surname>
            <given-names>Leskovec</given-names>
          </string-name>
          , Anand Rajaraman, and Jefrey David Ullman.
          <year>2014</year>
          .
          <article-title>Mining of massive datasets</article-title>
          . Cambridge university press.
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Edward</surname>
            <given-names>C</given-names>
          </string-name>
          <string-name>
            <surname>Malthouse</surname>
          </string-name>
          .
          <year>2010</year>
          .
          <article-title>Accounting for the long-term efects of a marketing contact</article-title>
          .
          <source>Expert Systems With Applications</source>
          <volume>37</volume>
          ,
          <issue>7</issue>
          (
          <year>2010</year>
          ),
          <fpage>4935</fpage>
          -
          <lpage>4940</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Edward</surname>
            <given-names>C Malthouse</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ewa Maslowska</surname>
          </string-name>
          , and Judy U Franks.
          <year>2018</year>
          .
          <article-title>Understanding programmatic TV advertising</article-title>
          .
          <source>International Journal of Advertising</source>
          (
          <year>2018</year>
          ),
          <fpage>1</fpage>
          -
          <lpage>16</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>Umberto</surname>
            <given-names>Panniello</given-names>
          </string-name>
          , Shawndra Hill, and
          <string-name>
            <given-names>Michele</given-names>
            <surname>Gorgoglione</surname>
          </string-name>
          .
          <year>2016</year>
          .
          <article-title>The impact of profit incentives on the relevance of online recommendations</article-title>
          .
          <source>Electronic Commerce Research and Applications</source>
          <volume>20</volume>
          (
          <year>2016</year>
          ),
          <fpage>87</fpage>
          -
          <lpage>104</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>Claudia</surname>
            <given-names>Perlich</given-names>
          </string-name>
          , Brian Dalessandro, Rod Hook, Ori Stitelman, Troy Raeder, and
          <string-name>
            <given-names>Foster</given-names>
            <surname>Provost</surname>
          </string-name>
          .
          <year>2012</year>
          .
          <article-title>Bid optimizing and inventory scoring in targeted online advertising</article-title>
          .
          <source>In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM</source>
          ,
          <volume>804</volume>
          -
          <fpage>812</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>Mario</surname>
            <given-names>Rodriguez</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>Christian</given-names>
            <surname>Posse</surname>
          </string-name>
          ,
          <string-name>
            <given-names>and Ethan</given-names>
            <surname>Zhang</surname>
          </string-name>
          .
          <year>2012</year>
          .
          <article-title>Multiple objective optimization in recommender systems</article-title>
          .
          <source>In Proceedings of the sixth ACM conference on Recommender systems. ACM</source>
          ,
          <volume>11</volume>
          -
          <fpage>18</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>Amin</given-names>
            <surname>Sayedi</surname>
          </string-name>
          .
          <year>2018</year>
          .
          <article-title>Real-Time Bidding in Online Display Advertising</article-title>
          . Marketing
          <string-name>
            <surname>Science</surname>
          </string-name>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <surname>Amin</surname>
            <given-names>Sayedi</given-names>
          </string-name>
          , Kinshuk Jerath, and
          <string-name>
            <given-names>Kannan</given-names>
            <surname>Srinivasan</surname>
          </string-name>
          .
          <year>2014</year>
          .
          <article-title>Competitive poaching in sponsored search advertising and its strategic impact on traditional advertising</article-title>
          .
          <source>Marketing Science</source>
          <volume>33</volume>
          ,
          <issue>4</issue>
          (
          <year>2014</year>
          ),
          <fpage>586</fpage>
          -
          <lpage>608</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <surname>Özge</surname>
            <given-names>Sürer</given-names>
          </string-name>
          , Robin Burke, and
          <string-name>
            <surname>Edward</surname>
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Malthouse</surname>
          </string-name>
          .
          <year>2018</year>
          .
          <article-title>Multistakeholder Recommendation with Provider Constraints</article-title>
          .
          <source>ACM Recommendation Systems Conference Proceedings</source>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>Wush</given-names>
            <surname>Chi-Hsuan</surname>
          </string-name>
          <string-name>
            <given-names>Wu</given-names>
            ,
            <surname>Mi-Yen Yeh</surname>
          </string-name>
          , and
          <string-name>
            <surname>Ming-Syan Chen</surname>
          </string-name>
          .
          <year>2015</year>
          .
          <article-title>Predicting winning price in real time bidding with censored data</article-title>
          .
          <source>In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM</source>
          ,
          <volume>1305</volume>
          -
          <fpage>1314</fpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>