<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>ACM RecSys Workshop on Recommenders in Tourism, October</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Lifecycle of promotional campaigns in the online travel industry</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Carlos Herrero-Gómez</string-name>
          <email>carlos.herrero@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Amit Livne</string-name>
          <email>amit.livne@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Itsik Adiv</string-name>
          <email>itsik.adiv@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Hugo Manuel Proença</string-name>
          <email>hugo.proenca@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Felipe Moraes</string-name>
          <email>felipe.moraes@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Javier Albert</string-name>
          <email>javier.albert@booking.com</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Dmitri Goldenberg</string-name>
          <email>dima.goldenberg@booking.com</email>
        </contrib>
      </contrib-group>
      <pub-date>
        <year>2024</year>
      </pub-date>
      <volume>18</volume>
      <issue>2024</issue>
      <abstract>
        <p>This paper provides a comprehensive review of the lifecycle of promotional campaigns within online travel platforms (OTPs) based on real world experience from Booking.com. It emphasizes the critical aspects that must be addressed to define, optimize and monitor successfully these promotions. Initially, we present an overview of promotions in OTPs, highlighting their unique characteristics in comparison to other industries. We continue by identifying the diferent aspects of a promotional campaign with focus on those that can be tuned for campaign optimization. Following this, we propose best practices for experimentation, evaluation and steering to make sure that running promotional campaigns stay compliant with the business requirements.</p>
      </abstract>
      <kwd-group>
        <kwd>Promotion optimization</kwd>
        <kwd>Tourism</kwd>
        <kwd>Uplift modelling</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        to a complex ecosystem in which an intervention in any of them—such as a promotion—
can afect the rest. These efects can be either positive synergies but also unwanted
negative results such as cannibalisation. Proper promotion management should be aware
of these efects and take actions to maximize positive outcomes and mitigate the negative
ones.
• Shared supply: Travel products are not only limited but also often listed on several
competing OTPs and on the business itself. Promotions can play a role in diferentiating
the value provided to the customer by ofering perks like Free Cancellation, Free Breakfast
or Room Upgrade.
• Promotional costs: In funnel OTP’s promotional campaigns only incur costs when the
customer materializes the given promotion by making a purchase. For example a 10 €
discount for a hotel room will only incur in a cost if the customer books that room. Unlike
other promotional channels such as Pay Per Click or marketing.
• Low interaction frequency: as a reflection of the tourism seasonality, customers don’t
engage with OTPs on a frequent basis [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. This is in contrast to other industries such
as online retail or streaming where the interactions are more recurrent. This limited
and sporadic amount of interactions plays a role in campaign personalisation eforts, as
described in section 2.
• International nature: A big proportion of travel products and their associated
promotional campaigns frequently span multiple countries, e.g. a family from the United States
traveling to Europe. This cross-border nature implies compliance with the regulations of
both the origin and destination countries, increasing the complexity of campaign
management. Furthermore, the geo-cardinality of potential market segmentation in the travel
industry is significantly higher than in non-travel sectors, due to the myriad combinations
of origin and destination countries.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Campaign set up and optimization</title>
      <p>Turning an hypothesis into a successful promotional campaign is not an easy task and typically
takes multiple iterations to optimize. There are multiple aspects which define a promotional
campaign, but in general, those can be divided between settings or constraints and levers. The
settings define the scope of the campaign, these are properties that cannot typically be modified
and any campaign must comply with them, these include:
• Goal: The business objective of the campaign. it can be a certain amount of profit, increase
in sales or user engagement, among others.
• Budget: the finite resource that the company allows itself to invest in the campaign.
• Eligibility: The subset of OTP’s trafic susceptible to the promotion. i.e in the case of
cross-sell campaigns, only customers that already purchased a product can be targeted.
The levers are those aspects that can be tuned to optimize the promotional campaign, inside the
framework defined by the constrains or settings. These include aspects of the campaign setup,
and audience targeting. The former drives coarse changes in volumes and economics whereas
the latter represents finer trade-ofs between the two</p>
      <sec id="sec-2-1">
        <title>Goal</title>
        <p>Placement
y
itt
n
a
u
q
a
t
a
D
y
ilt
a
u
q
a
t
a
D</p>
      </sec>
      <sec id="sec-2-2">
        <title>Settings and Constraints</title>
      </sec>
      <sec id="sec-2-3">
        <title>Budget</title>
      </sec>
      <sec id="sec-2-4">
        <title>Levers</title>
      </sec>
      <sec id="sec-2-5">
        <title>Benefit</title>
        <p>-15%</p>
        <sec id="sec-2-5-1">
          <title>2.1. Campaign set up</title>
          <p>
            The main levers of a promotional campaign set up are the following:
• Placement: Promotions can be ofered at various stages, such as the upper funnel of
the webpage, the lower funnel (post-purchase), or even through diferent contexts like
marketing channels. The placement of the campaign predominantly afects campaign
audiences and the quantity and quality of the customer information. Generally speaking
the audience is reduced along the funnel and the amount of the customer information
is increased. Therefore the campaign placement will determine target audience and the
quantity and quality of the customer data available
• Benefit type: A fixed discount is easier for the user to perceive and for predicting costs
due its linear nature. However, in the travel industry, transaction value may vary
significantly between diferent countries and destinations, thus making a fixed discount
very appealing for cheaper destinations, where the expected revenue is small, making
it economically ineficient. On the other hand, a percentage discount, or a benefit with
monetary equivalent which is proportional to the transaction value, e.g., free breakfast,
is more eficient across destinations but actual costs can be harder to predict. A fixed
discount is therefore more suitable where the expected transactions value and/or expected
revenue are relatively similar across items.
• Benefit size: This is the actual amount or percentage of discount. The law of demand
suggests that for most goods, there’s an inverse relationship between the price and
demanded quantity, or, in the context of promotions and benefits, the higher the benefit
size, the higher the demand. However, increasing benefit size is very costly, and is also
sufering from diminishing returns. Once the target audience and discount type are
set, setting the sweet spot for discount value or benefit size would be such that the
incrementality is maximized with economics (iROI, cost of acquisition, etc.) slightly
under-performing, leaving room for further optimization [
            <xref ref-type="bibr" rid="ref3">3</xref>
            ].
          </p>
        </sec>
        <sec id="sec-2-5-2">
          <title>2.2. Targeting</title>
          <p>
            In the context of promotional campaigns, the customer audience can be divided in four segments
according to their response to a given discount [
            <xref ref-type="bibr" rid="ref6">6</xref>
            ].
          </p>
          <p>• Complier: customers who respond positively only when targeted
• Always-taker: Customers who respond positively, regardless of targeting
• Never-taker: customers who won’t respond regardless of being targeted.</p>
          <p>• Defier: customers who are less likely to respond positively if they are targeted</p>
          <p>
            The ideal targeting, in the context of promotion and benefits, is giving the discount or benefit
only to the complier group. This way the promotion budget would be allocated in the most
efective way. To that end, Uplift models are commonly employed to predict the causal efect of
a treatment at the individual level based on data collected from a Randomized Control Trial
(RCT) [
            <xref ref-type="bibr" rid="ref7">7</xref>
            ]. In uplift modeling, the incremental response of a customer with covariates  is given
by the Conditional Average Treatment Efect ( () ), defined by the diference in outcome
had the customer been treated and not. high values of  () correspond to high incremental
response. Therefore the goal of uplift modelling is to model the  () per customer given
the available information (covariates  ). There are several techniques to estimate the  () ,
to name just a few:
• Meta-Learners is a family of standard ML models that are combined or modified to predict
 () . These include, the single model (S-Learner) [
            <xref ref-type="bibr" rid="ref8">8</xref>
            ] and two model estimator
(T-Learner) [
            <xref ref-type="bibr" rid="ref9">9</xref>
            ]among others.
• Uplift Trees and various deep learning based approaches, both uses modified loss functions
of ML algorithms to predict  () [
            <xref ref-type="bibr" rid="ref10">10</xref>
            ].
• Retrospective Estimator is a technique that uses data from converted-only users to predict
a proxy quantity to  () [
            <xref ref-type="bibr" rid="ref11">11</xref>
            ]. The main advantage of this technique is that it only
requires data from customers that did materialize a purchase, which makes it very suitable
for promotions placed at high end of the funnel. This technique has been proven to be
highly beneficial in an online setting for multiple promotional use cases at Booking.com.
In an online setting, such as in E-Commerce platforms, a decision rule is required, the estimated
 () is compared versus a threshold to decide whether to give a specific promotion to
the customer or not. Using  () as a sorting mechanism, one could then evaluate the
cumulative economics (iROI, Cost of acquisition, etc.) associated with treating all customers
with  () &gt;= ℎ ℎ , also known as QINI Curve. It is customary to bucketize  ()
by population density, e.g., percentiles, deciles instead of equally spaced bucket, which allows
to easily set the threshold that balances economics and incrementality.
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Campaign management</title>
      <sec id="sec-3-1">
        <title>3.1. A/B testing VS Continuous experimentation</title>
        <p>
          A/B testing is the standard methodology to evaluate any new promotion campaign [
          <xref ref-type="bibr" rid="ref12 ref13">12, 13</xref>
          ]. It
is a well-established method where incoming trafic is randomly assigned to one out of two
versions — ‘benchmark’ (A) and ‘experiment‘ (B) which are compared over a fixed period of time
to determine which one performs better based on specific metrics. Once a given policy proofs
to be superior in the A/B test it might be deployed as the new benchmark. While efective and
well-established, this method has several limitations:
• Time-Consuming: Each test cycle requires distinct setup, execution, and analysis phases,
leading to slower innovation cycles.
• Lack of Flexibility: Once the test is set, variations do not change until the test concludes,
which can limit the ability to adapt to new insights during the testing period.
Alternatively, continuous experimentation is a modern approach that ofers ongoing, iterative
testing and optimization which is specially well-suited. for the OTP’s, given the seasonality of
the travel market.
        </p>
        <p>Continuous experimentation can be defined as a diferent paradigm in which both
experimental features and consolidated policies are continuously running. All the running policies
conform a portfolio that is continuously monitored and modified if needed. This method ofers
several key advantages:
• Flexibility and Adaptability: Continuous experimentation allows real-time adjustments
to test parameters and variations. Leveraging adaptive algorithms and real-time data
analysis facilitates dynamic refinement based on user interactions, ensuring the testing
process evolves in response to emerging insights.
• Continuous baseline. By continuously allocating trafic to a baseline, campaign managers
can monitor and detect seasonality efects or any disruption in the market.
• Scalability and Comprehensive Insight: The methodology supports concurrent testing
of multiple variations, providing a holistic view of user preferences and interactions.
This capability enhances the depth of analysis and facilitates informed decision-making
regarding feature optimization and deployment strategies.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Experimental design</title>
        <p>Typically, in a continuous experimentation set up, we are working with three distinct sets
of policies: Baseline, Benchmark and Experimental Policies. Each one is designed to test the
efectiveness of discount strategies in a controlled and systematic manner. Below is a detailed
explanation of each approach:
1. Baseline — a control group of trafic that does not receive any discounts or treatments.</p>
        <p>This group remain untreated to provide a baseline for comparison.
2. Benchmark — Well established promotion campaigns.
3. Experimental Policy— Any experimental policy to be tested.</p>
        <p>Portfolio Average
(Arbitrary metric)</p>
        <p>Steer</p>
        <p>Steer</p>
        <p>Time</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Portfolio steering</title>
        <p>While each of these policies is important on its own, one of the significant advantages of using
this framework is the ability to manage all of them together as a portfolio of discounts strategies.
Similar to a stock portfolio, this framework enables us to create a mixture of policies with
diferent characteristics. Continuous monitoring of the portfolio allows promotional campaign
managers to steer the portfolio. Steering refers to the change of the amount of trafic allocated
to each of the policies of the portfolio in order to bring the portfolio average performance to
the desired value. Figure 3 illustrates a steering scenario in which trafic is reallocated from
one policy to another. This figure highlights two pivotal steering points where the portfolio
performance metric is adjusted in response to shifts in economic or business priorities.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Conclusions</title>
      <p>This work highlights the critical aspects of promotional campaigns in the online travel industry.
We describe the unique characteristics of this domain, such as seasonality and interconnected
verticals. We elaborate on the diferent features of a promotional campaign, emphasizing the
distinction between constraints and levers, and list several options for leveraging the levers,
such as targeting compliers. Additionally, we discuss the need for continuous
experimentation and metric monitoring in contrast to traditional A/B testing. Finally, we emphasize the
benefits of managing all promotions together as a single portfolio, which allows for balanced
experimentation and alignment with business goals through strategic steering.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>E. T.</given-names>
            <surname>Anderson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. J.</given-names>
            <surname>Fox</surname>
          </string-name>
          ,
          <article-title>How price promotions work: A review of practice and theory, Handbook of the Economics of Marketing 1 (</article-title>
          <year>2019</year>
          )
          <fpage>497</fpage>
          -
          <lpage>552</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>T.</given-names>
            <surname>Tong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Yan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <article-title>Impact of diferent platform promotions on online sales and conversion rate: The role of business model and product line length</article-title>
          ,
          <source>Decision Support Systems</source>
          <volume>156</volume>
          (
          <year>2022</year>
          )
          <fpage>113746</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>J.</given-names>
            <surname>Albert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Goldenberg</surname>
          </string-name>
          ,
          <article-title>E-commerce promotions personalization via online multiplechoice knapsack with uplift modeling</article-title>
          ,
          <source>in: Proceedings of the 31st ACM International Conference on Information &amp; Knowledge Management</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>2863</fpage>
          -
          <lpage>2872</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>A.</given-names>
            <surname>Badanidiyuru</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Kleinberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Slivkins</surname>
          </string-name>
          ,
          <article-title>Bandits with knapsacks</article-title>
          ,
          <source>in: 2013 IEEE 54th Annual Symposium on Foundations of Computer Science</source>
          , IEEE,
          <year>2013</year>
          , pp.
          <fpage>207</fpage>
          -
          <lpage>216</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>D.</given-names>
            <surname>Goldenberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Levin</surname>
          </string-name>
          , Booking.
          <article-title>com multi-destination trips dataset</article-title>
          ,
          <source>in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval</source>
          , SIGIR '21,
          <string-name>
            <surname>Association</surname>
          </string-name>
          for Computing Machinery, New York, NY, USA,
          <year>2021</year>
          , p.
          <fpage>2457</fpage>
          -
          <lpage>2462</lpage>
          . URL: https://doi.org/10.1145/3404835.3463240. doi:
          <volume>10</volume>
          .1145/3404835. 3463240.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Pearl</surname>
          </string-name>
          ,
          <article-title>Unit selection based on counterfactual logic</article-title>
          ,
          <source>in: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence</source>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>F.</given-names>
            <surname>Moraes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H. Manuel</given-names>
            <surname>Proença</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Kornilova</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Albert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Goldenberg</surname>
          </string-name>
          ,
          <article-title>Uplift modeling: From causal inference to personalization</article-title>
          ,
          <source>in: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>5212</fpage>
          -
          <lpage>5215</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>S. R.</given-names>
            <surname>Künzel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. S.</given-names>
            <surname>Sekhon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. J.</given-names>
            <surname>Bickel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <article-title>Metalearners for estimating heterogeneous treatment efects using machine learning</article-title>
          ,
          <source>Proceedings of the national academy of sciences 116</source>
          (
          <year>2019</year>
          )
          <fpage>4156</fpage>
          -
          <lpage>4165</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>N.</given-names>
            <surname>Radclife</surname>
          </string-name>
          ,
          <article-title>Using control groups to target on predicted lift: Building and assessing uplift model</article-title>
          ,
          <source>Direct Marketing Analytics Journal</source>
          (
          <year>2007</year>
          )
          <fpage>14</fpage>
          -
          <lpage>21</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>N. J.</given-names>
            <surname>Radclife</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. D.</given-names>
            <surname>Surry</surname>
          </string-name>
          ,
          <article-title>Real-world uplift modelling with significance-based uplift trees</article-title>
          ,
          <source>White Paper TR-2011-1</source>
          ,
          <string-name>
            <given-names>Stochastic</given-names>
            <surname>Solutions</surname>
          </string-name>
          (
          <year>2011</year>
          )
          <fpage>1</fpage>
          -
          <lpage>33</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>D.</given-names>
            <surname>Goldenberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Albert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Bernardi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Estevez</surname>
          </string-name>
          ,
          <article-title>Free lunch! retrospective uplift modeling for dynamic promotions recommendation within roi constraints</article-title>
          .,
          <source>in: Proceedings of the 14th ACM Conference on Recommender Systems</source>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>R. L</given-names>
            . Kaufman, J.
            <surname>Pitchforth</surname>
          </string-name>
          , L. Vermeer,
          <article-title>Democratizing online controlled experiments at booking</article-title>
          . com,
          <source>arXiv preprint arXiv:1710.08217</source>
          (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>R.</given-names>
            <surname>Kohavi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. M.</given-names>
            <surname>Henne</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Sommerfield</surname>
          </string-name>
          ,
          <article-title>Practical guide to controlled experiments on the web: listen to your customers not to the hippo</article-title>
          ,
          <source>in: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining</source>
          ,
          <year>2007</year>
          , pp.
          <fpage>959</fpage>
          -
          <lpage>967</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>