Tuning filters to refine Ads eligibility to surface in search results emerges as a pivotal problem. It often necessitates a nuanced approach to cater to diverse requirements from the customers. Adjusting these filters must judiciously balance the preferences of both advertisers and users in the online marketplace. Hence, it requires a multi-objective optimization which often turns out to be hard due to the conflicting nature of the objectives from these customers. In this paper we present AFA: Auto-tuning Filters for Ads - a novel application of Bayesian Optimization for auto-tuning these filters. We specifically develop AFA to employ a probabilistic model to navigate the intricate trade-ofs between multiple objectives. It iterates over a feasible solution space and quickly converges to an operating point which ensures showing well performing ads while increasing their scale. This ofers a substantial advancement in the automation for digital advertising campaigns. Our approach significantly reduces the reliance on manual adjustments and expensive A/B testing, as demonstrated by empirical results from a large-scale e-commerce platform.
Sponsored search is an advertising model used by search engines to display paid advertisements alongside organic search results. When a user enters a query, the search engine runs an auction among advertisers who have bid on ners of this auction have their ads displayed in prominent positions on the search results page, typically marked as ”Sponsored” or ”Ad” to distinguish them from other content.
While these advertisements (a.k.a. ads) benefit advertisers by increasing the visibility of their products to potential users, maintaining user satisfaction is also important for the e-commerce platform to monetize the search from showing ads. Thereby, it is crucial to ensure that the participating ads are of high quality to comply with user’s desirability. Generally, ad platform employs a variety of quality filters controlled by hyperparameters 1. Its core purpose is to eficiently navigate through extensive datasets, find content that aligns closely with the user’s search intent and personal preferences by providing more relevant outcomes.
The paper presents a innovative approach for automatically tuning quality filters in the search monetization domain using Bayesian Optimization. The approach aims to optimize the quality threshold to balance scale improvement and ad performance. An automated pipeline was developed to minimize human efort, time, and errors in this process.
E-commerce platforms face the challenge of setting the right quality threshold for ads while ensuring suficient ad exposure to increase scale. This issue arises from the difering needs of two main stakeholders: advertiser and user
Advertisers enlist their items on e-commerce marketplace to sell and expect greater visibility of their product in exchange 0000-0003-2264-8097 (J. Gharibshah) AdKDD ’24, August 26, 2024, Barcelona, Spain CEUR
ceur-ws.org quality filter: (1) Observed data points consist of thresholds and metrics lift (set of values for each objective function on that given threshold) denoted as < , ( ), ( ) > respectively, initial data points collected based on random thresholds from a feasible solution space. (2) Optimizer fits surrogate function on the observed data points and produce ∗ as the next threshold (T*) to be evaluated (3) Evaluator measures the impact of the new threshold ∗ on the objective value and produce another triple < ∗, ( ∗), ( ∗ ) > to be appended to the stored observed data points so that we can explore next point based on that for paying more to the platform. As they often prioritize visibility over the quality of their product, the platform needs to maintain a quality filter that strikes a balance between their need for exposure for the product (measured by scale) and its relevance to the user’s preferences.
The e-commerce platform is also committed to meeting the essential needs of user who uses this platform to find high quality product based on their search queries as input. It is the platform’s responsibility to provide them with relevant and high quality search results for their queries. To achieve this, an appropriate quality filter is required to surface products in the search results that are most relevant to the user’s query. User’s preference is calculated as the ratio of the click over number of impressions they have (to measure iflter’s performance).
This dilemma between the preferences of users and advertisers presents a challenging problem in e-commerce platforms. On one hand, they need to increase the quality threshold to meet users’ expectations of obtaining more relevant items. On the other hand, they must keep the quality filter to a degree where the platform can accommodate advertisers’ appetite for increased visibility. Therefore they aim to enhance ad scales without compromising the performance.
The dynamic nature of the marketplace, characterized by continuous growth, frequent updates, and new feature roll outs, further complicates the task by rendering the static thresholds obsolete in short order. Manual tuning of filters is not only time-intensive but also lacks eficiency, underscoring the need for an automated, scalable solution. Main inspirations for this work are:
Capturing and quantifying the intricate correlations between two diferent objectives (Scale and Performance), which often exhibit complex interactions, is a hard task. The absence of a formal function to encompass both the objectives, exacerbates this issue and hinders the systematic exploration of the trade-ofs inherent to the objectives. Consequently, without a clear mathematical framework to navigate the multi-dimensional objective space, manual optimization becomes not only cumbersome but also prone to sub-optimal decision-making, as it relies on intuition rather than analytical precision. This emphasize the necessity for a structured approach that can efectively balance these competing goals and facilitate the discovery of an optimal solution that results in a desired outcomes.
As a common practice, multiple experiments covering
various operating points and parameters via grid or random
search [
Our proposed system employs a probabilistic model that captures the complex relationship between the quality threshold and the multi-objective function, which includes both performance and scale lift. By utilizing Bayesian optimization, we iteratively update the quality filter settings, eficiently navigating the search space to identify the optimal threshold that satisfies the dual objectives as demonstrated in Figure 1. This approach not only accounts for the inherent uncertainty in user behavior but also reduces the need for extensive manual tuning and A/B testing.
To this end, we introduce AFA as an auto filter tuning
pipeline for ads to tune the quality filter. AFA consists of
three major steps that we illustrated in the figure 1 and we
introduce them here.
is a process to collect and store data points observed over
the course of running our system. We store a triplet
( , 1( ), 2( )) for each filter threshold that we
explore. This triplet has a threshold along with two
corresponding objective values, i.e., scale lift and performance
lift. The pipeline starts with some initial data points which
are collected via running A/B tests over various thresholds
within the exploration range to give the process a warm
start.
1.3.2. Optimizer
utilizes a Bayesian optimizer to solve multiple objectives
consisting of scale and performance. This optimizer will
read initial data points i.e ( , 1( ), 2( )) and build a
surrogate model based on those data points. Then it utilizes
an acquisition function [
is the component which runs an A/B test to evaluate the objective function with respect to the suggested threshold. Then, we compute a new triple as ( ∗, 1( ∗), 2( ∗)) which consists of the suggested threshold ∗ by the optimizer and corresponding objectives values evaluated by the evaluator. This newly explored data point is then feedback to the initilizer.
Thus, the pipeline constantly updates the quality threshold to respond to new changes, ensuring that search results remain relevant.
This paper presents several key contributions: • We formalize the process of general filter tuning problem by leveraging a Bayesian optimization method in AFA, considering the need of multiple competing stakeholders. • AFA provides a fast-converging approach that significantly reduces human efort and time for tuning quality threshold. • AFA has been successfully deployed within an ecommerce platform, demonstrating its efectiveness in refining quality filters for the search engine and attesting to its scalability in large-scale industry settings.
The structure of the paper is as follows: Section 2 describes implementation details in the optimization process and evaluation metrics. Section 3 analyzes the results of AFA. Section 4 reviews related literature. Section 5 concludes with a summary and future research directions.
In the pipeline presented in Figure 1, there are three components as we explained in section 1.3. In this section, we focus on the Optimizer and provide more details regarding its implementation and evaluation. where () represents ad scale changes.
Maintaining performance: AFA also looks for that quality thresholds to minimize performance lift via: ∈̄ ()
∈ Thresholds ∈̄ ∈ Thresholds − | () | subject to the constraint | () | < . where () represents performance changes.
We combine these two objective in one as follows: =̄
argmax ∈ Thresholds ( () − | () |) subject to the constraint | () | < . In this function, we apply the same weight to both objectives, although these weights may vary in diferent scenarios.
To impose the constraint on the objective function we employ a penalty and reward mechanism. We penalize and reward the objective function when the explored thresholds fails and succeeds to maintain the constraint respectively. Thereby, we formulate the optimization problem as follows:
In this section, we describe our objective function and its formulation.
Objective function In AFA, we aim to find a threshold value for quality filter that satisfies two objectives related to advertisers and users needs: maintaining performance lift and a positive scale lift in ads. The lift amounts calculated for this purpose are all relative diferences between the suggested value and the current value in the system as AFA updates the threshold in each iteration. (more detail in section 3 and equation 14).
We formulate these objectives as follows:
Increasing scale: AFA looks for quality thresholds which increase the ad scale in search result by: () = =̄ ∈ Thresholds ( () − ∗() ) where the modified performance will be defined as: ∗() = { () ∗ () () ∗ () − ≤ () ℎ < Here, () and () are the amount of penalty and reward we apply on the .
This generic formulation can accommodate additional objectives and constraints, as well as custom-defined penalty and reward values, according to business requirements. Although we have defined the constraint in closed-form, we have not devised a closed-form definition for the objective components that would allow us to use the Lagrange multiplier method. However, we discuss related work and alternative approaches in Section 4.
In our pipeline, at each iteration, the Optimizer solves for the objective function defined in Equation 4 by using a Bayesian optimization approach, as we describe below.
Bayesian Optimization employs a probabilistic model to represent the uncertainty about the objective function’s behavior, and it updates this model iteratively using Bayes’ Theorem as new data points are observed.
Bayes’ Theorem is formulated as follows: (| data) = ( data|) ()
( data) where: () is the prior probability of the hypothesis before seeing the data. ( data|) is the likelihood of the data under the hypothesis. ( data) is the marginal likelihood or evidence, the probability of the data under all possible hypotheses. (| data) is the posterior probability of the hypothesis after seeing the data. In our case, is representing the threshold
Based on this Bayes’ Theorem, we build a surrogate model which will rely on the observed data points (thresholds) and make a prediction for unobserved data points. The surrogate model is as follows:
a) Surrogate model is a probabilistic model used to approximate the unknown objective function () that we wish to optimize. The surrogate model, denoted as (̂) , is used to predict the output of () given new inputs , and to estimate the uncertainty of that prediction.
The surrogate model we used is a Gaussian Process (GP), which is defined by a mean function () and a covariance function (kernel) (, ′). The GP surrogate model for any input point is: (̂) ∼ ((), (, ′))
This formulation allows the Bayesian optimization
algorithm to not only predict the function value at unobserved
points but also quantify the prediction uncertainty, which
is crucial for balancing exploration and exploitation during
the optimization process. In our GP, we used a White Noise
as a kernel function in Bayesian optimization[
The White Noise kernel is defined as:
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
where 2 is the noise variance, a hyper-parameter that
represents the variance of the noise in the observations. Δ(, ′)
is the Kronecker delta function, which equals 1 if ( x = x’
) (i.e., the points are identical) and 0 otherwise [
b) Acquisition function is a function that guides the optimization process by determining where to sample next. The acquisition function balances exploration of the search space (sampling where the model is uncertain) with exploitation (sampling where the model predicts high performance).
Two common acquisition functions which we used in AFA are:
1) Expected Improvement (EI): This function measures the expected amount of improvement over the current best observation ( +) at a new point .
() = [max( () − ( +), 0)] where () is the objective function and ( +) is the best observed value so far.
2) Probability of Improvement (PI): This function measures the probability that sampling at a new point will lead to an improvement over the current best observation ( +).
() = ( () > ( +))
The selected acquisition function will generate a new threshold, which will be passed to the evaluator for testing in the next step.
To ensure fast convergence we adopted PI . We explain the evaluation process in Section 2.2.
We need to evaluate the optimizer’s eficiency using the Evaluator through an A/B test.
We execute this phase of the pipeline at the conclusion of each iteration, which, in our case, spans a period of six days. We set the recommended threshold by AFA as a treatment in an A/B test and get the real-time impact on both the scale and performance since metrics like performance are dependent to user behaviour and is not possible to be computed ofline precisely. The objectives calculated by AFA are:
Scale is calculated as the ratio of the total number of times ads are displayed to the number of qualified queries (i.e. buyer searches) that trigger the ads, expressed by the formula: = .
(11)
Performance is measured as the proportion of clicks an advertisement receives relative to the number of times it is shown (impressions), represented by the formula: = .
(12)
We introduce quality and revenue here as additional output metrics in order to track business impacts.
Revenue is the income earned from displaying ads on a platform. It can be calculated as: =
Buyers’ action (i.e clicks) × Revenue per action group (T), the formula
Quality is the indicator of the relevance of the ads with respect to query.
The amount of lift in an A/B test is calculated by comparing the evaluation metric (e.g., scale) between the treatment , and the control group (C), , using = ( − ) × 100%
The amount of lifts will be used by the Optimizer and also stored in the table introduced at Section 2 for the ongoing exploration. (13) (14)
Analyzing the outcomes of filter tuning to meet multiple objectives simultaneously, is not a trivial task. It requires a deep and precise understanding of the interplay between diferent objectives. However, in our algorithm, we have formalized this relationship in a robust manner, allowing measurable and accurate improvements. In this section, we discuss our experimental results.
Our experiments demonstrate that AFA could achieve the defined objectives within three iterations through our tuning pipeline. Figure 2 illustrates how the model evolved over multiple iterations.
One of the standout results of using AFA for automatic iflter tuning is its marked eficiency in both time and human efort. In a comparative analysis between manual tuning and AFA-assisted tuning, we observed a significant reduction in the number of iterations required to optimize the quality filter. Manual eforts necessitated 12 iterations to achieve satisfactory results, whereas AFA achieves comparable improvements in the ad scale metric and maintained performance within just 3 iterations.
Moreover, the total number of data points needed for AFA was halved, with only 6 data points (including 3 initial data points) compared to the 12 required for manual tuning. This reduction in data points translates directly into savings extensive AB testing spanning for multiple weeks as well as post test analysis eforts. AFA autonomously computes the next threshold to test, minimizing the need for manual evaluation.
Time eficiency was also significantly improved. Each data point in the AFA corresponded to outcome of one week of experimentation using our A/B testing platform, leading to an optimization timeline of just 4 weeks. This consisted of a week of initial data collection followed by 3 weeks of iterations. Conversely, the manual approach spanned over 12 weeks, with each iteration taking one week - yet resulting in a sub-optimal solution for the problem.
In summary, the deployment of AFA for quality filter tuning within our experimental framework yielded a 4X increase in eficiency in terms of both time and human efort. This demonstrates AFA’s potential to accelerate the tuning process, ultimately reducing the laborious and timeintensive nature of manual filter tuning.
In this section, we present the amount of improvement achieved by employing AFA in our real world production platform, targeting a large online user population over a two-week experiment period.
We compared two thresholds: the recommended threshold from AFA, which converged after three iterations via an automated pipeline, and the best threshold suggested by manual tuning, which required human adjustment over twelve iterations. As indicated in Table 1, both methods improved the scale, but AFA showed a higher impact on business metrics. AFA not only improves the quality score 80% but also we achieved a 0.72% higher performance compared to manual tuning, resulting in a mere 0.18% performance loss when using AFA, as opposed to a 0.66% performance loss with the manual approach. Considering that performance was our constraint during the optimization phase, a smaller loss signifies a better operating point. Moreover, we observed a 44% improvement in ad revenue with AFA; manual tuning resulted in a 0.22% loss in revenue, while AFA managed a 0.01% gain. We also observed a startling gain of 80% quality as well. In summary, our approach enabled the display of more ads without sacrificing performance.
Furthermore, it improved quality and revenue which are key business metrics.
In Figure 3, we provide an empirical example illustrating the search results page for a buyer query ”leather jacket”.
Using the threshold recommended by AFA, we were able to display more sponsored ads compared to the threshold determined by manual tuning. Specifically, AFA enabled the presentation of four sponsored ads, whereas the manual tuning approach yielded only two. This empirical evidence (a) Surrogate model and acquisition function after (b) Surrogate model and acquisition function after (c) Surrogate model and acquisition function after feeding initial data points second iteration third iteration (a) An example of items retrieved for a query using suggested threshold (b) An example of items retrieved for a query using suggested threshold by AFA by a manual approach supports our claim that AFA fine-tunes the threshold more precisely than manual methods, thereby improving ad scale.
Additionally, we observed that all sponsored items were relevant and of high quality, which correlates with the improved performance and ad revenue as detailed in Table 1.
There is limited research on automatic filter tuning in ads industry, and even fewer studies on pipelines to facilitate the process. Prior works can be categorized in the following categories.
Black box optimization (BBO) methods are essential
for optimizing functions without closed-form expressions,
which are often encountered in real-world scenarios
involving complex systems. Evolutionary algorithms (EAs) and
genetic algorithms (GAs), such as those described by [
To overcome the limitations of traditional EAs and GAs,
surrogate-based optimization (SBO) techniques have been
developed. Bayesian optimization was introduced by [
The incorporation of Bayesian optimization into
multiobjective optimization (MOO) has been an area of active
research. [
Multi-objective optimization (MOO) addresses
complex problems where multiple, often conflicting, objectives
must be optimized simultaneously. Researchers like [
Recent work in MOO has focused on improving the
efifciency and scalability of these algorithms. For instance,
[
Hyper-parameter tuning: Hyper-parameter tuning is
a critical step in machine learning that involves selecting
the optimal set of hyper-parameter for a learning algorithm
to maximize its performance. Hyper-parameter are the
conifguration settings used to structure the learning process, as
opposed to model parameters that are learned from the data.
There are several work focusing on search mechanism like
grid and random search which are the simplest and most
commonly used approaches [
Our research introduces an automated pipeline that applies new advancements to automatically tune thresholds for advertising systems, a problem not previously solved. It combines Bayesian Belief Optimization (BBO) and MultiObjective Optimization (MOO) in a novel way, providing a practical solution to a real-world challenge.
Our study conclusively demonstrates that AFA can autonomously and efectively tune quality filters in advertising programs, outperforming manual tuning methods in terms of both speed and precision. AFA significantly reduces the number of necessary evaluations, enabling faster convergence towards optimal settings. This automated approach streamlines the optimization process and mitigates the potential for human error and bias. The implications of our findings are suggesting that AFA could be instrumental in various optimization tasks across the digital advertising domain and potentially in other fields.
Looking forward, our future plans involve (a) extending the application of the proposed pipeline to other facets of our e-commerce platform and (b) incorporating additional dimensions into the optimization approach, thereby broadening the scope and impact of AFA.