In a large E-commerce platform, all the participants compete for impressions under the allocation mechanism of the platform. Existing methods mainly focus on the short-term return based on the current observations instead of the long-term return. In this paper, we formally establish the lifecycle model for products, by defining the introduction, growth, maturity and decline stages and their transitions throughout the whole life period. Based on such model, we further propose a reinforcement learning based mechanism design framework for impression allocation, which incorporates the first principal component based permutation and the novel experiences generation method, to maximize short-term as well as long-term return of the platform. With the power of trial-and-error, it is possible to optimize impression allocation strategies globally which is contribute to the healthy development of participants and the platform itself. We evaluate our algorithm on a simulated environment built based on one of the largest E-commerce platforms, and a significant improvement has been achieved in comparison with the baseline solutions.
• Computing methodologies → Reinforcement learning; Policy iteration; • Applied computing → Online shopping;
Nowadays, E-commerce platform like Amazon or Taobao has developed into a large business ecosystem consisting of millions of Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
SIGIR 2018 eCom, July 2018, Ann Arbor, Michigan, USA
© 2018 Copyright held by the owner/author(s).
customers, enterprises and start-ups, and hundreds of thousands of
service providers, making it a new type of economic entity rather
than enterprise platform. In such a economic entity, a major
responsibility of the platform is to design economic institutions to
achieve various business goals, which is the exact field of
Mechanism Design [
Existing work of impression allocation in literature are mainly
motivated and modeled from a perspective view of supervised
learning, roughly falling into the fields of information retrieval [
Although the properties of products can not be fully observed
or accurately predicted, they do share a similar pattern in terms
of development trend, as summarized in the product lifecycle
theory [
• Introduction: Also known as market development - this is when a new product is first brought to market. Sales are low and creep along slowly. • Growth: Demand begins to accelerate and the size of the total market expands rapidly. • Maturaty: Demand levels of and grows. • Decline: The product begins to lose consumer appeal and sales drift downward.
During the lifecycle, new products arrive continuously and outdated products wither away every day, leading to a natural metabolism in the E-commerce platform. Due to the insuficient statistics, new products usually attract few attention from conventional supervised learning methods, making the metabolism a very long period.
Inspired by the product lifecycle theory as well the reinforcement mechanism design framework, we consider to develop reinforcement mechanism design while taking advantage of the product lifecycle theory. The key insight is, with the power of trial-and-error, it is possible to recognize in advance the potentially hot products in the introduction stage as well as the potentially slow-selling products in the decline stage, so the metabolism can be speeded up and the long-term eficiency can be increased with an optimal impression allocation strategy.
We formally establish the lifecycle model and formulate the impression allocation problem by regarding the global status of products as the state and the parameter adjustment of a scoring function as the action. Besides, we develop a novel framework which incorporates a first principal component based algorithm and a repeated sampling based experiences generation method, as well as a shared convolutional neural network to further enhance the expressiveness and robustness. Moreover, we compare the feasibility and eficiency between baselines and the improved algorithms in a simulated environment built based on one of the largest E-commerce platforms.
The rest of the paper is organized as follows. The product lifecycle model and reinforcement learning algorithms are introduced in section 3. Then a reinforcement learning mechanism design framework is proposed in section 4. Further more, experimental results are analyzed in section 5. Finally, conclusions and future work are discussed in section 6. 2
Many researches have been conducted on impression allocation
and dominated by supervised learning. In ranking phase, search
engine aims to find out good candidates and brought them in front
so that products with better performance will gain more
impressions. Among which click-through rate is one of the most common
representation of products performance. Some research presents an
approach to automatically optimize the retrieval quality with
wellfounded retrieval functions under risk minimization frame-work
by historical click-through data [
One of the most related topics in user impressions allocation is
item cold-start problem [
The application of reinforcement learning in commercial system
such as web recommendations and e-commerce search engines has
not yet been well developed. Some attempts are made to model
the user impression allocation problem in e-commerce platform
such as Tabao.com and Amazon.com. By regarding the platforms
with millions of users as environment and treating the engines
allocating user impressions as agents, an Markov Decision Process
or at least Partially Observable Markov Decision Process can be
established. For example, an reinforcement learning capable model
is established on each page status by limit the page visit sequences
to a constant number in a recommendation scene [
Recently, mechanism design has been applied in impression
allocation, providing a new approach for better allocating user
impressions [
In this subsection, we establish a mathematical model of product lifecycle with noises. At step t , each product has an observable attribute vector xt ∈ Rd and an unobservable latent lifecycle state zt ∈ L, where d is the dimension of the attribute space, and L = {0, 1, 2, 3} is the set of lifecycle stages indicating the the introduction, growth, maturity and decline stages respectively. Let pt ∈ R be the CTR and qt ∈ R be the accumulated user impressions of the product. Without loss of generality, we assume pt and qt are observable, pt , qt are two observable components of xt , the platform allocates the impressions ut ∈ R to the product. The dynamics of the system where f can be seen as the derivative of the p, and д is the state transition function over L
.
According to the product lifecycle theory and online statistics, Introduction (1) (2) (3) qt +1 = qt + ut pt +1 = pt + f (zt , qt ) zt +1 = д(xt , zt , t ) the derivative of the CTR can be formulated as f (zt , qt ) = (2 − z)(1 + e−δ (qt ))2
(ch − cl )e−δ (qt ) ξ , + ξ , z ∈ {1, 3} z ∈ {0, 2} where ξ ∼ N (0, σ 2) is a gaussian noise with zero mean and variance σ 2, δ (qt ) = (qt − q˜t z − δµ )/δσ is the normalized impressions accumulated from stage z , q˜t z
is the initial impressions when the product is firstly evolved to the life stage z, δµ , δσ are two unobservable parameters for normalization, and ch , cl ∈ R are the highest CTR and the lowest CTR during whole product lifecycle, inferred from two neural networks, respectively:
cl = h(xt |θl ), ch = h(xt |θh ), obtained for the product in the decline stage. where h(·|θ ) is a neural network with the fixed parameter θ , indicating that cl , ch are unobservable but relevant to attribute vector xt . Intuitively, when the product stays in introduction or maturity stage, the CTR can be only influenced by the noise. When the product in the growth stage, f will be a positive increment, making the CTR increased up to the upper bound ch . Similar analysis can be t > t2, q < q2 g : z = 0 t > t1 z = 1
q > q2 z = 2 t > t3 z = 3 in to the maturity stage if its accumulated impressions q reaches q2, or the decline stage if the time exceeds t2 while q is less than q2. A product in the maturity stage will finally enter the last decline stage if the time exceeds t3. Otherwise, the product will stay in current stage. Here, t1, t2, t3, q2 are the latent thresholds of products.
We simulate several product during the whole lifecycle with diferent latent parameters (the details can be found in the experimental settings), the CTR curves follow the exact trend described in Fig. 2. can be written as Typical Lifecycle space S, action space A, a conditional probability distribution p(·) and a scalar reward function r = R(s, a), R : S × A For states st , st +1 ∈ S and action at ∈ A, distribution function p(st +1 |st , at ) denotes the transition probability from state st to st +1 when action at is adopted in time step t , and the Markov property → p(st +1 |st , at ) = p(st +1 |s1, a1, · · · , st , at ) holds for any historical trajectories s1, a1, · · · , st to arrive at status st . A future discounted γt = Í∞ k=t γ k−t R(sk , ak ), where γ is a scalar factor representing the discount. A policy is denoted as πθ (at |st ) which is a probability distribution mapping from S to A , where diferent policies are distinguished by parameter θ .
The target of agent in reinforcement learning is to maximize the expected discounted return, and the performance objective can be denoted as
π max J = E R
γ1 π ] = Es∼d π ,a∼πθ [R(s, a)] where d π (s) is a discounted state distribution indicating the possibility to encounter a state s under the policy of π . An action-value function is then obtained iteratively as
Q(st , at ) = E R(st , at ) + γ Ea∼πθ [Q(st +1, at +1)]
In order to avoid calculating the gradients of the changing state
distribution in continuous action space, the Deterministic Policy
Gradient(DPG) method [
∇θ µ Qw (s, a) = Es∼d µ
∇θ µ µ (s)∇aQw (s, a)|a=µ (s) where µ is the deep actor network to approximate policy function. And the parameters of actor network can be updated as θ µ
← θ µ + α E ∇θ µ µ (st )∇aQw (st , at )|a=µ (s) where Qw is an obtained approximation of action-value function called critic network. Its parameter vector w is updated according min L = Es∼d µ yt − Qw (st , at )) w 2 where yt = R(st , at ) + γ Qw′ (st +1, µ ′(st +1)), µ ′ is the target actor network to approximate policy π , Qw′ is the target critic network (4) (5) (6) (7) (8) 3.2
to objective methods Reinforcement learning maximizes accumulated rewards by trialand-error approach in a sequential decision problem. The sequential decision problem is formulated by MDP as a tuple of state to approximate action-value function. The parameters w ′, θ µ ′ are updated softly as In our scenario, at each step, the platform observes the global information of all the products, and then allocates impressions according to the observation and some certain strategy, after which the products get their impressions and update itself with the attributes as well as the lifecycle stages. Then the platform is able to get a feedback to judge how good its action is, and adjust its strategy based on all the feedbacks. The above procedures leads to a standard sequential decision making problem.
However, application of reinforcement learning to this problem encounters sever computational issues, due to high dimensionality of both action space and state space, especially with a large n. Thus, we model the impression allocation problem as a standard reinforcement learning problem formally, by regarding the global information of the platform as the state
s = [x1, x2, ..., xn ]T ∈ Rn×d where n is the number of the product in the platform, d is the dimension of the attribute space, and regarding the parameter adjustion of a score function as the action,
a = π (s |θ µ ) ∈ Rd where π is the policy to learn parameterize by θ µ , and the action a can be further used to calculate scores of all products 1 oi = 1 + e−aTxi
, ∀i ∈ {1, 2, ..., n} After which the result of impression allocation over all n products can be obtained by
eoi
ui = Íin eoi , ∀i ∈ {1, 2, ..., n}
Without loss of generosity, we assume at each step the summation
of impressions allocated is 1, i.e., Íin ui = 1. As is well known,
products number n(billions) is far bigger than products attributes
dimensions d(thousands) in large scale E-commerce platforms. By
such definition, the dimension of the action space is reduced to d,
significantly alleviating the computational issue in previous work [
The goal of policy is to speeded up the metabolism by scoring and ranking products under the consideration of product lifecycle, making the new products grow into maturity stage as quickly as possible and keeping the the global eficiency from dropping down during a long term period. Thus, we define the reward related to s and a as
R(s, a) = n1 Õni t1it∫=0 p(t )dqd(tt )dt (14) where ti is the time step of the i-th product after being brought to the platform, p(t ), q(t ) is the click through rate function and accumulated impressions of a product respectively. The physical meaning of this formulation is the mathematical expect over all products in platform for the average click amount of an product during its lifecycle, indicating the eficiency of products in the platform and it can be calculated accumulatively in the online environment, which can be approximately obtained by (15) (16) 1 Õn 1 Õti R(s, a) ≈ n i ti τ =0 pτi uτi
A major issue in the above model is that, in practices there will
be millions or even billions of products, making combinations of
all attribute vectors to form a complete system state with size n × d
computationally unafordable as referred in essays [
sˆ = [x1, x2, · · · , xns ]T ∈ Rns ×d where sˆ is the approximated state. Then, two issues arise with such sampling method: • In which order should the sampled ns products permutated in sˆ, to implement the permutation invariance? • How to reduce the bias brought by the sampling procedure, especially when ns is much smaller than n? To solve these two problem, we further propose the first principal component based permutation and the repeated sampling based experiences generation, which are described in the following subsections in details. 4.1
The order of each sampled product in the state vector has to be
proper arranged, since the unsorted state matrix vibrates severely
during training process, making the parameters in network hard to
converge. To avoid it, a simple way for permutation is to make order
according to a single dimension, such as the brought time ti , or the
accumulated impressions qi . However, such ad-hoc method may
lose information due to the lack of general principles. For example,
if we sort according to a feature that is almost the same among
all products, state matrix will keep vibrating severely between
observations. A suitable solution is to sort the products in an order
that keep most information of all features, where the first principal
components are introduced [
∥e ∥=1 eˆ = βeˆ + (1 − β ) (et − eˆ) ∥βeˆ + (1 − β ) (et − eˆ)∥ vi = eˆTxi , i = 1, 2, · · · , ns (17) (18) where et is the first principal component of system states in current step t obtained by the classic PCA method as in Eq. 17. eˆ is the projection vector softly updated by et in Eq. 18, with which we calculate the projected score of each products in Eq. 19. Here 0 < β < 1 is a scalar indicating the decay rate of eˆ. Finally, the state vector is denoted as sˆ = [xk1 , xk2 , · · · , xkns ]
T (20) where k1, k2, · · · , kns is the order of products, sorted by vi . 4.2
We adopt the classic experience replay technique [
To reduce the above bias, we propose the repeated sampling based experiences generation. For each original experience, we do repeated sampling st and st +1 for m times, to obtain m2 experiences of
(sˆti , at , rt , sˆtj+1), i, j ∈ 1, 2, · · · , m (21) as illustrated in Fig. 3. This approach improves the stability of ob(st,at,Rt,st+1)
(oi,t, at, Rt, oj,t+1) t Samplinagt+1
Batch Sliding Pool agent
Sliding Pool t SaBmatpclihngat+1 agent servation in noise environment. It is also helpful to generate plenty of experiences in the situation that millions of times repetition is unavailable.
It is worth noting that, the repeated sampling is conducted in the training phase. When to play in the environment, the action at is obtained through a randomly selected approximated state sˆt , i.e., at = π (sˆt1). Actually, since at does not necessarily equal to π (sˆti ), ∀i ∈ 1, 2, · · · , m, it can further help learning a invariant presentation of the approximated state observations.
The overall procedure of the algorithm is described in
Algorithm 1. Firstly, a random sampling is utilized to get a sample of
system states. And then the sample is permutated by the projection
of the first principal components. After that, a one step action and
multiple observations are introduced to enrich experiences in
experience pool. Moreover, a shared convolutional neural network is
applied within the actor-critic networks and target actor-critic
networks to extract features from the ordered state observation [
Repeated observation times
respectively. Apart from the normalized dynamic CTR p and the
accumulated impressions q, the attribute vector x is uniformly
sampled from [
The DDPG algorithm is adopted as the learning algorithm. The
learning rates for the actor network and the critic network are 10−4
and 10−3 respectively, with the optimizer ADAM [
Comparisons are made within the proposed reinforcement learning based methods as • CTR-A: The impressions are allocated in proportion to the
CTR score. • T-Perm: The basic DDPG algorithm, with brought time based permutation and a fully connected network to process the state • FPC: The basic DDPG algorithm, with first principal component based permutation and a fully connected network to process the state. • FPC-CNN: FPC with a shared two-layers convolutional neural network in actor-critic networks. • FPC-CNN-EXP: FPC-CNN with the improved experiences generation method. where CTR-A is the classic supervised learning method and the others are the proposed methods in this paper. For all the experiments, CTR-A is firstly applied for the first 360 steps to initialize system into a stable status, i.e., the distribution over diferent lifecycle stages are stable, then other methods are engaged to run for another 2k steps and the actor-critic networks are trained for 12.8k times. 5.2
We firstly show the discounted accumulated rewards of diferent methods at every step in Fig. 4. After the initialization with the CTR-A, we find that the discounted accumulated reward of CTR-A itself almost converges to almost 100 after 360 steps (actually that why 360 steps is selected for the initialization), while that of other methods can further increase with more learning steps. It is showed that all FPC based algorithms beat the T-Perm algorithm, indicating that the FPC based algorithm can find a more proper permutation to arrange items while the brought time based permutation leads to a loss of information, making a drop of the final accumulated rewards. Moreover, CNN and EXP algorithms perform better in extracting feature from observations automatically, causing a slightly 160 n r u tre140 d e t uon120 c s i d100 0.6 0.5
FPC-CNN-EXP FPC-CNN FPC T-Perm
Introduction Growth Maturity Decline 200 400 600 800 1000 1200 1400 1600 1800
step improvement in speeding up the converging process. Both the three FCP based algorithms converge to same final accumulated rewards for their state inputs have the same observation representation. 200 400 600 800 1000 1200 1400 1600 1800 step
Then we investigate the distribution shift of the impression allocation over the 4 lifecycle stages after the training procedure of the FPC-CNN-EXP method, as shown in Fig. 5. It can be seen that the percentage of decline stage is decreased and percentage of introduction and maturity stages are increased. By giving up the products in the decline stage, it helps the platform to avoid the waste of the impressions since these products are always with a low CTR. By encouraging the products in the introduction stage, it gives the changes of exploring more potential hot products. By supporting the products in the maturity stage, it maximizes the short-term eficiency since the they are with the almost highest CTRs during their lifecycle.
We finally demonstrate the change of the global clicks, rewards as well as the averaged time durations for a product to grow up into maturity stage from its brought time at each step, in terms of relative change rate compared with the CTR-A method, as is shown in Fig. 6. The global average click increases by 6% when the rewards is improved by 30%. The gap here is probably caused by the inconsistency of the reward definition and the global average click metric. In fact, the designed reward contains some other implicit objectives related to the metabolism. To further verify the guess, we show that the average time for items to growth into maturity stage has dropped by 26%, indicating that the metabolism is significantly speeded up. Thus, we empirically prove that, through the proposed reinforcement learning based mechanism design which utilizes the lifecycle theory, the long-term eficiency can be increased by speeding up the metabolism.
1.4 1.2 0.6 Click Rewards Time cost 6
In this paper, we propose an end-to-end general reinforcement learning framework to improve the long-term eficiency by speeding up the metabolism. We reduce action space into a reasonable level and then propose a first principal component based permutation for better observation of environment state. After that, an improved experiences generation technique is engaged to enrich experience pool. Moreover, the actor-critic network is improved by a shared convolutional network for better state representation. Experiment results show that our algorithms outperform the baseline algorithms.
For the future work, one of the promising directions is to develop a theoretical guarantee for first principal component based permutation. Another possible improvement is to introduce the nonlinearity to the scoring function for products.