From previous literature, we see that the researches in purchase intention prediction vary mainly with dataset’s attributes. Nevertheless, the approaches that employ queries to perceive purchase intention is still scarce. In this study, we collaborate with Yahoo Taiwan which incorporates its e-commerce platform’s data. In particular, we collect the dataset describing various behaviors made by its users, e.g., actions (click, view, buy), connection properties (timestamp, time spent), and the most critical piece of information entered queries. Diferent from most existing researches, this paper focuses on the construction of user representation via their online actions to cope with e-commerce purchase prediction. Moreover, we utilize the relationship between users’ textual inputs and their actions to realize his/her intention. Furthermore, we propose that users behavior are usually time-sensitive in two manners, that is, environment-shifting and experience-shifting. These phenomena illustrate the evolution of the user’s actions through the purchase activity, which occurs in actual e-commerce platforms.

According to the above observations, we establish Prediction framework that analyzes User’s Sequential Actions via Contextbased information (PSAC), a prediction system based on sessionbased representation which records the behavioral sequence of each user. Meanwhile, we apply corpus embedding techniques to encode the latent information of entered queries and derive related attributes from the sequence of entered queries. Finally, the system utilizes two deep learning frameworks to analyze, store, and predict a user’s intention under diferent scenarios. The simulation results indicate that the sequence of query contents is an essential feature in learning user’s shopping intention and points out the diference between the two frameworks. Our main contributions in this paper include: (1) We implement a Prediction framework that analyzes User’s Sequential Actions via Context-based information, which predicts user’s purchase intention via query-based information and their corresponding actions. (2) We investigate the behavior of the real-world user on ecommerce activity that provides further information in predicting the user’s purchase intention. (3) We confirm that query factors are critical in intent and action predictions, which can be utilized in future works and industries application.

The remainder of the paper proceeds as follows. Section 2 consists of three parts: a description of the datasets, the investigation of data, and the preprocessing procedures. Section 3 describes the implementation of e-commerce purchasing prediction system via time-shifting scheme and the deep learning frameworks. Finally, Section 4 examines the simulation results, and Section 5 presents the conclusions. 2

In this paper, we use a dataset provided by Yahoo Taiwan, which contains a variety of behaviors of its e-commerce platform users. Most importantly, the data not only document user’s activities on the platform but also save their searching queries, a textual expression used by the customers to find the products they want. 2.1

Our sample dataset consists of real-world transaction records from the e-commerce platform which spans six months. Specifically, the dataset consists of three types of information: click, view, and buy, of over a million unique user’s online actions, including 12M, 10M, and 1.2M records individually. Both click and view records specify the query terms the user entered before such actions. For further merging procedure, we label each record in the dataset with a session id s which specifies the belonging connection. The dataset contains the following information: 2.1.1 Click Records. Each click action represents the click action of a specific user after he/she enters an arbitrary searching query. Explicitly, numerous information will be stored, i.e., the user’s id, entered queries, product’s id, and the time that the action occurs. Because the click behavior demonstrates a one-to-one connection between entered queries and product, it can be seen as a strong assumption that the user has a great interest in the product. 2.1.2 View Records. Similar to click records, viewing data record the same set of features as clicks data. Nevertheless, there is a slight diference in the product’s attributes. In each record, view actions stand for the operation of a user skimming through a shopping page. Compared with click actions which cover only a single product, a view record saves plural items on a single page after searching for arbitrary keywords. 2.1.3 Buy Records. Finally, buy dataset describes each purchase occurrence that happens during the targeting period. As for the data information, each buy record consists of all previous attributes except for the entered query. We list each dataset’s attributes in Table 1.

To obtain a brief knowledge of users’ behavior on Yahoo Taiwan e-commerce platform, we conduct initial analysis on three topics: (i) query usage, (ii) action behavior, and (iii) session statistics. Furthermore, we examine the analysis under monthly-basis. 2.2.1 Query Usage. First of all, we find out that for each month in the dataset, the amount of unique queries ranges from 210K to 260K. Around 70% of queries were found unrepeated in the dataset. Beckmann et al. [ 3 ] also state this phenomenon that users tend to define personal keywords to describe the products. While an extensive data like this is possible for causing oversize dimension problems during the embedding process, we set two terms to tackle the issue, i.e., Count Ratio (CRm,C ) and Space Ratio (SRm,C ) of each month m. We represent the calculation in Eq. (1a) and (1b) and the bar charts in Figure 1(a) and 1(b).

CRm,C = number of qi in month m with ci,m ≥ C

nm SRm,C = Íin=m1 σi ∗ ci,m Íin=m1 ci,m , .

(1a) (1b) where nm is the amount of unique query in m, Qm = {qi | i = 1...nm } is the set of unique queries in m, ci,m is the number of occurance of qi in m, σi = 1 if qi in month m with ci,m ≥ C else σi = 0, and C is a predefined constant ranges from one to five.

Count ratio (CR) of each month

Space ratio (SR) of each month C >= 2

C >= 3 C >= 4 1 2 3 4 5 6 (a)

C >= 5

C >= 2 C >= 3 C >= 4 C >= 5 1 2 3 4 5 6 (b)

In 1(a), each segment in the chart represents the number of diferent query qi occurs more than C times in month m. We can 40.0% 30.0% 20.0% 10.0% 0.0% see that there is a similar distribution across six months. Around 30% of queries appear at least two times, 20% and 13% of queries occur at least three and four times, and lastly, 10% of queries appear ifve or more times. The gfiures are not only credible that most users use an independent query in searching actions but also demonstrate the diversity of e-commerce searching. Moreover, we define Space Ratio in 1(b) to illustrates the query’s coverage ratio. For instance, SRm,5 computes the coverage ratio of queries that appear at least ifve times in month m. We can see that across all months, although CRm,C diminishes when we increase C, the value of corresponding SRm,C retain 80% of the corpus space. The results indicate that users on e-commerce platforms tend to reuse queries during their searching events. Furthermore, it is conceivable and economical to learn the figuration of e-commerce queries data via a compact dataset by only preserving the queries with multiple counts. 2.2.2 Action Behavior. Customers’ action exhibits various clues as well. Figure 2 describes the time diference between the start of action (click/view) with the action to add products into the cart. Specifically, the diference is computed via action’s ts and corresponding buy’s ts (after matching with the same u and p).

Difference between click and buy (in minutes)

Difference between click and buy

(in hours) 80.0% 60.0% 40.0% 20.0% 0.0% 80.0% 60.0% 40.0% 20.0% 0.0% 40.0% 30.0% 20.0% 10.0% 0.0% 40.0% 30.0% 20.0% 10.0% 0.0% 0-10 10-20 20-30 30-40 40-50 50-60 1 2 3 4 5 6

(a) Difference between view and buy (in minutes) <1 1-3 3-6 6-12 12-24 > 24 1 2 3 4 5 6

(b) Difference between view and buy (in hours) 0-10 10-20 20-30 30-40 40-50 50-60 1 2 3 4 5 6 (c) <1 1-3 3-6 6-12 12-24 > 24 1 2 3 4 5 6 (d)

Furthermore, for both click and view actions, we separate the ifgures into minutes and hours sections. Additionally, the first bars within 2(b) and 2(d) include all elements of 2(a) and 2(c). Comparing 2(a) with 2(c), we can see that around 35% of click actions have a 10 minutes gap with the following buy action. On the other hand, nearly 20% of view actions lead to a buy action within 10 minutes. Moreover, a significant diference between 2(b) and 2(d) locates at the last bars, which depicts the portion of users carting the products after twenty-four hours. For view actions, there is a slight excess in carting after a day compare to click actions. These observations suggest two arguments: (i) A click action is faster to induce a buy action than a view action; and (ii) An user who is still browsing the platform might require more time to make the purchase decision. These indications are useful for companies to design diferent strategies for drawing customers’ intent. 2.2.3 Session Statistics. Lastly, we investigate the number of queries used by users having diferent intention in this section. To begin with, we combine our click and view records via session’s id s and mark each action as purchased or non-purchased according to the buying dataset. Details for merging will be presented in Section 2.3. Afterward, we divide all constructed sessions into two parts: purchased sessions and non-purchased sessions. Figure 3 describes the related information.

Length of a session (All session) 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0%

L = 1

L = 2

L = 3 L = 1

L = 2

L = 3

L = 4 L = 5 L = 6~10 L = 11~20 L = 21~30 L = 31~50 others

1 2 3(a) 4 5 6 Length of a session (buy=True)

L = 4 L = 5 L = 6~10 L = 11~20 L = 21~30 L = 31~50 others 1 2 3 4 5 6

(b)

Length of a session (buy=False) L = 1

L = 2

L = 3

L = 4

L = 5 L = 6~10 L = 11~20 L = 21~30 L = 31~50 others 1 2 3(c) 4 5 6

It is obvious to see the diference between 3(b) and 3(c) that about 40% of non-purchased users use only one query during their session. On the contrary, only 15-25% of purchased users use one query through their session. Moreover, for users who purchase the product eventually, around 50% of the customers generate a medium or large sequence of actions (L ≥ 6) within a single session. The appearance of session length demonstrates that customers who might make the deal at the end have more thoughts and reactions to achieve their goal in given operation time.

Like most financial issues, one of the primary problem in ecommerce purchase prediction is the existence of imbalance data [ 4 ]. Typically, most of the platform users have no absolute intention before they start searching for products. It is more probable that they were either entertained by other websites’ commercial or kept pondering over the purchasing decision. As a result, a majority of users leave the website without making a purchase. For example, the size of non-purchased records in our dataset is 22 to 30 times the size of purchased records. Without balancing the training data, preliminary results are likely to be overrated. Therefore, we perform the random over-sampling approach on the training data, that is, randomly multiplying true cases until the number of both cases are similar. As for the testing set, we do not balance the data since it needs to fit the circumstance in the real world. 2.3

To utilize the e-commerce data, we illustrate the procedure to merge separate datasets in this section. We first utilize the session id s to merge the action data into a unified user behavior dataset. In the Web industry, a session defines the login event after a user connects to the hosting server. Specifically, we record all of the activities during a given session to illustrate the user’s behavior, which is beneficial for learning the development of their intentions dynamically. According to our dataset, Yahoo Taiwan specifies its session as a fixed period after any user creates a connection to their server. If an individual user browses the pages more than the given period, they will create multiple sessions according to the actual spending time.

The reason why we construct the data on a session-level is for the rationale behind searching conventions. In the real world, a user’s searching habits and related actions are usually regarded as time-sensitive, closely connected with prior and latter instances. This phenomenon manifests in two aspects: (i) experience-shifting and (ii) environment-shifting. As for the first concept, it is convinced that human often generates interests as the time shift. For example, when a customer is entering a shop, he or she might initially search for the thing using a general description, such as shoes, bags, or computer. Typically, people take these actions as wandering and waiting; they either describe fewer details about the item needed in the first place or did not realize the subject they are searching. Furthermore, they might want to compare the same kind of products manufactured by a diferent company. Such actions are likely to last for a while until a suitable commodity is found. Usually, customers tend to check their favorite brand or look for specific functionality and appearance. In addition, some buyers will calculate their afordable budget. For example, a user who start the search of bags might end up checking Saint Laurent leather shoulder bag or backpack 13’ notebook waterproof. For consumers already have a preference, they are likely to finish the previous process in lesser time. Also, the behavior exists not only in of-line business but also the online platform. With the advantage of technology, customers have more resources and information from the Internet, which makes the experience cycle more transient. Moreover, this process often accompanies with several clicking actions which help the user for getting more product details. As for the second concept, we will describe the rationale in Section 3.3.

To construct a session-level behavior sequence, we perform several preprocessing steps on the raw data. At first, we need to define two groups of features from our dataset: the first set describes user’s actions after establishing a connection. The other group of features indicates the textual contents of users’ queries, which requires Natural Language Preprocessing (NLP) techniques to constructs the latent representation. 2.3.1 User Behavior Sequence (UBS). To illustrate history movements after each user logs onto the server, we transform the sessionbased data into the User Behavior Sequence (UBS) to represent each session’s properties. During the merging process, we first define the click and view records as primary actions and the buy records as supportive information. Then, we combine primary actions having the same session id s and sort all of the records by primary actions’ start time ts . As for the supportive information, we employ the time information to check whether each action leads to a purchase event. If any buy record’s start time ts is less than any primary action’s start time, and they share the same user id u and product id p, we mark the primary activities afterward as purchased. Moreover, we ignore repeated primary actions, which defined as two consecutive actions having a time gap of fewer than 5 seconds. In general, this phenomenon might be a consequence of the network error or double-clicking rather than actual intent. For each logged user, we also sort his/her purchasing records by start time before the merging process. 2.3.2 Context-aware Latent Representation. After constructing the structure of UBS, we turn our attention to the query information in the dataset. To utilize the semantic features of each query, we perform several NLP techniques to construct queries’ latent representation and integrate the features with UBS.

First, we would like to construct the query corpus from all sequences in UBS. For each sequence, we concatenate each primary action’s entered queries into a sequence and tokenize the sequence using white space. Besides, there is no stop words replacement or term normalization (stemming and lemmatization) during the preprocessing since the existent of redundant inputs are typical behavior in e-commerce. After these operations, we build up a corpus consists of over 165M searching sequences in a more structured status. Nevertheless, the corpus space is too large to utilize which requires further processing. To generate the embedded representation, we trained through word2vec [ 13 ] to project the input corpus into a lower dimension to recognize the latency representation. Based on the conclusion in Section 2.2.1, we only take the word which occurs at least five times in the corpus into consideration. Furthermore, we empirically set the target dimension as 100, which is significantly smaller than the original vector spaces.

After generating the representation of current corpus, we extend the constituents of UBS and compute several context-related attributes. For each occurence of searching, we assign two queryrelated features (i) query length li1 and (ii) query amount li2. As for the query contents, we concatenate the embedded representation of linked qi via two approaches. The first approach is to append the average of complete semantic embedding while the second approach considers the representation thoroughly, denoted as wavд and wf ull respectively. Since we like to investigate whether each action of the user gradually afects their final decision, we further define a parameter pos to state the query’s position inside the linked session. For a latter position, it is plausible that the user has a more precise thought on the product he/she wants and a more thoroughgoing intention in making the orders or not. The definitions of advanced terms and context-related attributes are demonstrated in Table 2. 2.3.3 The Input Layers Setings. To understand the eforts of each feature, we construct the input layer under the following context: For the baseline in comparison, we only consider UBS’s standard features, i.e., time gap td and the action type βi . As for remaining settings, we concatenate the baseline feature vector with the following combination of advanced terms.

(1) UBS: each record consists of n of actions and the time the user spent (td + βi ); (2) UBS+: each record is the concatenation of UBS, li1, and li2; (3) PSACa : each record is the concatenation of UBS+, wavд , ws , and wv ; (4) PSACf : each record is the concatenation of UBS+, wf ull , ws , and wv ; (5) PSAC+f: each record consists of PSACf and pos.

Since the main purpose of this paper is to probe the efectiveness of query-based dataset in perceiving user’s purchase intention, we ignore context-related features in the baseline method. For the second and third combination, we consider several query-related features, including query length and amount, latency representation, similarity, and variance. Moreover, we apply diferent computation approach to incorporate the semantic values. Finally, the position attribute is added in the vector to examine the process that the user reshape their thoughts during the connection.

In this section, we propose the Prediction framework that analyzes User’s Sequential Actions via Context (PSAC). Firstly, we define several situations to apprehend human’s behavior and the concept of time-shifting mentioned in previous sections, including (i) experience-shifting and (ii) environment-shifting. Specifically, we will consider the evolution of queries reshaping and a data-process that takes contemporary fashion into account. For the experiments, we apply two deep learning models: (i) Deep Neural Network (DNN) and (ii) Long Short-Term Memory Networks (LSTM), where we observed opposite views between two frameworks.

Concatenation Sequential Segmentation (UBS)

UBS & QLR

Input and Embedding Layer (N_GRAM and SUB_GRAM) DL Layer

In short, we illustrate the framework of PSAC in Figure 4. We ifrst combine the view and click instances with session id s. The preliminary output will be used to construct the UBS. Next, to deal with environment-shifting, we apply sequential segmentation and divide the data into training and testing sets to simulate the actual condition. For each segment, we extract two parts of data, namely, query corpus and action sequences. As for the query, we apply context embedding techniques to project the corpus into lower dimensional representation. As for action, we label each action as purchased or non-purchased based on the buy dataset’s start time t1, user id u, and product id p. After merging each action in UBS with its corresponding textual feature, we compute several advanced terms, e.q., similarity, variance, and position attributes. Finally, we deal with the imbalance in e-commerce data and establish diferent scenarios to investigate the eefct of query-based features. In the experiment, we implement the proposed system under two deep learning frameworks (DNN and LSTM) for purchase prediction.

Recall that in the real world, customers shape their idea through time and experience when searching on the internet. With the session-based data structure, we use the following settings to consider this property. After each construction of the input layer, we thus define two parameters: N_GRAM ( M) and SUB_GRAM (µ ) where µ = 1...M. With a given M, we first gather UBS with a length of n where n ≥ M. Then, each UBS will be separated into n − M + 1 of subsequences, and we denote the set of subsequences as UBSn,M . Also, each subsequence owns n actions. Furthermore, we regard each instance in UBSn,M as an independent action series. Meanwhile, the final position in each series is the prediction target for purchase intention. We illustrate the construction of UBS5,4 in Figure 5; we can see that a UBS with five actions separates into two subsequences if M is four (action1−4 and action2−5).

Secondly, both subsequences generate four action series and each series consist of µ = 1, 2, 3, 4 continuous actions. Such a design is capable for us to understand whether more actions benefit the proposed system in learning user’s intention. Eventually, the number of subsequences built from our data is represented in Table 3.

T iran iraT rT n ia n T

We then introduce the idea of environment-shifting, that is, the latest trend or craze toward certain products in the market. Practically, people are likely to put focuses on the hottest products or the latest fashion in the market. Furthermore, companies often enlarge the current by releasing commercials or discounts whenever they want to attract potential consumers. To simulate the real condition, we utilize the rolling-based approach to train the model separately. Explicitly, we segment the data into several pairs of subsets for the simulation: the first subset will cover x days of data to build UBS as the training set, and the other defines consecutive y days of data to construct UBS as the testing set. We execute the training process separately until all segments were done and aggregate the results to get the whole pictures. We demonstrate the process in Figure 6.

Moreover, we balance the training set to avoid overfitting and remain the imbalance in the testing set to fit reality. In sum, the advantages of sequential segmentation are 1) it is more reasonable for companies to use prior data as the training set of following End data because we can not foresee the behavior of future users in reality, and 2) it is more manageable since each training utilize a smaller dataset compared with the original one. In this paper, we empirically set the period of x , y as 28 and 14. Eventually, we divide our dataset into twelve pairs of subsets, and each pair has a month of data as the training set and the afterward half month of data as the testing set. In this paper, we want to explore whether the usage of queries help us to distinguish potential buyers from other users. To achieve the goal, we adopted two types of deep learning structure: Deep Neural Network (DNN) and Long Short-Term Memory Networks (LSTM) to predict a user’s purchasing intention. 3.4.1 Deep Neural Network, DNN. DNN is the most common model in the deep learning field. With the help of tuning parameters through gradient descent and a diferent activation function, it is possible to build up the optimal objective functions for targeting problems. 3.4.2 Long Short-Term Memory Networks, LSTM. LSTM is one of the extensions from vanilla RNN proposed by Hochreiter et al. [ 5 ] to deal with vanishing and exploding gradient problems. With the capacity of learning long-term dependencies, LSTM is widely used and demonstrates decent performance on many recent studies. To solve the problem of vanishing and exploding gradient, LSTM replaces the kernel of RNN structure with the idea of cell state, a representation of the state of input sequence transmitted information through the entire network. Moreover, LSTM demonstrates the capacity to add or remove information from the cell state by a mechanism called gates. 3.4.3 Hyper-parameter Seting and Implementation Details. To make sure that two models train under a similar condition, we construct both models with the structure having an approximate number of untrained nodes. For DNN, we build up a five layers network with one input layer of 1024 nodes, three hidden layers (1024, 256, and 256 nodes particularly), and one output layer of 2 nodes while our task is a binary classification. On the other hand, we construct the LSTM model as a four layers structure (one input layer of 256 nodes, two hidden layers of 256 nodes, and one output layer of 2 nodes). Moreover, layers in both models are fully connected, and the input layer’s shape will match the size of the embedding layer given in Section 2.3.3. As for the naive setting (under M = 1 and use UBSbase ), both models deploy roughly 1.3M untrained weights during the training process.

We evaluate the performance of PSAC with simulations utilizing the Yahoo! Taiwan dataset. For each simulation, we select the training and testing data sharing the same segment and extract the set of attributes under a given scenario. After constructing the input layer of each segment, we process the modeling on training data with 30% of instances as the validation set. Once finishing the training, we evaluate the model’s ability to predict testing data in the following sections. 4.1

According to [ 7 ], as the increase in data skewness, ROC would be a misleading indicator. Since the imbalance in our data distribution is highly significant (22x-30x), we employ the F1 score as the performance evaluation principle.

Pre ∗ Rec F1 = 2 ∗ Pre + Rec , (2) where Pre stands for the precision and Rec is the recall. 4.2

In the experiment, we analyze the efect of query contents with diferent environmental settings. Table 5 and 6 describe the numerical results of PSAC for purchase prediction under diferent scenarios and deep learning framework. Explicitly, the simulation results are computed by taking the average of all segments’ outcome. Moreover, the average improvement illustrates the enhancement between current input settings with baseline approach (UBS). Based on the results, we made the following observations.

First of all, we discover a slight increase when we add basic queryrelated features (query length and query number, UBS+) to the baseline approach. As for the PSAC which utilizes context-related features (PSACa and PSACf ), the performance of both methods outperform the baseline outputs; however, their results are entirely diferent. If we take averaged values as the input features, the improvement compared with the first approach is under 10%. On the other hand, if we extend the input layers with the complete queries information (PSACf ), the performance consistently surpasses the baseline outcomes around 20% to 30%. We conclude that a vast amount of information is missing during the regularization procedure. Furthermore, we can see that query information plays a vital role in perceiving the user’s intention. Next, we observe that the results demonstrate that the position attribute is another useful factor. We recognize one to two percent increase after considering the position attribute on most of the session sets with diferent N_GRAM M.

If we consider only the baseline approach and the enhanced version (UBS, UBS+), the prediction performance declines as the length of the input sequence (µ ) grow. After we take query-related features into account, the influences of µ on these two deep learning frameworks are diferent. As for DNN model, once we construct the input layer with query-related features (PSACa , PSACf , and PSAC+f), the model’s performance gradually raises as µ increases. Since DNN recognizes each action in the input sequence equally during the training process, it is possible for us to comprehend the evolution of the user’s query contents globally and to capture their intention more appropriately in an attention-like structure. Additionally, this phenomenon occurs in all M settings. Contrary to DNN, the score of LSTM decreases when applying query-related features if we utilize more µ in training. While the rationale behind RNN-based models pushes itself to put more focuses on closer steps, it is plausible that a more extended sequence might impair the performance. For example, a user might determine his/her shopping list in the middle of the searching operations and continues to browse other commodities afterward. If the user did not pass the thought onto the next action, it is hard for LSTM to perceive the conversion. Finally, we provide the comparison of DNN and LSTM with the PSAC+ in Table 7. We can see that LSTM outperforms f DNN for shorter action sequences. However, as the users begin to reshape their thoughts and to start a new searching action, DNN is more capable in realizing the connection between each action and query. 5

With the growing usage of e-commerce platforms, the analysis of user’s purchase behavior has attracted attention in the field. Through an integrated prediction framework, the numerical results in our study indicate that query-related features are essential in making purchase prediction on e-commerce platforms. To utilize query-related features correctly, we need to consider several query’s attributes, such as user’s behavior in reshaping their ideas, the relationship between continuous queries, and the approach to represent query’s embedded components. These results also support that both deep learning frameworks have their advantages; however, DNN’s performance is more robust in capturing customer’s purchasing intention as the searching sequence grow. Furthermore, the construction of session-based data structure is useful in storing customers’ behavior sequences, and we provide several mechanisms in explaining the phenomenon of time-shifting in the real world. Such evidence should be of importance in e-commerce purchase prediction.

This work was supported by the Ministry of Science and Technology under Grant MOST 105-2221-E-001-003-MY3 and the Academia Sinica under Grand Challenge Seed Project AS-GC-108-01.