Efective question-intent understanding plays an important role in enhancing the performance of Question-Answering (QA) and Search systems. Previous research in open-domain QA has highlighted the value of intent taxonomies in comprehending data and facilitating answer generation and evaluation. However, existing taxonomies have limitations for specific domains. We're interested in question intent for e-commerce scenarios where questions are specific to shopping activities.
CEUR ceur-ws.org
Question answering (QA) as a longstanding task in NLP has been pushed forward rapidly in
recent years with the development of language models. Transformer-based models perform well
in most factoid QA datasets, however, they tend to still perform poorly compared to humans
in datasets containing more complex problems. In closed-domain QA, such as AmazonQA [
In practice, Broder [
We propose EQA (E-commerce Question Answering) taxonomy, a tailored approach that
advocates for the creation and adoption of a bespoke taxonomy dedicated to e-commerce
questions. Specifically, recognizing the limitations of the existing NFQA taxonomy in accurately
reflecting the intent of e-commerce queries, we eliminate categories that show low
interrater agreement rates in the e-commerce context and introduce new categories that are more
contextually appropriate. EQA is designed to encapsulate the unique characteristics of
ecommerce data and user queries within this domain. Our taxonomy demonstrates that it can
represent users’ real information needs in the context of shopping scenarios. To operationalize
the EQA taxonomy, we leverage instruction fine-tuning [
Our contribution can be summarized as follows: • We propose a question intent taxonomy for e-commerce questions that can be used in diferent shopping scenarios. Our quantitative and qualitative analyses confirm the reliability of this taxonomy in e-commerce problems. • We describe how to build classifiers when introducing a new taxonomy. While EQA is based on e-commerce, we believe this methodology can be generalized to other domains.
NFQA is a comprehensive question intent taxonomy for open-domain question-answering tasks. To explore the suitability of NFQA trained from open-domain questions for e-commerce questions, two human annotators followed the NFQA taxonomy to label a set of in-domain questions. Meanwhile, using the pre-trained classifiers of NFQA, we obtained NFQA predictions from a deep learning model. The results show that human-to-human agreement stands at a mere 49.13%. These disagreements occur mainly in experience and evidence-based categories. Moreover, we note that some classes do not faithfully respond to the information needs of the question, e.g., debate. More quantitative analyses are covered in Section 4.
The EQA taxonomy is presented in Table 1. In this section, we describe how EQA fulfills users’ information needs in close domains and how to evaluate the taxonomy. Opinion Description Comparison Recommendation Factoid
Description Example The customer wants instructions, guidelines, Where is the doorknob? Once the code or procedures to achieve something with re- is entered, how do you open the door? spect to a product or service. How can I tell if this will work on
my TV or BluRay player? The customer wants a subjective piece of infor- Is this product worth buying or will mation about a product, service, or shopping I end up sending it back? category. Any defect complaints? The customer wants a definition, description, What are the sizes and types of explanation, or summary of a product, service, blades that come with the 5-blade or shopping category. package?
What are the measurements of this product? The customer wants a comparison of two or What's the difference between exclusive more products or services. Castiel and regular?
How does this knife compare with a
Kershaw? The customer wants recommendations for a Looking for a bag for golf for drinks product or service. and snacks. Would this be a good choice? I need a case for a .357 with a sixinch barrel. Suggestions? The customer wants an objective piece of infor- Is the price and shipping for one bar mation about a product, service, or shopping or a set of two? Does it fit on Honda category. CRV 2014?
The concept of information need refers to the foundational motivation driving users to
engage with search systems [
Building upon the existing NFQA taxonomy, we first focus on the fact that the debate category does not constitute a valid intent in e-commerce. When individuals take part in shopping, they are not likely to anticipate engaging in formal debates with others. Even inquiries that could potentially spark heated discussions on forums or in other contexts, such as “What's the best graphics card for non-gamers?”, where the user is looking for advice with a purchase. This observation consequently highlights the significance of recommendation, which is a prevalent intent within the e-commerce domain. Furthermore, opinions, representing subjective insights from other consumers, prove instrumental in guiding purchasing decisions. This intent commonly manifests in the form of queries seeking feedback on product usage or opinions about comparable items. Correspondingly, inquiries for objective information are encompassed by the description category. Answers to these queries can often be readily found on the product page, such as technical details provided by the seller. Alongside these, instruction and comparison persist as two enduring question types that are pertinent in e-commerce.
To quantitatively assess the eficacy of the question intent taxonomy, we adopt two distinct matrices that serve as indicators of its performance across specific datasets.
Distribution of categories We analyze the distribution of each intention as a percentage within the dataset. Although closely tied to dataset characteristics like data source, the distribution ofers valuable insights. Extremely unbalanced distributions often imply that current taxonomies struggle to establish efective boundaries for splitting the questions in the given dataset.
Human-to-human agreement Within a given dataset, a well-defined taxonomy should facilitate consistent labeling by diverse human annotators. Agreement implies that the categories in the taxonomy clearly delineate the problem intent. Minimize blurry classification regions as well as controversial labels.
We chose the encoder-decoder model T5x [
Motivated by the success of few-shot learning in NLP [
Factoid
Instruction Not-a-question
Experience Evidence-based
Comparison
Reason Variance
To better align with the downstream task, i.e., intent classification, we adopt an instruction
ifne-tuning paradigm [
For prediction trustworthiness, in addition to the intent label, we record the model transition probability at the generated token to approximate the confidence score. Particularly, the score , is determined by its conditional probability given all preceding tokens (the given question), < . The overall score is computed by Equation 1, where is the logit corresponding to , and the denominator is the sum of exponential logits for all tokens in the vocabulary, ensuring normalization.
( | < ) =
exp( ) ∑ exp( ) (1)
This formulation quantifies the model’s certainty or confusion in selecting given the preceding context, with lower scores indicating the model’s uncertainty about the prediction. In practical applications, setting a threshold could advise users against placing too much trust in uncertain predictions, thereby enhancing the reliability of classification results. In this work, the threshold is manually set at 0.6. We envision that future research could develop an adaptive, learnable threshold by training a simple neural network, such as a Multilayer Perceptron (MLP), to improve the discernment of prediction reliability.
In line with the prompt design from recent instruction fine-tuning works, our training utilizes an instruction prefix combined with dataset questions as input and human-annotated intent categories as expected labels. The fine-tuning is delivered by the cross-entropy loss and Adam optimizer through 20 training epochs. The entire fine-tuning process was completed in under Category Factoid
Debate Evidence-based
Instruction Experience Comparison
Reason Three-way Agreement
Distribution
17.43
0.57
40.29
4.29
12.86
18.57
6.00
49.13
two hours on a single NVIDIA A-100 GPU, following the hyperparameter settings recommended
by T5x [
Built on the top of a product review-based e-commerce dataset, Amazon review data [
Table 2 presents the intent distribution of questions in the AmazonQA dataset using EQA and NFQA. Our analysis aimed to establish which taxonomy better represents the nature of queries in an e-commerce context. The EQA taxonomy revealed a predominant focus on factoid questions, constituting 51.98% of the dataset. This result aligns well with the nature of e-commerce inquiries, where customers often seek specific, factual information about products. The next significant category in the EQA taxonomy was opinion (16.97%) and description (15.07%), reflecting the customer’s interest in reviews and detailed product descriptions. In contrast, the NFQA taxonomy’s most prominent category was debate, accounting for 55.70%. However, the concept of debate is less relevant in an e-commerce setting, as customers typically seek concrete information rather than engage in discussions of a contentious nature. The factoid category in NFQA, while still significant, was markedly lower at 16.03%, suggesting a less precise alignment with the nature of e-commerce queries. Of the remaining categories with relatively low occupancy rates, Not-a-question accounts for 7.56% of the NFQA and is higher than any other category. This finding points to many cases where the NFQA classifier fails, which did not Question I need to replace a defective Julie from brake on a Cannondale Scalpel MTB. Is this a good replacement and will it bolt right on? Compare best 3G/4G internet access plan? 10′ is L, H or W? What is the protein in this? Wish they would call that out on the site. Trying to decide between this and Kay's naturals which is protein packed! For a more permanent solution, do you super glue the pads along with the sticky adhesive onto the glass or without the sticky adhesive? EQA Recommendation (99%) Comparison (99%) Factoid (99%) Description (99%) Opinion (88%)
Debate Not-Question Not-Question Instruction occur with EQA. In both taxonomies, instruction and comparison share similar proportions. NFQA presents experience and evidence-based with high rates. While these categories are relevant in broader information contexts, their specific applicability in e-commerce is less direct compared to the recommendation in EQA, which reflects a customer’s desire for guidance to make informed purchasing decisions.
Further statistical analysis revealed that the variance for the intent distribution in EQA taxonomy was approximately 275.76, while for NFQA, it was slightly higher at 284.78. This higher variance in the NFQA taxonomy indicates a broader spread in the distribution of question types, which may imply less consistency in categorization relevance for e-commerce data. Our experiment result from AmazonQA emphasizes the adaptability of the EQA taxonomy in capturing the intent of e-commerce customers, providing a more relevant and practical categorization framework for analyzing customer queries on e-commerce scenarios.
We performed human annotation of a random sample of 350 data points, where half were from AmazonQA, and another half were from our internal unpublished real e-commerce data. To ensure reliability and reduce the subjectivity inherent in manual labeling, we recruited three independent annotators and adopted a two-stage majority voting process for deriving the final label. In the initial stage, data points where at least two annotators agreed on the label were directly accepted, and these consensus labels were deemed final for those specific data instances. Next, to address the cases with complete annotator disagreement, we calculated the individual accept rate of each annotator, defined as the proportion of their labels being accepted in the ifrst stage. For data points with divergent annotations, the label proposed by the annotator with the highest accept rate was chosen as the final label. We analyzed the distribution of each intent category and calculated the rate of three-way agreement, which is the proportion of three labelers providing the same label.
As reported in Table 3, the human evaluation results reveal a significant diference in the faithfulness of the EQA and NFQA taxonomies in e-commerce contexts. While the EQA taxonomy achieved a substantial three-way agreement rate of 76.88%, NFQA’s agreement rate was notably lower at 49.13%. This diference highlights a key challenge with NFQA in e-commerce: its categories are less tailored to the specific types of queries that arise in this domain. For instance, NFQA’s broader categories, like Evidence-based and Reason, may lead to varied interpretations among annotators when applied to the more focused needs of e-commerce customers. The disagreement in NFQA suggests that its categories, possibly well-suited for open-domain questions, are less intuitive and coherent for e-commerce queries, leading to more subjective and inconsistent categorization. In contrast, EQA, with its higher agreement rate, demonstrates a clear alignment with the distinct, often more pragmatic and product-focused nature of e-commerce questions.
This section outlines three patterns in which the NFQA and EQA yield divergent outcomes.
As mentioned in Section 4.3.1, one notable issue with NFQA is its tendency to classify a large
number of questions as Debate, which is a less reasonable intent in the online shopping context.
For example, as illustrated in the first two examples in Table 4, questions that may contain a
debating intent in daily discussion (e.g., debates over “the best”) [
We introduce the EQA taxonomy, tailored specifically for e-commerce queries. Our research highlighted the limitations of generic taxonomies like NFQA in the e-commerce context and demonstrated the need for a domain-specific solution. The development and validation of EQA, coupled with an intent classifier trained using instruction fine-tuning, shows a lot of promise for question intent understanding in e-commerce. This approach ofers a more accurate framework for question categorization in e-commerce and sets a precedent for developing domain-specific taxonomies in other specialized areas.
EQA has been practiced reliably on e-commerce data; however, the efectiveness of its intent label for downstream tasks is still unproven. Moving forward, we anticipate the integration of EQA classifiers into operational pipelines, enabling systematic evaluation of their eficacy in supporting downstream tasks. Furthermore, the prospect of extending the methodologies employed in this study to other domains, such as healthcare, by adapting domain-specific intent taxonomies for classifier training points in an exciting direction for future research.