Recent advances in the fields of Information Retrieval and Machine Learning have focused on improving the performance of search engines to enhance the user experience, especially in the world of online shopping. The focus has thus been on leveraging cutting-edge learning techniques and relying on large enriched datasets. This paper introduces the Shopping Queries Image Dataset (SQID), an extension of the Amazon Shopping Queries Dataset enriched with image information associated with 190,000 products. By integrating visual information, SQID facilitates research around multimodal learning techniques that can take into account both textual and visual information for improving product search and ranking. We also provide experimental results leveraging SQID and pretrained models, showing the value of using multimodal data for search and ranking. SQID is available at https://github.com/Crossing-Minds/ shopping-queries-image-dataset.
CEUR ceur-ws.org
In the age of online shopping, eCommerce platforms must help customers find what they are looking for with the least amount of efort. Product search allows users to enter a search query, and get back a list of results matching that query. An efective product search should be able to understand exactly what a user is looking for, and retrieve the most relevant results from a catalog of available items. To efectively fulfill a user’s shopping needs, a search engine must draw on all the information it has available, including textual, visual, and contextual metadata associated with the user, the search query, and the products in the catalog.
In particular, visual information can be very useful to identify characteristics of products that may not be well represented in textual metadata. To illustrate this point, consider the product listing for a men’s dress shirt1, which includes textual metadata such as: • Description: “This Stylish Men’s Collared Dress Shirt Comes in a Modern Fit Which is Slightly More of a Tailored Fit Than a Regular Fit. It Also Features Slim Fit, Vertical Striped Printed Pattern, Buttoned Up Closure, Turn Down Collar, Single Breasted Buttons, Convertible Double French Cuf, Round Curved Shirttail Hem” • Size Options: Small, Medium, Large, Extra Large • Color Options: Black, Blue, Navy Blue White, Khaki, White Stripe Black, White/Purple
Stripe, Grey Plaid
If a user is looking for a “men’s dress shirt with thin vertical stripes”, they might expect that this product is a relevant match based solely on the textual metadata. However, when looking at the product images, they would quickly notice that the stripe pattern on the shirt is not “thin” but rather “thick” stripes. Not only that, but many of the diferent color options in fact have a completely diferent design and thickness of stripes, while some color options have a checkered pattern instead of stripes (Figure 1).
(a) A striped shirt
(b) Same shirt, color “Khaki” (c) Same shirt, color “Gray Plaid”
None of these options would be a great match for the “thin vertical stripes” the user is looking for. However, search engines that rely only on textual metadata are likely to return these shirts as relevant results. If, on the other hand, the search engine leveraged multimodal information such as the product images, it might not have made that mistake.
In order to support research on improving product search by leveraging image
information, we are releasing the Shopping Queries Image Dataset (SQID ) - an augmented
version of the Amazon Shopping Queries Dataset (SQD)2 [
The paper is structured as follows. Section 2 presents related work around SQD and pretrained models used to embed multimodal data. Section 3 provides the details of the data covered in SQID, as well as the methodology followed for data collection. Section 4 presents the experimental setting used in this paper to highlight the benefit of using multimodal data for ranking, followed by the experimental results provided in Section 5. 2https://github.com/amazon-science/esci-data
This section covers work related to SQD based on which SQID is built, on one hand, and multimodal learning techniques allowing to leverage image and text data for representing items and products, on the other hand.
In 2022, Amazon released the Shopping Queries Dataset (SQD) [
In the context of this challenge, a variety of techniques were explored to improve the score
for each of the tasks, including self-distillation, data augmentation, and adversarial training,
among others [
SQD has also been used to support other use cases. For instance, Tang et al. [
On another note, the TREC Product Search Track of 2023 [
Multimodal pretrained models emerged as a powerful paradigm allowing to learn joint
representations capturing the relationships between diferent types of data such as images, text, and
audio. In particular, Contrastive Language-Image Pre-training (CLIP) [
Several extensions of multimodal models were made to specifically address the item retrieval
and ranking problems in the e-commerce domain, among others, and to take into account
characteristics of user behavior. One such approach is the CLIP-ITA model [
In this paper, we rely on CLIP to embed queries and products based on text and image data. While fine-tuning pretrained models on a dataset specific to the task is very beneficial to improve the performance, we consider it outside of the scope of this paper and only use pretrained models in our experiments (more details in Section 5).
The Shopping Queries Image Dataset (SQID) builds upon SQD by including image information and visual embeddings for each product, as well as text embeddings for the associated queries which can be used for baseline product ranking benchmarking. The image information can be used to enhance or improve the accuracy of product search algorithms by allowing them to leverage multimodal machine learning techniques.
The image information included in this dataset includes:
1. Image URL
2. Image Embeddings extracted using a CLIP model [
The reduced set consists of 1,118,011 <query, rating> judgements, out of which 601,354 are from locale “us”. These judgments contain references to 482,105 unique products (with a unique product_id).
We then mainly focus on the products found in the test set of SQD’s Task 1 (i.e., having “split” = “test”). The total number of products appearing there is 181,701, out of which 164,900 are unique. While the rest of the paper focuses on this set of products, SQID also includes supplementary data, covering additional products appearing in at least 2 query judgements in the “us” locale subset of Task 1. There are 27,139 unique products meeting this criteria and that are not in the test split.
Overall, therefore, SQID covers 164,900 products, with a supplementary part covering an additional set of 27,139 products.
Image URLs. We scraped the Amazon website to retrieve the URL to the main product image displayed on the product page of 164,900 products, resulting in 156,545 product_id’s having an image URL (95%). We focused on the following domains, attempting to retrieve product pages from each of these successively: .com, .ca, .com.au, .cn, .fr, .de, and .co.jp. There are two main cases for when a product does not have an image URL: • The product_id failed to return a valid product page, usually when the product is no longer ofered on Amazon, or • There was no image associated with the product - to be precise, the main image of the product is a blank image that says “No image available”.
There are 442 products where the image URL contains this particular URL: https://m.mediaamazon.com/images/G/01/digital/video/web/Default_Background_Art_LTR._SX1080_FMjp_.jpg. These are “generic” product images for digital video products where there is no product-specific image.
Textual and visual embeddings. In addition to product image URLs, SQID also includes visual and textual embeddings of products. These were obtained using CLIP pretrained model, specifically clip-vit-large-patch14 7, and based on product image URLs and product titles. To address the product ranking task, we also include query embeddings obtained based on the query text. 6https://huggingface.co/openai/clip-vit-large-patch14 7https://huggingface.co/openai/clip-vit-large-patch14
In order to illustrate the value of using multimodal data for product ranking, we leverage SQID
for the Task 1 of the KDD Cup 2022 consisting of query-product ranking [
We evaluate the performance of several ranking approaches for the Task 1 (“small_version” = 1) on the test set (“split” = “test”) and for the US locale (“product_locale” = “us”). The evaluation dataset consists of 181,701 judgements, 8,956 queries, and 164,900 products. The average number of judgements per query is around 20.
We only rely on pretrained models and consider that fine-tuning models on the ESCI training
data as well as other more advanced techniques used by winning solutions of the challenge
(e.g., [
Following the setting of the challenge, the ranking quality is measured using the Normalized
Discounted Cumulative Gain (NDCG) [
We first include in our evaluation two baselines for reference and to allow comparing with the main ranking approaches considered.
Random baseline. The random baseline is included to provide a lower-bound of NDCG for
the ranking task considered, and consists of randomly ranking products for each query.
ESCI_baseline. The _ is the standard baseline introduced in the initial ESCI
benchmark [
CLIP_image. We use CLIP [
We also consider ranking approaches that combine both product text and images. This is done by either combining query-product similarities or directly combining ranking lists, using a weighted average. These approaches are designated by the notation 1_ _ 2, where 1 and 2 are the two approaches combined, and is the method used to combine the results from 1 and 2 ( when combining rankings and when combining scores). A weight parameter is used to counterbalance the impact of text versus images. In terms of notation, is associated with 1 and (1 − ) with 2.
Using the Terrier IR platform to compute NDCG for the _ leads to an NDCG of
0.83, as reported in the ESCI benchmark [
Figure 2 shows the results for approaches combining both text and image. The performances of _ , _ , and _ are visualized as dashed horizontal lines on the plot, for reference. The points at = 0.0 correspond to the performance of _ (with a weight of 0.0 for the text-based approach), and the points at = 1.0 correspond to the performance of the text-based approach (with a weight (1 − ) of 0.0 for the image-based approach). By varying the value of , the weight of 1, the results show that combining image 14https://github.com/amazon-science/esci-data/blob/main/ranking/prepare_trec_eval_files.py and text outperforms the approach using only text data.
More specifically, _ _ _ _ results in a 2.41% improvement compared to the text-only approach (i.e., _ ), _ _ _ _ results in a 2.1% improvement, _ _ _ _ results in a 0.82% improvement, and _ _ _ _ results in a 0.22% improvement.
This paper presents the Shopping Queries Image Dataset (SQID), building upon the Amazon Shopping Dataset and enriching it with image information for products. We present the dataset and its characteristics, and provide experimental results showing the value of incorporating image data for the task of product search. We hope that this data will support further research around product search and ranking using multimodal data.
SQID can be leveraged in the context of the ESCI benchmark, by evaluating the performance of models using images on the ESCI test set. The data can also be used to fine-tune pretrained models, outside of the ESCI benchmark. In addition, and as mentioned throughout the paper, the availability of text together with images allows investigating diferent techniques around multimodal learning relevant to the eCommerce space.
This dataset would not have been possible without the Shopping Queries Dataset by Amazon.