Intermediate Fusion for Multimodal Product Matching

Abstract

Web-based entity resolution, particularly in the context of online marketplaces and e-commerce ecosystems, is a critical task for accurately identifying and matching similar product ofers across the web. Traditional approaches to entity resolution have primarily relied on textual information, but the increasing availability of diverse data modalities has led to the adoption of a multimodal approach. This paper introduces an innovative intermediate fusion architecture for multimodal product matching, efectively combining textual information from RoBERTa embeddings and visual information from Swin-Transformer embeddings. Our approach enhances matching accuracy by leveraging the complementary nature of text and image modalities. Experimental results on the WDC Shoes and Zalando datasets show the superiority of our proposed approach compared to unimodal models and multimodal baselines. The outcomes highlight the potential for multimodal product matching to improve entity resolution in online marketplaces, thereby enhancing the user shopping experience.

Keywords

• eol>entity resolution
• product matching
• multimodal data
• intermediate fusion

1. Introduction

ships. Intermediate fusion strikes a balance by merging modality-specific features at an intermediate stage, Web-based entity resolution, particularly in product allowing for a nuanced integration of complementary matching, serves as a cornerstone for online market- information while retaining modality-specific details. places, aiding in price comparison, reducing duplicate This paper focuses on the concept of intermediate fulistings, and ensuring a seamless shopping experience. sion, where textual and visual features are combined at However, traditional approaches relying solely on tex- an intermediate representation level. We make the foltual data encounter limitations due to ambiguous or in- lowing contributions: complete descriptions and the vast expanse of online marketplaces. In response, the adoption of a multimodal • We introduce a publicly available, high quality, approach, integrating both textual and visual data, has and challenging benchmark dataset for multigained traction to enhance matching accuracy. modal product matching.

Multimodal product matching presents a unique oppor- • We propose an intermediate fusion architecture tunity to leverage the complementary nature of textual combining textual information from RoBERTa [ 1 ] descriptions and visual images. While textual data pro- and visual information from Swin-Transformer vides context and details about products, visual informa- embeddings [ 2 ]. tion captures nuances and attributes that are challenging • We comprehensively evaluate our intermediate to convey through text alone. This fusion of modalities fusion approach against a range of unimodal and holds promise for achieving more precise and reliable multimodal approaches. matches.

Fusion strategies play a pivotal role in multimodal product matching, determining how information from diferent modalities is integrated. Early fusion combines raw data from text and images at the onset, potentially losing modality-specific details. Late fusion defers integration until each modality has been independently processed, possibly missing subtle inter-modal relation

The remainder of this paper is structured as follows.

In Section 2 we discuss related work. We introduce the benchmark datasets that we utilize to evaluate the efectiveness of our proposed intermediate fusion architecture in Section 3. Section 4 introduces our intermediate fusion architecture. Subsequently, we present our evaluation results in Section 5. Lastly, Section 6 summarizes our findings and outlines po35th GI-Workshop on Foundations of Databases (Grundlagen von Daten- tential avenues for further research in this field. The *baCnokrernes),pMonadyin2g2-a2u4,th2o02r.4, Herdecke, Germany. full code implementation, data, and accompanying $ [e-mail] (J. Pollack); documentation can be accessed at the following [e-mail] (H. Köpcke); itory: https://git.informatik.uni-leipzig.de/jp31zusu/ [e-mail] (E. Rahm) intermediate-fusion-for-multimodal-product-matching.

0000-0003-2501-2609 (H. Köpcke); 0000-0002-2665-1114 (E. Rahm) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License

Attribution 4.0 International (CC BY 4.0).

2. Related work

the fusion process. Intermediate Fusion involves additional processing steps after feature fusion, while Late The field of Entity Resolution (ER) has been a subject of Fusion makes the decision directly after fusion without active research since the 1950s [ 3 ]. To gain an in-depth additional processing. understanding of ER, readers are directed to recent books With these clarified definitions in place, we proceed and surveys [ 4, 5, 6, 7 ]. An overview of the (increasing) to categorize existing multimodal product matching apusage of neural networks and deep learning for entity proaches according to these distinct fusion strategies. resolution can be found in [ 8 ]. The combination of text Wilke et al. [ 17 ] apply late fusion enhancing the Deepand image data in deep learning systems is called multi- Matcher framework [ 18 ]. They use FastText for tokenmodal deep learning. An overview on tasks, datasets and level text embeddings, which are then aggregated via problems in this new field can be found in [ 9 ]. Recurrent Neural Network (RNN). For images, they em

Applying ER on e-commerce data has been explored ploy a pre-trained Residual Neural Network (ResNet) with various approaches and methods within the last with a fully connected layer. Similarity is computed with years. The initial approaches for product matching uti- Euclidean distances and concatenation, followed by claslize only textual information and supervised machine sification using two fully connected layers. learning to categorize and match products [ 10, 11, 12, 13 ]. Ali Mazhar et al. [ 19 ] propose two intermediate fuAn early approach to also consider image data for prod- sion architectures for multimodal networks, one using uct matching is [ 14 ] but with the goal of enriching textual element-wise multiplication to combine pre-trained improduct descriptions with images (the images are only in age embeddings (ResNet, VGG, or MobileNet) with text one of the two datasets). embeddings from a character-level CNN model, followed

Recently several multimodal product matching ap- by logistic regression for classification, and the other proaches have emerged, employing diferent fusion relying on bidirectional triplet loss. strategies. It’s worth noting that the terminology and Gupte et al. [ 20 ] propose a weighted hybrid fusion definitions of these fusion strategies can vary consider- strategy, combining BERT text embeddings and ResNet ably across the literature [ 15, 16 ]. To ensure clarity and image embeddings in Siamese Networks. Their approach consistency, we establish the following definitions: outperforms text-based and late fusion methods. • In Early Fusion, data from diferent modalities are Das et al. [ 21 ] employ intermediate fusion, prioritizdirectly combined. An example would be overlay- ing product images as their primary data source, using ing text onto an image and then processing both title information to emphasize relevant image regions. the text and image simultaneously using a CNN. Their model consists of a global branch for overall im• In Intermediate Fusion, features from diferent age feature extraction, a local branch for specific feature modalities are processed separately, combined, derivation, and incorporates self-attention mechanisms. and then further processed before a final decision While showing promise on a multi-category dataset, it is made. This means that after combining the fea- exhibits potential performance variations in low-data tures, additional processing steps, such as passing scenarios. them through a neural network or applying fur- Valenciano et al. [ 22 ] delve into both intermediate ther transformations, are carried out before the fusion and late fusion strategies. The first method inifnal decision-making. tegrates modalities early, combining ResNet image embeddings with joint BERT text embeddings, followed by • In Late Fusion, features from diferent modalities BiLSTM and hybrid pooling. The second method utiare still processed separately, but a decision is lizes a Siamese Network, computing Euclidean distances made directly after the fusion step, without fur- between ResNet image embeddings, and later concatether processing of the combined features. This nating the outputs with joint BERT text embedding. The means that once the features are combined, a ifrst strategy outperforms the second, indicating that redecision-making mechanism, such as applying a lying solely on Euclidean distances of images may not be threshold or weighted averaging, is immediately efective for multimodal matching.

applied to determine the final output. Feng et al. [ 23 ] take a late fusion approach, integrating • Hybrid Fusion integrates modalities at multiple text embeddings generated by Robustly Optimized BERT stages within the processing pipeline. For ex- Pre-training Approach (RoBERTa) [ 1 ] models with imample, combining text and image features at an age embeddings from Hierarchical Vision Transformer intermediate stage, then further processing them, (Swin-Transformer) [ 2 ] models, leveraging a K-gram Exand finally combining the results again with those ponential decay scheme for text embeddings. Notably, from unimodal methods. they opt for Swin-Transformers instead of ResNets for The key diference between Intermediate Fusion and Late image embeddings, capitalizing on self-attention mechaFusion lies in the timing of decision-making relative to nisms for comprehensive context capture.

Early fusion can lead to larger input vectors with redundancies, causing increased computational complexity [ 24 ], while late fusion may struggle to capture complex inter-modal relationships, limiting efectiveness in multimodal product matching [ 20 ]. Therefore, we employ an intermediate fusion approach, inspired by the methodologies of Valenciano et al. [ 22 ] and Feng et al. [ 23 ].

Our method enhances text embeddings with a BiLSTM layer and hybrid pooling, while also integrating higherlevel image embeddings from Siamese Networks in an intermediate training step. Notably, we depart from the conventional use of ResNets for image embeddings, opting for Swin-Transformers, as demonstrated in the work of Feng et al. [ 23 ].

3. Benchmark datasets

In the realm of research on multimodal product matching,

one of the primary challenges has been the limited availability of publicly accessible datasets containing groundtruth information for product matching tasks [ 17 ].

To address this challenge, Primpeli et al. [ 25 ] generously shared their WDC Training Dataset and Gold Standard for Large-scale Product Matching with the scientific community. Originally designed for text-based unimodal product matching, these datasets have been instrumental in advancing the field. Building upon this foundation, Wilke et al. [ 17 ] undertook the task of augmenting the WDC product matching dataset with images sourced from the internet. They meticulously verified a Figure 2: Hard non-matching (negative) pair from Zalando subset of products in the shoe category from the original dataset datasets to create multimodal datasets. In these datasets, each product is associated with an image, and no erroneous or distracting images are included.

More recently, Lamm et al. [ 26 ] introduced a publicly prove model performance, we curated non-matching available large-scale dataset for visual entity matching, (negative) pairs. To accomplish this, we leveraged based on a real-world use case within the retail domain. text and image embeddings generated by pre-trained Unfortunately, this dataset does not provide textual infor- RoBERTa and Swin-Transformer models, incorporatmation for the products, rendering it unsuitable for the ing them into multimodal embeddings. Subsequently, evaluation of multimodal approaches that rely on both we employed an approximate nearest neighbor search text and images. within the resulting embedding space to identify, for

While the WDC Shoes dataset is primarily centered each Tommy Hilfiger and Gerry Weber product, the most on footwear, we introduce an additional dataset that en- similar non-matching Zalando product. As a result, our compasses a wide range of clothing items. This dataset negative samples included products that exhibited visual is constructed using product titles and images extracted similarities in their images or shared textual characterisfrom three online stores: Zalando, Tommy Hilfiger, and tics but ultimately did not constitute a match. Exemplary Gerry Weber. We crawled all available products from the matching and non-matching pairs represented in terms of Tommy Hilfiger and Gerry Weber online stores. When their pre-processed product titles and images are shown collecting data from Zalando, our focus was on products in Figures 1 and 2. from the Tommy Hilfiger and Gerry Weber brands. Es- We divided the resulting pairs into both training and tablishing ground truth for this diverse dataset relied on test datasets, with a specific condition: the Zalando proda semi-supervised approach, combining manufacturer ucts in both sets of pairs were mutually exclusive. This part (MPN) matching with manual inspection. separation allowed for a more accurate assessment of In order to create a challenging dataset that can im- model performance on previously unseen data. For a quantitative summary of the datasets, refer to Table 1 which outlines the total unique products, matching (positive) pairs, and non-matching (negative) pairs for each dataset.

4. Intermediate Fusion

Architecture

Our intermediate fusion architecture strategically inte

grates text-based information and image-based information, harnessing the unique strengths of both modalities and efectively capturing intricate relationships between them.

We utilize RoBERTa embeddings to ensure a robust representation of textual information, backed by their exceptional performance across various natural language processing tasks, including multimodal product matching [ 23 ].

We utilize the Swin-Transformer architecture for visual information representation, known for its eficiency in capturing spatial dependencies and long-range dependencies in various image processing tasks, including multimodal product matching [ 23 ]. Swin-Transformer, a recent advancement in vision transformer architecture, employs shifted windows and hierarchical transformer blocks to replace traditional convolutional layers, facilitating parallel processing and superior extraction of visual features, contextual information, and semantic relationships within images compared to established techniques like ResNet.

Our multimodal model combines RoBERTa-based text information with Swin-Transformer-based image data using an intermediate fusion strategy. We chose intermediate fusion over early or late fusion because it enables nuanced integration of high-level representations from diferent modalities, preserving their distinctive characteristics and adapting to the complexities of multimodal data, as empirically supported by prior research [ 20, 22 ].

We build upon Valenciano et al.’s multimodal models, making specific modifications to address their limitations.

While Valenciano et al. found Siamese Network-based models superior in unimodal image contexts, the concatenation of Euclidean distances with text embeddings did not improve performance in multimodal settings. Consequently, for better multimodal performance, they resorted to direct classification. In this work, we introduce an intermediate training step designed to seamlessly incorporate higher-level image embeddings obtained from Siamese Networks. These embeddings contain rich information as Siamese Networks inherently learn to identify similarities and diferences between inputs, especially The outcomes show that our text-based model consisthose critical for making matching decisions. Our archi- tently outperforms its image-based counterpart across tecture is depicted in Figure 3 and comprises two pri- both datasets. This observation underscores the potent mary branches: a text branch and an image branch. Each role of textual data in product matching, aligning with the branch undergoes individual training in an intermediate textual richness often present in product descriptions and step before subsequent fully connected layers carry out titles. However, our central contribution lies in the amalthe final classification. gamation of both unimodal models into a multimodal

In the text branch, input texts are structured as se- model (Intermediate Fusion), resulting in a significant quences in the format "[CLS] text 1 [SEP] text 2 [SEP]" to enhancement in performance. leverage RoBERTa’s ability to handle text pairs. The out- It’s worth noting that the ability of our models to put of the [CLS] token encapsulates the sentence-level efectively harness complementary information from embedding for the entire text pair. This embedding is re- each modality proves especially advantageous when conifned through a Bidirectional LSTM (BiLSTM) layer that fronted with ambiguous or incomplete data. By intelconducts bidirectional analysis, encompassing both pre- ligently fusing textual and visual features, the model ceding and subsequent context. The BiLSTM layer’s out- demonstrates a capacity to compensate for missing inputs are then channeled through a hybrid pooling layer, formation, promoting more accurate and robust product amalgamating max-pooling and mean-pooling to capture matching. Our multimodal model not only achieved the key and average representations, respectively. This pro- highest F1-score but also demonstrates a more balanced cess transforms the embeddings into one-dimensional trade-of between precision and recall, which is often vectors. During the intermediate training step, these out- desirable in real-world applications. puts from the hybrid pooling layer are further processed We compared our intermediate fusion approach with by multiple fully connected layers for classification. multimodal models by Wilke et al. [ 17 ] and Valenciano

In the image branch, Swin-Transformer embeddings et al. [ 22 ]. Our model showed significant performance are fed into an average pooling layer, transforming improvements over Wilke et al.’s approach, mainly due them into one-dimensional vectors, which are then fine- to our efective fusion strategy. Additionally, we retuned using multiple fully connected layers. Adopting a implemented Valenciano et al.’s ImageBERT model and Siamese Network approach, both sub-networks share the its unimodal counterparts [ 22 ], further highlighting the same parameters. The Euclidean distance between the performance gains achieved by our intermediate fusion resulting embeddings of both sub-networks is employed architecture, primarily attributed to our strategic choice during the intermediate training step for classification. of Swin-Transformer embeddings and enhanced fusion

The outputs of both sub-networks in the image branch strategy. serve as higher-level image embeddings, which are con- The Zalando dataset demonstrated lower performance catenated with the fine-tuned RoBERTa embedding from than the WDC Shoes dataset, attributed to greater varithe text branch. This combined representation is then ability in both textual descriptions and visual attributes passed through multiple fully connected layers. Our ar- among matching products. Additionally, the WDC Shoes chitecture is meticulously designed to capitalize on in- dataset features overlapping products between training formation from both textual and visual modalities, ef- and testing sets, potentially biasing model evaluations fectively capturing the intricate relationships between by assessing performance on partially seen data. Morethem. over, the WDC Shoes test set has a higher proportion of matching pairs compared to non-matching ones, which may not accurately mirror real-world scenarios where 5. Evaluation non-matching products typically outnumber matching ones. Thus, the Zalando dataset provides a more realistic We evaluate the performance of our proposed intermedi- assessment of model performance in practical settings. ate fusion approach on two distinct datasets: WDC Shoes Products in the Zalando dataset varied more in both and Zalando product matching datasets. Our evaluation textual descriptions and visual representations, making entails a comprehensive comparison against a range of it challenging for models to establish consistent relationunimodal and multimodal baseline models, with Table ships, especially with limited training data. Images of 2 ofering a detailed summary of the experimental out- matching pairs in the Zalando dataset could difer signifcomes. In order to provide a thorough evaluation, we icantly, presenting varied contexts like diferent clothing report the F1-score, precision, and recall metrics for each combinations (See Figure 1). Similarly, non-matching case. The reported results are an average computed over product images could appear very similar, making accuthree training and test runs to ensure robust estimates of rate determination challenging without additional conperformance, efectively mitigating the impact of stochas- text (See Figure 2). Some non-matching pairs difered ticity. only in minor variants, leading to nearly identical images

6. Conclusion and Future work

In this paper, we propose an intermediate fusion architecture for multimodal product matching, enhancing webbased entity resolution. Our model efectively captures complex relationships between product descriptions and images by merging fine-tuned RoBERTa textual embeddings with higher-level Swin-Transformer visual embeddings from Siamese Networks. Experimental results on WDC Shoes and Zalando datasets demonstrate the superiority of our approach over unimodal models, highlighting its potential for enhancing entity resolution in online marketplaces and e-commerce. Future work will focus on scaling the approach, incorporating advanced data augmentation techniques, and exploring cross-modal attention mechanisms for improved performance.

Acknowledgments

The authors acknowledge the financial support by the

Federal Ministry of Education and Research of Germany and by the Sächsische Staatsministerium für Wissenschaft Kultur und Tourismus in the program Center of Excellence for AI-research "Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig", project identification: ScaDS.AI. Computations for this work were done (in part) using resources of the Leipzig University Computing Centre.

References

[1] Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , O. Levy , M. Lewis , L. Zettlemoyer , V. Stoyanov , Roberta: A robustly optimized bert pretraining approach , ArXiv abs/ 1907 .11692 ( 2019 ).

[2] Z. Liu , Y. Lin , Y. Cao , H. Hu , Y. Wei , Z. Zhang , S. Lin , B. Guo , Swin transformer: Hierarchical vision transformer using shifted windows , 2021 IEEE/CVF International Conference on Computer Vision (ICCV) ( 2021 ) 9992 - 10002 .

[3] H. B. Newcombe , J. M. Kennedy , S. J. Axford , A. P. James , Automatic linkage of vital records , Science 130 ( 1959 ) 954 - 959 .

[4] O. Binette , R. C. Steorts, (almost) all of entity resolution , Science Advances 8 ( 2022 ) eabi8021 .

[5] P. Christen , T. Ranbaduge , R. Schnell , Linking Sensitive Data - Methods and Techniques for Practical Privacy-Preserving Information Sharing , Springer, 2020 .

[6] X. L. Dong , D. Srivastava , Big Data Integration, Synthesis Lectures on Data Management , Morgan & Claypool Publishers, 2015 .

[7] G. Papadakis , E. Ioannou , E. Thanos, T. Palpanas, The Four Generations of Entity Resolution, Synthesis Lectures on Data Management , Morgan & Claypool Publishers, 2021 .

[8] N. Barlaug , J. A. Gulla , Neural networks for entity matching: A survey , ACM Trans. Knowl. Discov. Data 15 ( 2021 ) 52 : 1 - 52 : 37 .

[9] A. Mogadala , M. Kalimuthu , D. Klakow , Trends in integration of vision and language research: A survey of tasks, datasets, and methods , J. Artif. Intell. Res . 71 ( 2021 ) 1183 - 1317 .

[10] H. Köpcke , A. Thor , S. Thomas, E. Rahm, Tailoring entity resolution for matching product ofers , in: Proc. 15th Int. Conf. on Extending Database Technology (EDBT) , 2012 , pp. 545 - 550 .

[11] C. d'Amato , P. Ristoski , P. Petrovski , P. Mika , H. Paulheim, A machine learning approach for product matching and categorization , Semant. Web 9 ( 2018 ) 707 - 728 .

[12] J. Li , Z. Dou , Y. Zhu , X. Zuo , J. Wen , Deep crossplatform product matching in e-commerce, Inf . Retr. J. 23 ( 2020 ) 136 - 158 .

[13] R. Peeters , C. Bizer , Supervised contrastive learning for product matching , in: Companion Proceedings of the Web Conference 2022 , WWW '22, Association for Computing Machinery, New York, NY, USA, 2022 , p. 248 - 251 .

[14] P. Ristoski , P. Petrovski , P. Mika , H. Paulheim, A machine learning approach for product matching and categorization , Semantic Web 9 ( 2018 ) 707 - 728 .

[15] S. R. Stahlschmidt , B. Ulfenborg , J. Synnergren , Multimodal deep learning for biomedical data fusion: a review, Briefings in Bioinformatics 23 ( 2022 ) bbab569 .

[16] S. Poria , E. Cambria, R. Bajpai , A. Hussain , A review of afective computing: From unimodal analysis to multimodal fusion , Information fusion 37 ( 2017 ) 98 - 125 .

[17] M. Wilke , E. Rahm, Towards multi-modal entity resolution for product matching , in: GvDB, 2021 .

[18] S. Mudgal , H. Li , T. Rekatsinas , A. Doan , Y. Park , G. Krishnan, R. Deep , E. Arcaute , V. Raghavendra , Deep learning for entity matching: A design space exploration , Proceedings of the 2018 International Conference on Management of Data ( 2018 ).

[19] K. A. Mazhar , M. Brodtbeck , G. Gühring, Similarity learning of product descriptions and images using multimodal neural networks , Natural Language Processing Journal 4 ( 2023 ) 100029 .

[20] K. Gupte , L. X. Pang , H. Vuyyuri , S. Pasumarty , Multimodal product matching and category mapping: Text+image based deep neural network , 2021 IEEE International Conference on Big Data (Big Data) ( 2021 ) 4500 - 4505 .

[21] N. Das , A. Joshi , P. Yenigalla , G. Agrwal, Maps: Multimodal attention for product similarity , 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) ( 2022 ) 2988 - 2996 .

[22] R. Estrada-Valenciano , V. Muñiz-Sánchez , H. D. la Torre-Gutiérrez , An entity-matching system based on multimodal data for two major ecommerce stores in mexico , Mathematics ( 2022 ).

[23] C.-F. Feng , W. Chen , C. Chen , T. Xu , E. Chen, Multimodal representation learning-based product matching , in: China Conference on Knowledge Graph and Semantic Computing , 2022 .

[24] D. Ramachandram , G. W. Taylor , Deep multimodal learning: A survey on recent advances and trends , IEEE Signal Processing Magazine 34 ( 2017 ) 96 - 108 .

[25] A. Primpeli , R. Peeters , C. Bizer , The wdc training dataset and gold standard for large-scale product matching , Companion Proceedings of The 2019 World Wide Web Conference ( 2019 ).

[26] B. Lamm , J. Keuper , Retail-786k: a large-scale dataset for visual entity matching , 2023 .