In human communication, messages are often conveyed through a combination of different modalities, such as visual, audio and linguistic modalities. In order for an automatic understanding of the multimodal messages, one needs to fuse the information from different modalities to construct a joint multimodal representation. The challenge falls on how to characterise the interactions between different modalities, which can be complicated in some scenarios. In the example shown in Fig. 1, the visual-linguistic query cannot be understood solely by either the text or image individually, but the relation between the image and text must be correctly recognized. This is where the significance and challenge sit for multimodal representation learning.
Most existing research focus on neural network-based fusion strategies for constructing multimodal representation, ranging from the earlier Hidden Markov Model (HMM)-based models [7] to different RNN variants [1] and more recently tensor-based approaches [12] and seq-to-seq structures [8]. Despite their strong accuracy performances, the interplay between different data modalities are encoded in an inherent way by the neural network components, making it difficult for humans to understand the contributions of each modality in a particular task.
Quantum Theory (QT) provides a well-established theoretical and mathematical formalism for describing the physical world on a microscopic level. Beyond physics, QT frameworks have been applied to many research areas [2,9,10,15], among which successful
Today's consumers have become very exigent. When shopping online, they have in mind a specific clothing item in a particular color and style, and they want to find it without too much effort. However, current e-commerce search mechanisms are often too limited to provide this kind of service. A common way for searching 1 Image material adapted from www.amazon.com Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. products in a webshop is to navigate through a product category hierarchy. Users end up in a subcategory which they need to search completely, which is time-consuming. They need to go through many irrelevant products, without the guarantee of actually finding the product they are looking for. To narrow down the search, users can sometimes select certain filters. However, desired product attributes might not be amongst the available filters. With a text-based search approach, users can describe the desired product by entering keywords into a search bar. The webshop then finds relevant products by matching these keywords to the words in the product descriptions. It is often difficult for users to write the right keywords that will induce the search engine to provide the products they are interested in. For example, some users might be interested in "jeans with holes", but the relevant products are described as "distressed jeans". Additionally, this approach hampers the search for product attributes which are not mentioned in the product descriptions. Alternatively but rarely, webshops offer image-based search where the user uploads an image of the desired product and receives visually similar products. Recently, there is an increasing interest of users in this kind of image-based search. One of the main reasons is the growing usage of visual social media such as Pinterest and Instagram, where users see products they want to buy. Using an image as a query allows users to convey much more information about the desired product than with a textual query. Additionally, another advantage of image-based search over text-based search is that the language of images is universal. However, the user might be interested in changing or adding attributes to the product in the query image to obtain very specific results. For example, for an image of a red dress, the user likes the sleeve length to be different.
WSDM'18, February 5-9, 2018, Marina Del Rey, CA, USA 342 results have been observed in text-based language understanding [10,15]. In this context, quantum-inspired frameworks have the natural advantage of explicitly capturing the correlations between features by the concepts quantum superposition and quantum entanglement. This inspires us to adopt quantum-inspired frameworks for constructing multimodal representation, and propose effective and interpretable multimodal fusion methods. In particular, we propose to capture the interactions within a single modality with quantum superposition, and model the cross-modal interactions by means of quantum entanglement. In this way, we establish a pipeline that extracts the interactions within multi-modal data in a way understandable from a quantum perspective. We expect to obtain comparable performances to state-of-the-art systems in concrete multimodal tasks.
Here we introduce our proposed approach for building multimodal data representation inspired by quantum theory. Essentially, we represent multimodal data as many-body systems composed of different data modalities as subsystems. The interaction of different modalities is inherently captured by the notion of entanglement between the subsystems. We propose to build a complex-valued learning network to implement the quantum theoretical framework, which facilitates learning the cross-modal interactions in a datadriven fashion.
Complex values are essential for the mathematical formalism of quantum physics. However, most existing quantum-inspired models for text representation are based on the real vector space, ignoring the complex-valued nature of quantum notions. Recently, our prior works [3,4,11] leveraged quantum superposition to model correlations between textual features, and the complex-valued representation leads to improved performance and enhanced interpretability. We attempt to employ the concept of superposition for modeling intra-modal interactions, and investigate the complexvalued embedding approach to capture the interactions within other modalities.
We represent multimodal data as a many-body quantum system in entanglement. The mathematical formulation will be a complexvalued tensor constructed from unimodal complex-valued vectors by tensor-based approaches. Tensor-based models have been applied for classification [5] and matching [14] tasks, but they avoid directly computing the tensor by decomposing the tensor and learning the decomposed weights via neural networks. Explicit tensor combination of different data modalities into a holistic multimodal representation remains unexplored. [6] proposed a framework to investigate the entanglement between user and document for relevance feedback, but it remains a challenge on how to apply it to multimodal data.
We focus on the multimodal sentiment analysis task, and work with the benchmarking datasets CMU-MOSI [13] and CMU-MOSEI [1].
The task is to classify sentiment of a video into 2, 5 or 7 classes with textual, visual and acoustic features. As is shown in Fig. 2, our framework represents unimodal data as a set of pure states through complex-value embedding, and constructs the many-body state of an video utterance through tensor-based approaches. Finally, quantum-like measurement operators are implemented for sentiment classification. The whole process is implemented into a complex-valued neural network, and the parameters in the pipeline can be learned from labeled data in an end-to-end manner. The network is born with an advantage in interpretability compared to classical neural networks, in that the role of each component is made explicit prior to the network training phase. We are working on deploying the network to CMU-MOSI and CMU-MOSEI, and we expect to see comparable values to state-of-the-art models in terms of effectiveness.
This PhD project is supported by the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 721321.