Recommender systems now span the entire customer journey. Amid the multitude of diversified experiences, immersing in cultural events has become a key aspect of tourism. Cultural events, however, sufer from fleeting lifecycles, evade exact replication, and invariably lie in the future. In addition, their low standardization makes harnessing historical data regarding event content or past patron evaluations intricate. The distinctive traits of events thereby compound the challenge of the cold-start dilemma in event recommenders. Content-based recommendations stand as a viable avenue to alleviate this issue, functioning even in scenarios where item-user information is scarce. Still, the efectiveness of contentbased recommendations often hinges on the quality of the data representation they build upon. In this study, we explore an array of cutting-edge uni- and multimodal vision and language foundation models (VL-FMs) for this purpose. Next, we derive content-based recommendations through a straightforward clustering approach that groups akin events together, and evaluate the eficacy of the models through a series of online user experiments across three dimensions: similarity-based evaluation, comparison-based evaluation, and clustering assignment evaluation. Our experiments generated four major findings. First, we found that all VL-FMs consistently outperformed a naive baseline of recommending randomly drawn events. Second, unimodal text-based embeddings were surprisingly on par or in some cases even superior to multimodal embeddings. Third, multimodal embeddings yielded arguably more fine-grained and diverse clusters in comparison to their unimodal counterparts. Finally, we could confirm that cross event interest is indeed reliant on the perceived similarity of events, resonating with the notion of similarity in content-based recommendations. All in all, we believe that leveraging the potential of contemporary FMs for content-based event recommendations would help address the cold-start problem and propel this field of research forward in new and exciting ways.
Over the last years, the application of recommender systems in the tourism industry has
expanded to cover more and more facets of the overall customer experience, ranging from
recommending trips [
Yet, the task of recommending events poses, arguably, greater challenges compared to
recommending standardized mass services like transport or accommodation [
In scenarios characterized by limited or constrained data about item-user relationships (e.g.,
purchases, ratings), the system’s ability to proficiently represent the content of these items
is vital. Yet, as described above, the efective modeling of event contents hinges on encoding
them into semantically meaningful and expressive representations. Recent advances in Deep
Representation Learning (DRL), notably through foundation models (FMs) pretrained on massive
amounts of broad data and adaptable to a wide range of specific tasks [
Considering that the intangible nature of events begs for multimodal content descriptions,
it seems promising to evaluate the capacity of VL-FMs for integrating multimedia data into
content-based event recommendations [
In light of the above, we set out to explore the potential of harnessing both uni- and multimodal VL-FMs to learn informative representations of cultural events. The resultant event embeddings formed, in turn, the foundational cornerstone for content-based recommendations. We derived these content-based recommendations through a simple clustering approach that groups the events into semantically related cluster.
We conducted our computational experiments on an event dataset sourced from the eventbased platform Meetup.com1. The dataset comprises 10, 658 distinct cultural events from the ten largest cities in the US, each accompanied by its respective descriptions and corresponding images. To evaluate the usefulness of diferent VL-FMs for generating content-based recommendations on this dataset, we conducted a series of online user experiments, in which we presented users with recommendations based on diferent VL-FMs and asked them to evaluate these recommendations with regards to three dimensions: similarity-based evaluation, comparison-based evaluation, and clustering assignment evaluation.
Our experiments generated four major findings. First, we found that all VL-FMs consistently outperform a naive baseline of recommending randomly drawn events. Second, surprisingly we found that unimodal text-based embeddings matched or in some cases even outperformed multimodal embeddings in terms of perceived similarity. Third, multimodal embeddings yielded arguably more fine-grained and diverse clusters in comparison to their unimodal counterparts. Finally, we could confirm that cross-event interest is reliant on the perceived similarity of events.
Our contributions can be summarized as follows: First, to the best of our knowledge, we present a novel exploratory method for grasping the impact of FMs on event recommendation systems. Second, to tackle the cold-start problem, we provide a straightforward clusteringbased approach aimed at automatically identifying semantically relevant events based on their multimedia content. Third, we conduct a sequence of user experiments to confirm the results across three diferent dimensions. In doing so, we assess the eficacy of modern unimodal and multimodal FM techniques for cultural events representation.
The remainder of the paper is organized as follows: The next section ofers an overview of related work on using embeddings for content-based event recommendations. The subsequent section imparts foundational knowledge regarding the technical background. Following that, section four delves into the approaches for generating uni- and multimodal embeddings and introduces representative FMs for each one. In section five, we outline the general approach, including the clustering framework, data and user experiments. Section six elaborates on and discusses the results, while section seven wraps up the study by addressing its limitations, implications, and providing a perspective on potential future research directions.
For decades, there has been a growing interest in incorporating richer information beyond
numerical ratings to promote recommendations. Over time, the field has evolved significantly
with the development of various frameworks and methodologies [
Several works considered content-based event recommendations. Authors in [
Some studies tackled the cold-start problem in event-based social networks (EBSNs). Authors
in [
While these studies collectively exploit content based signals, they frequently rely on user features or contextual input and overlook scenarios where such information might be unavailable. Furthermore, most of these works typically start from scratch, without exploring the possibilities ofered by large-scale state-of-the-art FMs for eficiently encoding event data. They also overlook including images as valuable signals in the recommendation process.
Motivated by these factors, we shift our focus towards modeling event data by leveraging accessible and advanced uni- and multimodal VL-FMs. By doing so, we (i) intend to experiment with integrating image information into event embeddings and (ii) probe into the potential of these models for enhancing content-based event recommendations against cold-start scenarios.
The key to enabling an intelligent system to comprehend the world around us lies in its ability to
identify and separate the fundamental explanatory factors that are hidden within the low-level
sensory data it observes. Representation Learning (RL) is a subarea of machine learning (ML)
that aims to accomplish precisely that. At its core, RL learns a set of meaningful features from a
given collection of data, making it simpler to derive valuable information when performing
diferent ML tasks [
More recently, the growing popularity of unimodal FMs has sparked increased research
interest in the integration of multiple modalities together. Indeed, at the most basic level, it is
theoretically insuficient to imitate a whole range of human perceptions and understanding
through only one modality [
The overarching goal for multimodal FMs thus becomes to capture the joint distribution
of multiple modalities and to learn a shared representation space that reduces the semantic
heterogeneity gap between the modalities while preserving the modality-specific semantics
[
Pre-training VL-FMs typically involves three main steps: (i) encoding images and text into
latent representations, (ii) designing a high-performing architecture for modeling interactions
between modalities, and (iii) devising efective pre-training tasks [
The main architectural distinction lies in the interaction step, and there are several strategies to generating multimodal embeddings. However, the literature lacks agreement on the optimal design. For cultural event data, we find it intriguing to experiment with a single-stream fusion encoder approach, a dual encoder approach, and a two-step approach.
In the single-stream fusion encoder approach, the text embeddings and image features are
concatenated together before feeding them into a transformer-based encoder to model the
vision and language (VL) interaction [
In the dual encoder approach, the text and image modalities are encoded separately through
two single-modal encoders. As next, the embeddings are projected through a shallow interaction
module to the same semantic space to compute VL similarity scores [
Lastly, in the two-step approach, one modality is translated into the other and the multimodal embeddings are generated using an unimodal FM based on the combined resulting set.
As is widely recognized, pre-trained FM models amass broad knowledge through extensive
pre-training on numerous source tasks with abundant labeled and unlabeled data [
We experiment with a separate unimodal FM for each individual modality– i.e. a language model and a vision model. For the multimodal VL-FMs, we evaluate the three approaches introduced in the previous section 3 using corresponding VL-FMs. Throughout this study, we denote embeddings generated from text only as text-based embeddings and those generated solely from vision as vision-based embeddings. An overall overview of the all approaches is provided in Figure 1. The forthcoming subsections delve deeper into the architectures, pre-training paradigms, and adaptation processes of the models for each approach.
To create unimodal embeddings that rely exclusively on text but are adjusted to suit the event
data domain, we employ the methodology described in [
The method combines two domain adaptation strategies, namely, Transformer-based
Denoising AutoEncoder (TSDAE) [
Event Images Event Descriptions
Event Images Event Descriptions
Event Images Event Descriptions
Event Images
Text Transfromer
Encoder Vision Transfromer
Encoder Text Transfromer
Encoder Vision Transfromer
Encoder Text Transfromer
Encoder Vision Transfromer
Encoder Text Transfromer
Encoder Vision Captioning model Vision-Based Embeddings Multimodal Embeddings Multimodal
Embeddings Text Embeddings Image Features Text Embeddings Image Features Image Captions
Concatenate Embeddings Concatenate Embeddings
Multimodal Transfromer
Encoder Preappend
Combined Textual
Descriptions
Multimodal
Embeddings
cross-encoder. The model is then trained on these synthetically generated labels, utilizing the
MarginMSE loss as introduced in [
Our implementation heavily relies on the publicly available GitHub repositories2,3 of the two methods and adapts largely the same training arguments.
Until recently, CV applications were dominated by convolutional neural networks (CNNs), while
transformer architectures were completely absent from the field. In 2020, this architectural
gap was bridged with the introduction of ViT -i.e. the first Vision Transformer Model by
authors in [
In 2022, the Masked Autoencoder (MAE) for CV was proposed [
The unsupervised nature of autoencoders renders MAE as a viable tool for pre-training
based domain adaptation on in-domain data. As such, we utilized the Huggingface library4 and
continued training the (facebook/vitmae-base) model for further 30 epochs and with the same
default hyper-parameters as set in the original implementation. Finally, during inference, we
extract akin to authors in [
As a representative for the single-stream architecture, we selected the Vision-and-Language
Transformer (ViLT) [
The image and text embeddings are concatenated into a single sequence, that undergoes
multiple interaction block layer updates up until the final contextualized sequence. The pooled
multimodal representation is then obtained by linear projection upon the first index of this
sequence. The model is pre-trained on Image text matching (ITM) and masked language
modeling (MLM) [
Since ViLT is pre-trained on smaller public paired datasets, we opted for domain adapting a base ViLT model on the event data following the same methodology as in 4.2. For the implementation, we leveraged the publicly available ViLT repository5 and trained (vilt_200k_mlm_itm.ckpt) with the default settings for further 3 epochs.
Contrastive Language-Image Pre-training (CLIP) [
CLIP undergoes pre-training on an immense private image-and-text dataset, comprising 400
million (image, text) pairs and sourced from publicly available web data. In line with other
4Last visited 10.08.2023: https://github.com/huggingface/transformers
5Last visited 10.08.2023: https://github.com/dandelin/vilt
VL-FMs in this class [
Taken the above into account, there appears to be no need for any further training. Instead, we apply the CLIP model7 directly to the event texts and images to obtain their respective embeddings. As per the authors’ approach, we extract the features before the linear projection layer, and fuse them into a unified vector representation through simple concatenation.
Up until now, we have covered two VL-FMs, each with a diferent architecture. However, another viable approach is to convert one modality into the other. Once both modalities are represented in the same modality form, multimodal representations can be obtained by applying an unimodal FM on the combined information. By adopting such a strategy, one can benefit from the strengths of unimodal models in handling singular modalities, while still achieving multimodal VL understanding.
Converting an item from one modality into another would require creating an adequate
representation of the same item in the other modality. For this purpose, VL generation techniques,
such as visual captioning (VC) or text-to-image synthesis [
Here, we employ VC to automatically generate textual captions for the event images. In
particular, we implement the generative image-to-text transformer (GiT) proposed in [
The obtained multimodal representations are achieved by applying the domain-adapted language model as discussed in subsection 4.1 on the combined set of event captions and event descriptions.
As already discussed, content-based recommendations can mitigate the cold-start problem in event recommenders by delivering suggestions solely grounded in event content. The idea behind clustering-based recommendations on the other hand builds upon similarity within clusters to provide alike suggestions. Events within the same cluster are assumed to have shared characteristics, making it likely that users showing interest in a specific event will tend to also show interest in others from the same cluster. Alternately, cluster-based recommendations can serve to diversify suggestions by ensuring that events from various clusters are suggested, 6For detailed experiments and results, please refer to the supplementary material provided by the authors. 7Last visited 10.08.2023: https://github.com/huggingface/transformers thereby expanding the spectrum of choices available to users and enhancing overall exposure to diferent events.
This study is primarily concerned with exploring whether contemporary FMs are suitable candidates for event content representations. As such, our emphasis lies in evaluating a select set of unimodal and multimodal VL-FMs for event representation within a simple clustering framework designed for content-based recommendations.
Embeddings generated by large-scale FMs are typically high-dimensional. However, as the
number of dimensions in data increases, the proximity of a random data point to its nearest
neighbor and its farthest neighbor approximate each other. In spaces with high dimensionality,
the notion of spatial locality therefore loses its clarity [
One of the available techniques is Uniform Manifold Approximation and Projection (UMAP)
[
We proceeded by clustering the embeddings with Hierarchical Density-Based Spatial
Clustering (HDBSCAN) [
In the last step, we draw on approaches proposed by [
It is well established that the eficacy of clustering algorithms is contingent upon the inherent structure of the data. By the same token, we argue that the quality of representations can be assessed by evaluating the quality of the resultant clusters. That is, provided that the experimental setting and clustering procedure remain consistent across all models. Figure 3 illustrates the clustering framework visually.
We fill in the footsteps of previous research about event recommendations [
We used the Meetup GraphQL API in November 2022 to perform a comprehensive crawl of all publicly available activity on the platform from the ten largest cities located in the USA8, namely, New York, Los Angeles, Chicago, Houston, Phoenix, Philadelphia, San Antonio, San
Empirical evaluation
Content-Based Recommendations
Unimdal
Type? Vision Only Vision-Based Embedings Preprocess event
data Modality? Embeddings with reduced dimensions
using UMAP Unified Represenatation of Events for each Cluster using
MMR Text Only SinAgplep-rSoatrceham DuAaplpErnocaocdher TAwpop-rSoatecph
Diego, Dallas, and San Jose, California. The resulting collection served as the training dataset, on which we trained all FMs models. In April 2023, we performed another crawl to create a second independent set for testing purposes. This way, we emulate real-world conditions by delivering recommendations for events that are scheduled to occur in the future.
We selected these particular cities for the analysis based on two primary reasons: Firstly, as the largest cities in the United States, they are considered to be among the most popular and vibrant locations, which often corresponds with a high level of event activity. Secondly, these cities are spread across diferent states, which provides a level of cultural diversity 9. In total, the collection comprises 13, 685 distinct events, with 10, 658 in the training set and 3, 027 in the testing set. For each event, we obtained (i) the event image and (ii) the event description and its title. The events do not have true labels, thus, our setting is unsupervised.
Prior to the analysis, we carried out several standard NLP preprocessing steps such as
excluding non-English texts and removing special characters HTML tags and hashtags. Additionally,
we applied regex [
Regarding the images, we applied data augmentation including resizing, cropping, and normalization techniques for the vision-based and VL-FMs separately as outlined in their
original research papers.
One significant limitation of clustering research is the lack of validation methods for unlabeled
data. Many studies in this area of research attempt to solve the clustering problem on datasets
with known true labels. However, in real-life situations, having prior knowledge of the ground
truth is often the exception rather than the norm. There are few external evaluation methods
available that can be applied in such settings, like the silhouette coeficient [
These reasons motivated us to validate the results by justifying them with human perception through a series of online user experiments. We leveraged the crowdsourcing service Prolific 10 to recruit 300 human evaluators, all of which passed an attention check. We presented users with content-based recommendations founded on the VL-FMs introduced in this study and prompted them to evaluate these recommendations along three dimensions: similarity-based evaluation, comparison-based evaluation, and clustering assignment evaluation.
All in all, the recommendations were based on one of the following types: (i) Text-based embeddings. (ii) Vision-based embeddings. (iii) VL-based embeddings obtained through singlestream fusion encoder approach. (iv) VL-based embeddings obtained through the dual encoder approach. (v) VL-based embeddings obtained through the two-step approach. As a baseline, we match and recommend events randomly.
In the first set of experiments, participants were presented with two events displayed side by side. For each event, participants viewed the corresponding image, title, and description. Both events were sampled randomly from the same cluster and had the same embedding type. The events were randomly placed on either the left or right side and were not presented in any particular order.
As previously mentioned, clustering-based recommendations rely on the assumption of similarities among events within a cluster and consider other events within the same cluster as relevant. We attempt to examine this assumption by modeling participants’ perceived similarity and cross event interest between two such events. We refer to Figure 5 in Appendix for an illustrated example.
In the second set, participants were shown one event and two clusters, with one cluster being the correct cluster to which the event belonged. The events and clusters were randomly sampled and based on the same embedding type. We created the cluster visualizations by drawing representative exemplars from each cluster (as explained in subsection 5.1), and then consolidating them into one single visual representation. To evaluate the representations quality, we gauge the agreement between the clustering assignments and the participants’ perceived memberships by asking them to indicate the cluster to which they believed the event belonged. We refer to Figure 8 in Appendix for an illustrated example.
In the last set, we asked the respondents to choose the form of modality (text,image or both) that they believe had the most significant impact on their judgment and to motivate their choice. Here, we attempted to model the participants’ preferences and perceptions regarding the influence of diferent modalities on their decision-making process.
Furthermore, to ensure that the participants’ attention is not diverted to or biased by secondary information such as the varying length of descriptions, we constrained the length of the description for the sampled events to one standard deviation around the mean length of all deceptions. We reconstructed the events by parsing the raw HTML data and converting it into a structured HTML document. Before this process, we took measures to protect privacy by removing sensitive personal data and anonymizing images containing individuals using Gaussian blurring. The remaining settings, including image resolution and font size, were left unaltered. In the last step, we integrated the reconstructed events into windows of equal sizes and blended them into the experiment’s user interface.
In this section, we detail the findings of the study. We commence by delivering quantitative measures and further support these findings with a concise qualitative analysis.
For the similarity-based evaluation (corresponding to the first set of experiments), we carried out a series of simple linear regressions in which we regressed the variable of interest against the type of embeddings used. Table 1 displays the regression results of the perceived similarity degree across the reference groups, random, vision-based and text-based, respectively.
First and foremost, the results indicate that recommendations based on event representations, as encoded by the FMs, performed on average significantly better than a naive random baseline. The two-step approach emerged, notably, as the most superior among all models.
The results also suggest, to our surprise, that text-based embeddings led to recommendations that were either comparable or in some cases even superior in the perceived similarity degree to their multimodal counterparts. This strongly suggests that text-based clusters seem to satisfy the assumption of semantic similarity and demonstrate comparable traits.
Additionally, when pitted against vision-based embeddings, all models once again achieved superior performance with the two-step approach surpassing all others. Comparing the random baseline with vision-based embeddings however yielded insignificance. This implies that there was minimal distinction in the perceived similarity between recommendations based solely on visual similarities and a randomly selected sample.
To explore this in more detail, we looked into cases where the similarities between events were rated high by the study participants and compared them to those rated low11.
11In all examples, we retrieved similar images under Creative Commons licenses due to copyright reasons from https://commons.wikimedia.org (last visited on 10.09.2023)
Vision-based clusters can be interpreted along two dimensions: content and context. The
content dimension reflects the entities that are visible in the images, and it is the most intuitive
dimension. In contrast, the context dimension reflects the circumstances in which the content
occurs [
Further, the instances where participants rated the events as highly similar were relatively limited. These cases featured events that seem to have images matching across the content and the context dimensions. As an example for such a case we provide Figure 6, in which the event images of a sample recommendation are shown. Both images depict a book and both events were about literary gatherings. On the other hand, instances rated as dissimilar by the participants appeared to feature events with images that matched only in terms of their content. Figure 6 in the Appendix visually displays such an example. While both images share similar attributes, there is an absence of contextual similarity between both events. To elaborate, the event depicted on the left pertains to travel and tourist excursions, whereas the event on the right is centered around meditation journeys. These observations could potentially therefore reafirm the notion that vision models encode images primarily based on content, lacking the incorporation of suficient semantic signals within them.
Furthermore, and perhaps most importantly, we conducted a regression analysis to examine the relationship between cross-event interest as a dependent and embeddings type as an independent variable. The rationale behind this is that the perceived similarity may not always convey a complete picture. For instance, two events with the same image color are likely to be perceived as similar but the two events may still be completely diferent.
Table 2 demonstrates that cross interest is significantly higher to the text-based and two-step approach, thereby confirming that instances with higher perceived similarity indeed correspond to higher cross interest levels.
We also found that all multimodal embeddings resulted in clusters that are more varied and detailed in comparison to those achieved by their unimodal counterparts, all the while maintaining comparable perceived similarity and cross-event interest as the text-based ones. This observation could hint that the VL-FMs provided richer and more ample event representations, as Figure 7 in Appendix shows.
In the second set of experiments, we attempted to assess the clustering quality by measuring the agreement degree between human judgment and clustering assignment.
As can be seen in Table 5 in Appendix, the agreement metric appears to vary across the diferent embeddings’ types. In particular, recommendations founded on embeddings created by the VL two-step approach showed the highest agreement frequencies. To test for statistical significance, we opted for a simple logistic regression, modeling the variable agreement as the dependent variable and the type of embedding as the independent variable. The two-step approach was set as the reference level.
The coeficient estimates in Table 3 suggest that clusters formed using the two-step approach tend to exhibit notably higher agreement probabilities in comparison to other VL and text-based approaches, although not when compared to the vision-based approach. Treatment(reference=Vision-Based) C(Random) C(Text-Based) C( VL Single-Stream Approach) C(VL Dual Encoder Approach) C(VL Two-Step Approach) Treatment(reference=Text-Based) C(Random) C(Vision-Based) C( VL Single-Stream Approach) C(VL Dual Encoder Approach) C(VL Two-Step Approach)
This observation aligns with the principles of Gestalt [
A Chi Square test of independence revealed that among participants, the form of modality that they believe had the most significant impact on their judgment and embeddings type were significantly associated, 2(10) = 29.36, < 0.001. Post hoc comparisons in Table 4 for each pair, with FDR correction applied, revealed that participants who were shown random events were more likely to answer based on event descriptions. In comparison, statistical similarity was observed among all other cases.
Intercept Treatment(reference=Random) C(Vision-Based Approach) C(Text-Based Approach) C(VL Single-Stream Approach) C(VL Dual Encoder Approach) C(VL Two-Step Approach) Treatment(reference=VL Two-Step Approach) C(Vision-Based) C(Text-Based) C(VL Single-Stream Approach) C(VL Dual Encoder Approach) Intercept Observations Note:
Dependent variable: Cross Event Interest
(1) 3.360* (0.355)
The results suggest therewith that both the textual and visual stimuli were perceived as equally important regardless of the embedding type.
This work has showcased the potential of harnessing contemporary uni- and multimodal FMs as a means for event data representation. We contend that (i) these models present themselves as efective tools for appropriately capturing event content, and that (ii) contentbased recommendations grounded on these models can efectively address the cold-start problem in event recommendations.
We started of by building a collection of events from the event-based platform Meetup. Subsequently, we transformed these into informative vector representations using a selected set of FMs. The resultant representations were arranged into clusters, bringing together events with common characteristics. The clusters served in turn as the corner stone for generating content-based recommendations. we then executed a series of user experiments to ascertain the quality of these recommendations with regards to three dimensions: similarity-based evaluation, comparison-based evaluation, and clustering assignment evaluation. The findings of this study can be summarized to : (i) Recommendations generated through the use of FMs outperformed a naive baseline that randomly drew events, with the VL two-step approach emerging as the top performer. (ii) Surprisingly, text-based embeddings performed on par or even surpassed multimodal VL embeddings in some cases. (iii) All multimodal VL-FMs yielded more diverse clusters, each with diferent scopes, while still demonstrating comparable perceived similarity to the text-based clusters. (v) The agreement between cluster assignments and human judgment revealed that the two-step approach was statistically superior to all other models except for vision-based embeddings, aligning with the principles of Gestalt theory. (iv) The results indeed verified that cross interest is associated with perceived similarity, confirming that similar content-based recommendations are more likely to be regarded as relevant.
Nonetheless, the empirical results reported herein should be considered in light of some limitations. First, the baseline is limited to a naive random approach. Additional standard content-based extraction methods could be considered to substantiate the superior performance of FM models. Second, the dataset was sourced from Meetup during the months of November and April of 2023, limiting the scope of our data to the type of events commonly promoted on the platform during these months. In order to ofer a more thorough representation of the broader spectrum of events, further experiments spanning extended time frames are imperative.
In conclusion, our experiments indicate that leveraging the capabilities of modern FMs for content-based event recommendations could efectively address the cold-start problem. It would be interesting to test these models in a real-world recommendation setting with postrecommendation human feedback. We view our study as a step in this direction and hope that our results can support the development and application of future technologies in this field.