The advancement of social networks has placed the analysis and study of social network data at the forefront of priorities. User-representation learning is a powerful tool to solve many critical problems in social media studies. Reasonable user representations in vector space could help build a recommendation system [1,2], conduct social analysis [3,4,5], detect bot accounts [6,7,8], and so on. To obtain userembeddings of higher quality, many multimodal methods are proposed to fully utilize all types of available information from the social networks, including interactive graphs, user pro les, images, and texts from their posts [9,10,11,12]. Compared with models using single modality data, multimodal methods utilize more information from social media platforms. Hence they usually achieve better results in downstream tasks.
Among all modalities in social networks, user-interactive graphs (i.e., what they do) and text content (i.e., what they say) are the two most frequently used options, due to their good availability across di erent datasets and large amounts of observations. The graph-neural-network (GNN) models [13,14,15] makes it more convenient to fuse both the text information and graph-structure information of social-network users, where text-embeddings from language-models such as GloVe [16] or BERT [17] are usually directly incorporated into GNNs as node attributes. Although those approaches have achieved great performance in a bunch of downstream tasks [18], the text information and graph-structure information are fully entangled with each other, which makes it hard to illustrate the two modalities' respective contributions to learning each user's representation.
Researchers have already found that di erent groups of users can behave very di erently on social media [19]. If such di erences are not correctly captured, it might cause signi cant bias in the user attribute prediction (e.g., political stance prediction) [20]. Hence, when learning multi-modal user representation for di erent users, it is not only important to ask what the prediction results are, but also important to ask why we are making such predictions (e.g. Are those predictions due to the same reason?). Only in that way, we could provide more insights into the user modelings, and potentially enable unbiased and personalized downstream analysis for di erent user groups.
On the other hand, under a multi-modality setting, if one aspect of a user's data is not trustworthy and misleading, it might still be fused into the model and make the performance lower than single-modality models [21]. Consider the case when we want to make a political ideology prediction for Elon Musk based on his Twitter content before the 2020 U.S. presidential election (Figure 1), when he has not revealed his clear Republican political stance yet. If we trust the follower-followee graph structure information, we can see that he is likely to be a Republican, since he follows more Republicans than Democrats, and has more frequent interactions with the veri ed Republicans accounts. However, in his tweet content, his word choice also shows some Democratic traits. Due to the existence of such con icting information, being able to automatically identify which modality is more trustworthy for each individual becomes essential in building an accurate social media user embedding for di erent groups of users.
To address the above two shortcomings of text-graph fusion in social networks, we propose a simple yet e ective framework called Contribution-Aware Multimodal User Embedding (CAMUE), which can identify and remove misleading modality from speci c social network users during text-graph fusion, in an explainable way. CAMUE uses a learnable attention module to decide whether we should trust the text information or the graph structure information when predicting individual user attributes, such as political stance. Then, the framework outputs a clear contribution map for each modality on each user, allowing personalized explanations for downstream analysis and recommendations. For ambiguous users whose text and graph structure information disagree, our framework could successfully mitigate unreliable information among di erent modalities by automatically adjusting the weight of that information accordingly.
We conduct experiments on the TIMME dataset [21] used for a Twitter political ideology prediction task. We observed that our contribution map can give us some interesting new insights. A quantitative analysis of di erent Twitter user sub-groups shows that link information (i.e., interaction graph) contributes more than text information for most users. This provides insights that political advertising agencies should gather more interaction graph information of Twitter users in the future when creating personalized advertisement content, instead of relying too much on their text data. We also observe that when the graph and text backbone are set to R-GCN and GloVe respectively, our approach ignores the unreliable GloVe embedding and achieves better prediction results. When the text modality is switched to a more accurate BERT embedding, our framework can assign graph/text weights for di erent users accordingly and achieve comparable performance to existing R-GCN-based fusion methods. We pick 9 celebrities among the 50 most-followed Twitter accounts1 , such as Elon Musk. A detailed qualitative analysis of their speci c Twitter behaviors shows that our contribution map models their online behaviors well. Finally, we run experiments on the TwiBot-20-Sub dataset [22] used for a Twitter human/bot classi cation task, showing that our framework could be generalized to other user attribute prediction tasks. By creating social media user embeddings that are more explainable, reliable, and e ective, our framework enables improved customized content delivery.
Social network user embedding is a popular research eld that aims to build accurate user representations. A desirable user embedding model should accurately map sparse user-related features in high-dimensional spaces to dense representations in low-dimensional spaces. Multimodal social network user embedding models utilize user di erent types of user data to boost their performance. Commonly-seen modality combinations include graph-structure (i.e. link) data and text data [23], graph-structure data and tabular data [24,25,9], and graph-structure data, text data and image data altogether [26] [27], etc.
Among those multi-modality methods, the fusion of graph-structure data and text data has always been one of the mainstream approaches for user embedding. At an earlier stage, without much help from the GNN models, most works trained the network-embedding and text-embedding separately and fused them using a joint loss [28,29,30,31]. With the help of the GNN models, a new type of fusion method gained popularity, where the users' text-embeddings are directly incorporated into GNNs as node attributes [23,32,33].
Despite their good performance, all existing models do not explain how much the graph structure and the text information of speci c users contribute to the nal prediction results, making it di cult to give customized modality weight for downstream analysis or recommendations. Also, if one modality is poorly learned, it can be counter-e ective to the user embedding quality, making it even worse than their single-modality counterparts [21]. How to address this problem in a universally-learned way instead of heuristic-based information ltering, has largely gone under-explored. Hence, we propose a framework that not only utilizes both text and graph-structure information, but also reveals their relative importance along with the prediction result.
Graph Neural Network (GNN) is a collection of deep learning models that learn node embedding through iterative aggregation of information from neighboring nodes, using a convolutional operator. Most GNN architectures include a graph convolution layer in a form that can be characterized as message-passing and aggregation. A general formula for such convolution layers is:
where H (l) represents the hidden node representation of all nodes at layer l, operator is a non-linear activation function, and the graph-convolutional lter A ˜is a matrix that usually takes the form of a transformed (e.g., normalized) adjacency matrix A, and the layer-l's weight W (l) is learnable.
In the past few years, GNN models have reached SOTA performances in various graph-related tasks. They are widely regarded as promising techniques to generate node embedding for users in social-network graphs. [13,34,14,15].
The eld of natural language processing has undergone a signi cant transformation with the advent of neural-network-based language models. Word2Vec [35] introduced two architectures: Continuous Bag-of-Words (CBOW) and Skip-Gram. CBOW predicts a target word given its context, while Skip-Gram predicts context words given a target word. GloVe [16] model went beyond by incorporating global corpus statistics into the learning process. ELMo [36] was another signi cant step forward, as it introduced context-dependent word representations, making it possible for the same word to have di erent embeddings if the context is di erent. BERT [17] is a highly in uential model that is built on the transformer architecture [37], pre-trained on large text corpora using, for example, masked language modeling and next-sentence prediction tasks. Recently, large language models like GPT-3 [38], InstructGPT [39], and ChatGPT have achieved signi cant breakthroughs in natural-language-generation tasks. All those models are frequently used to generate text embedding for social network users.
Our framework does not rely on any speci c language model, and we do not have to use LLMs. Instead, we use language-models as a replaceable component, making it possible for both simpler ones like GloVe and more complicated ones like BERT to t in. We will explore some di erent options in the experimental section.
In the past, several methods have been proposed to improve the interpretability and explainability of multimodal fusions [40]. Commonly used strategies include attention-based methods [41,42,43], counterfactual-based methods [44,45], scene graph-based methods [46] and knowledge graph-based methods [47]. Unfortunately, most of them focus on the fusion of image modality and text modality, primarily the VQA task, while to the best of our knowledge, no work focuses on improving the explainability between the network structure data and text data in social-network user embedding.
Our general goal is to propose a social network user embedding fusion framework that could answer: 1. which modality (i.e. text or graph structure, saying or doing) contributes more to our user attribute prediction, hence allowing more customized downstream user behavior analysis and 2. which modality should be given more trust for each user, and automatically lter out the untrustworthy information when necessary, in order to achieve higher-quality multi-modal user-embedding.
A general framework of our problem could be formulated as follows: given a social media interaction graph G = (V, E) with node set V representing users and edge set E representing links between users.
be the adjacency matrices of G, m be the number of link types and A i 2 R n⇥n , our training objective is:
Here, L is the loss of our speci c downstream task, and f is some function that combines the graph structure information and text information, producing a joint user embedding.
To investigate the e ectiveness of the existing GNN-based multimodal fusion methods in ltering the unreliable modality when the graph structure and text contradict, we run experiments using a common fusion method that feeds the ne-tuned BERT features into the R-GCN backbone, similar to the approaches in [23] and [10]. We observe that this conventional fusion method fails to lter the unreliable information for some of those ambiguous users. Table 1 shows two politicians whose Twitter data contains misleading information, either in the graph structure or text data. While the singlemodality backbones which are trained without misleading information give the correct predictions, the multi-modality fusion method is confused by the misleading information, hence it is not able to make correct predictions.
These insights revealed the importance of having a more exible and explainable framework for learning multimodal user embedding.
We propose a framework of Contribution-Aware Multimodal User Embedding (CAMUE), a fusion method for text data and graph structure data when learning user embedding in social networks. The key ingredient of this framework is an attention gate-based selection module which is learned together with the link and text data, and decide which information we want to trust more for each particular user.
Our framework has three main parts: a text encoder, a graph encoder, and an attention-gate learner. The text content of each user passes through the text encoder and generates a text embedding for that user. The embedding is then passed through a three-layer MLP for ne-tuning. The adjacency matrix of the users passes through the graph encoder and generates a node embedding for that user. At the same time, both the text embedding and the graph adjacency matrix pass through our attention gate learner. The output of this module is two attention weights, ↵ and , which control the proportion of our graph structure information and text information. Without loss of generality, if we make R-GCN our graph encoder and BERT our text encoder, our model will be trained in the following way (Equation 3-6, also illustrated in Figure 2):
H (2) = (H (1) W (2) ) (4)
[e ↵ , e ] = H (2) W (3) (5)
Where H and W are hidden layers and weights of our attention gate learner,
is the adjacency matrices of G and m is the number of link types.
Then, our overall training objective becomes:
Here, acts as a regularizer to ensure our model is not overly dependent on a single modality.
Our methods o er two levels of separation. First, we separate the text encoder and the graph encoder to allow better disentanglement on which data contributes more to our nal prediction results. Second, we separate the learning of the downstream tasks and the learning of which data modality (i.e. text or graph structure) we can rely on more. This makes our framework adaptable to di erent downstream social media user prediction tasks. The learned trustworthiness of di erent modalities allows for autoadjustment of the weight between graph structure and text modalities, hence ltering any unreliable information once they are discovered.
Figure 2 shows the overall architecture of our framework, note that the graph structure encoder and text encoder could be replaced by any other models that serve the same purposes. We give a short complexity analysis of our architecture for the case of R-GCN + BERT: Since we are using sparse adjacency matrix for R-GCN, the graph encoder part has a complexity of O(L graph EF graph + L graph NF 2 graph ) (according to [48]), where L is the number of layers, E is the number of edges, N is the number of nodes, and F is the feature dimension. Since we xed the maximum text length to be a constant for the text encoder, it has a complexity of O(F 2 text ) (based on [49]). Since F text and F graph are about comparable size, our fusion module has the complexity of O(F
We run experiments on two Twitter user prediction tasks: 1. Predicting the political ideology of Twitter users (Democrat vs Republican) and 2. Predicting whether a Twitter user account is a human or a bot.
TIMME [21] introduced a multi-modality Twitter user dataset as a benchmark of political ideology prediction task for Twitter users. TIMME contains 21, 015 Twitter users and 6, 496, 112 Twitter interaction links. Those links include follows, retweets, replies, mentions, and likes. Together they form a large heterogeneous social network graph. TIMME also contains 6, 996, 310 raw Twitter content from those users. Hence, it will be a good dataset to study di erent fusion methods of text features and graph structure features. In TIMME, there are 586 labeled politicians and 2, 976 randomly sampled users with a known political a liation. Some of them are ambiguous users we investigated before. Labeled nodes belong to either Democrats or Republicans. Note that the dataset cut-o time is 2020, so the political polarities of many public gures (e.g. Elon Musk) have not been reviewed at that time.
TwiBot-20 [22] is an extensive benchmark for Twitter bot detection, comprising 229, 573 Twitter accounts, of which 11, 826 are labeled as human users or bots. The dataset also contains 33, 716, 171 Twitter interaction links and 33, 488, 192 raw Twitter content. The links in TwiBot-20 include follows, retweets, and mentions. To further examine the generalizability of our method, we run experiments for Twitter bot account detection on the TwiBot-20 dataset. To reduce the computation cost of generating node features and text features, we randomly subsample 3, 000 labeled users and 27, 000 unlabeled users from the TwiBot-20 dataset, and form a new dataset called TwiBot-20-Sub. In this way, the size and label sparsity of the TwiBot-20-Sub dataset becomes comparable with the TIMME dataset.
We split the users of both datasets into an 80%:10%:10% ratio for the training set, validation set, and test set respectively.
To test the e ectiveness of our framework across di erent models, we choose two single-modality text encoders, GloVe and BERT, and two single-modality graph encoders, MLP and R-GCN.
The GloVe embedding refers to the Wikipedia 2014 + Gigaword 5 (300d) pre-trained version. 2 The BERT embedding refers to the sentence level ([CLS] token) embedding of BERT-base model [50] after ne-tuning the pre-trained model's parameters on the tweets from our training set consisting of 80% of the users. We chose a max sequence length of 32. After the encoding, we have a 300-dimension text embedding for GloVe and a 768-dimension text embedding for BERT.
We choose a modi ed version of R-GCN from TIMME [21] as an R-GCN graph encoder. R-GCN [34] is a GNN model speci cally designed for heterogeneous graphs with multiple relations. In the TIMME paper, it is discovered that assigning di erent attention weights to the relation heads of the R-GCN model could improve its performance. Hence, we adopt their idea and use the modi ed version of R-GCN. We did not use the complete TIMME model since it is designed for multiple tasks outside our research scope, and will overly complicate our model.
We also choose a 3-layer MLP as another graph encoder for comparison, the adjacency list for each user is passed to the MLP.
Large language models (LLMs) like ChatGPT are powerful in understanding texts, but they usually have a great number of parameters, making traditional supervised ne-tuning a hard and costly task [38]. Instead, less resource-intensive methods like few-shot learning, prompt tuning, instruction tuning, and chain-of-thought are more frequently used to adapt LLMs on speci c tasks [51]. We do not use large language models as one of the options for the text encoder since those methods are not compatible with our framework -they do not provide a well-de ned gradient to train our attention gate learner.
We run experiments on a single NVIDIA Tesla A100 GPU. We used the same set of hyper-parameters as in the TIMME paper, with the learning rate being 0.01, the number of GCN hidden units being 100, and the dropout rate being 0.1, on a PyTorch platform. For a fair comparison, we run over 10 random seeds for each algorithm on each task.
To show that our framework could e ectively provide personalized explanations during the fuse of modalities, we draw the contribution map based on ↵ (graph weight) and (text weight) attention for users from each dataset. The darker the color, the weight of the corresponding modality is closer to 1.
In the contribution map, pure white indicates a zero contribution (0) from a modality, while pure dark blue indicates a full contribution (1).
The top gure of Figure 3 shows the contribution map output when the text encoder is BERT and the graph encoder is R-GCN, on a subgroup of the TIMME dataset consisting of some politicians and some random Twitter users. As we can see, there is a clear cut between the percentage of contributions from di erent modalities to the nal prediction. It is notable that for the two ambiguous politician users we have mentioned earlier (Ryan Costello and Sheldon Whitehouse), CAMUE could give correct attention, where we should trust more text data from Mr. Costello while trusting more graph structure data from Mr. Whitehouse. To avoid any misuse of personal data information, we hide the names of random Twitter users and only include politicians whose Twitter accounts are publically available at 3 .
The bottom gure of Figure 3 shows the contribution map output when the text encoder is GloVe and the graph encoder is R-GCN, on the same subgroup of the TIMME dataset. Note that for all shown users text information does not contribute to the nal prediction. This could be attributed to the fact that GloVe is not very powerful for sentence embedding, especially when the text is long. This contribution map shows that our framework lters out the text modality almost completely when it is not helpful for our user embedding learning. As we can see from table 2, the traditional fusion method for GloVe+R-GCN only yields an accuracy of 0.840, which is much lower than the single graph structure modality prediction (0.953) using R-GCN, due to unreliable GloVe embedding. In contrast, our CAMUE method obtains a higher accuracy (0.954) than all single modality models, since it disregards unreliable information.
Figure 4 shows the contribution map output for the same set of encoders on a subgroup of the Twibot-20-Sub dataset. There is also a clear cut between the percentage of contributions from di erent modalities, for both the human Twitter accounts and bot accounts.
Hence, we verify that our framework could both provide personalized modality contribution and drop low-quality information during the fuse of modalities. Some quantitative analysis of how this low-quality information ltering could bene t the general model performance could be found in the next section, and some qualitative analysis about what new insights we could gain from the output of our framework could be found in the case study section.
Table 2 shows the performance of CAMUE on di erent combinations of encoders. The traditional fusion method in Figure 2 is denoted as "simple fusion". For MLP, we do not have such a natural fusion method. We also add "CAMUE, xed params" as an ablation experiment to prove the e ectiveness of our attention gate-based selection module.
We observe that within those combinations, sometimes simple fusion methods are signi cantly worse than single-modality methods (e.g. GloVe+R-GCN vs R-GCN only) due to some untrustworthiness in one of the modalities. However, any fusion under our CAMUE framework always performs better than the corresponding single modality methods. That suggests that our algorithm can bene t from attending to the more reliable modality between text and graph structure, if one particular modality is not trustworthy (e.g. GloVe embedding), and learning not to consider it when making predictions (as we can see in Figure 3, bottom).
It is also notable that our CAMUE method outperforms "CAMUE, xed params". These results suggest that adjusting the weight of di erent modalities dynamically yields better performance than xed weights of modalities. Finally, when the text modality is switched to a more accurate BERT embedding, our framework still gives comparable performance to its corresponding simple fusion methods.
User Sub-groups Table 3 gives a quantitative analysis when the text encoder is BERT and the graph encoder is R-GCN, for di erent sub-groups of Twitter users we are interested in. In general, graph structure information contributes the most when comes to bot accounts. One possible explanation for this is the variety of bot accounts on Twitter, such as those for business advertising, political outreach, and sports marketing [22]. Bots with di erent usage might talk very di erently, however, they may share some common rule-based policies when interacting with humans on Twitter [52,53]. Graph structure information contributes the second highest when it comes to politicians. This is also not surprising since politicians are generally more inclined to retweet or mention events related to their political parties [21]. It is also notable that the weight of text information for Republicans is slightly less than that for Democrats. This aligns with the ndings in [54] that Democrats have a slightly more politically polarized word choice than Republicans. For random users, the weight of text information is the largest, although still not as large as the weight of graph structure information. This could be attributed to the pattern that many random users interact frequently with their non-celebrity families and friends on Twitter, who are more likely to be politically neutral. Table 4 shows some predicted political stances and the main contributing modalities of a group of news agencies. We can see that most of them have more reliable information about graph structure than text information. This is not surprising since most news agency tends to use neutral words to increase their credibility, hence it is hard to gather strong political stances from their text embedding, except for some of them like Fox News and Guardian which are known to use political polarized terms more often [54,55]. Our framework is able to capture this unique behavior pattern for Fox News and Guardian, meanwhile giving mostly accurate political polarity predictions aligning with results in [54] and 4 .
To conclude, we are able to obtain customized user behavior patterns through our multi-modality fusion. Those patterns could provide insights on which modality we should focus on more for di erent types of users, for downstream tasks such as personalized recommendations, social science analysis, or malicious user detection. of the two aspects of user pro les (text and graph structure) on their nal predictions, we do case study by evaluating a subset of users qualitatively to validate our frameworks' capability to give personalized explanations. We selected 9 celebrities among the top 50 most followed Twitter accounts from 5 , whose Twitter accounts appear in the TIMME dataset, as we are not allowed to disclose regular Twitter users' information. We obtain the political polarity predictions of those celebrities and record the percentage of text/graph structure information that contributes to their political polarity predictions (See Figure 5).
• Elon Musk: Before 2020 (dataset cut-o ), Elon Musk's political views in his tweet text content are often complex. He claimed multiple times not to take the viewpoints in his tweets too seriously 6 . This aligns with the low contribution weight of his texts on his political stand prediction. However, on the graph level, 66.67% of the politicians Elon Musk liked more than one time, have also liked Trump at least once. This is signi cantly larger than the average number in the TIMME dataset (23.67%). This could be a strong reason why our graph structure weight is so high and why we predict Elon Musk to be Republican-leaning. Our prediction is proved correct when in 2022 (which is beyond our dataset cut-o time, 2020 [54]), Elon Musk claimed that he would vote for Republicans in his tweet 7 . This is a strong indicator that our framework is using correct information. • LeBron James: In his tweets, LeBron James frequently shows his love and respect to Democratic President Obama8 . Our prediction for him to be Democrat-leaning with a strong text contribution aligns with this observation. • Lady Gaga: Similarly to James, Lady Gaga also expresses explicitly in her tweets about her support of Democratic candidates 9 . Our graph weight is 0, meaning that the text alone is su cient to predict that she is Democrat-learning. Clinton during the 2016 election, a reason why we predict her as Democrat-leaning from the graph structure. Although she supports some republican politicians in 2022 11 , that is beyond the dataset cuto . • Justin Timberlake: Justin Timberlake has frequent positive interactions with President Obama 12and rmly supports Hillary Clinton in his tweets 13 , both suggesting that he is Democrat-learning.
Our model assigns a similar weight to text and graph structure, suggesting that both contribute to that prediction equally. • Taylor Swift: In the case of Taylor Swift, the model fails to give the correct prediction. Her tweets show that she voted for Biden during 2020 14 , but the prediction is Republican. One reason is that at the graph structure level, the majority of Taylor Swift's followers are classi ed as Republican (67.09%) in the dataset, which can mislead the graph encoder.
Overall, we conclude that graph structure information is usually more useful when predicting the political polarities of those celebrities. That aligns with the quantitative results in table 3. As we can see, di erent celebrities may have very di erent behavior patterns. Those patterns can be correctly captured and explained by our contribution weight. That con rms the e ectiveness of our framework.
In this paper, we investigate some potential limitations of existing fusion methods for text information and graph structure information in user representation learning on social networks. We then propose a contribution-aware multimodal social-media user-embedding with a learnable attention module. Our framework can automatically determine the reliability of text and graph-structure information when learning user-embeddings. It lters out unreliable modalities for speci c users across various downstream tasks. Since our framework is not bound to any speci c model, it has great potential to be adapted to any graph-structure-embedding component and text-embedding component, if a ordable. More importantly, our models can give a score on the reliability of di erent information modalities for each user. That gives our framework great capability for personalized downstream analysis and recommendation. Our work can bring research attention to identifying and removing misleading information modality due to di erences in social network user behavior, and paves the way for more explainable, reliable, and e ective social media user representation learning. Some possible future extensions include adding more modalities other than text and graphs (e.g., image and video data from user's posts). Also, we consider the user identities to be static throughout our analysis, which might not be the case in many scenarios. We can bring time as a factor to produce a multi-modality dynamic social media user embedding. For example, a user's text content may be more trustworthy in the rst few months, and interactive graph structure information becomes more reliable in longer terms.