Introduction

In recent years, knowledge graphs (KGs) have become increasingly recognized for their ability to organize structured data in a semantically significant way, allowing them to effectively support various AI systems [1]. Large-scale KGs are usually generated through a semi-automated process, utilizing both structured and unstructured data. Some prominent examples include DBpedia [2] 1 , Google Knowledge Graph2 , BabelNet3 , and YAGO 4 . A few proposals have been recently put forth for generating organized, interconnected, and machine-readable data frameworks of content found within text from microblogging platforms [3,4,5], using Semantic Web technologies such as ontologies and knowledge graphs [6,7,8]. However, creating extensive and high-quality knowledge graphs from social media is still an open research problem. Existing solutions either depend on systems that aid social media experts in structuring their knowledge, therefore suffering from scalability problems, or rely on information extraction pipelines [9,10,11]. Generating large-scale, coherent, and semantically sound representations of social media texts drawn from millions of posts has proven to be challenging, as existing methods for entity and relationship extraction typically focus on specific domains [4].

In this paper we present Triplétoile, an enhanced information extraction architecture designed to extract and merge instances of open-domain entities from social media text and to identify and generalize various relationships among these entities by using hierarchical clustering, word embeddings and dimensionality reduction techniques.

The designed architecture is scalable and introduces a novel approach for entity extraction that leverages the dependency tree of an input sentence and a list of patterns validated by experts. It incorporates a module for unifying and grounding entity instances using external resources such as DBpedia. We also provide a use case application of the proposed architecture to a set of around 100k tweets extracted from the X/Twitter platform5 from 2022 and concerning the digital transformation domain and we released the resulting knowledge graph. Finally, we conducted an assessment of Triplétoile by comparing it to several alternative solutions using a benchmark dataset consisting of 500 triples and show that it outperforms them in terms of accuracy, while at the same time generating a relatively higher number of triples.

Related Work

Numerous scholarly articles delve into the methodologies for generating knowledge graphs across different domains and under various constraints [12,7,13,14,15,16]. These knowledge graphs are often enhanced and refined using link prediction techniques [17,18]. The extraction of knowledge graphs from web sources to answer questions related to social networks [3], such as Twitter or Facebook, has been widely discussed in literature [19,20,4]. He et al. [5] described how to build knowledge graphs for social networks by developing deep Natural Language Processing models. A number of information extraction pipelines have been proposed to create high-quality knowledge graphs within the social network analysis domain ( [9,4,10,11]).

Haslhofer et al. [21] have emphasized the importance of connected knowledge graphs and discovery, whereas Hyvönen and Rantala [22] have highlighted the significance of new relationships extracted from the original dataset. In recent years there has been also an increasing research focus on ontologies and interoperable data [23]. In particular, Dessì et al. [14] have proposed an information extraction method that combines data from different tools using a domain ontology, enabling the creation of expansive knowledge graphs. This first approach has been a source of inspiration for further research in the field [24,25,26,27,28]. Recently, approaches leveraging fine-tuning of pre-trained large language model such as GPT-3 have been proven to be effective in performing joint named entity recognition and relation extraction for complex hierarchical information [29,30]. Some recent solutions also augment large language model by using knowledge injection methods in order to improve their performance in specific domains [31].

Methodology

Figure 1 shows the workflow of the pipeline that we propose in this paper. We describe in more detail in the following the main component processing blocks and modules.

Data Preprocessing

Prior to extracting triples, we follow a two-fold approach to tweet normalization. On the one hand, we remove tokens and token sequences encoding platform-specific metadata or denoting communicative conventions that (typically) do not carry any syntactic function in the tweet sentence, namely sentiment emoticons and smileys, reserved tokens (e.g., RT for 'retweet') and URLs. On the other hand, we keep by default other platform-specific tokens that can carry syntactic functions in some contexts, like the tokens (e.g. #digitaltransformation, #SME, @NASA). Then, we apply a number of heuristics for capturing and removing token patterns that typically disrupt the syntactic parsing of the sentence 6 . This results in fixing noisy parser edges induced for example by trailing hashtags sequences. The preprocessing step is carried out using the output of Spacy's English transformer pipeline en_core_web_trf-3.6.1 after customizing the default Tokenizer in order to parse tweet metadata (e.g., mentions and hashtags) 7 .

Triple Extraction

In the triple extraction block, preprocessed tweets are sentence split and each sentence is fed to the Spacy pipeline. Building upon the works in [32] and [33], we define a set of procedures to extract candidate nominal entities and predicative triples connecting them out of dependency parse trees.

Entity extraction module: It detects local nominal phrases with a restricted range of syntactic modifications (e.g., compound nouns and adjectives). It then connects and expands them with a. non-recursive attached prepositional phrases; b. quantity-type entities (MONEY, PERCENT, QUANTITY, CARDINAL); c. entity spans linked via pronominal anaphoras, resolved using the Spacy pipeline component coreferee 8 . Overall, the module ends up with a set 𝐸 = {𝑒 0 , ..., 𝑒 𝑛 } of non-unified, candidate entity phrases (e.g. #digitaltransformation and digital transformation are not mapped to the same general concept digital transformation at this stage.

Relation extraction module:

For each sentence 𝑠 𝑖 all the shortest paths of the dependency tree between each pair of entities (𝑒 𝑚 , 𝑒 𝑛 ) containing a verb and matching any of a shortlist of expert validated patterns 9 are selected. The target pattern set has been selected through an expert validation process over a sampling of the most frequent patterns matched in an external, opendomain text corpus. The entire updated process generates a set of verbal relations 𝑉 = 𝑣 0 , ..., 𝑣 𝑘 and a set of triples 𝑆 = 𝑠 0 , ..., 𝑠 𝑘 of the form < 𝑒 𝑚 , 𝑣, 𝑒 𝑛 > where 𝑣 ∈ 𝑉 and 𝑒 ∈ 𝐸.

The final goal of the pipeline is to allow to generalize from the set 𝑆 of surface form triples to the lower sized set 𝑇 = 𝑡 0 , ..., 𝑡 ℎ of triples of the form < 𝜖 𝑚 , 𝑟, 𝜖 𝑛 > where each 𝜖 𝑖 ∈ 𝐸 is a unified entity and 𝑟 is a label in a generalized relation vocabulary 𝑅.

Entity Refining

Entities are first cleaned up by removing leading/trailing punctuation marks as well as stopwords. Then, hashtags and @ mentions are normalized and lower-cased and "camel case" forms resolved (e.g. from #SmartCities to 'smart cities'), while we lemmatize and lowercase all other component tokens whose POS tag is neither Verb nor Proper Noun.

We leverage the linking of these normalized candidate entities to DBpedia entries via DBpedia Spotlight library 10 in order to merge them. To this purpose, we run the library over modified tweet sentences with the original subjects and objects entity spans replaced with their normalized forms, and then merge entities that get linked to the same DBpedia entries. For example, the two candidate entities 'Gartner' and '@Gartner_inc' are merged as they get linked to the DBpedia entry of the Gartner consulting firm http://dbpedia.org/resource/Gartner). This is then formalized with a relation owl:sameAs in the final knowledge graph. In case only the first condition is met, we assign a semantic 'relatedness' link between the candidate entity and the DBpedia entry, indicating that the former is not an instance of, but rather related to the latter. For example, the span '@gartner_survey' is considered only 'related' (encoded as skos:related) to the DBpedia entry for Gartner.

Relation Refining

In order to find the best predicate label 𝑟 for each relation verb 𝑣 in a triple < 𝑒 𝑚 , 𝑣, 𝑒 𝑛 > and to map 𝑣 to 𝑟 in the resulting triple, we first derive a word embedding representation of the verb predicates from a pre-trained model, then we compute an optimized clustering of the relation vectors, and finally use a representative instance of each cluster to map verb predicates.

Relation Embeddings:

For each single or multi-token relation predicate, we use the static, 300-dimensional word embeddings learned with GloVe [34] and made available for text Span objects in the Spacy en_core_web_lg-3.6.0 pipeline 1112 .

Dimensionality Reduction and Clustering:

We used the HDBSCAN clustering algorithm enhanced by previously applying UMAP dimension reduction technique on the word embeddings vectors 13 . HDBSCAN is a hierarchical version of the popular density-based DBSCAN algorithm, which is characterized by considering outliers and leaves unclustered the data points lying in low-density regions [35]. Consequently, high dimensional data require more observed samples to produce the suitable level of density for HDBSCAN to work properly. However, applying UMAP to perform non-linear, manifold aware dimension reduction [36] has been proven to transform the datasets down to a dimension small enough for HDBSCAN to cluster the vast majority of instances. In order to optimize the combination of UMAP and HDBSCAN, we perform a grid search over the hyper-parameters of both algorithms and evaluate the clustering using the score: 𝑆 = 𝑠𝑖𝑙ℎ𝑜𝑢𝑒𝑡𝑡𝑒 𝑋 • 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑒𝑑 𝑋 , where 𝑠𝑖𝑙ℎ𝑜𝑢𝑒𝑡𝑡𝑒 𝑋 is the mean silhouette coefficient over all the instances of the dataset 𝑋 that were actually clustered by HDBSCAN [37] and 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑒𝑑 𝑋 is the fraction of instances of 𝑋 that were actually clustered. In practice, we optimize for the classical measure of cluster cohesion and separation while penalizing the configurations with low coverage of the dataset. We finally chose a subset of best-scoring hyper-parameter configurations and plotted their 𝑆 score over the number of output clusters they generate, so that we are able to pick a sub-optimal configuration that balances between generalization (fewer clusters) and accuracy (cluster number closer to the dataset size).

Relation Mapping: Finally, for each relation verb 𝑣 in the dataset, we replace it with the predicate label 𝑟 consisting of the lemma of the most frequent relation in the cluster of 𝑣.

Otherwise, we map it to itself if 𝑣 was an outlier.

Evaluation

We first evaluate the precision by manually assessing the truthfulness of a test set of statements. Second, we evaluate our pipeline's precision against a number of alternative tools.

Human Expert Assessment:

We randomly selected about 500 statements extracted by our pipeline and ask three domain expert evaluators to assess each statement as True or False 14 . Average pair-wise Cohen 𝜅 inter-rater agreement was 0.61, while Fleiss 𝜅 𝐹 agreement score over all 3 raters (ranging in [−1, +1], [38]) reached 0.558 (substantial agreement). The accuracy of the majority vote assessments over the 500 triples was 0.96, indicating that the pipeline is able to extract triples with good precision.

Comparative Evaluation: Successively, we randomly sampled 500 tweets from the 100ksized original dataset and used our pipeline to extract candidate entities. We then merge this set of entities with the ones generated by the DyGIEpp Extractor [39]. Finally we deploy four alternative methods to identify relationships between these entities and extract statements from the 500 tweets. Specifically, we compared with: a. OpenIE Extractor, the IE tool of the Stanford Core NLP suite [40]; b. PoST Extractor, a module that searches for all verbs in a 15 token window between two candidate entities in a sentence to extract verb relations; c. Dependency-based Extractor, a module that exploits 12 hand-crafted paths 15 over Stanford Core NLP dependency parses to find verbs that connect DyGIEpp entities; d. Entity and Relationship Refiner, described in [33]. Precision (P) of the triples extracted from a set of alternative methods from a set of 500 tweets, using a combination of Triplétoile and DyGIEpp candidate entities.

Table 1 reports the number of extracted triples for each of these methods. In order to use these numbers as an indirect assessment of the relative recall levels of the different pipelines, we also report the expert-assessed precision on a limited random sample of 150 triples generated by each method. It can be seen how the precision of our pipeline on this smaller sample largely outperforms all the alternative methods, while also yielding the second largest number of triples, interestingly outperforming the Dependency-based Extractor method, which deploys very similar syntactic information from the sentence 16 .

Digital Transformation Monitoring Knowledge Graph

The presented prototype pipeline was deployed as part of a Digital Transformation monitoring system, targeting specifically its capacity to link and extend existing knowledge graphs generated from conventional sources (scientific papers, patents) with continuous updates from news and social media. Therefore, we have generated a knowledge graph from around 100k topicspecific tweets, sampled from 4M English language tweets from 2022 containing the hashtag #DigitalTransformation, excluding retweets 17 .

The generated DTSMM (Digital Transformation Social Media Monitor) knowledge graph comprises approximately 22,270 (non-reified) triples, connecting a total of 22597 nodes via 43428 edges. A number of sample triples are shown in Table 2.

We reified then each claim into dtsmm-ont:Statement class instances, where dtsmmont:Statement defines a specific claim extracted from a given number of tweets. Figure 2 shows an example of claim reification having the instance multi_page_document_classification as rdf:subject. DTSMM features a total of 18693 unique detected entities, whose 33.9% and 6.44% included hashtags and @ entity mentions, respectively, 3.34% were complex noun phrases with prepositional attachments, while around 16.6% contained quantitative modifiers of any type (currency, percent, etc.). Out of all the generated triples, a 5.98% had either the subject or object entity made by a resolved pronominal anaphora.

Around 8% of all unique entities were linked to DBpedia entries via 2,857 owl:sameAs and 3,309 skos:related predicates in order to encode entity equality and relatedness, respectively. Centre (JRC) 19 , whose goal is to create a tracker that monitors societal and economic activities through European countries using unconventional data [41].

Therefore, DTSMM has been made publicly accessible both through a SPARQL endpoint and a serialization file. Through the Virtuoso SPARQL endpoint https://api-vast.jrc.service.ec.europa. eu/sparql/ DTSMM can be queried using the graph name 'DTSMM_KG'20 as shown in Figure 3, where we retrieve all the statements in DTSMM having the target entity dtsmm:microsoft as object.

Finally, a Turtle format serialization of DTSMM has been publicly released 21 within the Joint Research Centre Data Catalogue 22 , as well as within the European Data portal 23 . The direct link is: https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/CC-COIN/se-tracker/DTSMM_KG.ttl.

Conclusions

We presented an approach to specifically extract a knowledge graph comprising open-domain entities from micro-blogging posts on social media platforms. In a topic-specific test collection of Digital Transformation tweets the pipeline proved to outperform some of the state-of-the-art methods, generating mostly valid triples. Moreover, around 12% of entity mentions are linked to DBpedia entries, suggesting that the method is potentially useful for tracking relevant entities in the target social media text collection. A current limitation is that the entity and relation extraction processes are not backed by an underlying ontology specification. Therefore, on one hand, the extracted entities are not natively typed and no domain-specific classification schema for relations is available for setting up a supervised learning of relation mapping. We plan to work on an enhanced version of the pipeline that builds upon the current entity and relation spans and further classifies them into domain-specific categories, leveraging fine-tuning of contextual word embedding representations from Large Language Models [42]. Simultaneously, we aim to capitalize on the resultant knowledge graph to develop knowledge plugins [43], thus augmenting the proficiency of these language models across various natural language processing tasks.