This research explores whether Masked Language Models (MLMs) can understand semantic containment relations, such as sub-class and instance-of relationships, which are crucial for Semantic Web applications. The study introduces PRONTO, a novel approach that leverages MLM predictions to discover semantic containment relations in unstructured text by translating the model's internal predictions into classification labels. The efectiveness, reliability, and interpretability of PRONTO are assessed through a comprehensive probing procedure. The findings demonstrate that MLMs can capture semantic containment relationships, which has significant implications for ontology construction and aligning text data with ontologies. For the sake of reproducibility, we make our code, datasets, and evaluation tools available at https://github.com/sisinflab/PRONTO.
Pre-trained Language Models (PLMs) have become essential in Natural Language Processing
(NLP) due to their ability to capture complex language patterns through extensive training
on large text datasets. Studies show PLMs efectively capture factual [
Brooklyn sports
Nets team Brooklyn Nets is [MASK] of sports team
LM
Encoder Vanilla PLM (Frozen)
P r e d iit c o n H e a d
Containment verbalizer
V _ d i m reflects a hierarchical "is a" relationship between entities. The study explores whether PLMs can identify semantic containment when two entities are present in a prompt (e.g., "Paris [MASK] city"), to determine if PLMs are zero-shot semantic containment learners. We propose PRONTO, a novel procedure aimed at the extraction of semantic containment relations from bidirectional PLMs based on the examination of their masked language modeling prediction head. Our key contributions can be summarized as follows: • We propose a general procedure to probe semantic containment knowledge from MLMs by means of automatically learned verbalizers, i.e. mappings between a MLM prediction head and a label. • Through extensive analysis, we reveal how vanilla (i.e., not fine-tuned) MLMs exhibit an inner awareness with respect to semantic containment.
To the best of our knowledge, this is the first attempt to use the knowledge stored in PLMs to detect ontological containment through relation prediction with automatically extracted verbalizers. Finally, we present practical applications in zero-shot entity typing.
In this section, we formally introduce our containment prediction task that we schematize in Figure 1. Let = {1, 2, . . . , } represent the set of classes in a reference ontology . Each class is a node within the ontology graph. Let be the set of edges representing the subclass relations among these classes, where each edge (, ) ∈ denotes that class is a subclass of class . Let = {1, 2, . . . , } denote the set of instances of classes in , and be the set of edges denoting the instance of relation, where each edge (, ) ∈ indicates that instance is of type , linking instances to their respective classes. We define the semantic containment graph as the union of the two sets of edges and , combined with their respective node sets and . Formally, = ⟨ ∪ , ∪ ⟩. For any two nodes , ∈ , we aim to determine whether there exists a path from to that signifies an “is-a” relationship within the ontology . This relationship is characterized by a sequence of edges each representing either a direct subclass relation between classes or an instance belonging to a class, thereby forming a chain of semantic containment. Formally, we aim to learn a model : (, ) ↦→ ˆ with ˆ being 1 if there exists a path from to in and 0 otherwise. The function is parameterized by the parameters derived from a vanilla PLM (e.g., BERT). The aim of is to learn the mapping : (, ) ↦→ ˆ, where ˆ represents the predicted probability that a containment relationship exists between the concepts and . Let us define a function that constructs a prompt for a pre-trained MLM, given two nodes and . The function obtains the verbalized forms of and through (·) and inserts a mask token [MASK] between them to form the prompt. Formally, the prompt construction can be represented as (, ) = () ⊕ "[MASK] " ⊕ ( ), (1) where V() and V( ) are two natural language representations for the nodes and , respectively. The symbol ⊕ stands for string concatenation, and V( ) is the rdfs:label associated to .
Automatic Extraction of a Containment Verbalizer. Given the prompt (, ) as input
to a bidirectional PLM capable of mask-filling, the output of its MLM prediction head consists
of the predicted probability distribution over possible tokens that could replace the [MASK]
(Fig. 1). We propose investigating whether these predicted probabilities can help determine the
existence of a containment relationship between and . Given a PLM capable of mask-filling
trained on a vocabulary of size and a prompt function (·, ·) , we aim to create a mapping
between the prediction head output and a discrete label . Prior work formulate the concept of
verbalizer [
1 ∑︁ ([MASK] = |), =1 with being the number of tokens in and being the prompt. The construction of is often done manually: for instance, if y="city", a reasonable although simplistic verbalizer construction could be = {, }. In this work, we formulate the construction of a verbalizer as a search problem over the whole vocabulary. This enables our verbalizer to fully exploit the expressiveness of such a large vocabulary and possibly capture associations between labels and tokens that could be not easily identifiable even for domain experts. We want to design a verbalizer as a direct mapping function between the PLM prediction head and a label. An implementation of such a verbalizer is the following:
(|) = ∑︁ ([MASK] = |) = ∑︁ ( )([MASK] = |),
=1 =1
where ∈ [
P2 Sparsity: We aim to enforce a sparsity constraint on the weights to promote
interpretability. This constraint facilitates the identification of the most influential tokens minimizing
the influence of less relevant ones. In fact, a PLM vocabulary is highly populated even for
smaller models (30000+ tokens). Therefore, sparsity can aid interpretability for humans,
which can only realistically focus on a smaller set of informative tokens simultaneously.
To satisfy P1, MLM prediction head logits pass through a weighted-softmax [
1 exp(1)
∑︀=1 exp()
, . . . , ∑︀=1 exp()
exp() ︂)
where are parameters learned jointly in the optimization process and constrained in the [
(|) = ( 2 · ℎ( 1 · (logits MLM))).
In summary, we experiment with diferent verbalization strategies: • PRONTO-VF: a verbalizer-free baseline approach, where the hidden state of the [MASK] token is fed into two fully connected layers with a final sigmoid activation, as in Equation (5); • PRONTO-LIN: a naive linear direct-mapping approach, based on Equation (3); • PRONTO-WS: a direct-mapping approach where logits are re-weighted before the Softmax as in Equation (4), and the final label probability is obtained as in Equation (3); • PRONTO-MAV: a mapping-free approach where logits are fed into two fully connected layers as in Equation (5).
The direct-mapping verbalizers (PRONTO-WS, PRONTO-LIN) are inherently interpretable, since each can measure the contribution of the -th token for the final label prediction. On the other side, PRONTO-MAV and PRONTO-VF can give an indication on more subtle patterns in the prediction heads that can only be acquired by means of non-linearities. Data Preparation. Given a semantic containment graph = ⟨ ∪ , ∪ ⟩, we denote Π + as the set of all the pairs of nodes (, ) that can be found along a path of . In other (4) (5) words, we compute the transitive closure of each node in G.Since does not contain negative information, this leaves an important decision: how to extract useful negative pairs. This decision is crucial since it impacts both the eficacy and generalizability of our learned verbalizers and the reliability of our evaluation. Intuitively, we want our model to be capable of distinguishing between semantically similar classes, although disjoint ones (e.g., "city"/"region"). However, we want it to be also able to distinguish among completely unrelated classes (e.g. "city"/"person"). Furthermore, we want it to correctly model a semantic containment relationship that is noncommutative, instead of just discriminating based on word similarity. We devise three strategies to build the set Π − of negative samples: • Reverse negatives: given a positive pair (, ) we obtain a negative pair by inverting subject and object (, ); • Soft negatives: given a positive pair (, ), we replace with a random class sampled based on the class distribution in the data; • Hard negatives: given a positive pair (, ) ∈ Π +, we build the two sets +(, ) = { | (, ) ∈ Π +} and ˆ+(, ) = {ˆ | (ˆ , ) ∈ Π + and ˆ ̸∈ +(, ) and ∈ +(, )}. While +(, ) represents the set of nodes along a path starting from in the original graph , namely all the nodes in a semantic containment relation with , the set ˆ+(, ) contains the nodes on the paths arriving in +(, ). These nodes are not in a semantic containment relation with but are semantically "close" to it. Given a node , the hard negatives are then built as (, ˆ ) with ˆ ∈ ˆ+(, ).
Prompt Construction. Prior work has demonstrated the sensitivity of PLM outputs to prompt
selection [
This section outlines the experimental setup to probe the ability of PLMs to understand
ontological containment relationships. We specifically focus on evaluating the inherent capacity
of vanilla pre-trained MLMs prediction heads to recognize the hierarchical relation between
instances and classes. The experiments are structured around three core research questions:
RQ1: Do Masked Language Models (MLMs) capture semantic containment?
RQ2: How does contextual information influence MLM in semantic containment prediction
tasks?
RQ3: Can MLMs generalize their semantic containment reasoning abilities to new data and
tasks?
Dataset. We base our study on the dataset introduced by Wu et al. [
Probed PLMs. It is worth noticing that the proposed probing procedure is versatile and can be
readily applied to any bidirectional PLM with mask-filling capabilities. For this investigation,
we focus on two prominent encoder-only PLMs,1 BERT [
To evaluate the efectiveness of the probed PLMs in identifying semantic containment relationships, we analyze the performance of various combinations of verbalization strategies, templates, and PLMs (the interested reader may take a look to Section 2 for further details). We report the results in Table 2, presenting accuracy, precision, recall, and F1-score for each combination. A decision threshold of 0.5 was used for all models.
PLM Comparison. The analysis reveals several interesting trends. The first finding is that the verbalization strategy matters. The Mapping-Free Automatic Verbalizer (MAV) consistently outperforms those based on direct mapping (LIN and WS). This suggests token probabilities likely contain complex relationships that direct mapping approaches might miss. The MAV strategy seems to capture these more efectively. The RoBERTa-Large model generally achieves better and more consistent results, particularly with direct-mapping verbalizations (LIN and WS). For the MAV verbalizer, RoBERTa-Base outperforms the larger model with specific template choices (h_4, h_2, s_1, and s_2). This suggests that prompt design plays a crucial role in performance, even for larger models. There is no clear correlation between model size and the 1For all the adopted PLMs, we employ the pre-trained checkpoints available at https://huggingface.co/.
PLMs’ discriminative ability to distinguish containment relationships. MAV verbalizers show
similar performance across model sizes while direct-mapping variants tend to improve with
larger models. We hypothesize that smaller PLMs may exhibit more nuanced activation patterns
for containment, requiring a non-linear verbalizer like MAV to capture them. This result is in
line with previous works that reached conflicting conclusions on this matter: indeed, Petroni et
al. [
Figure 2 shows the Area Under the ROC Curve (AUC) scores for all verbalizer-prompt combinations using the RoBERTa-Large PLM. These scores reflect the model’s ability to distinguish between positive and negative containment pairs. While overall performance varies with prompt choice for the same verbalizer, the results indicate some general trends. While PRONTO-LIN achieves the lowest accuracy and F1 scores for hard prompts, it exhibits good AUC scores, particularly for the h_1 and h_2 prompts. This suggests that PRONTO-LIN might benefit from optimizing the decision threshold used to classify positive and negative pairs. A potential explanation is in its underlying architecture. Indeed, PRONTO-LIN computes the label probability as a linear sum of individual token probabilities. These token probabilities can be noisy and potentially influenced by irrelevant factors, especially as vocabulary size increases. However, adjusting the decision threshold could help mitigate the impact of this noise and potentially improve PRONTO-LIN’s performance. The interested reader may find an additional comparison with GPT-3.5 turbo in the extended version of this paper.
Additional analyses. In the extended version of this paper, the interested reader may find the experiments regarding the sensitivity to the relative positioning of instances and classes to determine if the models’ predictions were based on memorizing word co-occurrences rather than understanding the underlying meaning of containment relationships. The evaluation set consisted of positive and "reverse negative" examples, with the less specific concept appearing on the right-hand side of the prompt, and the verbalizers performed better on the reverse negative set. This suggests that the verbalizers could distinguish between the relative specificities of concepts, with models sensitive to the order in which concepts are presented. Moreover, we have performed an analysis of the PRONTO-WS verbalizer that revealed both interpretable and less intuitive top tokens, suggesting the model captures nuanced patterns beyond human comprehension. These findings support the idea that containment relationships are intricate and that the model uses a wide range of cues within the vocabulary, highlighting the need to explore the full vocabulary for developing efective verbalizers.
Building upon the learned verbalizers, we investigate the feasibility of leveraging textual descriptions from our knowledge graph (KG) to potentially improve their performance in addressing RQ2. This exploration is rooted in the hypothesis that enhancing our prompts with relevant context about the entities involved can reinforce the model’s understanding of the underlying semantic relationships and lead to better discrimination between positive and negative containment pairs.
To address RQ2, we reformulate the original containment prediction task as a textual
entailment task [
Table 3 presents the final results on the Eval (hard) dataset after incorporating textual descriptions from the knowledge graph (KG). The results reveal an interesting trend. Contrary to expectations, adding context generally leads to a decrease in performance across most verbalizerprompt combinations. This suggests that the KG descriptions might be introducing noise rather than providing beneficial information. This negative impact can be attributed to architectural factors. The prediction heads of the PLMs used may be sensitive to variations in input data, struggling to integrate the additional context efectively. Moreover, the verbalizers themselves might be susceptible to changes in the input, hindering their ability to leverage the supplementary information. Interestingly, direct-mapping verbalizers (like PRONTO-LIN) are less afected by the inclusion of context, showing improvements for specific prompts (h_3, h_4, h_5). This experiment highlights the need for further investigation into effective strategies for incorporating contextual information from knowledge graphs.
Generalizability to Unseen Instances. To address RQ3, we examine how training data size afects verbalizers’ generalizability to unseen entities, simulating ontology completion. We adopt an inductive setting, where the model predicts relationships for unseen entities. We modify our training data by removing all training pairs containing randomly selected entities from 80% of the Eval (hard) dataset. We retrain verbalizers on this split and report results in Table 4. Reducing training size negatively impacts performance across metrics, though not substantially. Interestingly, PRONTO-MAV outperforms its full-dataset counterpart in F1 score and accuracy with the ℎ4 prompt, showing strong generalization.
Zero-shot Entity Typing. We evaluate the generalizability of verbalizers through a zero-shot entity typing task, assigning an entity type to a mention based on its context. Reformulating this as a textual entailment task, we construct a cloze prompt for each type in and select the one with the highest probability. For experiments, we use the Few-NERD dataset [17], a manually annotated NER dataset with fine- and coarse-grained tags. Due to type overlaps (e.g., "Living Thing" under "Person"), we focus on well-defined, disjoint categories: Person (7 types), Location (6 types), and Organization (9 types), excluding ambiguous types like MISC. PRONTO-MAV, our top-performing model, is used without additional training, leveraging the verbalizer from our containment prediction task.
This study investigated the ability of pre-trained Masked Language Models (MLMs) to understand hierarchical semantic relationships. The findings suggest that MLMs exhibit some grasp of ontological containment, as evidenced by consistent patterns in the prediction heads. We explored the generalizability of this approach, including learning specific verbalizers, inductive containment prediction, and zero-shot entity typing. While non-linear verbalizers showed remarkable performance, there is room for further exploration on developing more advanced verbalization strategies to better integrate textual information with structured ontological frameworks.
The authors acknowledge partial support of the following projects: OVS: Fashion Retail Reloaded and ePansa. We acknowledge the CINECA award under the ISCRA initiative for the availability of high-performance computing resources and support.
The authors have not employed any Generative AI tools. G. Stoilos, L. Hollink, Z. Kaoudi, G. Cheng, J. Li (Eds.), The Semantic Web – ISWC 2023, Springer Nature Switzerland, Cham, 2023, pp. 80–100. [17] N. Ding, G. Xu, Y. Chen, X. Wang, X. Han, P. Xie, H. Zheng, Z. Liu, Few-NERD: A few-shot named entity recognition dataset, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 3198–3213. URL: https://aclanthology.org/2021.acl-long.248. doi:10.18653/v1/2021.acl-long.248.