Dependency-based Semantic Role Labeling (SRL) is bound to dependency parsing, as the arguments of a predicate are identified through the token that heads the dependency relation subtree of the argument span. However, dependency-based SRL corpora are susceptible to the dissociated nucleus problem: when a subclause's semantic and structural cores are two separate words, the dependency tree chooses the structural token as the head of the subtree, coercing the SRL annotation into making the same choice. This leads to undesirable consequences: when directly using the output of a dependency-based SRL method in downstream tasks it is useful to work with the token representing the semantic core of a subclause, not the structural core. In this paper, we carry out a linguistically-driven investigation on the dissociated nucleus problem in dependency-based SRL and propose a novel algorithm that aligns predicate-argument structures to the syntactic structures from Universal Dependencies to select the semantic core of an argument. Our analysis shows that dissociated nuclei appear more often than one might expect, and that our novel algorithm greatly increases the richness of the semantic information in dependency-based SRL. We release the software to reproduce our experiments at https://github.com/SapienzaNLP/semdepalign.
Algorithm 1: SemDepAlign input: the role node role_node; the root node of the UD dep-tree root_ud. output: the head node of the role in the UD dep-tree.
Both SRL and Dependency Parsing investigate how
words in the same sentence relate to each other,
respectively in a semantic or syntactic sense. The Conference
on Computational Natural Language Learning (CoNLL) role_tokens ← get _tokens(role_node)
organized several Shared Tasks regarding both tasks, cul- ud_role_subtree ← root_ud
minating in the CoNLL-2008 Shared Task [
The dependency relation annotation scheme adopted 3. Re-associating Dissociated
in both CoNLL-2008 and its multilingual successor Nuclei
CoNLL-2009 [
A way to quantify this phenomenon is to look at the nucleus phenomenon. We leverage this characteristic of frequency of part-of-speech (POS) tags of role tokens UD parsers to automatically annotate the whole CoNLLin the corpus. We are interested in the POS label of 2009 corpus using trankit [16], which emerges as the “Preposition or subordinating conjunction”, which is the strongest UD parser in the comparison we include in second-most frequent tag with 76,821 role tokens out of Appendix B. a total of 475,069, or ~17% of all the role tokens. Table 5 in the Appendix provides a complete breakdown over all 3.1. SemDepAlign: subtree alignment POS classes in the English-split of CoNLL-2009.
We argue that both the training corpora and dependency-based SRL systems should identify the semantic core of an argument span as the head of the argument. In Appendix A we provide further examples of this phenomenon in non-English partitions of CoNLL-2009.
tactic parse semi-alignment from the dependency annotations in CoNLL-2009 to UD, described in Algorithm 1. SemDepAlign is a deterministic subtree aligning algorithm that, for each role token associated with a predicate, finds the UD subtree that most closely matches the original subtree headed by in the original dependency tree of CoNLL-2009. It then returns the head node ′ of AM-LOC
At last night ’s
rally
AM-LOC pmod nmod sufix
loc nmod amod case case nmod:poss
A0 they A0 obl
root sbj
loc call.05
called call.05 nsubj root oprd pmod
nmod A1 on their followers
A1 nmod obl case xcomp
A2 to im
prd be firm
A2 cop mark
dissociated nucleus phenomenon, obtaining the Aligned- Although we demonstrate that re-associated nuclei in CoNLL2009 dataset. After the application of SemDe- dependency-based SRL provide additional semantic inpAlign, the number of role token annotations that are formation, an important research question is whether modified is considerable over all CoNLL-2009 languages integrating our proposal into current systems can lead (between 21% and 32% of the total roles), except for Czech to a change in performance. Therefore, we build on top (~7%). of the strong SRL model proposed by Conia and Navigli
To gain a better understanding of the diferences that [18] and design a new approach that jointly learns both the alignment process introduces, we consider the an- types of role annotations, i.e. the original role tokens and notations of the original tokens that are modified by the UD subtree, which will be assigned the role label in the aligned SRL annotation.
As shown in Algorithm 1, SemDepAlign starts from the UD root node (root_ud), loops over the nodes of the tree through a breadth-first search (BFS), and finds the node which heads the subtree with minimal symmetric set diference ( SymDif ) between its tokens and the set of tokens in the original role span (role_tokens). The symmetric diference between two sets of tokens 1 and 2 is defined through the set operations diference (∖) and union (∪) like so: (1∖2) ∪ (2∖1). Intuitively, if the symmetric diference between the original and the UD subtree is the empty set, they match exactly and we can simply select the head of the UD subtree as the role token.
Otherwise, selecting the head of the UD subtree with the minimal symmetric diference compared to the original subtree is equivalent to selecting the subtree with the most overlap with the original span.
Figure 1 gives an example of the output of SemDepAlign: at the top of the figure we display the original annotation of the sentence derived from the English split of CoNLL-2009, with the presence of a dissociated nucleus in three of the four roles for the predicate “called”; in the bottom part we show the output of our alignment procedure, which moves the role annotations to the tokens that perform the semantic function.
SemDepAlign and the resulting aligned role tokens. We measure three metrics on these two sets to evaluate their semantic richness: • Number of content words, i.e. words that are either nouns, adjectives, adverbs, or verbs, which indicates that the heads identified by SemDepAlign are more varied (2713 vs. 680 for English, 3.99× ); • Number of unique tokens, which indicates that the heads identified by SemDepAlign are less repetitive (1906 vs. 477, 4× ); • Number of unique synsets, which indicates that the heads identified by SemDepAlign are associated with diferent meanings (1387 vs. 481, 2.88 × ) according to a Word Sense Disambiguation system [17].
increases the semantic content of role tokens in English, Spanish and German, identifying more than 4× the number of content words, more than 2.5× the number of unique tokens and around 3× the number of unique synsets compared to the original annotations. We find a smaller but consistent increase of semantic content in Catalan and Chinese, whilst in Czech all metrics are similar, indicating a reduced efect of SemDepAlign.
# modified roles Content words Unique tokens Unique synsets
Catalan O A the aligned ones. In brief, this architecture derives a con- unchanged. We conduct our experiments on all of the textualized word representation for each word in a sen- language splits of CoNLL-2009, namely, Catalan, Czech, tence from a BERT-like Pretrained Language Model [19, German, English, Spanish, and Chinese. PLM]. It then applies a custom “fully-connected” stackedBiLSTM sequence encoder to derive a predicate-aware Results Table 2 compares the results of our jointrepresentation, which is in turn used to derive a predicate- modeling alignment system against our baseline on the and argument-specific embedding for each word in the CoNLL-2009 validation and test sets. Importantly, we sentence. Finally, an argument-specific fully-connected observe that the additional task of modeling the semantic BiLSTM is applied to further encode each word with re- core of an argument does not significantly alter the perspect to a specific predicate, from which it derives the formance (very similar F1 score on the test), despite the ifnal score distribution over the role vocabulary through a added dificulty brought by the identification of semansimple linear classifier. The model is trained to minimize tic cores. Table 3, instead, provides a breakdown of the the sum of categorical cross-entropy losses on predicate F1 scores on predicate, role and aligned role predictions. identification, predicate disambiguation and argument The aligned system is in line with the baseline despite identification and classification. being tasked with a more complex objective. More inter
To adapt this model for our joint modeling task, we estingly, we observe that the F1 score on the semantic duplicate the linear classifier for the semantic roles and heads is comparable, indicating that the model is able to set two diferent targets for the two role classifiers: the identify UD-aligned roles efectively. original role token and label from CoNLL-2009 and the aligned role token and label obtained with SemDepAlign.
Our final loss adds terms for UD-aligned argument iden- 5. Semantic roles in AMR graphs tification and classification to the original loss.
Test dataset LORELEI Weblog and WSJ Xinhua MT BOLT DF MT BOLT DF English Proxy reports Average
tant for recognizing semantic frames in SRL. Marcheggiani and Titov [24] were among the first to model the dependency information provided in dependency-based SRL, followed most recently by Xia et al. [25], Fei et al. [26]. These works difer in respect of modeling choices and in the kind of extra syntactic data to be included (e.g. constituency trees, POS tags).
We also considered other syntactic frameworks, such as HPSG [27], to align the role annotations. HPSG robustly models the relationship between semantic cores of a sentence, but the lack of automatic tools with an acceptable performance and the dificulty in aligning dependency-based subtrees to HPSG spans compelled us to use UD. tasks aim to construct a semantic representation of a sentence, although SRL, covering only surface-level semantic frames, is more superficial than AMR, which aims to provide a more complete and in-depth structured repre- 7. Conclusion sentation that can interconnect diferent semantic frames.
Given that AMR aims to abstract away from the specific In this paper, we conducted an in-depth investigation syntax of a sentence to focus only on its semantic content, on the dissociated nucleus issue in dependency-based our intuition is that a dependency-based SRL system is SRL. We introduced SemDepAlign, a novel method to more “semantic” if its predictions of predicate-role pairs align predicate-argument structures in SRL with synare contained in the AMR annotation for the same sen- tactic parses from the Universal Dependencies project, tence. which addresses the dissociated nucleus phenomenon.
Therefore, we devise the AMR-precision metric: Our analyses and experiments in SRL modeling demongiven a sentence , its golden annotated AMR graph strate that our approach to dissociated nuclei brings more AMR with token-node alignments available and a set semantic richness whilst remaining competitive on stanof dependency-based SRL predictions, we filter the pre- dard benchmarks. dicted semantic frames so that the predicate of each frame is present in the golden AMR graph. We then compute 8. Limitations the ratio between the number of role tokens that are connected to their predicate in the AMR graph over the total number of roles predicted.
Given the SRL system introduced in Section 4, we apply it to the AMR3.0 (LDC2020T021) test datasets, keeping both the standard and the aligned role predictions. We then compute the AMR-precision for both sets of predicted roles, and compare them in Table 4. It is clear that aligned roles are more likely to be present in the corresponding AMR graph of a sentence, with a consistent diference in AMR-precision in all test datasets except Proxy reports. This particular dataset has a “templatic, report-like structure” as mentioned in the AMR3.0 guidelines, so it is possible that the reduced performance is due to this particular characteristic.
This finding can pave the way for future work exploring the linkage between these two fundamental semantic tasks, as also suggested in the multi-layer annotation provided in MOSAICo [23].
dependency parsers trained on Universal Dependencies. These parsers have reached high robustness across many languages, between 85 and 93 in Labeled Attachment Score (LAS) on the languages present in CoNLL-2009. But the error that these automatic methods necessarily encounter propagates directly to our alignment algorithm, with no way of recovering from the mistake. This limitation would be even more impactful in languages where the automatic dependency parser performed worse, presumably in low-resource settings, preventing a robust expansion of our work to these settings.
A more methodological limitation of our contributions concerns the availability of the CoNLL-2009 dataset. Although it is a well-established corpus in the SRL literature, it has a proprietary licensing scheme and one must acquire the resource from the Linguistic Data Consortium (LDC). We trust that, given the importance of the corpus, this will not limit the relevance of our work.
port of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020). International Conference on Language Resources 747. doi:10.18653/v1/2020.acl-main.747. and Evaluation (LREC’14), European Language Re- [21] L. Chen, P. Wang, R. Xu, T. Liu, Z. Sui, B. Chang, sources Association (ELRA), Reykjavik, Iceland, ATP: AMRize then parse! enhancing AMR parsing 2014, pp. 4585–4592. URL: http://www.lrec-conf. with PseudoAMRs, in: M. Carpuat, M.-C. de Marnorg/proceedings/lrec2014/pdf/1062_Paper.pdf . efe, I. V. Meza Ruiz (Eds.), Findings of the Associa[14] M.-C. de Marnefe, C. D. Manning, J. Nivre, tion for Computational Linguistics: NAACL 2022, D. Zeman, Universal Dependencies, Com- Association for Computational Linguistics, Seatputational Linguistics 47 (2021) 255–308. tle, United States, 2022, pp. 2482–2496. URL: https: URL: https://aclanthology.org/2021.cl-2.11. //aclanthology.org/2022.findings-naacl.190. doi: 10. doi:10.1162/coli_a_00402. 18653/v1/2022.findings-naacl.190. [15] J. Grimshaw, Argument Structure, The MIT Press, [22] R. Navigli, Natural Language Understanding: In
Cambridge, MA, 1990. structions for (Present and Future) Use, in: Proceed[16] M. V. Nguyen, V. Lai, A. P. B. Veyseh, T. H. Nguyen, ings of the Twenty-Seventh International Joint ConTrankit: A light-weight transformer-based toolkit ference on Artificial Intelligence, IJCAI-18, Internafor multilingual natural language processing, in: tional Joint Conferences on Artificial Intelligence Proceedings of the 16th Conference of the Euro- Organization, 2018, pp. 5697–5702. URL: https://doi. pean Chapter of the Association for Computational org/10.24963/ijcai.2018/812. doi:10.24963/ijcai.
Linguistics: System Demonstrations, 2021. 2018/812. [17] R. Orlando, S. Conia, F. Brignone, F. Cecconi, [23] S. Conia, E. Barba, A. C. Martinez Lorenzo, P.-L.
R. Navigli, AMuSE-WSD: An all-in-one mul- Huguet Cabot, R. Orlando, L. Procopio, R. Navtilingual system for easy Word Sense Disam- igli, MOSAICo: a multilingual open-text semanbiguation, in: Proceedings of the 2021 Confer- tically annotated interlinked corpus, in: K. Duh, ence on Empirical Methods in Natural Language H. Gomez, S. Bethard (Eds.), Proceedings of the Processing: System Demonstrations, Association 2024 Conference of the North American Chapter for Computational Linguistics, Online and Punta of the Association for Computational Linguistics: Cana, Dominican Republic, 2021, pp. 298–307. Human Language Technologies (Volume 1: Long URL: https://aclanthology.org/2021.emnlp-demo.34. Papers), Association for Computational Linguisdoi:10.18653/v1/2021.emnlp-demo.34. tics, Mexico City, Mexico, 2024, pp. 7990–8004. [18] S. Conia, R. Navigli, Bridging the gap in URL: https://aclanthology.org/2024.naacl-long.442. multilingual semantic role labeling: a language- doi:10.18653/v1/2024.naacl-long.442. agnostic approach, in: Proceedings of the [24] D. Marcheggiani, I. Titov, Encoding sentences with 28th International Conference on Computa- graph convolutional networks for semantic role lational Linguistics, International Committee beling, in: Proceedings of the 2017 Conference on Computational Linguistics, Barcelona, on Empirical Methods in Natural Language ProSpain (Online), 2020, pp. 1396–1410. URL: cessing, Association for Computational Linguistics, https://aclanthology.org/2020.coling-main.120. Copenhagen, Denmark, 2017, pp. 1506–1515. URL: doi:10.18653/v1/2020.coling-main.120. https://aclanthology.org/D17-1159. doi:10.18653/ [19] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: v1/D17-1159.
Pre-training of deep bidirectional transformers for [25] Q. Xia, R. Wang, Z. Li, Y. Zhang, M. Zhang, language understanding, in: Proceedings of the Semantic role labeling with heterogeneous 2019 Conference of the North American Chap- syntactic knowledge, in: Proceedings of the ter of the Association for Computational Linguis- 28th International Conference on Computatics: Human Language Technologies, Volume 1 tional Linguistics, International Committee (Long and Short Papers), Association for Com- on Computational Linguistics, Barcelona, putational Linguistics, Minneapolis, Minnesota, Spain (Online), 2020, pp. 2979–2990. URL: 2019, pp. 4171–4186. URL: https://aclanthology.org/ https://aclanthology.org/2020.coling-main.266.
N19-1423. doi:10.18653/v1/N19-1423. doi:10.18653/v1/2020.coling-main.266. [20] A. Conneau, K. Khandelwal, N. Goyal, V. Chaud- [26] H. Fei, S. Wu, Y. Ren, F. Li, D. Ji, Better comhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, bine them together! integrating syntactic conL. Zettlemoyer, V. Stoyanov, Unsupervised cross- stituency and dependency representations for selingual representation learning at scale, in: Pro- mantic role labeling, in: Findings of the Association ceedings of the 58th Annual Meeting of the Associa- for Computational Linguistics: ACL-IJCNLP 2021, tion for Computational Linguistics, Association for Association for Computational Linguistics, OnComputational Linguistics, Online, 2020, pp. 8440– line, 2021, pp. 549–559. URL: https://aclanthology. 8451. URL: https://aclanthology.org/2020.acl-main. org/2021.findings-acl.49. doi: 10.18653/v1/2021.
findings-acl.49. Original sentence: [27] C. Pollard, I. Sag, Head-Driven Phrase Structure “Don Antonio se encontraba en su casa cuando sonó el Grammar, Studies in Contemporary Linguistics, timbre de la puerta.” University of Chicago Press, 1994. URL: https:// Translation: books.google.it/books?id=Ftvg8Vo3QHwC. “Don Antonio was at his home when the doorbell rang.” [28] M. Straka, UDPipe 2.0 prototype at CoNLL 2018 Dissociated nucleus:
UD shared task, in: Proceedings of the CoNLL 2018 The role “en su casa” (“at his home”) for predicate ‘enconShared Task: Multilingual Parsing from Raw Text traba’ (‘was’) is tagged as arg2-loc on the token ‘en’ to Universal Dependencies, Association for Com- (‘in’) instead of the semantic nucleus ‘casa’ (‘home’). putational Linguistics, Brussels, Belgium, 2018, pp. 197–207. URL: https://aclanthology.org/K18-2020. A.4. Chinese doi:10.18653/v1/K18-2020. [29] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Man- Original sentence: ning, Stanza: A Python natural language processing 巴拉克 在 民意 测验 中 一直 表现 不 佳 。 toolkit for many human languages, in: Proceed- Transliteration: ings of the 58th Annual Meeting of the Associa- “Barak in public opinion test in continuously performance tion for Computational Linguistics: System Demon- no good.” strations, 2020. URL: https://nlp.stanford.edu/pubs/ Translation: qi2020stanza.pdf . “Barak has consistently underperformed in the polls.” Dissociated nucleus: In the clause 在 民意 测验 (“in the public opinion polls”) for the nominal predicate 佳 (‘good’), as the token 在 (‘in’) is tagged as the LOC role, instead of the more semantic 测验 (‘polls’).
“Piqué recomana les fusions entre empreses per millorar la rendibilitat.” Translation: “Piqué recommends mergers between companies to improve profitability.” Dissociated nucleus: In the clause “per millorar” (“to improve”), ‘per’ (‘to’) is tagged as argM-fin for predicate ‘recomana’ (‘recommends’) instead of the head of the subclause ‘millorar’ (‘improve’).
dency parsers, namely, trankit [16], UDPipe [28] and Stanza [29]. Table 6 compares the reported evaluation of each parser on standard treebanks for Catalan, Czech, German, English, Spanish and Chinese. We choose trankit as it achieves a higher UAS and LAS than the two alternatives in all languages except Spanish (slightly worse than UDPipe), with a considerable margin in Chinese. Catalan AnCora
Czech PDT German GSD English EWT Spanish AnCora
Chinese Simplified GSD
Average
System