Modeling and Evaluation for Perspectivist NLP
                         Valerio Basile1
                         1
                             University of Turin, Italy




                         1. Abstract
                         A recent line of research in NLP proposes to never aggregate human annotations1 , but rather to leverage
                         the worth of knowledge found in label variation for building models [1] and evaluating them [2]. This
                         approach is particularly relevant when dealing with highly subjective aspects of natural language such
                         as irony or undesirable language. In this talk, I will present the perspectivist paradigm in NLP, and the
                         results of recent and ongoing research focusing on building perspective-aware predictive models and
                         automatically extract human perspectives from annotated data. Particular emphasis will be given to
                         language resources, for training or few-shotting models, but also for benchmarking in the perspectivist
                         framework.


                         2. Short Biography
                         Valerio Basile is an Assistant Professor at the Computer Science Department of the University of Turin,
                         Italy, member of the Content-centered Computing group and the Hate Speech Monitoring group. His
                         work spans across several areas such as: formal representations of meaning, linguistic annotation,
                         natural language generation, commonsense knowledge, semantic parsing, sentiment analysis, and hate
                         speech detection, perspectives and bias in supervised machine learning, from data creation to system
                         evaluation. He is currently PI of the project BREAKhateDOWN “Toxic Language Understanding in
                         Online Communication”, and among the main proponents of the Perspectivist Data Manifesto.


                         References
                         [1] F. Cabitza, A. Campagner, V. Basile, Toward a perspectivist turn in ground truthing for predictive
                             computing, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023, pp.
                             6860–6868.
                         [2] V. Basile, M. Fell, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio, A. Uma, et al., We need to
                             consider disagreement in evaluation, in: Proceedings of the 1st workshop on benchmarking: past,
                             present and future, Association for Computational Linguistics, 2021, pp. 15–21.




                          NL4AI 2024: Eighth Workshop on Natural Language for Artificial Intelligence, November 26-27th, 2024, Bolzano, Italy
                          $ valerio.basile@unito.it (V. Basile)
                                        © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
                         1
                             https://pdai.info/

CEUR
                  ceur-ws.org
Workshop      ISSN 1613-0073
Proceedings