The phenomenon of multimodality is currently gaining increased attention across diverse fields of study concerned with communication, interaction and meaning-making, which range from linguistics and semiotics to artificial intelligence research and cognitive science. However, just how multimodality is understood and conceptualised is largely dependent on the discipline in question. Whereas AI research often adopts a sensory perspective to multimodality, in which the definitions of 'modalities' are based on the senses, semiotic approaches to multimodality generally work with more fine-grained definitions that seek to capture how diferent modalities co-operate with each other. In this presentation, I explore how semiotic approaches to multimodality could be used to inform computational models of multimodal discourse, particularly in terms of identifying meaningful units of analysis and describing their interrelations.
1. Short Biography Tuomo Hiippala is Associate Professor in English Language and Digital Humanities in the Department of Languages at the University of Helsinki, Finland (2018-). He holds a PhD in English Philology from the University of Helsinki (2014) and a Title of Docent in Multimodality Research and Digital Methods from the University of Turku (2022). His current research interests include the multimodality of diagrams and information graphics, and the development of computational methods for building and analysing multimodal corpora. He currently directs two research projects that explore the use of crowdsourcing as a method for creating multimodal corpora and the role of crowdsourcing as a form of academic infrastructure.