The goal of this workshop is to discuss the potential benefits and the open challenges of giving a physical form to Artificial Intelligence (AI), towards the definition of tangible, or graspable AI. In the workshop we will focus on the use-case of a convolutional Neural Network for image analysis and we will carry out a hands-on paper prototyping activity to imagine tangible interactive AI-powered systems where critical parameters of the Neural Network are physicalized. Such systems have the potential to lower the barrier for AI education and for making AI systems more trustable and explainable.
Amershi et al. [
Tangible User Interfaces are good candidates to address these challenges. Indeed, Tangible interaction
proposes to use the affordances of physical objects to interact with data, exploiting metaphors and
interaction concepts borrowed from the physical world. Using physical objects for explaining and
interacting with AI is a very recent research trend. To the best of our knowledge, there have been only
a couple of workshops on this subject [
In our workshop, we would like to explore further such topics, opening new research pathways in collaboration with the workshop participants. The central question that would be tackled in the workshop is:
● Can we use physical representations and controls to make AI more explainable and trustable?
Therefore, the main goals of the workshop will be to:
● Explore how the principles of tangible interaction could increase trust and understandability of
deep learning algorithms
● Reflect on how such principles could be included in the design of future XAI systems
In the workshop, we will focus on the use case of image analysis with Convolutional Neural Networks
(CNNs). CNNs are a popular network topology for image classification, constituted by several layers
of convolutional filters and subsampling filters (e.g., max pooling). A typical representation of a CNN
is shown in Figure 1.
To better understand how CNNs work, Zeiler and Fergus [
The workshop will build on previous research carried out by the organizers in the field of the Internet
of Tangible Things [
30 minutes: Analysis of the main aspects that should be physicalized and the expected impact
on the users. Using IoTT cards as starting point of the reflection [
About 15 people attended our workshop. We formed three groups, ensuring the presence of at least
one knowledgeable person in the domain of machine learning and in particular of CNNs. To get a better
understanding of how CNNs work we used four online tools for the visualization of CNNs networks
[
Two groups explored the potential of tangible interfaces for representing the CNNs structure, enabling the possibility of manipulating physical artifacts to manipulate the architecture of the network.
The first group explored the metaphor of a transparent box (in contrast to the black box representation usually perceived with deep learning algorithms) to represent a layer of the network. In the box a transparent filter would overlap with an image in order to give the different output of the layer (Figure 3, left). Stacking more boxes would be equivalent to adding more layers to the network (Figure 3, center).
The second group used instead cardboards of different colors to represent different operations (i.e., convolution and max pooling). Stacking cardboards on a table was perceived as an intuitive interaction for building the different layers of a CNN (Figure 3, right).
Pulling a thread from a wool yarn was also identified as another tangible interaction metaphor for exploring and visualizing the features learned by a CNN.
The third group worked on an application of tangible XAI for the healthcare domain. In the context of CNN for breast cancer diagnosis, doctors may struggle in understanding, and trusting, the model predictions. In order to make informed decisions, workshop participants proposed an interactive machine learning system where the doctor can use playdough to cover parts of the x-ray images and see in real-time what would be the effect of ignoring that part on the algorithm prediction (Figure 4, left). One of the participants of group 3 presented also a system developed few years before (unpublished work) to explore the functioning of CNNs through a tangible tabletop installation (Figure 4, right).
Based on the ideas and paper prototypes presented by each group, we identified several insights that could inspire designers of tangible interfaces for explainable AI: - Tangible interaction could help understanding the structure and functioning of deep learning algorithms, facilitating group discussion and giving multiple controls for shared interaction. - Most representations of CNNs layers were done using paper or cardboard sheets. This may be due to the fact that the workshop focused on the example of image recognition. Nevertheless, participants found interacting with paper sheets intuitive and easy. - Tangible interaction encourages exploration, which could help to understand the functioning of deep learning algorithms
Leonardo Angelini is Assistant Professor at the School of Management of Fribourg (HES-SO) and affiliated to the HumanTech Institute (HES-SO) as postdoctoral researcher. Leonardo has a PhD in Computer Science and his thesis investigated gesture interaction with tangible objects in the context of smart environments. Leonardo has several years of experience in the field of Human-Computer Interaction and Artificial Intelligence, with a particular focus on application for seniors’ well-being, eHealth and automotive interfaces.
Nadine Couture is Professor in Computer Science at ESTIA Institute of Technology in Biarritz (France). Her research is conducted at ESTIA-Recherche, which she leads. Nadine is interested in Tangible Interaction, from the physical embodiment of data to interaction with the whole body, applied to Affective Computing and Mixed Reality. She develops research-enterprise links and, as such, she co-founded the PEPSS platform about Human Factors for Interactive Technology, she is involved in the Aerospace-Valley Cluster on data economy and artificial intelligence, and vice-president of the Aquitaine Centre for Information Technologies and Electronics.
Elena Mugellini is Professor at the University of Applied Sciences of Western Switzerland in Fribourg (HES-SO). She holds a Diploma (Bsc and Msc) in Telecommunications Engineering and a Ph.D. in Computer Sciences from University of Florence, Italy. Elena is the leader of HumanTech Institute (Technology for Human well-being, humantech.eia-fr.ch/). The institute aims at improving the quality of life and well-being of human beings thanks to the ingenious use of new technologies, in order to strengthen their abilities as individuals, as well as members of an increasingly dynamic, nomadic and globalized society. Her research expertise lies in Human Computer Interaction (tangible, multimodal and natural interaction, conversational interface and empathic interaction) and Intelligent Data Analysis (machine learning, artificial intelligence, human analytics).
Mira El Kamali is a phD candidate at the University of Fribourg. She is also a scientific collaborator at the University of Applied Sciences of Western Switzerland in Fribourg (HES-SO). Her main research interests are in human computer interaction, multimodal systems, multimodal conversational agents. She worked in a European project “Nestore” with 15 partners where she designed a virtual coach for older adults’ wellbeing. She is also currently working in an Innosuisse project that aims at building recommender systems for smart homes. She holds a master of Science in Computer Engineering where her master thesis was about activity recognition using machine learning.
Quentin Meteier is a PhD candidate at University of Fribourg. He is also a scientific collaborator at the University of Applied Sciences of Western Switzerland in Fribourg (HES-SO). He is working on conditionally automated driving and focuses his research on evaluating the physiological state of the driver using machine learning techniques.
The organizers have collaborated in the past to organize workshops in the field of Tangible Interaction, and in particular in the field of tangible interaction with the Internet of Things.