In this paper, we discuss the impact of Deep Learning (DL) techniques in the present and future of the interactive system engineering. On the one hand, the support for more complex vocabularies o ers opportunities in better shaping the communication between the user and the system. On the other hand, we identify challenges related to the lack of transparency and explainability in the trained models, which have a negative impact on system understanding for both developers and users.
In the later years, Deep Learning techniques provided the research community
with robust solutions for challenging problems in di erent elds, such as
object and speech recognition, text generation and analysis, shape understanding
etc. [
DL techniques contributed to x many usability problems, for instance in the
eld of speech recognition [
Using a pre-trained model for classifying or analysing data is an example of a black-box component. Indeed, many highly accurate and robust pre-trained models are available for di erent tasks, and they relieve the development team from the burden of creating their own Neural Network model. However, a pretrained model does not support the inspection by developers or by end-users.
This may cause unwanted behaviours, for instance when the considered application uses frequently a rare word (e.g., a jargon word in speech recognition) or by a speci c user.
Another common case is having a model developed or trained by the same team that creates the application. Developers have more control over the input distribution and they may ne-tune the model on the application at hand. However, the learnt function remains di cult to understand for developers, and this creates di culties in tracking unwanted behaviours. The problem is even worse if we consider approaches based on reinforcement learning: the system learns guided by human feedback (reward), which may be completely out of the engineer's control.
In this paper, we will refer to the interaction framework proposed by Abowd
and Beale [
In this section we summarise the main concepts introduced in the interaction
framework by Abowd and Beale [
The interaction process requires four translations between these languages. Each translation corresponds to one of the four phases of an interaction cycle (articulation, performance, presentation, observation ), which we can summarise as follows. The User formulates her goals in the task language and articulates them in the input language. The interface translates such information into operations to be performed by the System. After the computation, the System changes its state and it presents such information using concepts or features of the output language. Finally, the User observes such output and assesses the result according to her original goal.
In this paper, we analyse the impact of exploiting Deep Learning techniques in each one of these phases. 3
In this section, we will analyse the engineering challenges and opportunities in the articulation and performance phases.
On the one hand, we are positive that Deep Learning nears the expressive gap between the Input and the Task language. Indeed, DL is the technology that grounds the communication between users and the system using natural language, hand or body gestures or a combination of modalities. This is an important opportunity that is currently fostering the development of conversational interfaces, guided both by speech recognition and/or natural language processing.
On the other hand, an increase in the input language expressiveness increases also its complexity and, consequently, it causes challenges in the performance phase (see Figure 1). Higher complexity means a larger vocabulary for expressing the user's intent, and higher complexity in the translation to the core language. Deep Learning may help again in this task (e.g. in Natural Language Understanding), but this reduces the developer control over the overall process and the interface ability to provide intermediate feedback.
The following sections provide two sample modalities that had a di erent level of success in applying Deep Learning techniques for their intrinsic characteristics. 3.1
In the last years, the development of conversational interfaces such as
textbased chat-bots or vocal assistant applications (e.g., Alexa or Google Home)
dramatically decreased their complexity. Di erent services exist that provide
both speech recognition and the natural language interpretation. They require
an easy con guration if we take into account the complexity of natural language
itself. Instances of such services are Facebook Wit.ai [
A conversation naturally requires communication turns. Therefore, it is acceptable for both the users and the system to wait until the entire sentence is available before starting the processing phase. People do not need to learn which word or phrases are available in a system since we reached good language coverage. This allows using libraries based on DL for both speech recognition and interpretation. The Core receives a simpli ed representation of the user's commands including part of speech tagging, handling synonyms etc.
In this eld, the DL techniques t particularly well, since the pipeline does not require any intermediate guidance and turns are strictly de ned. Applying a black box approach is acceptable for both developers and users as long as the accuracy and the robustness to the input uctuation of the model is good enough. In addition, using a pre-trained model does not require sharing the training set, so the big data owners such as Google, Amazon and Facebook provide APIs and software components that exploit them, increasing the overall usability of the applications built on top of such components. 3.2
Deep Learning techniques do not have the same seamless integration in gestural interaction. First of all, in this eld, the tracking sensors limit the interaction vocabulary, if we compare it against the natural expressiveness of the human body. Therefore, users need guidance for discovering which movements the system is able to interpret and how to perform them.
This requires an accurate design for both the feedback and feedforward
components in a gestural interface. They respectively support the users in
understanding the e ect of their previous actions (feedback) and to foresee the e ect
of future actions before performing them (feedforward). The design space for
both components has been widely investigated in the literature, (e.g., by Luyten
et al. [
However, how to map the structure of highly-accurate classi er into a gesture
description language is an open challenge. Solutions for speci c techniques exist
(e.g. Hidden Markov Models [
Finally, the lack of a common vocabulary and of a clear winner among the tracking sensors hinders the availability of big training datasets for reaching the same interface design exibility we have for natural language. In general, datasets are tailored for the interface at hand and, consequently, their size is much smaller. 4
While the Input elements may employ Deep Learning techniques for supporting the user in controlling the application, easing the articulation and the performance phase, the System and the Output components may exploit DL for performing computations on data or for presenting the results to the user. 4.1
In the System component, Deep Learning techniques help in solving highlydimensional problems, approximating functions that are di cult to de ne through the training data. DL provides a robust solution to di cult problems, but it creates challenges when the user requires information on the computation procedure. This a ects the observation phase, where the user assesses the results according to her goals. Without proper explanations, users may perceive the results as unreliable. However, the lack of information lays on the System component rather than on the Output since usually DL models do not support straightforward ways for creating labelling explanations.
A typical example is supporting transparency and explainability in
Recommender Systems. The user may increase her trust in the system when it provides
a list of suggested items together with the explanation of why the system
considers them useful [
As already discussed for gesture interfaces, such fragmentation into domainspeci c solutions requires also domain-speci c and high quality data. This narrows the applicability of the techniques to datasets available for the research community and the general public. In other domains, it is a complete prerogative of big data owners. 4.2
At the Output component, Deep Learning techniques may be employed for generating human understandable representations of the internal system state, or to provide information on how to perform speci c tasks through the interface.
Such advice remains underinvestigated in the literature, to the best of our knowledge. In our opinion, it represents an opportunity for lowering the barrier for systems requiring a relevant initial learning phase: DL techniques may be employed for acquiring knowledge on the di culties in the observation phase (for instance analysing help tickets) and generating suggestions when the problems are detected.
Such a process may be enhanced exploiting explicit models representing the
output language. We plan to investigate this in the End User Development [
techniques for supporting both goal-oriented suggestions and question answering, tailoring the answers in the context of the developed game, as described by its rules.
In this paper, we presented a set of challenges and opportunities in engineering
interactive systems, analysing the integration of Deep Learning techniques into
the Abowd and Beale interaction framework [