Hybrid Procedural Semantics for Visual Dialogue: An Interactive Web Demonstration

Abstract

Visual dialogue refers to the task in which a conversational agent needs to hold a meaningful and coherent conversation with a human interlocutor about a scene they observe. To tackle this task, we introduce a novel methodology that makes use of (i) a novel data structure, called the conversation memory, which holds information that is incrementally conveyed in the conversation and (ii) a hybrid procedural semantic representation that is grounded in both the visual input and the conversation memory. In this paper, we present a demonstration that showcases this novel methodology. In this demonstration, a user can interact with a visual dialogue agent and discuss an image of their choice. While the agent is answering questions, the user can follow the agent's reasoning process. Due to its explainable and interpretable nature, the novel methodology can be used in a wide range of application domains, especially when it is important that the system is human-interpretable. We believe that this novel methodology of hybrid procedural semantics combined with a conversation memory paves the way for building truly intelligent and explainable systems that are able to hold human-like conversations.

1. Introduction and background

The task of visual dialogue as introduced by [1]

requires an agent to correctly answer a series of questions about visual input. However, these questions are not independent from each other. In many cases, answering these questions involves resolving coreferences with respect to earlier dialogue turns. Compared to the task of visual question answering, the interdependent questions are an extra complexity inherent to the task of visual dialogue. Here, the answers do not only have to be grounded in the visual context, but also in the conversational context.

The CLEVR-Dialog dataset [ 2 ] was especially designed to be a diagnostic benchmark for the visual dialogue task. The dataset consists in dialogues discussing images from the CLEVR dataset [ 3 ]. In the course of a dialogue, an agent initially receives an image and a caption describing some parts of the contents of the image. Then, ten questions are respectively asked and answered. The goal for the agent is to answer each question correctly. An example dialogue about the image in the upper right corner of Figure 1 would be:

C: There is a green object in the middle.

Q: What is its shape? A: Sphere Q: And its size? A: Small Q: Is there an object to its left? A: Yes Q: How many other objects are in the image? A: 4

A visual dialogue agent must thus be capable on

the one hand of solving coreferences in the conversation, and on the other hand of grounding references in the image. Traditionally, the task of visual dialogue is tackled with neural network approaches.

For example, [ 1 ] introduced an encoder-decoder architecture with encoders that have late-fusion, hierarchical encoding and memory networks. Other approaches take a more explicit approach and use mechanisms that explicitly represent the dialogue history. For example, [ 4 ] and [ 5 ] make use of an associative memory to represent the previous questions with their corresponding attentions. In combination with this associative memory, [ 5 ] use a neural module networks architecture [ 6 ].

Taking inspiration from this last approach, we introduce a novel methodology, based on two conboth the conversational context and the visual con- demonstration is for the user on the one hand to text. To represent the history of a dialogue, the reflect on what it takes to hold a meaningful and coconversation memory keeps track of relevant infor- herent conversation, and on the other hand to gain a mation conveyed in the dialogue. To understand deep understanding of our innovative methodology a question correctly, the agent maps the question based on hybrid procedural semantics. onto a meaning representation in terms of procedural semantics [ 7, 8, 9 ]. This meaning representation consists of the conceptual operations that an agent 2. Methodology needs to execute in order to answer a question. Procedural semantics has been used successfully for the The novel methodology that we designed for solving task of visual question answering [10, 11, 12]. We visual dialogue tasks builds on two foundational have extended the procedural semantics that was ideas: the use of a conversation memory and a designed for the task of visual question answering hybrid procedural semantics that is grounded in [12] to include operations that cover the incremental both visual input and the conversation memory. nature of dialogues. Moreover, the meaning repre- The conversation memory is a data structure sentation is executed in a hybrid way, where some that keeps track of all relevant knowledge that is steps are executed symbolically (in particular opera- built up during the dialogue. It is composed of tions related to reasoning processes) and others are a number of turns, which represent the turns in executed subsymbolically (in particular operations the dialogue (i.e., the observation and the questionresponsible for perception). The hybrid method answer pairs). After each turn of the dialogue, contrasts with the approach of [ 5 ], who use a neural information is added to the conversation memory. module networks approach with queries that are Each turn in the conversation memory consists of executed subsymbolically. its timestep, the utterance type of the turn, the

In the demonstration that is introduced in this current topic of the conversation, and a symbolic paper, our novel methodology is explained didac- representation of the mentioned attributes of the tically in a step-wise fashion. The interactive web objects that were discussed. demonstration can be found at https://ehai.ai.vub. The procedural semantics consists of operations ac.be/demos/visual-dialog/1. The main interface that can interact with the conversation memory, of the demo is shown in Figure 1. Through this when coreferences between turns (e.g., ‘it’) need to interface, the user can discuss an image with a vi- be resolved. For example, the primitive operation sual dialogue agent. The user chooses an image and get-last-topic returns the topic of the previous selects a question. While the agent computes an turn, which was stored in the conversation memory. answer to the question, the reasoning steps that Figure 2 shows the conversation memory after the the agent is performing are shown. The goal of the second turn. The question corresponding to this turn was ‘What is its shape?’, the answer that the 1A video accompanying the demonstration can be found at: agent computed ‘sphere’. The utterance type of https://youtu.be/D3Ny6kta5d8 the question is ‘query’. ‘Attention-4 ’ is the visu- this topic in the image, which is itself the output alisation of the topic of the turn. The attributes of get-context. Then, the symbolic operation that are conveyed in the dialogue are symbolically unique checks whether the input attention contains added to the memory. In this case, the attribute just one object. Lastly, the subsymbolic operation ‘green’ was known from the previous turn and the at- query-shape will use a classifier to classify the intribute ‘sphere’ was added in the second turn. The put attention into a shape category. The output of demonstration gives the user the option to view this last step is the answer to the question. Thus, the previous turns in the dialogue, so that it be- the answer to the question ‘what is its shape?’ is in comes clear how the information is built up over this case ‘sphere’. the dialogue turns.

The hybrid procedural semantics represents the meaning of utterances. The meaning representation 3. Visual dialogue demo is expressed in terms of the conceptual operations that need to be performed to obtain an answer to a Figure 1 shows the main interface of the demonquestion. This makes the meaning representation stration. The interface consists of two main parts. directly executable. Each operation in the meaning On the right, there is a chat window in which the representation is performed either symbolically or user can interact with the visual dialogue agent, subsymbolically. The symbolic operations are oper- by choosing images and asking questions. On the ations that typically represent reasoning processes, left, the user can follow the reasoning process of such as comparisons or operations on the conversa- the agent. This part is itself divided in an uttertion memory (e.g., get-last-topic). Subsymbolic ance, execution network and conversation memory. operations are linked to perception and are exe- Utterance shows the utterance under consideration, cuted directly on the image (e.g., find-cubes). In which can be an observation (i.e., a statement about most cases, these subsymbolic primitives are imple- the image) or a question. The utterance is then mented by neural networks, in other cases matrix mapped onto a procedural semantic representation, operations on attentions are used (e.g., binary and, which is shown under execution network. The meanor, ...). ing representation can be viewed before execution

An example of the meaning representation of the and after execution. Figure 3 shows the execution question ‘what is its shape?’ is shown in Figure process of the meaning representation. The last 3. It consists of the steps get-memory, get-last- window shows the conversation memory, which is topic, get-context, find-in-context, unique updated after the agent computes the answer. and query-shape. The output of an operation be- The demonstration proceeds as follows. First, the comes the input to another operation. For example, user chooses an image. Then, the visual dialogue the output of the operation get-memory becomes agent processes the image and utters a statement the input to the operation get-last-topic. The about the image. The underlying meaning represenoperation find-in-context gets as input the out- tation of the statement and the hybrid execution put of both get-last-topic and get-context. network are shown. Meanwhile in the chat, the The first operation get-memory is a symbolic op- agent provides more information about the methoderation that returns the conversation memory. This ology. Next, the agent updates its conversation conversation memory is the input to the next sym- memory with information from the observation. Afbolic operation get-last-topic, which returns the terwards, a few questions appear for the user to symbolic representation of the topic of the previ- choose from. These questions are based on the ous turn. The operation find-in-context retrieves questions from the CLEVR-Dialog dataset [ 2 ]. The

4. Contribution

The goal of this didactic demonstration is twofold.

On the one hand, it aims to let users experience the challenges involved in understanding grounded natural language conversations. On the other hand, it aims to showcase our novel hybrid procedural semantics-based methodology for solving visual dialogue tasks. The demonstration especially focusses on showcasing the explainability and interpretability of the reasoning processes performed by the agent, which is one of the major advantages of our approach as compared to other state-of-the-art approaches. This makes our approach particularly interesting to human-centric AI applications, in which explainability and interpretability are a major concern. Possible applications include safety-critical systems such as emergency response platforms and decision support systems that are required to motivate decisions in a human-understandable fashion.

5. Conclusion

The demonstration that is proposed in this paper shows the dynamics of two novel techniques for the task of visual dialogue: (i) a conversation memory that is incrementally updated and (ii) a procedural semantics that is executed in a hybrid way. The conversation memory is a representation of all previous turns of the dialogue and is used to solve coreferences with respect to previous turns. The procedural semantics is a meaning representation for utterances in terms of steps that need to be performed. The meaning representation is executed in a hybrid manner with symbolic primitives to execute reasoning operations and subsymbolic primitives that are responsible for operations related to perception. In the interactive web demonstration, the user can ask questions to the agent. While the [8] T. Winograd, Understanding natural language,

Cognitive Psychology 3 (1972) 1–191. [9] P. N. Johnson-Laird, Procedural semantics,

Cognition 5 (1977) 189–214. [10] J. Andreas, M. Rohrbach, T. Darrell, D. Klein, Learning to compose neural networks for question answering, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1545–1554. [11] J. Johnson, B. Hariharan, L. van der Maaten, J. Hofman, L. Fei-Fei, C. Lawrence Zitnick, R. Girshick, Inferring and executing programs for visual reasoning, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2989–2998. [12] J. Nevens, P. Van Eecke, K. Beuls, Computational construction grammar for visual question answering, Linguistics Vanguard 5 (2019) 20180070.

References

[1] A. Das , S. Kottur , K. Gupta , A. Singh , D. Yadav , J. M. Moura , D. Parikh , D. Batra , Visual Dialog, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017 , pp. 1080 - 1089 .

[2] S. Kottur , J. M. Moura , D. Parikh , D. Batra , M. Rohrbach, CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics , Minneapolis, Minnesota, 2019 , pp. 582 - 595 .

[3] J. Johnson , B. Hariharan , L. van der Maaten, L. Fei-Fei , C. Lawrence Zitnick , R. Girshick , CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning , in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017 , pp. 2901 - 2910 .

[4] P. H. Seo , A. Lehrmann , B. Han , L . Sigal, Visual reference resolution using attention memory for visual dialog , in: Proceedings of the 31st International Conference on Neural Information Processing Systems , Curran Associates Inc., Red Hook , NY , USA, 2017 , p. 3722 - 3732 .

[5] S. Kottur , J. M. F. Moura , D. Parikh , D. Batra , M. Rohrbach , Visual coreference resolution in visual dialog using neural module networks , in: Proceedings of the 15th European Conference on Computer Vision (ECCV) , 2018 , pp. 153 - 169 .

[6] J. Andreas , M. Rohrbach , T. Darrell , D. Klein , Neural module networks , in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016 , pp. 39 - 48 .

[7] W. A. Woods , Procedural semantics for a question-answering machine , in: Proceedings of the December 9-11 , 1968 , Fall Joint Computer Conference, Part I , New York, NY, USA, 1968 , pp. 457 - 471 . URL: https://doi.org/10. 1145/1476589.1476653.