We propose rule-based search systems that outperform not only the state-of-the-art but the human performance, measured in accuracy, in GuessWhat?!, a vision-language game where either of two players can be a human. Although those systems achieve the high accuracy, they do not meet other requirements to be considered as an AI system that communicates effectively with humans. To clarify what they lack, we suggest the use of three criteria to enable effective communication with humans in vision-language tasks. These criteria also apply to other two-player vision-language tasks that require communication with humans, e.g., ReferIt.
Recent advances in computer vision and natural language processing have led researchers’ attentions to the intersection of these two areas, vision-language tasks. An initiative to this kind of task was image description [Kiros et al., 2014; Vinyals et al., 2015; Xu et al., 2015; Johnson et al., 2016b; Mao et al., 2016]. In the image description task, an image is given, and the model is supposed to generate descriptions or captions on the image. However, generated descriptions have been difficult to evaluate, and results have not been directly related to how well the model understand the image. To show the comprehension of the model, visual question answering (VQA) was introduced [Antol et al., 2015; Johnson et al., 2016a; Agrawal et al., 2017; Fukui et al., 2016; Kim et al., 2016]. In VQA, an image and a question about the image is given, and the model is supposed to answer the question. However, the communication occurs one-way, and the model has the passive role only to answer questions.
Some vision-language tasks require active bidirectional communication between two (or possibly more) agents. To interactively communicate over an image, visual dialogues were introduced [Lazaridou et al., 2016; Das et al., 2016; Mao et al., 2016; de Vries et al., 2017; Strub et al., 2017; Das et al., 2017]. In visual dialogues, an image is given, and y authors contributed equally. two (or possibly more) agents communicate over the image. Visual dialogues involving agents with specific roles or tasks were mainly studied up to the present.
ReferIt game [Kazemzadeh et al., 2014] is an example of visual dialogue. It is a two player game referring to objects in an image of natural scenes. One player is shown an image with a target object and has to explain it to distinguish from the other, where what it says is called the referring expression. The other player is shown the same image and the referring expression written by the other player and guesses the target object.
GuessWhat?! game is also an example of visual dialogue (Fig. 1). It contains dialogue on question answering about the given image over two players. Its goal is to locate an unknown object in a rich image scene by asking a sequence of questions. One player is randomly assigned an object in the image and the other player is to locate the hidden object with a series of Yes/No questions.
On the two tasks mentioned above, each player also can be human or agent. If one player is human and the other is agent, the agent must generate meaningful dialog with humans in natural and conversational language about a visual image. This aspect is crucial to solve the task successfully. Such tasks have been evaluated in terms of task-specific performance metrics such as accuracy or success rate of the task. However, they are not the only criteria to enable efficient bidirectional communication with humans.
In this paper, we pose the problem that we need more criteria to measure and analyze the bidirectional communication between human and agent other than metrics. To demonstrate that, we first try to tackle GuessWhat?! game which we mentioned above and show that our proposed rule-based search systems outperformed not only state-of-the-art, but the human performance measure, the success rate of the task. Then, we suggest the use of some criteria to enable efficient bidirectional communication between human and agent in visionlanguage tasks such as GuessWhat?! game.
The rest of the paper is organized as follows. First we review related works to vision-language tasks on Section 2 and we proposed rule-based search systems which outperformed state-of-the-art performance on Section 3. Then, we suggest the use of some criteria for measure and analyze the bidirectional communication between human and agent on Section 4. Finally, we discuss conclusions and future work on Section 5. 2 2.1
Automatic image description is a challenging problem that involves analyzing an image, reasoning contextual information between existing objects in the image and generating textual descriptions. It has been first stage on research about vision-language grounding. [Vinyals et al., 2015] proposed neural image caption (NIC) generator inspired by advances in machine translation. They replaced encoding step which extracts abstract representations of source language using RNN to using CNN fed into given image. Encouraged by advances in employing attention in machine translation and object recognition, attention mechanism is introduced by [Xu et al., 2015]. The mechanism can attend to salient part of given image while generating its caption so that demonstrated the learned alignments correspond very well to human intuition.
Many previous papers on image description have focused on describing the entire image. On the other hand, [Johnson et al., 2016b] address a new task, dense captioning, which requires a model to predict a set of descriptions of regions of given image. It is important to understand each object or part not an entire image for high-level scene understanding. [Mao et al., 2016] also focused on generating an unambiguous description of a specific object or region in an image. They considered both description generation and description comprehension and jointly modeled both tasks combining CNN with RNN.
These models are just passive roles only to generate description about given image on the task, not bidirectional communication. And, generated descriptions have been difficult to evaluate, and results have not been directly related to how well the model understand the image. Therefore, the task could be extended for bidirectional communication task such as ReferIt or GuessWhat?! game and it needs further consideration for evaluation and analysis. 2.2
ReferIt [Kazemzadeh et al., 2014; Lazaridou et al., 2016] is a two player game referring an object in an image of natural scenes. One player is shown an image with a target object and has to explain it to distinguish from the other, where what it says is called the referring expression. The other player is shown the same image and the referring expression written by the other player and guesses the target object. To accomplish this game, both agents should cooperate and learn the relation between vision and language. [Lazaridou et al., 2016] designed referIt game between two agents, constitute the referring expression by a binary vector between two agents, formulated the game as classification, and solved the classification problem by neural networks. On the task, these agents develop their own artificial language from the need to communicate in order to succeed at the game. It showed some correlation with human language, but also showed some mismatches. If one player is agent and the other is human, the referring expression may not carry the exact meaning and cause the confusion between the players. In terms of meaninful vision-language integration between human and agent, we argue that we need to analyze details of these referring expressions other than metrics such as success rate of game and accuracy. 2.3
GuessWhat?! is a cooperative two-player guessing game proposed by [de Vries et al., 2017]. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. One player who called Oracle is randomly assigned an object in the image and the other player who called the Questioner does not know the object assigned to the Oracle. The goal of the Questioner is to locate the hidden object with a series of Yes/No questions which are answered by the Oracle. If the Questioner selects a right object, we consider the game successful.
[de Vries et al., 2017] collect a large-scale humanplayed GuessWhat?! game dataset consisting of 800K visual question-answering pairs on 66K images and propose baseline deep learning model. To solve the proposed task successfully, [de Vries et al., 2017] suggested that an agent is required higher-level image understanding, like spatial reasoning, visual properties, object taxonomy, and interaction. The authors also proposed that the agent should understand the relationships between objects and how they are expressed in natural language. The baseline model consists of 3 parts: Oracle, Guesser and Question generator. Oracle is a modelbased a simple neural network, which fed embedded inputs and classifies answer among Yes/No or N/A. The role of guesser is to predict one hidden object. It compares dotproducts of embedded vectors of image, dialogue, and information of candidate objects and classifies the most probable object among the candidates. Question generator is to generate questions reflecting the context of the previous question answering pairs based on the Hierarchical Recurrent Encoder Decoder(HRED) model.
As a follow-up research, [Strub et al., 2017] present endto-end reinforcement learning optimization for question generation task to find the correct object efficiently. They define GuessWhat?! game as a Markov Decision Process: A state xt is the tokens generated on the dialogue until time t and an action ut is to select a new word with zero-one reward depending on the Questioner’s choice. They train the question generator with policy gradient and obtain about 17% improvement of accuracy as compared with the baseline model.
GuessWhat?! game have been measured only by the success rate of the game. However, as following Section 3, we show rule-based search system which attains not only stateof-the-art but also human performance. At the point, we underline that it is not enough to measure the bidirectional communications only by the metrics such as success rate of the game or accuracy and propose criteria for meaningful evaluation on Section 4. 3 3.1
We constructed rule-based search systems using only the spatial information of the target object for GuessWhat?!. We can divide an image evenly into three parts by two vertical lines and divide each part continuously by horizontal lines or vertical lines in turns (Fig. 2). Then in a current region of interests divided by two vertical lines, we say the left of the left-side region, the right of the right-side region, and N/A of the middle region. Similarly, we say the top, the bottom, and N/A when a region is divided by horizontal lines. Through this protocol or language, we can ask and answer for the location of the center of the target object. Rule-based search systems use this simple language to locate the target object. We may also utilize statistics or the distribution of the spatial information. For the first turn, we may divide an image not evenly. We found that the range of the middle side of 0.18 covers 1/3 target objects, and the other sides of 0.82 covers 2/3 target objects, which means the distribution of target objects was denser in the evenly-divided middle side. Therefore, we set the vertical lines to locate at 0.41 and 0.59.
Given a segmentation model, we can further improve our system. To explore this case, we stole a look at the segmentation information of the candidates, which is supposed to be known when the guesser selects a candidate after a series of question-answer pairs. Then we can implement the binary search based on the spatial information of the candidates. If a segmentation model gives the segmentation similar with that of candidates given in the dataset, we can get an algorithm close to the binary search, which is optimal.
Proposed rule-based search systems do not break the rule of GuessWhat?! game. The spatial information is commonly used by humans as appears in the dataset. Moreover, we can substitute the spatial information with other features or properties. We can choose a real-valued feature without the point mass like the area and the color. Then the feature gives another rule-based search algorithm. This search algorithm is optimal (for the brightness of the center of the target object) or near-optimal (given an optimal segmentation model for the area.) We can also use real-valued features with the point mass or unordered (nominal or categorical) features by choosing a feature that evenly divide the candidates, which gives a near-optimal algorithm. 3.2
Results [de Vries et al., 2017] and [Strub et al., 2017] constructed the question generator by the hierarchical recurrent encoderencoder (HRED) and recurrent neural network (RNN) with reinforcement learning, respectively (Table 1). The oracle and the guesser was trained before interacting with the question generator. Therefore, the oracle and the guesser do not benefit from interaction. In contrast, in proposed rule-based search systems, the oracle and the questioner share a set of strict rules. Proposed systems outperformed the state-of-the-art accuracy in two turns and the human accuracy in four or five turns. This result is remarkable improvement of accuracy considering that [de Vries et al., 2017] and [Strub et al., 2017] generated 5 and 8 questions respectively to get their accuracy and that the minimum, mode, mean, and maximum of the number of questions by humans are 1, 3, 5.2, and 24, respectively. Furthermore, we improved the system by tuning the first division of an image, utilizing statistics on spatial information (denoted Fine-Tune). To explore the effect of segmentation, we stole a look at the segmentation information of the candidates (denoted Segment Info). 4
The search system exploits low-level features and strictly follows a predefined set of rules without considering the uncertainty or ambiguity. If either the oracle or the questioner is a human, the search system may not successfully communicate with the human. If the search system takes the part of the questioner, then it may explain how it works at the beginning. If the search system plays the role of the oracle, the search system is unlikely to take the initiative, so it may not have a chance to show how it works. To develop more satisfying systems, we investigate characteristics of the effective AI system for bidirectional communication with humans in vision-language tasks. We will present some considerations in designing such systems. Then we will review some criteria in vision-language tasks and suggest the use of some criteria to be adopted in GuessWhat?!.
To develop effective AI systems for bidirectional communication with humans in vision-language tasks, we need to formulate a problem or task first. Many vision-language tasks have been proposed for this purpose. A task naturally gives a main set of criteria, called objective functions and constraints in optimization. However, this main set may not be enough, then we need to add more criteria. Several criteria in visionlanguage tasks have been made. We may group these criteria into subjective, task-specific, or similarity criteria.
The subjective evaluation by humans is widely performed in many AI researches as well as vision-language areas. People observe or interact with systems and then evaluate how well or alike humans systems behave. The subjective evaluation is not objective so may not be considered scientific, but the subjective evaluation is crucial because it tells how humans actually feel about the system. However, the subjective evaluation is costly in general.
Task-specific criteria are essential to show how well the system performs in each task. In vision-language tasks, some cross-modal classification or retrieval metrics have been used including accuracy, median rank (mRank), and precision / recall at k (P/R@k) [Kent et al., 1955]. These criteria are measured on data that was collected before constructing the system, so they cost less than subjective criteria, which require additional human efforts.
Similarity criteria evaluate how human-like systems behave. In language generations like machine translations and text summarizations, some language similarity metrics have been used including bilingual evaluation understudy (BLEU) [Papineni et al., 2002], metric for evaluation of translation with explicit ordering (METEOR) [Banerjee and Lavie, 2005], recall-oriented understudy for gisting evaluation (ROUGE) [Lin, 2004], and consensus-based image description evaluation (CIDEr) [Vedantam et al., 2015]. These similarity criteria are calculated systematically, so they cost less than subjective criteria.
Adversarial evaluation through neural networks [Bowman et al., 2015; Kannan and Vinyals, 2017; Li et al., 2017] was suggested as a similarity criterion. Unlike previous similarity criteria, the adversarial evaluation is not fixed. Instead, it changes when a neural network called a discriminator learns whether the speaker is a human or not. After learning, the discriminator tells the human from the other for the test data. Since the discriminator is a neural network, it may catch various complex patterns which could not be found by other similarity metrics. However, training the discriminator is necessary unlike other similarity metrics.
Beyond the accuracy, we need to adopt more criteria in GuessWhat?!. Basically, more criteria are considered the better in evaluating vision-language tasks, which is partially because we do not have a unique criterion that everyone agrees with. However, we have limited resources, so we have to choose some criteria among them. The accuracy is the obvious task-specific criterion, but people would not be satisfied only with the accuracy achieved by two AI players. We need to solve the following constrained optimization problem to develop systems that can communicate with humans in GuessWhat?!. Given an objective functional f (the accuracy in GuessWhat?!), human oracle Oh, and human questioner Qh, the optimal oracle O and questioner Q are given by solving max f (O; Q) O;Q s.t. O is compatible with Qh
where, for a threshold t > 0,
if f (O; Qh) > (1
if f (Oh; Q) > (1 t) max f (O; Qh)
O t) max f (Oh; Q)
Q This problem is a joint optimization problem and is difficult to solve because it involves two optimization functions and the interaction with human oracles as well as the observation of human questioners. Instead, a two-phase greedy optimization was commonly used in previous works. It involves the observation of human questioners like the previous joint optimization problem, but it involves only one optimization function at each phase and the interaction with an oracle model instead of human oracles.
O^ = argmax f (O; Qh)
O Q^ = argmax f (O^; Q)
Q However, O^ and Q^ are marginally optimal, and Q^ may not be compatible with Oh. We may employ human oracles to determine whether Q^ is compatible with Oh even if it costs large. Under the assumption that a questioner Q similar with the human questioner Qh is compatible with Oh, similarity criteria may complement or substitute with human evaluation. Criteria that distinguish the AI system from humans are necessary for this purpose. The adversarial metric is a promising criterion among them because it involves the neural network, which can learn complex patterns, as a discriminator or judge who determines whether two players are human-like (Fig. 3).
We reviewed some criteria in vision-language tasks and suggested the use of some criteria to be adopted in GuessWhat?!. We grouped the criteria into subjective, task-specific, or similarity criteria and gave some examples. In GuessWhat?!, we need more criteria other than the accuracy. If we are affordable, then we can employ human evaluation. Otherwise, we may choose similarity criteria. We recommended the adversarial evaluation as a promising similarity criterion. 5
We proposed rule-based search systems which used only the spatial information of the target object for GuessWhat?! game. It can be regarded as the artificial language between agents on the GuessWhat?! task. Our rule-based search system outperformed the state-of-the-art accuracy in two turns and the human accuracy in four or five turns. In the view of (1) (2) (3) (4) the results, we argue that we need to measure the performance of the system and analyze details of the results with more concrete criteria not just task-specific metrics such as the success rate of game and accuracy. We suggested the use of criteria for bidirectional communication between humans and agents in vision-language tasks. The adversarial evaluation can be considered as a promising similarity criterion.
This work was supported by the Institute for Information & Communications Technology Promotion (2015-0-00310SW.StarLab) and Korea Evaluation Institute of Industrial Technology (10044009-HRI.MESSI, 10060086-RISF)