We investigate using Amazon Mechanical Turk (AMT) for building a low-level description corpus and populating VigNet, a comprehensive semantic resource that we will use in a text-to-scene generation system. To depict a picture of a location, VigNet should contain the knowledge about the typical objects in that location and the arrangements of those objects. Such information is mostly common-sense knowledge that is taken for granted by human beings and is not stated in existing lexical resources and in text corpora. In this paper we focus on collecting objects of locations using AMT. Our results show that it is a promising approach.
Our aim is to populate VigNet, a comprehensive semantic resource that we will
use in a text-to-scene generation system. This system follows in the footsteps
of Coyne and Sproat's WordsEye [
VigNet is based on FrameNet[
This paper deals with the collection of spatial information to populate VigNet. Even though VigNet contains vignettes for actions and other events, complex objects and situations, this paper focuses only on the induction of location vignettes. Knowledge about locations is of great importance to create detailed scenes because locations de ne the context in which an action takes place. For instance when someone takes a shower he usually does so in the bathroom, interacting with the `a ordances' provided by this room (i.e. shower cabin, curtain, shower head, shower tap etc.) in a speci c way. Note that location vignettes can, but do not have to be evoked by lexical items. We can say John took a shower in the bathroom, but this seems redundant because bathrooms are the preferred location for taking a shower. VigNet records knowledge of this type that can be accessed in the text-to-scene generation process.
In this paper we propose a methodology for collecting semantic information for locations vignettes using Amazon Mechanical Turk (AMT). The next section rst discusses location vignettes in more detail. We then review related work in section 3. We describe how we use AMT to build an image description corpus and collect semantic information for locations in section 4 and compare di erent methods in an evaluation. Section 5 concludes. 2
As mentioned before, location vignettes are important because they provide the context in which actions can take place. Locations involve the spatial composition of several individual objects. For example, in `John sat in the living room', we might expect the living room to contain objects such as a sofa, a co ee table, and a replace. In addition, these objects would be spatially arranged in some recognizable manner, perhaps with the replace embedded in a wall and the coffee table in front of the sofa in the middle of the room. In order to represent such locations graphically we are adding knowledge about the typical arrangements of objects for a wide variety of locations into VigNet.
Any given location term can potentially be realized in a variety of ways and hence can have multiple associated vignettes. For example, we can have multiple location vignettes for a living room, each with a somewhat di erent set of objects and arrangement of those objects. This is analogous to how an individual object, such as a couch, can be represented in any number of styles and realizations. Each location vignette consists of a list of constituent objects (its frame elements) and graphical relations between those objects (by means of frame decomposition). For example, one type of living room (of many possible ones) might contain a couch, a co ee table, and a replace in a certain arrangement.
living-room 42(left wall, far wall, couch,
co ee table, replace) touching( gure:couch, ground:left wall) facing( gure:couch, ground:right wall) front-of( gure:co ee table, ground: sofa) embedded( gure: re-place, ground:far wall)
The set of graphical primitives used by location vignettes control surface properties (color, texture, opacity, shininess) and spatial relations (position, orientation, size). This set of primitive relations is su cient to describe the basic spatial layout of most locations (and scenes taking place in them). Generally we do not record information about how the parts of a location can be used in an action, but rather consider this knowledge to be part of the action. 3
Existing lexical and common-sense knowledge resources do not contain the
spatial and semantic information required to construct location vignettes. In a few
cases, WordNet [
More relevant to our project is OpenMind [
LabelMe [
One of the well-known approaches for building lexical resources is automatic
extracting lexical relations from large text corpora. For a comprehensive review
of these works see [
Furthermore, much of the information that we are looking for is commonsense knowledge that is taken for granted by human beings and is not explicitly stated in corpora. Although structured corpora like Wikipedia do mention associated objects, they are often incomplete. For example in the Wikipedia entry for kitchen there is no mention of a counter or other surface on which to prepare food but the picture that goes with the de nition paragraph (labeled \A modern Western kitchen") clearly has one.
In this paper we investigate using Amazon Mechanical Turk (AMT) for building a low-level description corpus for locations and for directly collecting objects of locations vignettes. We will compare the accuracy of collected data to several gold standard vignettes generated by an expert. We show that we can tune our information collection method to scale for large number of locations. 4
In this section we discuss how we use Amazon Mechanical Turk (AMT) to build a location description corpus and for collecting the typical objects of location vignettes. AMT is an online marketplace to co-ordinate the use of human intelligence to perform small tasks such as image annotation that are di cult for computers but easy for humans. The input to our AMT experiments are pictures of di erent rooms. By collecting objects and relations grounded to speci c rooms we capture coherent sets of dependencies between objects in context and not just generalized frequencies that may not work together. In each task we collected answers for each room by ve workers who were located in the US and had previous approval rating of 99%. Restricting the location of the workers increases the chance that they are native speakers of English, or at least have good command of the language. We carefully selected input pictures from the results of image searches using the Google and Bing search engines. We selected photos that show `typical' instances of the room type, e.g. room instances which include typical large objects found in such rooms. Photos should show the entire room. We then de ned the following task:
Task 1: Building low-level location description corpus: In this task, we asked AMT workers to provide simple and clear descriptions of 85 pictured room. We explicitly asked AMT workers that their descriptions had to be in the form of naming the main elements or objects in the room and their positions in relation to each other, using verbs such as is or are (i.e. linking verb). Each description had to be very precise and 5 to 10-sentence long. Our collected description corpus contains around 11,000 words.
In order to extract location information from the low-level location description corpus, the text is rst processed using the NLP module of WordsEye. We extracted the objects and other elements of locations which are mainly in the form of relation{ground{figure and extract the objects and elements which are represented as figure or ground. We then further processed the extracted locations as is explained in sub-section 4.1.
Task 2: Listing functionally important objects of locations: According to this criterion, the important objects for a room are those that are required in order for the room to be recognized or to function in this way. One can imagine a kitchen without a picture frame but it is rarely possible to think of a kitchen without a refrigerator. Other functional objects include a stove, an oven, and a sink. We asked workers to provide a list of functional objects using an AMT hit such as the one shown in gure 1. We showed each AMT worker an example room with a list of objects and their counts. We gave the following instructions: \ Based on the following picture of a kitchen list the objects that you really need in a kitchen and the counts of the objects. 1. In each picture, rst tell us how many room doors and room windows do you see. 2. Again, don't list the objects that you don't really need in a kitchen (such as magazine, vase, etc). Just name the objects that are absolutely required for this kitchen. "
Task 3: Listing visually important objects of locations: For this task we asked workers to list large objects (furniture, appliances, rugs, etc) and those that are xed in location (part of walls, ceilings, etc). The goal was to know which objects help de ne the basic structural makeup of this particular room instance. We used the AMT input form shown in gure 1 again, provided a single example room with example objects and and gave the following instruction: \ What are the main objects/elements in the following kitchen? How many of each? 1. In selecting the objects give priority to: { Large objects (furniture, appliances, rugs, etc).
{ Objects that are xed in location (part of walls, ceilings, etc). The goal is to know which objects help de ne the basic makeup and structure of this particular kitchen. 2. In each picture, rst tell us how many room doors and room windows do you see. " 4.1
Post-processing of the extracted object names from AMT We post-processed the extracted objects from the location description corpus and the objects that were listed in tasks 2 and 3 in the following steps: 1. Manual checking of spelling and converting plural nouns to singular. 2. Removing conjunctions like\and", \or", and \/". For example, we converted \desk and chair" to \desk" and \chair". 3. Converting the objects belonging to the same WordNet synset into the most frequent word of the synset. For example we converted tub, bath, and bathtub into bathtub with frequency of three. 4. Finding the intersection from the inputs of ve workers and selecting the objects that listed three times or more 5. Finding major substrings in common: some input words only di er by a space or a hyphen character such as night stand, night-stand, and nightstand. We convert such variants to the simplest form i.e. nightstand. 6. Looking for head nouns in common: if the head of the compound noun input such as projector screen can be found in another single-word input i.e. screen, we assume that both refer to the same object i.e. screen. 7. Recalculating the intersections and selecting the objects with frequency of three or more. 4.2
Evaluation For evaluating the results we manually built a set of gold standard vignettes (GSVs) for 5 rooms which include A) a list of objects in each room, and B) the arrangements of those objects. Selected objects for GSVs are the ones that help de ne the basic makeup and structure of the particular room. We are comparing the extracted object from AMT tasks against the list of objects in the GSVs.
Table 1 shows the comparison of the AMT tasks against GSVs. The \Extracted Objs" row shows the number of objects we extracted from each AMT tasks for 5 rooms. The \Correct Objs" row shows the number of extracted objects from AMT that are present in our GSVs of each room and the precision score derived based on that. The \Expected Objs" row shows the number of all the objects in GSVs that we expected the workers to list, and the recall score based on that.
Pre: 87% Rec: 85%
In this paper we explored di erent approaches to populate VigNet, a resource containing spatially grounded lexical semantics, with locational information (location vignettes) using Amazon Mechanical Turk. In one approach we used AMT to collect a low-level description corpus for locations. We then used the WordsEye NLP module to extract the objects from each description. For comparison we asked AMT workers to directly list objects of locations shown in photographs, either based on visual or on functional criteria. We then post-processed the extracted objects from each experiment and compared them against gold standard location vignettes.
We have shown that we can extract reasonably accurate objects from processing the description corpus as well as spatial relations and arrangements of objects. The results achieved using the functional and visual object listing tasks approximate the gold standard even better, with the visual elicitation criterion outperforming the functional one.
In current work, due to the good results on the small training set we are using the visual object listing paradigm to induce descriptions of 85 rooms. We are planing to collect vignettes for a variety of other indoor and outdoor locations.
Location vignettes also contain the spatial arrangement of objects. In addition to the extracted relations from the description corpus, we also designed a series of AMT tasks for determining the arrangements of objects in di erent locations using the objects that we collected in the present work. For each room we ask AMT workers to determine the arrangements of the previously collected objects in that particular room. For each object in the room, workers have to determine its spatial relation with A) one wall of the room and B) one other object in the room. We did not include the results in this paper since we are still exploring methods to evaluate the spatial arrangements task. The gold standard location vignettes include arrangements of objects, but it is di cult to directly compare the gold standard to the AMT workers' inputs as there are di erent possibilities to describe the same spatial layout.
This material is based upon work supported by the National Science Foundation under Grant No. IIS-0904361.