The Holodeck, a virtual reality simulator from the television show Star Trek, is known as the “holy grail” of interactive narrative experiences. However, while there have been approaches to various components of a theoretical Holodeck, scene generation from dialogue is often overlooked. This paper introduces a prototype AI Holodeck application for scene generation, demonstrating the use of Natural Language Processing and a corpus of spatial data. The application creates scenes from user input text and fills those scenes with objects and relationships not explicitly defined by the user. This paper discusses potential use cases of scene generation in creating environments for interactive narrative, virtual reality, and other development opportunities.
The Holodeck is a fictional virtual reality device in the
television show Star Trek, taking the form of a blank room
that generates interactive characters and objects dictated by
voice commands from people inside of it. It represents a
“holy grail” of interactive virtual reality
For example, a Holodeck-inspired scene generation system could be used in the following scenario:
Sarah has an idea about a game about a detective. She is thinking of the events in the game but having a hard time imagining the space, so she uses the AI Holodeck to see how the space would look. “Holodeck, give me a detective’s office”. The Holodeck renders an office using objects sourced from a database. There is a desk and a chair, a window behind the desk, and a shelf in the corner. She then thinks that a desk lamp could make it more mysterious at night. “Holodeck, put a lamp on the desk.” The Holodeck adds a lamp on the desk and a notebook beside it.” She thinks “Yes, there should be a notebook on the desk!” But she feels the metal desk looks really rough in the office. She says, “Holodeck, give me a wooden desk”. The Holodeck renders the new wooden desk by adding “wooden” to its initial database search of desks and replacing the original desk.
To facilitate the creation of scenes – and to populate them
– some semantically annotated datasets are currently
available that categorize objects with relative positions and sizes
Scene generation applications are able to parse these
datasets to create a “scene template”, a constrained mapping
of a scene’s objects and their basic spatial relationships
between them
However, the addition of elements that are both 1) unspecified by a user and 2) gathered from semantically annotated datasets in order to maintain relevance to the user-described scene is missing from current research. Therefore, this paper aims to address the following research question: Can semantically annotated datasets be used to extract contextinformed scene templates in a text-to-scene generation application, including items not specified in the input text?
In this paper, we introduce an AI Holodeck application. This system draws from previous NLP and scene generation work in order to create scenes with appropriate elements that were not specified by the user. For example, if the user inputs a farm, then the application may populate the scene with things commonly found in a farm such as cows, hay barrels, and fields of crops. The AI Holodeck can be used in a variety of scenarios: examples include visual story generation, game design and prototyping, interior design sketching or idea generation, and creating virtual or fantasy worlds.
The remainder of this paper explains how our system draws from and synthesizes existing works in the domains of natural language processing and scene generation, details the design of our AI Holodeck application, and discusses potential applications for integrating scenes generated by this system with other interactive mediums. 2.1
Scene graphs have been extensively used and explored in
the contexts of scene understanding and semantic image
captioning
Depending on the data format, different approaches have
been devised to extract the scene content. SceneGen focuses
on novel representations of scene graphs which embed
positional and orientation information of a set of objects present
in a given room to achieve the most realistic placement
Other literature has investigated learning the implicit
positional relationship between objects using transformers and
attention mechanisms. In SceneFormer
A recent transformer based model CLIP connects text and
images to understand the image content. This
discriminative model can be used to predict image content at scale by
encoding the image and caption in parallel
We infer the object categories and their positional relationships directly from images for two reasons: 1) images are more accessible than 3D models and there are more datasets available. 2) images function as representations of real world situations and convey properties that may be manipulated in 3D scenes such as messy desks or cluttered rooms. 2.2
WordsEye
Notating 3D datasets with natural language descriptions
is one approach to improve the unnatural experience
compared to the prior work
To avoid unnatural languages caused by strict input language requirements like with WordsEye, they also bring objects that are not mentioned but are relevant to the mentioned objects. They select inferred objects by searching the object hierarchy and bringing the explicit object’s parent objects with the highest probability (Chang et al. 2017). In addition to this, our approach considers the environment and objects that have the higher probability of co-appear with the explicit objects in that specific context. Our focus is not to produce the most sophisticated interior layout possible, but rather to demonstrate a natural language-based scene generation system that creates appropriate scenes fitting the user’s input contextually and thematically. 2.3
We explored scene manipulation research from the perspectives of content and interaction. In terms of content, research on scene manipulation focuses on scene level manipulation or object level manipulation which targets different purposes such as object removal or image blending.
Recent research investigates scene manipulation through
updating the existing scene graph
In terms of interaction, prior research explored different
methods or modes of interaction allowing users to
manipulate scenes. SceneSeer (Chang et al. 2017) enables users
to manipulate the scene with textual commands like
“replace the bowl with a red lamp”. In Scribbling Speech
Unlike the above tools, which visually render a scene, our approach also hosts the Holodeck component models in an API-like format, allowing for integration into a variety of applications such as Unity.
3
As seen in Figure 1, our Holodeck scene generator contains a full pipeline to collect input text and form a visual representation of a scene. Our system generates a scene template for any input text, determining objects and their locations in order for them to be placed in a scene. Then, implicit connections between objects in our semantically annotated datasets are used to add additional nodes to the scene template, creating a more vibrant scene. Objects are then mapped to a graphical representation of a scene in sequence, while resolving any collisions between them. Finally, a lightweight interface was created to allow for easily understood demonstration.
User input utterances are parsed through the CoreNLP
library
CoreNLP has made it possible to use an in-depth
dependency parser, allowing for parsing complex sentence
structures. However, we are also offering an offline version using
the NLTK library
In order to create a more fleshed-out scene, our system adds additional nodes which are not explicitly mentioned by the user. Figure 3 provides an example of this concept. In this example, the user is creating an office space and has used the text “There is a couch and a table” as the first input. Since the “couch” and “table” are explicitly mentioned by the user, these nodes are created and used in the scene template. The addition of these implicit nodes allows the system to bring in other objects such as a rug, a window, and a book because these are objects usually found near a couch or table in an office. If the object mentioned by the user is not usually found in such environment, eg. “a horse in an office”, our system searches for other objects that have a high probability of being found in an office space rather than objects that are typically found near a horse.
With this method, the objects surrounding an object are dependent on which environment that object is in. Figures 3 and 4 show this difference by using the same input sentence “There is a couch and a table” in different environments of “bedroom” and “library”.
Finally, our system prioritizes the explicitly defined positional relations over the implicit relations created by the system. Figure 5 shows that a computer is implicitly brought to the scene after 2 input sentences “There is a table” and “There is a chair”. In Figure 6, the user adds the input “The computer is on top of the table”, moving the computer to the explicitly specified position.
Datasets Used for Extracting Implicit Relations In
order to create a dataset of potential positional relations, we
used the MIT Indoor Scenes Dataset
When generating a scene, the system places bounding boxes representing each object in the scene template into a 3D graph. It searches for sizes for each object in the ShapeNetSem metadata. If none are found, they are replaced with default values for the output graph. The algorithm used to prioritize object placement queues objects on the bottom of a scene (objects with no “below” parameter), and recursively adds objects in those objects’ “above” property on the graph, stacking objects on top of each other.
As each object is added to the graph, collisions are detected. Objects with lower priority (determined by their place in the scene template) are shifted in the direction corresponding to their property name the until their bounding boxes no longer overlap with the other object. For example, if one object is “above” the other, it will be shifted vertically. 3.4
The AI Holodeck application uses a Tkinter interface 1 (see Figure 7), activated from the command line.
The application opens a window with a menu for selecting a scene found in the Indoor Scenes dataset, a prompt for entering in text or microphone input, and a display of the objects currently registered from the scene. When a user selects the “Create Graph” button, objects found in the input text are added to the list of objects. The scene is then displayed as a movable, 3D matplotlib 2 graph in a separate window. 1https://docs.python.org/3/library/tkinter.html 2https://matplotlib.org/
The application allows for a number of command line
arguments. Mode selects either text or voice input. When vocal
input is activated, a recording button is added to the
interface beside the text box. Pressing this button activates a
continuous microphone stream until a sentence is recognized,
which then populates the text field and automatically
activates the graph creation function. Model selects either NLTK
This system is also limited in the fidelity of the visualizations it is able to create. Objects are represented only as a bounding box labeled with the object’s name. A more sophisticated visualization application would include indexing of a database of 3D models, in order to dynamically populate generated scenes with appropriate representations of the objects inside.
Additionally, we plan on modifying the system to allow for the manual removal and repositioning of objects. As users correct the system output to fit the needs of the scene they are trying to create, the stored database will update with new spatial relationships and as such the system will be able to learn from generated scenes.
Our system is robust in terms of affording future modifications. For example, in this phase of the project, we have used the MIT dataset to extract possible positional relationships between various objects. These positional relationships can be constantly modified/added to by using other datasets not limited to the visual datasets, such as datasets of narrative texts. Other techniques – such as deep learning – could also be beneficial in removing our need for annotated visual datasets by extracting spatial relations automatically from other collections of images or narrative text.
The visualization of objects can be modified and extended to various platforms. Our system provides a scene template and graphical representation as formatted data and data structures extracted from the input text, which can be used by various platforms to create a detailed visualization. As explained in the system design of this paper, this data includes the various objects in the scene, their properties, positional relations and center-points for placement of the objects in the scene. Hence, the visual modification could be either in the form of changing visualization platform or in the form of adding new objects to the dataset of 2D/3D models used in these platforms.
5
In future iterations of the system, we will include a separate narrative text interpretation module. This module will be comprised of a series of models trained on literature, which will provide additional scene details given a user’s starting input. The current implementation, for both the NLTK and CoreNLP models, primarily uses simple sentences comprised of subject-object pairs in each clause. Training models on literature will enhance the system’s ability to capture information from input sentences with a higher structural variety.
Other prospective improvements in the software are documentation and user functionality to facilitate connections to software such as Unity and virtual reality for integration with game development. This addition will be used to explore manipulation of objects already in a scene, and the placement or movement of objects with both input vocal commands and gestures. Additionally, once objects are generated in a higher-fidelity graphical environment, modifiers can be extracted from the input. These modifiers include adjectives describing the scene and the objects within. They will be added to the scene template and transmitted to the external environment in order to generate visuals more appropriate for the user query.
Future work will also include evaluation of the application. The first evaluation will include separate analysis of the NLP and scene generation models, in terms of precision in the sentences they parse as well as the relevance of objects generated for a scene. System performance will also be measured in terms of stability and framerate, while generating large amounts of objects. Finally, we will conduct user studies measuring the strength of the system as a piece of interactive technology as well as users’ experience.