This paper proposes an interface that uses automatically-generated Natural Language (NL) descriptions to describe indoor scenes based on photos taken of that scene from smartphones or other portable camera-equipped mobile devices. The goal is to develop a non-visual interface based on spatio-linguistic descriptions which could assist blind people in knowing the contents of an indoor scene (e.g., room structure, furniture, landmarks, etc.) and supporting efficient navigation of this space based on these descriptions. In this paper, we concentrate on understanding the most salient content of a stereotypic indoor scene that is described by an observer, categorizing the description strategies employed in this process, and evaluating the best presentation of directional information using NL descriptions in order to support the most accurate spatial behaviors and mental representations of these scenes by means of human behavioral experiments. This knowledge will then be used to develop a domain specific indoor scene ontology, which in turn will be used to generate automated NL descriptions of indoor scenes based on their photographs, which will finally be integrated into a real-time non-visual scene description system.
Navigation involves a process of controlling and monitoring the movement of any
physical entity from one place to another [
The majority of the extant literature and technology development on accessible
navigation devices relates to technology for detecting and avoiding obstacles to the
path of travel or speech-enabled GPS systems for street navigation (see [
Where navigation assistance of the network structure of large-scale environments
benefits both blind and sighted users alike, sighted individuals rarely need assistance
gaining information about these small-scale environments as the information is
directly perceived through visual access to the scene. However, for a blind navigator
relying on non-visual sensing, which is generally more proximal and less spatially
precise, we argue that lack of access to spatial information about these local
environments can be equally detrimental to accurate navigation, spatial learning, and
cognitive map development. To date, limited research has been conducted to investigate the
description of indoor scenes or how knowledge of the spatial distribution of
architectural elements and salient objects in these spaces can be best imparted to blind people
through non-visual channels. The limited research that has been done in this domain
has required the use of expensive wearable specialty devices for acquiring spatial
information. For example [
Natural Language represents an intuitive interface as it is innate to most humans and
is easy to generate using text-to-speech engines. It is often used in spatial contexts,
e.g. direction giving, and has the advantage of being equally accessible to both
sighted and blind users. Owing to the sparse information content that can be specified
using a serial, temporally extended, and low-bandwidth medium, NL is considered a
limited information display. In addition, NL involves more cognitive load in working
memory than perceptual interfaces as it requires cognitive mediation to interpret the
verbal information being described, such as metric, topological, and other spatial
information [
Because of recent technological advancements and promising results in the field of Natural Language processing and generation, we argue that NL is one of the most important modes of information access for incorporation in non-visual interfaces and intelligent devices used by blind people. One problem is that NL description generation primarily concentrates on the semantic and syntactic aspects of the linguistic description in order to mimic human speech patterns. However, there is little formal research on understanding the ways in which a human summarizes the information they directly perceive, especially when looking at an indoor scene, through spatial verbal descriptions in order to convey survey knowledge of the scene.
Hence, in order to generate a NL description of an indoor scene, we argue that it is important to first understand the ways in which humans would naturally describe (e.g., verbally narrate) the space. NL generated without understanding of this narration is only a formal arrangement of words into sentences which abides the syntactic and semantic rules of a language following a specific architecture. To gain this knowledge, behavioral experiments must be conducted in order to understand the logic behind the human-generated NL description of an indoor scene. These results can then be compared to descriptions generated by a machine-generated NL description to assess where differences and similarities arise. The following human experiments are proposed to address this question. 4.1
The end goal is for blind persons to use photos taken with their smartphones in order
to obtain information about the spatial configuration of indoor scenes, including the
location of its constituent objects, delivered via NL descriptions. Photos taken using
smartphone cameras will inherently have a limited field of view (FOV). Hence it is
important to compare the spatial information obtained from photographic
observations of an indoor scene against the spatial information obtained from direct
observations of the same scene to evaluate whether this limited FOV leads to exclusion of
important environmental details in the ensuing spatial verbal descriptions. A
behavioral experiment was conducted to evaluate whether there is a significant difference of
observation by comparing the accuracy of scene re-creation based on previously
generated scene descriptions from both modes. Supporting the efficacy of camera-based
photos in our system, results revealed no significant differences between spatial
information acquired from human or camera-based observations or re-creation accuracy
based on descriptions generated from these two modes [
Flexibility is one of the most important features of a Natural Language. One challenge is that NL descriptions of spatial information of objects in an indoor scene could be structured in different ways following different strategies. For example, a description could begin by describing the name and spatial locations of objects in one corner of the room and then follow a cyclic clockwise strategy of describing the other objects around the room. Alternatively, a description could combine objects based on their functionality, e.g. describing the spatial location of all the tables that are present in a room, then describing the chairs, etc.
Research conducted in [
For a linguistic scene description to generate a mental map that is comparable in
function with mental maps developed from visual perception, it is extremely important to
have an accurate method for specifying directional cues about the spatial locations of
objects within the scene. As with scene description strategies, there are different ways
to verbally present these directional cues to the user. The work done in [
Degree measurements and clock face directions are the most common ways to present angular information using a relative frame of reference. For example, “a desk is at your 1 O’ clock position” and “a desk is at 30 degrees on your right” both specify the same spatial location of the desk. But it is important to know which of these presentation methods leads to the most accurate perception of directional information. To address this question, a behavioral study is currently being conducted to compare the accuracy of angular perception based on these two types of directional cues. 5
Natural Language descriptions of indoor scenes will be generated based on the results of the above mentioned behavioral experiments. They are expected to provide access to accurate spatial information for use by a blind user when made available. However, it is also important to compare the NL descriptions created by an automated machine with the NL descriptions created by a human user in order that results from the latter can guide development of the former. This could be tested by asking participants to reproduce the scenes based on the NL descriptions from both human generated and machine generated descriptions. The accuracy of re-created scenes should be tested for any significant differences in the ensuing re-creations before being implemented in a real time indoor scene description system. 6
Natural Language Generation architecture involves a procedural and formal way of
arranging raw spatial information that must then be converted to a NL [
This paper proposes a NL user interface for describing indoor scenes to visually impaired people. While current natural language systems concentrate on the semantic and syntactic components of natural language, we propose an automated NL system that is aimed at mimicking accurate descriptions delivered by humans in terms of spatial knowledge acquisition and information delivery as established from our human behavioral experiments. Several experiments are proposed to better understand the ways in which humans perceive and describe indoor scenes in order to establish the most salient information content and description strategies. Finally, we propose the construction of an indoor scene ontology to formally represent the knowledge acquired from the results of our behavioral experiments.
Acknowledgement: This project was supported by NSF grant CDI-1028895. Thanks also to Bill Whalen for comments on the manuscript.