In this paper we outline how to translate verbal subjective descriptions of spatial relations into metrically meaningful positional information, and extend this capability to spatiotemporal monitoring. Document collections, transcriptions, cables, and narratives routinely make reference to objects moving through space over time. Integrating such information derived from textual sources into a geosensor data system can enhance the overall spatiotemporal representation in changing and evolving situations, such as when tracking objects through space with limited image data. We focus on landmark identification, since it proves to be a more tractable problem than open-domain image recognition.
The relation between language and space has long been an area of active research. Human languages impose particular linguistic constructions of space, of spatiallyanchored events, and of spatial configurations that relate in complex ways to the spatial situations in which they are used. Establishing tighter formal specifications of this relationship has proved a considerable challenge and has so far eluded general solutions. One reason for this is that the complexity of spatial language has often been ignored. In much earlier and ongoing work, language is assumed to offer a relatively simple inventory of terms for which spatial interpretations can be directly stated. Examples of this can be found not only in accounts that focus on formalizations of particular tasks, such as path and scene descriptions, navigation and way-finding, but also in foundational work on the formal ontology of space, on qualitative spatial calculi, and on cognitive approaches.
Visual information in human experience is frequently accompanied by a linguistic
description of the image or scene. Consider, for example, the image in Figure 1. If the
goal is to identify the region of the image where one should look for the lost keys, one
first must identify the correct tree. If this image is automatically segmented using a
stock library of images for trees and entrances
Over the past decade, image annotation has been the focus of attention within
several research areas, in particular, in the context of content-based image retrieval
(CBIR). Some research, including the work done within the TRIPOD project at
Sheffield, examines the different ways that geo-referenced images can be described
Let us consider the scene depicted in Fig. 2, taken from Google StreetView, of the intersection of Huntington and Mass. Avenues in Boston. In it we can identify reference landmarks, namely three buildings: Horticultural Hall (HC), Prudential Center (PC) and the Christian Science Monitor building (CSM). It also comprises various other objects, for example a white van, a black truck, and a red car. Our interest is in geolocating the observer of this scene by using orientational descriptions of the relative appearance of the landmarks contained in it.
Assuming that a narrator is familiar with these three landmarks, he/she could describe the scene as follows:
I see the SW and SE sides of Horticultural Hall, and to the right of it I see the SW and SE sides of the Prudential Center, and to the right of it I see the Christian Science Monitor Building.
In this situation the narrator has described the scene through three types of statements: • • • explicit reference to specific landmarks, positioning the scene in their vicinity, explicit description of orientational properties expressing the relative positions of these landmarks in an observer-centric system1, and implicit visibility declarations, whereby she indicates that she can observe specific façades of landmark buildings.
The orientational properties are modeled using ISO-Space
Relations such as “to the right of” are annotated as a relative spatial relation
between two elements, the figure and the ground, and the viewer perspective is
accounted for by two further attributes on the link tag: rframe, with values absolute,
relative and intrinsic, and viewer, which contains a variable indexed to the viewer
For every visibility statement we can identify a visibility zone through viewshed
analysis, using the local GIS information
F1 is shown as the gray-shaded area. From any point outside this area it would be impossible to see façade F1.
Each additional visibility statement introduces additional visibility zone information, and the location of the narrator can be eventually determined through the intersection of the corresponding visibility zones through polygon clipping techniques, such as Weiler-Atherton (1977). Fig. 5 shows the implementation of this process for the scene of Fig. 1, through a progressive assessment of visibility conditions for HC, CSM, and PC.
Visibility zones (Z 1 in red, Z 2 in blue) of two visible facades of Horticultural Hall
Visibility zones (Z 3) of one visible facade of GSN
Visibility zones (Z 4) of anther visible facade of GSN sub-region=Z 1 ! Z2 sub-region =Z 3!(Z1 ! Z2)
sub-region = Z3!(Z1 ! Z2 ) Visibility zones (Z 5) of one visible facade of Prudential Center Visibility zones (Z 6) of one visible facade of Prudential Center sub-region = Z 5! (Z3!(Z1 ! Z2 ))
sub-region = Z 5! (Z3!(Z1 ! Z2 ))
Fig. 5. The progressive visibility intersection process
The narrator position estimated through the process visualized in Fig. 5 is shown on the local map in Fig. 6, marked as a red triangle. The triangle corresponds to all positions from which the narrator would have a view of our scene that would be comparable to the one depicted in Fig. 2 in terms of the orientational relationships of the three depicted landmarks. In this paper, we discuss the integration of multi-source data analysis for spatial knowledge extraction from images. In particular, we focused on the specific contribution of verbal subjective descriptions of spatial relations involving orientation, and how these can be translated into metrically interpretable positional statements within a GIS environment. We concentrated on the more tractable subproblem of landmark identification. Orientational information in language was modeled with ISO-Space annotation, providing both qualitative spatial relations and anchored GPS values, once geolocating is performed.
This work is ongoing research aimed to allow for the integration of information available from different sources, better addressing the evolving needs of the geoinformatics community. Our preliminary results suggest that scene content information provided by verbal description can be mapped faithfully to metrically grounded information. As this is preliminary work, there are clearly many details to be worked out. For example, we have not yet precisely defined how orientational relations are used to identify a landmark in a GIS, especially with anonymous landmarks, a problem exacerbated when an ambiguity between intrinsic and observerbased relative orientation cannot be easily resolved.
One of the ultimate goals of this research is the development of algorithms that take an image and accompanying verbal utterances and maps these to a partition of a 2D grid. This application would be tuned to deal with more natural utterances than the somewhat stilted verbal descriptions given with Fig. 2 above.
This research was funded under the NURI grant HM1582-08-1-0018 by the National Geospatial Agency. Steck S., Mallot, H.: The Role of Global and Local Landamarks in Virtual Environment Navigation. Presence, 9(1), 69-83 (2000).
Weiler K., P. Atherton. Hidden Surface Removal using Polygon Area Sorting. ACM SIGGRAPH Computer Graphics, 11(2), 214-222 (1977).
Werner S., Krieg-Brueckner B., Mallot H., Schweizer K., Freska C.: Spatial Cognition: The Role of Landmark, Route, and Survey Knowledge in Human and Robot Navigation. In: Jarke M., Pasedach K., Phl K. (eds.) Informatik ’97, pp. 41-50, Springer Verlag (1997).