The multi-layered structure of geographic information can be used to provide a rich description for geo-referenced video sequences. Furthermore, the complex structure of video, with its temporal dimension, makes this integration challenging. In this work we define a method for indexing and retrieving georeferenced video sequences based on the geographic content. This method is the basis for building a digital video library, and relies on a semantic framework that has been defined to represent and query the thematic information in a repository of geographic datasets. The meta-information structure used to describe video sequences is also described.
In a geo-referenced video sequence, some properties of the camera regarding its location are captured during the video recording. These properties include position (typically obtained from a GPS receiver), orientation with respect to the North (for instance captured from a digital compass), and often vertical tilt (the angle with respect to the horizon). Focal length, which determines the angle of vision, and receiver (negative or CCD in a digital camera) dimensions are also recorded. All these properties make it possible to obtain the geographic area (area of vision from now on) that can be seen in each frame of the video sequence. Our approach is based on describing a video sequence according to the area of vision of its frames, and according to the geographic information corresponding to this area of vision.
The geographic information used to describe videos is extracted from a repository
of geographic datasets. A geographic dataset represents the real world by assigning
thematic classes (for instance “forest”, “agricultural land”, “natural park” or
“motorway”) to spatial elements. However, different producers structure their datasets
in terms of different sets of thematic classes, which are often not precisely defined
and which may be understood in different ways by different subjects. An effective
method for integrating geographic information from diverse sources has to address
this semantic heterogeneity. In [
In Section 2 we provide a brief overview of different approaches using georeferenced video. Section 3 describes our semantic-based proposal for segmenting and indexing video sequences based on thematic information, which is the basis for the definition of a digital video library. It also describes how video is modelled following a stratification-based approach. Section 4 enunciates the main types of queries in the digital library, while Section 5 formulates how meta-information describing video segments is structured. Finally, in Section 6 we discuss some possibilities for continuing this work.
The use of geo-referenced video has become relatively frequent in the last years. This has been especially remarkable in organizations that maintain linear infrastructures as streets, roads and railroads. Special systems with video and location capturing devices are designed to be mounted on vehicles. This way, the vehicle can easily obtain geo-referenced videos for the whole network. Once processed and integrated in a GIS environment, these geo-referenced videos provide a rich and intuitive support for monitoring and decision making.
Several commercial systems exist to record geo-referenced videos and to add them to GIS environments through a simple post-processing procedure, which basically includes a synchronization of the video and the location data, and an interpolation to obtain a location for each frame (note that GPS receivers get measures slower than the video frame rate). Probably the most extended software tools are MediaMapper and GeoVideo, both developed by Red Hen Systems1, which also provides different hardware systems for recording geo-referenced video that can be mounted on vehicles or on aircrafts, or that can be carried by pedestrians. These software tools enable the user to view the path of the camera on a map. They also show a cursor on that path that indicates the exact location of the camera during video playback. The user may control the video by moving this cursor on the path, as well as from a typical video control with play/stop/rewind/fast-forward buttons. An analogous commercial tool is CamNavMapper, by BlueGlen2, that also covers acquisition, integration in a mapping environment and similar playback controls. A similar playback functionality is provided by ImageCat’s Views3. Other simpler software tools like GeoMovie by
Magic Instinct Software4 or VideoMapper5 provide a postproduction process that superimposes the coordinates of the camera and other meta-information on the video image.
As an indication of the increasing importance of geo-referenced video, the Open GeoSpatial Consortium has proposed a geo-video service for its OWS-3 (OGC Web Services Phase 3) interoperability initiative. The aim of this initiative is to develop “a web service for access to video data including geo-location information”. This service will provide an interface for requesting stream video, that can be controlled through play-back commands from a web service client. The service also will provide metadata in the video stream sufficient for a client to geo-locate the video.
Some examples of the use of geo-referenced video, apart from linear
infrastructures management, are [
Although from a different approach, it is worth mentioning the Aspen Movie Map
Project [
It has to be remarked that we do not consider here those approaches consisting in
video clips with an overall spatial reference for the whole clip. Instead, we focus on
geo-references at the level of frames or segments. A non-exhaustive list of “classical”
examples of the use video clips in geographical multimedia applications are BBC
Doomsday Project [
5 http://www.videomapper.com
visualize videos and other multimedia elements from certain localities; [
We can observe that the systems discussed in the previous section can be classified in two main groups: a first one that is mainly oriented to provide visual information for geographic features in a GIS environment, namely in contexts like road management, fire decision support or land-cover maps validation; and a second one that focuses on the enhancement of the video images with information of buildings or other elements. Our proposal follows a different direction: it uses the thematic geographic information that can be extracted from a GIS or spatial database, once a given video sequence is geo-referenced, in order to segment and index the video sequence. This process of segmentation and indexing is the basis for the definition of a digital video library, and consequently our work is more related to video information retrieval than to the abovementioned systems. Our library will enable external clients to access the video collection in order to retrieve elements satisfying thematic criteria. The main type of query that the system has to support is, given a thematic class, to retrieve the video segments depicting that theme. The results of a query will be a collection of video fragments (segments) extracted from different video sources. For example, the query asking “forests” would return all the fragments of videos in the video base containing forests or related themes (subclasses of forest). We have called VideoGIS to the system that implements this type of integration of geo-referenced video and geographic information. It has to be remarked that our semantic framework for geographic information provides a formal definition for the thematic classes that describe video sequences. Furthermore, the semantic services defined in the framework will be used in the segmentation and indexing process.
Each video segment is indexed according to a set of thematic classes from the ontology, T1,...,Tn (referred to as indexing themes from now on). They have to be previously selected by the user responsible of the indexing process. Each indexing theme gives rise to a different layer of meta-information of the video segment. This way, each layer of meta-information corresponds to a particular view of the thematic information of the video segment, focusing on one of the indexing themes.
It has to be noted that, although it is not compulsory, all the videos in the collection are usually indexed according to the same set of themes. However, there is no problem on indexing different videos with different focus, i.e. using different sets of indexing themes. Furthermore, T1,...,Tn are usually disjoint, or at least they do not have common subclasses in the thematic ontology. But again, this is not compulsory and the user may select themes with non-empty intersection.
It is also important to note that the indexing process is driven by this set of indexing themes, and not by datasets. This way, the user selects the themes that s/he is interested in, but s/he must not know in which datasets this thematic information can be found. The system will be responsible for finding the involved datasets and dataset values.
Our video model is based on a stratification approach [
Key-frame X: ...
Y: ...
Orientation: ...
Amplitude angle: ...
Area of vision Dataset i class 1: ... m2 class 2: ... m2 ...
Dataset j class 1: ... m2 class 2: ... m2 ...
Dataset i
Dataset j
Each frame of a geo-referenced video V has several properties related to the location and focal properties of the camera. Some of them are constant along the video, as receiver length (s) and width (w), while others may change frame-by-frame, as camera position (X,Y,Z), orientation (ρ), tilt (θ) and focal length (l). Furthermore, the angle of vision (α) of each frame can be computed from l, s and w. All these properties make it possible to obtain the area of vision Aj for a given frame fj. We have also used a Digital Elevation Model and a viewshed analysis algorithm to determine which areas can be seen from the camera position and which are hidden by elevations, up to a certain distance d from the camera.
Segmentation is carried out according to the thematic information in the area of
vision of each frame of the video. The process is equivalent to the feature-based
segmentation described in [
The segmentation algorithm firstly obtains the set of datasets where information related to the indexing theme Ti can be found in the repository. This is done through the service findDatasetsForTheme defined in the semantic framework. Once the area of vision is obtained, the obtained datasets are integrated using the service integrateDatasetsByTheme. As a result, a new virtual dataset dsi summarizing the information related to Ti in the source datasets will be created. This virtual dataset contains p values {Cij1,...,Cijp}, where i indicates the layer or indexing theme and j the number of frame, and where each of these values Cijk is a reference to a thematic class in the ontology. If this set of values is different from the one from the previous frame, it indicates a change of segment, and consequently the previous segment is finished and a new one is created. All this process is shown in the segmentation algorithm just below, where index i is used for indexing themes and j for frames.
Segmentation algorithm for each Ti ∈{T1,...,Tn} do
DsSeti = findDatasetsForTheme(Ti)
//where DsSeti is a set of m datasets {dsi1,..., dsim} for each frame fj in V do for each dsiu∈DsSeti do
dsAiuj = region of dsiu contained in Aj end for dsij = integrateDatasetsByTheme(dsAi1j,...,dsAimj, Ti) // where dsij has p values {Cij1,...,Cijp} if {Cij1,...,Cijp} ≠ {Ci(j-1)1,...,Ci(j-1)q} then if j ≠ 0 then
finish segment at layer i end if start new segment at layer i end if end for finish segment at layer i end for
Each segment obtained through this algorithm has now to be indexed according to the set of thematic classes in its corresponding area of vision. The following tuple represents an index entry for a given segment at layer i: < V, T, sf, ef, C, area > where V is the video (identified through its URI); T is one of the indexing themes; sfi and efi are respectively the starting and ending frame of the segment being indexed, which belongs to the layer related to T; C is one of the visible thematic classes at the segment; and area is the average of the areas of the spatial extent of C at every frame in the segment. Note that the set of visible thematic classes at a given layer may be empty. In this case, the index entry would not be created.
The following algorithm includes both segmentation and indexing. We use index i for indexing themes, index j for frames, and index k for visible thematic classes in a frame or segment. The variable areaCik is used to store the sum of the areas of the spatial extent of Cijk at every frame j along the current segment at layer i, while the variable sfi stores the frame where the current segment at layer i started. Note that segments with an empty set of visible themes are not indexed.
Segmentation and indexing algorithm for each Ti ∈{T1,...,Tn} do
DsSeti = findDatasetsForTheme(Ti) for each frame fj in V do for each dsiu∈DsSeti do
dsAiuj = region of dsiu contained in Aj end for dsij = integrateDatasetsByTheme(dsAi1j,...,dsAimj, Ti) // where dsij has p values {Cij1,...,Cijp} if {Cij1,...,Cijp} ≠ {Ci(j-1)1,...,Ci(j-1)q} then if j ≠ 0 then // finish segment at layer i: for each Ci(j-1)k∈{Ci(j-1)1,...,Ci(j-1)q} do add index entry
<V, Ti, sfi, j-1, Ci(j-1)k, areaCik/(j-sfi)> end for end if // start new segment at layer i: sfi = j forareeaacCihk C=ijk|∈e{(CCiijj1k,,.d.s.ij,)|Cijp} do end for else // continue segment at layer i: for each Cijk∈{Cij1,...,Cijp} do
areaCik = areaCik + |e(Cijk,dsij)| end for end if end for //finish segment at layer i: for each Cijk∈{Cij1,...,Cijp} do add index entry
<V, Ti, sfi, j, Cijk, areaCik/(j-sfi+1)> end for end for
Since two consecutive frames usually have the same thematic information, a simple optimization can be done to this algorithm by only checking sets of themes each F frames. F is a parameter that the user may modify, and its default value is the same as the property fps. This indicates that, by default, a comparison is made for each second of video. If the comparison between frames fa and fa+F results in different sets of themes, fa will be then compared with fa+F/2. The distance between frames being compared is successively divided by two until the exact frame for the end of segment is found.
The main query that has to be implemented in this digital library is the retrieval of the video segments that depict a given thematic class T from the ontology. The implementation of this query consists in obtaining those video segments with one index entry containing a theme C subclass of T in its fifth element.
< ..., ..., ..., ..., C, ... > C ⊑ T
In case of several segments to be returned, they are also ordered according to a simple ranking algorithm that prioritizes those index entries that maximize the ratio between areaCik and the average area of vision of the segment.
Another significant type of query is related to the playback of a whole source video sequence. For this video (first element of the index), and for each indexing theme or layer (second element), its corresponding index entries are retrieved and ordered according to the starting frame (third element). This produces a metadata structure of n layers that will be attached to the video in order for clients be able to interpret the related thematic information, as it will be described in the next section.
Finally, it is worth clarifying that this thematic-based approach does not exclude other information being used in parallel segmentations and indexes. For instance, another layer could be used to include manual annotations, which would determine a separate segmentation layer with its own indexing structure permitting non-thematic types of queries.
The video segments that are returned as the result of a query have to be provided with meta-information describing its thematic content. This way, clients can be developed to take profit of this meta-information. Spatial information will only be processed at the time of indexing the video sequence, and consequently, clients do not need to have access to geographic datasets and, since they do not have to deal with spatial data, can be more generic.
A video segment comprises only a layer of meta-information, containing its indexing theme and a list of visible themes (related to the indexing theme) with their spatial extent area. The segment meta-information also includes the camera properties of a representative frame, which will enable clients to find spatial relations between video segments according to their camera location. The following XML fragment exemplifies how the meta-information of a video segment is structured. <VideoSegment video="..." indexingTheme="Ti" startFrame="s" endFrame="e" averageVisibleArea="..."> <CameraPropertiesAtFrame frameNumber="..."> <CameraPosition> <X>X</X> <Y>Y</Y> <Z>Z</Z> </CameraPosition> <CameraOrientation>ρ</CameraOrientation> <CameraTilt>θ</cameraTilt> <CameraAngleOfVision>α</CameraAngleOfVision> </CameraPropertiesAtFrame> <VisibleTheme theme="C1e1" spatialExtentArea="area11" /> <VisibleTheme theme="C1e2" spatialExtentArea="area12" />
... <VisibleTheme theme="C1ep
spatialExtentArea="area1p /> </VideoSegment >
It has also to be noted that translations of the thematic classes to classes in a vocabulary understandable by the client may be needed. This would be done using one of the services of the semantic framework, translateThemeToDataset.
In the case of visualizing a complete video, the Video element contains a list of VideoSegment elements, as the following fragment of XML code exemplifies: <Video uri="..." fps="fps"> <VideoSegment video="..." indexingTheme="T1" startFrame="s11" endFrame="e11" averageVisibleArea="...">
... </VideoSegment > <VideoSegment video="..." indexingTheme="T1" startFrame="s12" endFrame="e12" averageVisibleArea="...">
... </VideoSegment > ... <VideoSegment video="..." indexingTheme="T2" startFrame="s21" endFrame="e21" averageVisibleArea="...">
... </VideoSegment > ... </Video>
The schema for the XML documents appearing in this section can be found in [
At the moment of writing this paper, we have conducted some initial tests of the digital video library. The segmentation and indexing algorithms have been implemented as a plug-in of a GIS. In these tests, for simplicity reasons, we have used virtual video sequences obtained from a 3D environment. To do so, we have developed a tool that uses a 3D digital elevation model (with the orto-photo as texture) as a virtual world. Once the user draws a path on a map, the camera is moved in the 3D world according to the path, generating both a video sequence and its georeference information.
Apart from a deeper evaluation with “real” videos, our plans for future work are mainly focused on exploring how the results of a query can be better presented in a more usable way.
Storyboard is the most widely used metaphor for showing the results of searches.
Following this metaphor, a video is represented by means of a set of temporarily
ordered shot’s key-frames that are simultaneously presented on screen as thumbnails.
This type of interface avoids downloading big amounts of video data and provides a
quick general view of the video. In the case of queries returning several video
segments, each one is represented by means of a thumbnail of one of its frames.
Storyboards are used in the majority of video retrieval systems. For instance, [
To partially avoid these problems, another type of interface was developed, called video collage. It aims at visualizing news video segments that results from a query, together with their respective contexts in Informedia [23, 24]. A video collage contains a rectangular panel where news video segments are placed by means of thumbnails, as well as text lists showing the “who,” “what,” “when,” and “where” information of the selected video segments. The panel may be organized either spatially or temporarily. In the spatial distribution, segments are placed on a map according to the location names that they contain. In the temporary distribution, video segments are organized according to two axes: the vertical axe is the relevance of the news video segment in respect to the query, while the horizontal axe corresponds to the date when the news were broadcasted. The user may change the granularity of the panel in both cases in order to properly visualize crowded spatial regions or time periods.
For our system, we would like to explore a different approach that goes beyond a retrieval system aiming at a hypermedia presentation generator. This way, given a thematic query, the system returns not just a list of resulting video segments, but a hypermedia presentation dynamically generated. This presentation contains two main parts: on the one hand, a video sequence dynamically generated from a selection among the resulting video segments, and on the other hand, a textual information describing the thematic information attached to video segments and suggesting links to other information or segments. We call video-itinerary to this dynamic presentation, since the result of searching a certain theme is a kind of visual itinerary along the locations where this theme is present. How video-itineraries are dynamically generated and how they can be navigated to build a hypervideo network are the two key issues here.
Regarding the first issue, the generation of the video-itinerary, the most relevant research topic is how the composition of segments can provide a narrative thread to the final sequence. The two main semiotic elements of films, mise-en-scene (what can be seen in a shot) and montage (how shots are sequenced, also called editing) influence this process. In our case, mise-en-scène is described through the thematic information associated to segments, but the way they are selected and temporarily organized may provide extra narrative elements to the final sequence. Montage treatises contain different techniques that have to be considered for giving continuity to sequences and providing them with the desired narrative (for instance temporal or spatial change). Although techniques based on sound and on characters (for instance in a dialogue sequence) cannot be applied in our context, other techniques could be considered. Particularly relevant for our context is documentary montage theory. In our case, montage may be influenced by different factors, including the relevance of the searched theme in the obtained video segments, or the location of the camera in video segments in order to group them according to spatial criteria providing the feeling of an itinerary. In this context, the user may also be interested in specifying an expected duration for this video and a type of rhythm, factors that would also have influence on the selection and composition of segments. Detection of zoom operations or other typical camera movements could also be considered.
The other relevant research issue is navigation, which is determined according to two main axes: thematic information and spatial location. This way, links can be automatically created from a segment to other segments containing related geographic themes or to segments that belong to the same spatial area. Other spatial relations between segments could also determine links, like segments closer to a certain distance or even those with a location that can be reached following a certain direction (North, South, East or West) from the current segment. The result is a hypervideo network that is dynamically built. 22. Smeaton, A.F. and H. Lee, Designing the User Interface for the Físchlár
Digital Video Library. Journal of Digital Information, 2002. 2(4). 23. Ng, T.D., et al. Collages as Dynamic Summaries of Mined Video Content for Intelligent Multimedia Knowledge Management. in AAAI Spring Symposium Series on Intelligent Multimedia Knowledge Management. 2003. Palo Alto, California, USA. 24. Christel, M., et al. Collages as Dynamic Summaries for News Video. in ACM Multimedia. 2002. Juan-les-Pins, France.