The videos recorded by video surveillance systems represent a key element in a police inquiry. Based on a spatio-temporal query specified by a victim, (e.g., the trajectory of the victim before and after the aggression) the human operators select the cameras that could contain relevant information and analyse the corresponding video contents. This task becomes cumbersome because of the huge volume of video contents and the cameras' mobility. This paper presents an approach, which assists the operator in his task and reduces the research space. We propose to model the cameras' network (fixed and mobile cameras) on top of the city's transportation network. We consider the video surveillance system as a multilayer geographic information system, where the cameras are situated into a distinct layer, which is added on top of the other layers (e.g., roads, transport) and is related to them by the location. The model is implemented in a spatio-temporal database. Our final goal is that based on a spatio-temporal query to automatically extract the list of cameras (fixed and mobile) concerned by the query. We propose to include this automatically computed relative position of the cameras as an extension of the standard ISO 22311.
The number of video surveillance cameras increases in public and private areas (e.g., in train and metro stations, onboard of buses and trains, inside commercial areas, inside enterprises buildings). For example, some estimations show that there are more than 400000 cameras in London and that only the RATP also known as Régie Autonome des Transports Parisiens (English: Autonomous Operator of Parisian Transports) surveillance system comprises around 9000 cameras in Paris. In these conditions, any person that lives and walks in those two big European capitals is likely to be captured many times during a day (up to 300 times in London) by several video surveillance systems (e.g., the traffic surveillance cameras, the cameras in the subway, and the cameras of a commercial centre). The only markers
This standard does not consider the video surveillance cameras’ mobility or their fields’ of view modification. In fact, at the beginnings of video surveillance systems the cameras were placed in fixed locations in order to monitor indoor and outdoor places. With the improvements in the hardware and software technologies, on-board cameras are more and more employed in mobile vehicles (e.g., buses, police cars). This cameras’ mobility makes the task of security agents even more difficult in the context of an inquiry, when they have to analyse a huge amount of video contents and to have supplementary knowledge on the system’s characteristics (e.g., the bus timetables, the city transport plan) in order to select the most appropriate video contents.
In this context, our goal is to provide users with tools that could assist them in their research and reduce the research space. In order to achieve this objective, in this article, we propose an extension of the ISO 22311 standard in order to take into account the cameras’ mobility. We consider the video surveillance system as a multilayer geographic information system, where the cameras are situated on a distinct layer, which is added on top of the other layers (e.g., roads, transport) through the location. We implemented our solution using a spatial database in order to select the cameras that might have acquired video contents corresponding to a user’s spatio-temporal query.
The remainder of this paper is organized as follows. After a review of related work concerning the three aspects addressed in this paper, video surveillance systems, standard ISO 22311 and mobile objects modelling in the Section 2, Section 3 presents our multilayer modelling approach. This model is implemented using a spatio-temporal database. Some queries that can be answered based on this database are presented in Section 4. Finally, Section 5 concludes and discusses possible future research.
The generic schema of a video surveillance system is illustrated in Figure 1. The content is captured and stored in a distributed manner and analysed in a control centre by human operators that watch a certain number of screens displayed in a matrix (the Video Wall in Figure 1).
There is a big diversity of cameras and sensors that constitute
the acquisition part of surveillance systems and a heterogeneity of
their installation contexts (e.g., on the halls or platforms of
railway or metro stations, on-board of trains and buses, on the
streets, in commercial centres or office buildings). Therefore, we
have fixed and mobile cameras having different technical
characteristics (most of the time dynamic) (see Figure 2 for an
example of such cameras) [
Sensor type and dimension: CMOS, CCD Transmission type: analogous/ IP Angle of view (horizontal and vertical), focal distance, pan-tilt-zoom, field of view orientation, visible distance etc.
We started by analysing the way a query is processed in a video surveillance system today. When a person (victim of an aggression for example) files a complaint, he is asked to fill a form describing the elements that could help the investigators to find the relevant video segment (the Figure 3 illustrates an example of such form). Based on the spatial and the temporal aspects of the query, the surveillance operator uses his own knowledge concerning the spatial disposal of the cameras’ network in order to select the most relevant video contents. Then he analyses these contents by playing them on the different screens that he has in front of him. The monitors themselves show no spatial relationship of any kind, only the numbering of the cameras is in a somewhat logical order.
Therefore, the operators’ tasks become cumbersome taking into consideration the huge volume of video contents to be analysed, the mobility and the different characteristics of cameras. Moreover, in the current systems, most of the stored contents is not exploitable because of the recording’s low quality. This lack of quality is often caused by inappropriate installation of cameras, bad shooting, bad illumination conditions etc. The operator has no a priori knowledge on the quality of the video contents and thus he loses time by visualizing the low quality contents also.
The video surveillance domain has seen a big number of
commercial systems developed [
We can summarize by saying that there is a growing concern
in the research and industrial environments for developing
algorithms for video content analysis (VCA) in order to
automatically index content and detect objects (e.g., abandoned
packets or luggage) and events (e.g., intrusions, people or vehicles
going the wrong way) [
In this article, we are going to address also the lack of interoperability between different surveillance systems. In the context of an inquiry, the police might need to analyse data from different sources (systems), so it is important that the different outputs of the systems to be interoperable. As a consequence, the big actors of the domain started to unify efforts in order to standardize the structure of folders and of metadata files generated by video surveillance systems. A result of these efforts is represented by the ISO 22311 standard that proposes a structure 2 http://cordis.europa.eu/ist/kct/caretaker_synopsis.htm
for the data issued from video surveillance systems and the metadata needed to exploit that data.
In the following, we are going to present the ISO 22311 standard, especially the part concerning the description of the cameras characteristics and mobility. We are going to highlight the interesting elements which relate to our research. 2.2 Standard ISO 22311
The Standard ISO 22311 defines an interoperability format for the data generated by video surveillance systems and for the metadata needed to exploit these huge volumes of data.
The audio visual packages (containing audio, video or metadata files) have to be structured hierarchically (in files, folders and groups of folders) according to time intervals in Coordinated Universal Time (UTC). For each group of folders it is mandatory for the system to provide a XML description of the source(s) (e.g., cameras, GPS, video analysis tools), codec(s), file formats and a temporal index enabling an easy access to the content.
The current technologies and processing power enable the analysis of video content and the extraction of metadata describing objects, events, scenes etc. This analysis depends on the acquisition context (e.g., the position of the camera, the image quality, the type of sensors). Therefore, the standard distinguishes between the systems, those that can generate such metadata (i.e., level 2 systems) and provides a general structure and dictionary for describing sensors and events (i.e., metadata).
As in this paper we are going to address the problem of cameras’ geo-localization we present the schema for the sensors description in Figure 4.
Each camera has an absolute location (GPS coordinates) as more and more of the installed cameras have an embedded GPS transmitter. But, there are many cases when the GPS is not enough because: (1) we need to model the position of the camera with regards to the video surveillance system and not to the world; (2) in some situations, for example in indoor environments, the GPS positions do not provide a good precision.
The mobile cameras are embedded in buses, train, police cars; The movement of these vehicles is constrained by a road network and a transportation network.
By analysing the standard, we can notice that it defines a relative position for a camera that is today a simple link to an image (the plan of the network of cameras or of a building). This kind of location is not easily exploitable. Furthermore, the standard does not consider the video surveillance cameras’ mobility. In order to overcome these issues, we propose to extend this standard through a multilayer modelling approach, where the network of cameras is put on top of a transportation network.
In the following, we present a state of the art of the mobile objects modelling as the cameras’ mobility management represents the main focus of this paper.
With the technology’s evolution, the mobility became very
important in the context of video surveillance systems. Not only
the objects (e.g., persons, cars) are moving in the monitored
scene, but also the surveillance cameras are moving. The great
majority of the research papers concerning the mobile objects in
the video surveillance domain concentrate on the video content
analysis in order to detect and track the objects, to interpret their
behaviour and to understand the visual events of the monitored
scene [
In the field of moving objects, a mobile object means the
continuous evolution of any object over the time, in terms of
position and dimension [
In the video surveillance domain, the objects are moving in a
constrained environment, mainly by the road network. This
environment is represented as a graph-based model [
In the management of mobile objects, a major issue is the
storage of the objects’ spatio-temporal positions. Several
strategies can be considered: using the spatio-temporal data types
defined by [
As far as we know, the video content’s mobility is not taken into account in the video surveillance domain. In this article, we want to exploit the advances in the field of mobile objects and apply them in the video surveillance domain in order to consider the mobile aspect of surveillance cameras.
As you could see in Section 2.2, the Standard 22311, defines a fix position of video surveillance camera, through the GPS coordinates and a link to an image containing the plan of the network. In order to overcome this issue, we propose to compute a relative position with regards to a map which will enable us to: Model the distances between the cameras and select the relevant cameras for a certain trajectory; Model the connections between the cameras ( e.g., possible path between camera1 and camera2 but not between camera2 and camera3 ); Model trajectories for mobile cameras; Model the fields of view and the maximum detection distances of fixed and mobile cameras.
In order to achieve this goal we took our inspiration from the
domain of GIS (Geographical Information Systems) [
By considering the video surveillance system as a GIS we benefit from the separation between the conceptual layers. Thus at any time, a new layer can be added without modifying the existing layers.
In our approach, we propose a four layer model: (1) Road network, (2) Transportation network, (3) Objects and (4) Cameras network. The Figure 5 illustrates the UML model for the first three layers.
The “Road network” layer, presented in blue in Figure 5, is based on the graph modelling approach well-known in the literature. The road network is considered as an undirected graph G= (V, E), with V a set of vertices and E a set of edges defined according to the granularity level that we want to consider (for a big boulevard of a European capital for example we can consider each segment of the road, each segment between two intersections or the entire boulevard). Each vertex has an identifier and a 2D position. Each edge is determined by two vertices.
The “Transportation network”, presented in yellow in
Figure 5, is also based on a graph model. At this level, the vertices
of the transportation network are intersections between roads, and
bus stations. Each transportation vertex has a position with
regards to a road segment. Ordered sequences of transportation
vertices constitute sections, which form lines (e.g., bus lines). The
advantage of our approach with regards to the ones proposed in
the state of the art [
The “Objects” layer, presented in red in Figure 5, models the positions of fixed and mobile objects with regards to the underlying layers.
The Fixed Object has a position on a road segment. Its
position is defined as a distance from each end of the segment.
For this kind of objects, we adopt the same localisation as the one
proposed by [
In the case of Mobile Objects (e.g., buses, police cars,
persons), the position changes in time. Each object will
periodically transmit its position using different strategies (e.g.,
each Δt seconds, each time the object is changing the segment,
when the object's position predicted by the motion vector deviates
from the real position by more than a threshold [
We distinguish two types of mobile objects: objects that move freely within the road and transportation networks (e.g., car, person) and objects of which trajectories are constrained by a “line” (e.g., buses).
On top of all these layers, we model a video surveillance cameras’ network. A simplified schema of this model is illustrated in Figure 6.
The cameras’ network is composed of fixed and mobile cameras. The fixed cameras have a 2D position that is given at installation time. The mobile cameras are associated with mobile objects (e.g., buses) and their trajectory is the same as the object’s one.
The new generation of digital surveillance cameras has embedded GPS transmitters and even compasses. The technologies developed around these cameras make it possible to automatically extract information from the camera related to its orientation, pan, tilt, zoom, focal distance, compression parameters etc.
Based on all these elements it is possible to model the field
of view for each camera and track its modifications in time. The
field of view is computed based on four parameters [
In order to select the most appropriate attributes to describe a video surveillance camera, we studied the sensor description proposed by the ISO 22311 standard, SensorGML6, KML7. We separated the identified camera’s properties in two categories: properties that could be modified over the time, and fixed characteristics.
Thus, the extension of the standard ISO 22311 is realised at three levels: Taking into account the road and transportation networks as a graph and not as an image; Taking into account the camera’s relative position and its mobility on the networks; Taking into account the camera’s characteristics change over the time.
Our model is implemented in a spatio-temporal database that can be queried by users in order to retrieve the relevant cameras for a given trajectory. The originality of our research work is given by: the fact that it combines different spatio-temporal information (e.g., road network, transportation network, objects’ positions) and computation (e.g., trajectories, field of view) within the same database; the twofold mobility, of the target objects and of the cameras.
In the next section we present the general architecture of the tool that could assist the video surveillance operators in their research based on our spatio-temporal database and some examples of queries.
Based on the presented model, our goal is to automatically select the cameras (fixed and mobile) that could contain relevant video content with regards to the user query (their field of view intersected the query trajectory).
More precisely, the idea is to compare a spatio-temporal query of the user (e.g., Rivoli Street from Louvre to Metro Chatelet the 14th of July between 10h and 14h) with the trajectories stored in our database and, for a better precision, with the cameras fields of view. The Figure 8 illustrates the generic architecture of a system based on our spatio-temporal database for assisting the video surveillance in their research.
From the Figure 8 it is easy to observe that there are two main questions when developing such system: How to query the system? and How to update the system?. As explained in the previous section our work addresses only the querying aspect that we are going to describe in the following.
First, a Query Interpreter module will transform the user query (e.g, Rivoli Street from Louvre to Metro Chatelet the 14th of July between 10h and 14h) in a spatio-temporal query. By spatiotemporal query we understand a sequence of road segments and a time interval that will be further transformed in a SQL query, by the SQL Query Generator module. The SQL query is executed on the database having as a result a list of cameras. Based on some image quality parameters a score per camera can be computed and the initial list can then be ranked according to this relevance score.
In the following we present two examples of spatiotemporal queries executed on our database implemented in Oracle Spatial 8:
The first selects the fixed cameras of which geometry (field of view) intersects the geometry of the Rivoli street; SELECT IdCamera FROM FixedCamera WHERE SDO_RELATE( camera_geom, (SELECT street_geom
FROM Road
WHERE Name ='Rivoli‘ ), 'mask=OVERLAPBDYDISJOINT querytype=WINDOW' )='TRUE'; 8http://www.oracle.com/fr/products/database/options/spatial/index .html
The second selects the mobile cameras that are associated with the buses that crossed the street within the given time interval.
Another perspective of our work is the improvement of the resulted cameras list by re-ranking it based on cameras’ characteristics (e.g., image quality, visible distance).
LET TimePeriod = Timestamp(hour(2013,1,14,10), hour(2013,1,14, 12)); SELECT ObjetID
FROM ConstrainedObject WHERE Type.MobileObject= “Bus” AND TimePeriod.ConstrainedObject (atperiods (Timestamp, TimePeriod)); SELECT DISTINCT IdMobileCamera
FROM ConstrainedObject, FreeObject, MobileCamera WHERE Intersect (MobileCamera.geom, ConstrainedObject.geom) AND Intersect (MobileCamera.geom, FreeObject.geom);
In this paper, we presented a spatio-temporal modelling approach of fixed and mobile cameras within a common transportation network. Taking our inspiration from the multilayer representation of the geographical information systems, we model spatial information about the road and transportation infrastructures and mobile objects’ trajectories in four independent layers: (1) Road network, (2) Transportation network, (3) Objects and (4) Cameras network.
Based on this modelling approach we also proposed a generic architecture for a system that could assist the video surveillance operators in their research. Starting from a sequence of trajectory segments and a temporal interval, such system generates the list of cameras that could contain relevant information concerning the query (that “saw” the query’s trajectory).
The need of such assisting tools was identified within the French National Project METHODEO. Among the project’s partners, we mention the French National Police, Thales and the RATP also known as Régie Autonome des Transports Parisiens (English: Autonomous Operator of Parisian Transports). Our approach has been validated and will be evaluated within the project.
Obviously, many questions are still left with no answer giving way to a large number of perspectives. We will present several of them in the following.
For now, our model considers only outdoor transportation and surveillance networks. We plan to extend our model to indoor spaces also in order to model cameras inside train or subway stations for example.
Our work is situated in the context of the a posteriori research in the case of a police inquiry. We would like to extend this context in the future in order to be able to process real time queries or to predict trajectories based on some statistics realized based on the stored data (e.g., average speed on some road segments).
This work has been supported by the ANR CSOSG-National Security (French National Research Agency) project METHODEO. 7 58 8 59