The number of video surveillance cameras increases in public and private areas (e.g., in train and metro stations, onboard of buses and trains, inside commercial areas, inside enterprises buildings). For example, some estimations show that there are more than 400000 cameras in London and that only the RATP also known as Régie Autonome des Transports Parisiens (English: Autonomous Operator of Parisian Transports) surveillance system comprises around 9000 cameras in Paris. In these conditions, any person that lives and walks in those two big European capitals is likely to be captured many times during a day (up to 300 times in London) by several video surveillance systems (e.g., the traffic surveillance cameras, the cameras in the subway, and the cameras of a commercial centre). The only markers available for all these videos are the id of the camera (eventually GPS coordinates) and a local date/timestamp that are not homogenous throughout the different systems.
A great majority of the existing video surveillance systems are manual or semi-automatic (they employ some form of video processing but with significant human intervention) [11]. Taking into account the huge amount of video contents that need to be handled, the purely manual approach (agents watching the videos and detecting events) becomes insufficient. The main objective in the video surveillance domain is to provide users with tools that could assist them in their research by reducing the research space and therefore the response time. These tools depend on the research context and complexity (e.g., real time surveillance of big events, police inquiry) [22].
Our work is situated in the context of the police inquiry which involves an a posteriori processing of the data in order to help the investigator to highlight (isolate) the relevant elements (e.g., persons, events). To do that, the investigators dispose of the set of recorded videos from different video surveillance systems (e.g., public, private, RATP). In order to assist the investigators in their tasks, it is important that the different outputs of the systems are interoperable, which is not currently the case. The interoperability between any video surveillance systems from the simple ones with only few cameras to the large scale systems is the main goal of the standard ISO 22311 1 . It specifies a format for the data which can be exchanged between the video surveillance systems in the inquiry context. This standard does not consider the video surveillance cameras' mobility or their fields' of view modification. In fact, at the beginnings of video surveillance systems the cameras were placed in fixed locations in order to monitor indoor and outdoor places. With the improvements in the hardware and software technologies, on-board cameras are more and more employed in mobile vehicles (e.g., buses, police cars). This cameras' mobility makes the task of security agents even more difficult in the context of an inquiry, when they have to analyse a huge amount of video contents and to have supplementary knowledge on the system's characteristics (e.g., the bus timetables, the city transport plan) in order to select the most appropriate video contents.
In this context, our goal is to provide users with tools that could assist them in their research and reduce the research space. In order to achieve this objective, in this article, we propose an extension of the ISO 22311 standard in order to take into account the cameras' mobility. We consider the video surveillance system as a multilayer geographic information system, where the cameras are situated on a distinct layer, which is added on top of the other layers (e.g., roads, transport) through the location. We implemented our solution using a spatial database in order to select the cameras that might have acquired video contents corresponding to a user's spatio-temporal query.
The remainder of this paper is organized as follows. After a review of related work concerning the three aspects addressed in this paper, video surveillance systems, standard ISO 22311 and mobile objects modelling in the Section 2, Section 3 presents our multilayer modelling approach. This model is implemented using a spatio-temporal database. Some queries that can be answered based on this database are presented in Section 4. Finally, Section 5 concludes and discusses possible future research.
The generic schema of a video surveillance system is illustrated in Figure 1. The content is captured and stored in a distributed manner and analysed in a control centre by human operators that watch a certain number of screens displayed in a matrix (the Video Wall in Figure 1).
There is a big diversity of cameras and sensors that constitute the acquisition part of surveillance systems and a heterogeneity of their installation contexts (e.g., on the halls or platforms of railway or metro stations, on-board of trains and buses, on the streets, in commercial centres or office buildings). Therefore, we have fixed and mobile cameras having different technical characteristics (most of the time dynamic) (see Figure 2 We started by analysing the way a query is processed in a video surveillance system today. When a person (victim of an aggression for example) files a complaint, he is asked to fill a form describing the elements that could help the investigators to find the relevant video segment (the Figure 3 illustrates an example of such form). Based on the spatial and the temporal aspects of the query, the surveillance operator uses his own knowledge concerning the spatial disposal of the cameras' network in order to select the most relevant video contents. Then he analyses these contents by playing them on the different screens that he has in front of him. The monitors themselves show no spatial relationship of any kind, only the numbering of the cameras is in a somewhat logical order.
Therefore, the operators' tasks become cumbersome taking into consideration the huge volume of video contents to be analysed, the mobility and the different characteristics of cameras. Moreover, in the current systems, most of the stored contents is not exploitable because of the recording's low quality. This lack of quality is often caused by inappropriate installation of cameras, bad shooting, bad illumination conditions etc. The operator has no a priori knowledge on the quality of the video contents and thus he loses time by visualizing the low quality contents also.
The video surveillance domain has seen a big number of commercial systems developed [8]. In the research area, many projects were developed as well: CROMATICA [5], CARETAKER 2 [3], VANAHEIM 3 for the indoor static video surveillance, and SURTRAIN [20], BOSS 4 [13], PROTECTRAIL 5 projects for the on-board mobile surveillance. All these heterogeneous projects concentrate on the development of the system's physical architecture and of better detection algorithms in order to obtain a fully automatic system [12], [24].
We can summarize by saying that there is a growing concern in the research and industrial environments for developing algorithms for video content analysis (VCA) in order to automatically index content and detect objects (e.g., abandoned packets or luggage) and events (e.g., intrusions, people or vehicles going the wrong way) [16] or to draw operators' attention to events of interest (e.g., alarms). However, solutions for assistance to a posteriori investigation are at a lesser stage of maturity, and to date most of the data remain unexploited.
In this article, we are going to address also the lack of interoperability between different surveillance systems. In the context of an inquiry, the police might need to analyse data from different sources (systems), so it is important that the different outputs of the systems to be interoperable. As a consequence, the big actors of the domain started to unify efforts in order to standardize the structure of folders and of metadata files generated by video surveillance systems. A result of these efforts is represented by the ISO 22311 standard that proposes a structure 2 http://cordis.europa.eu/ist/kct/caretaker_synopsis.htm 3 http://www.vanaheim-project.eu/ 4 http://celtic-boss.mik.bme.hu/ 5 http://www.protectrail.eu/ for the data issued from video surveillance systems and the metadata needed to exploit that data.
In the following, we are going to present the ISO 22311 standard, especially the part concerning the description of the cameras characteristics and mobility. We are going to highlight the interesting elements which relate to our research.
The Standard ISO 22311 defines an interoperability format for the data generated by video surveillance systems and for the metadata needed to exploit these huge volumes of data.
The audio visual packages (containing audio, video or metadata files) have to be structured hierarchically (in files, folders and groups of folders) according to time intervals in Coordinated Universal Time (UTC). For each group of folders it is mandatory for the system to provide a XML description of the source(s) (e.g., cameras, GPS, video analysis tools), codec(s), file formats and a temporal index enabling an easy access to the content.
The current technologies and processing power enable the analysis of video content and the extraction of metadata describing objects, events, scenes etc. This analysis depends on the acquisition context (e.g., the position of the camera, the image quality, the type of sensors). Therefore, the standard distinguishes between the systems, those that can generate such metadata (i.e., level 2 systems) and provides a general structure and dictionary for describing sensors and events (i.e., metadata).
As in this paper we are going to address the problem of cameras' geo-localization we present the schema for the sensors description in Figure 4.
Each camera has an absolute location (GPS coordinates) as more and more of the installed cameras have an embedded GPS transmitter. But, there are many cases when the GPS is not enough because: (1) we need to model the position of the camera with regards to the video surveillance system and not to the world;
(2) in some situations, for example in indoor environments, the GPS positions do not provide a good precision.
In the context of a video surveillance system: The mobile cameras are embedded in buses, train, police cars; The movement of these vehicles is constrained by a road network and a transportation network.
By analysing the standard, we can notice that it defines a relative position for a camera that is today a simple link to an image (the plan of the network of cameras or of a building). This kind of location is not easily exploitable. Furthermore, the standard does not consider the video surveillance cameras' mobility. In order to overcome these issues, we propose to extend this standard through a multilayer modelling approach, where the network of cameras is put on top of a transportation network.
In the following, we present a state of the art of the mobile objects modelling as the cameras' mobility management represents the main focus of this paper.
With the technology's evolution, the mobility became very important in the context of video surveillance systems. Not only the objects (e.g., persons, cars) are moving in the monitored scene, but also the surveillance cameras are moving. The great majority of the research papers concerning the mobile objects in the video surveillance domain concentrate on the video content analysis in order to detect and track the objects, to interpret their behaviour and to understand the visual events of the monitored scene [10]. Thus, the mobility of the cameras is not exploited.
In the field of moving objects, a mobile object means the continuous evolution of any object over the time, in terms of position and dimension [21]. This movement of the mobile objects can be effectuated in an unconstrained environment [18] (e.g., for hurricanes, fires) or in a constrained environment [17] (e.g., cars move on road and transportation networks).
In the video surveillance domain, the objects are moving in a constrained environment, mainly by the road network. This environment is represented as a graph-based model [6], [15], [25], where the vertices are junctions and the edges are the roads between the two junctions. [9] considers also the connectivity at each junction in order to represent the road network. [19] extends the model proposed by [9] in order to consider the predefined trajectories that some objects could have (e.g., buses). [7] proposes a mobile object data model where they consider the road and rail networks. [2] takes into account the transport network in a city as a graph and they add to each graph vertex the transport modes available (i.e., pedestrian, auto, urban rail, metro, bus).
In the management of mobile objects, a major issue is the storage of the objects' spatio-temporal positions. Several strategies can be considered: using the spatio-temporal data types defined by [9] (e.g., moving points, moving lines, moving regions), or using the dynamic attributes [23] (e.g., motion vector) which enables to limit the size of the data that has to be stored and queried.
As far as we know, the video content's mobility is not taken into account in the video surveillance domain. In this article, we want to exploit the advances in the field of mobile objects and apply them in the video surveillance domain in order to consider the mobile aspect of surveillance cameras.
As you could see in Section 2.2, the Standard 22311, defines a fix position of video surveillance camera, through the GPS coordinates and a link to an image containing the plan of the network. In order to overcome this issue, we propose to compute a relative position with regards to a map which will enable us to: Model the distances between the cameras and select the relevant cameras for a certain trajectory; Model the connections between the cameras ( e.g., possible path between camera1 and camera2 but not between camera2 and camera3 ); Model trajectories for mobile cameras; Model the fields of view and the maximum detection distances of fixed and mobile cameras.
In order to achieve this goal we took our inspiration from the domain of GIS (Geographical Information Systems) [4] and mobile objects modelling.
By considering the video surveillance system as a GIS we benefit from the separation between the conceptual layers. Thus at any time, a new layer can be added without modifying the existing layers.
In our approach, we propose a four layer model: (1) Road network, (2) Transportation network, (3) Objects and (4) Cameras network. The Figure 5 illustrates the UML model for the first three layers.
The "Road network" layer, presented in blue in Figure 5, is based on the graph modelling approach well-known in the literature. The road network is considered as an undirected graph G= (V, E), with V a set of vertices and E a set of edges defined according to the granularity level that we want to consider (for a big boulevard of a European capital for example we can consider each segment of the road, each segment between two intersections or the entire boulevard). Each vertex has an identifier and a 2D position. Each edge is determined by two vertices.
The "Transportation network", presented in yellow in Figure 5, is also based on a graph model. At this level, the vertices of the transportation network are intersections between roads, and bus stations. Each transportation vertex has a position with regards to a road segment. Ordered sequences of transportation vertices constitute sections, which form lines (e.g., bus lines). The advantage of our approach with regards to the ones proposed in the state of the art [9] is that we have two independent graphs that are connected to each other through the positions of transportation vertices. That way if the buses stations are modified or new buses lines are introduced we do not have to recompute the underlying road graph.
The "Objects" layer, presented in red in Figure 5, models the positions of fixed and mobile objects with regards to the underlying layers.
The Fixed Object has a position on a road segment. Its position is defined as a distance from each end of the segment. For this kind of objects, we adopt the same localisation as the one proposed by [9].
In the case of Mobile Objects (e.g., buses, police cars, persons), the position changes in time. Each object will periodically transmit its position using different strategies (e.g., each Δt seconds, each time the object is changing the segment, when the object's position predicted by the motion vector deviates from the real position by more than a threshold [23]) that are out of the scope of this article. We suppose that we periodically receive updates containing time-stamped GPS points that we transform into a relative position with regards to the road network (i.e., the segments). We use this information to reconstitute object's trajectory.
We distinguish two types of mobile objects: objects that move freely within the road and transportation networks (e.g., car, person) and objects of which trajectories are constrained by a "line" (e.g., buses).
On top of all these layers, we model a video surveillance cameras' network. A simplified schema of this model is illustrated in Figure 6.
The cameras' network is composed of fixed and mobile cameras. The fixed cameras have a 2D position that is given at installation time. The mobile cameras are associated with mobile objects (e.g., buses) and their trajectory is the same as the object's one.
The new generation of digital surveillance cameras has embedded GPS transmitters and even compasses. The technologies developed around these cameras make it possible to automatically extract information from the camera related to its orientation, pan, tilt, zoom, focal distance, compression parameters etc.
Based on all these elements it is possible to model the field of view for each camera and track its modifications in time. The field of view is computed based on four parameters [1]: the 2D position, the viewable angle, the orientation and the visible distance. A schema of a 2D field of view proposed by [1] is shown in Figure 7. In order to select the most appropriate attributes to describe a video surveillance camera, we studied the sensor description proposed by the ISO 22311 standard, SensorGML6 , KML 7 . We separated the identified camera's properties in two categories: properties that could be modified over the time, and fixed characteristics.
Thus, the extension of the standard ISO 22311 is realised at three levels: Taking into account the road and transportation networks as a graph and not as an image; Taking into account the camera's relative position and its mobility on the networks; Taking into account camera's characteristics change over the time.
Our model is implemented in a spatio-temporal database that can be queried by users in order to retrieve the relevant cameras for a given trajectory. The originality of our research work is given by: the fact that it combines different spatio-temporal information (e.g., road network, transportation network, objects' positions) and computation (e.g., trajectories, field of view) within the same database; the twofold mobility, of the target objects and of the cameras.
In the next section we present the general architecture of the tool that could assist the video surveillance operators in their research based on our spatio-temporal database and some examples of queries.
Based on the presented model, our goal is to automatically select the cameras (fixed and mobile) that could contain relevant video content with regards to the user query (their field of view intersected the query trajectory).
More precisely, the idea is to compare a spatio-temporal query of the user (e.g., Rivoli Street from Louvre to Metro Chatelet the 14 th of July between 10h and 14h) with the trajectories stored in our database and, for a better precision, with the cameras fields of view. The Figure 8 illustrates the generic architecture of a system based on our spatio-temporal database for assisting the video surveillance in their research.
From the Figure 8 it is easy to observe that there are two main questions when developing such system: How to query the system? and How to update the system?. As explained in the previous section our work addresses only the querying aspect that we are going to describe in the following.
First, a Query Interpreter module will transform the user query (e.g, Rivoli Street from Louvre to Metro Chatelet the 14 th of July between 10h and 14h) in a spatio-temporal query. By spatiotemporal query we understand a sequence of road segments and a time interval that will be further transformed in a SQL query, by the SQL Query Generator module. The SQL query is executed on the database having as a result a list of cameras. Based on some image quality parameters a score per camera can be computed and the initial list can then be ranked according to this relevance score.
In the following we present two examples of spatiotemporal queries executed on our database implemented in Oracle Spatial8 :
The
In this paper, we presented a spatio-temporal modelling approach of fixed and mobile cameras within a common transportation network. Taking our inspiration from the multilayer representation of the geographical information systems, we model spatial information about the road and transportation infrastructures and mobile objects' trajectories in four independent layers: (1) Road network, (2) Transportation network, (3) Objects and (4) Cameras network.
Based on this modelling approach we also proposed a generic architecture for a system that could assist the video surveillance operators in their research. Starting from a sequence of trajectory segments and a temporal interval, such system generates the list of cameras that could contain relevant information concerning the query (that "saw" the query's trajectory).
The need of such assisting tools was identified within the French National Project METHODEO. Among the project's partners, we mention the French National Police, Thales and the RATP also known as Régie Autonome des Transports Parisiens (English: Autonomous Operator of Parisian Transports). Our approach has been validated and will be evaluated within the project.
Obviously, many questions are still left with no answer giving way to a large number of perspectives. We will present several of them in the following.
For now, our model considers only outdoor transportation and surveillance networks. We plan to extend our model to indoor spaces also in order to model cameras inside train or subway stations for example.
Our work is situated in the context of the a posteriori research in the case of a police inquiry. We would like to extend this context in the future in order to be able to process real time queries or to predict trajectories based on some statistics realized based on the stored data (e.g., average speed on some road segments).
Another perspective of our work is the improvement of the resulted cameras list by re-ranking it based on cameras' characteristics (e.g., image quality, visible distance).
This work has been supported by the ANR CSOSG-National Security (French National Research Agency) project METHODEO.