The paper introduces a video surveillance and event detection framework and application for semi-supervised surveillance use. The system development follows the guidelines of national strategy for surveillance systems. The aim is to generalize the interoperability, compatibility and legality in camera surveillance systems in Hungary. The system's intended use is in automatic mode on camera feeds that are not actively watched by surveillance personnel, and should raise alarms when unusual events occur. We present the current detector filters, and the extendable modular interface. Filters include local and global unusual motion detectors, left/stolen object detector, motion detector, tampering/failure detector, etc. It has been tested in real life situation for police street surveillance.
Visual surveillance and activity analysis has attained great interest in the field of
computer vision research [
Post-proceedings of the 2nd International Conference on ICT Solutions for Justice (ICT4Justice 2009)
The system consists of a user interface and a database backend. The interface runs on a dual display workstation, which also contains the frame grabber cards that accept the camera feeds. The feeds come from large camera matrix multiplexers. The operators monitor and use the system through this interface, while the database backend can be anywhere.
A single interface instance can handle several live feeds simultaneously, and each can be assigned a chain of freely combined filters (i.e. filter chains). The handlers and the filters were written to be real time, with SMP and heavy multithreading in mind. Increasing the number of processor cores can increase the possibility of using more filters on more feeds. Modules/filters can be added easily, either by coding them by using a provided class template as internal filters, or by a provided a library template, as a plugin. Either way, the coder needs only focus on the core algorithm, interfacing is seamless.
In this section we describe some of the more important modules/filters currently deployed in the framework. The system also contains classical surveillance functions. The need of constructing and displaying a panorama image of the scene arose since there have been a lot of panning cameras that cover a large field of view. This module allows us to construct and display the full field of view of a camera for the operator, and also to identify the actual camera position. The method continuously registers the incoming frames and builds a mosaic. The properties of these cameras are totally unknown and different on each camera source.
For this filter, the goal was to detect unusual large motion patterns [
The second filter also detects unusual motion patterns [
The algorithm of this filter detects fighting people (a few people or small group), and raises an alarm when such disorderly motion patterns are detected in the video stream. The tuning of this algorithm is easy and mostly invariant to the characteristics of video (spatial resolution, refresh rate, view parameters, etc.). This filter runs in real time.
In conventional video surveillance applications, the aims of background modeling and
background subtraction modules are usually limited to moving object detection and
analysis. However, relevant information can be exploited by following the changes in
the background as well. We implemented a filter [
For indexing and retrieval of frame and video data we created a searchable video database application, on which textual and content-based searches can be performed, and also an annotation application which can be used to assign annotations to archived video footage. After a video – with or without annotations – is imported into the database, several content based feature extractors are automatically run (i.e. average color, color structure, texture, motion, edge histogram, day/night, grass/sky, etc.) that will be indexed (by a modified BK-tree based approach) and used when searches based on a model will be performed. Textual data, which comes from the annotations, is also stored into the database, thus textual and content based queries can both be performed.
All the modules of the unusual event detection framework produce alarm and module-output data in XML markup which is imported into a relational database with native markup language support. This provides us with the event data and details that can be later looked through and searches can be performed upon. The main data that gets stored into the database are the id of the alarm module, the time of the occurrence, and description of the event, associated filter data for later processing, the id of the camera feed on which the event occurred, optional annotations that can manually be assigned to events, and a frame taken at the time of the event from the live camera feed.
Besides presenting the results in decreasing order of relevance to the user on the web interface above, we created a tool which provides a flexible way for browsing among database shots and images, easily selecting query shots/images, performing textual queries, viewing/editing/assigning annotations and categories to shots/images, 2D and 3D visualization of results and images belonging to a specific category.
Most surveillance systems are subject to privacy control legal issues. Unusual events generating alarms are very rare, but watching and recording usual human activities are not accepted by legal authorities keeping the human rights for privacy. Implementing automatic alarming plug-ins into a surveillance system can help to control scenes of sensitive private areas as well without hurting the privacy. The above mentioned automatic capabilities make the system privacy-friendly, avoiding the unnecessary human watching.
The development of the introduced system is synchronized to the Hungarian
videosurveillance strategy [
We presented an automatic surveillance system, which is intended to be an aid for surveillance operators who handle hundreds of feeds, thus being physically unable to watch all feeds at once. We presented some of our current filters that are the base of the unusual event signaling and review framework. The actual version has been tested at local police surveillance stations, and we are working towards creating a finalized deployable version. The system is being developed to be as much modular and extendable as possible, with easy integration with existing systems.
The most promising result is the creation of national strategy for admissible surveillance systems. The further developments will support ASP (Application Service Provider) as a centralized event and quality monitoring and retrieval backend.