The paper will present a video surveillance and event detection and annotation framework for semi-supervised surveillance use. The system is intended to be used in automatic mode on camera feeds that are not actively watched by surveillance personnel, raising alarms and enrolling annotation data when unusual events occur. We present the current detector filters, and the easily extendable modular interface. Current filters include local and global unusual motion detectors, left object detector, motion detector, tampering/failure detector, etc. The system stores the events and associated data, which can be organized, searched, annotated and (re)viewed. It has been tested in real life situation for police street surveillance, and we are working towards developing a deployable version.
While the literature of content based image search contains quite a large number of
proposed and in some cases practical systems, the open literature of real time
surveillance feed analysis is much more limited. Also, processing surveillance feeds
for event detection is mostly done as a post-processing feature, working with archived
footage [
In simple terms, the system consists of a user interface and a database backend. The interface runs on a workstation with dual display, which also contains the multihead frame grabber cards that accept the camera feeds. The feeds come from large camera matrix multiplexer stations. The users monitor and use the system through this interface, while the database backend can be anywhere, given the proper internet connections. Figure 1 shows a simplified diagram of the general system architecture, while Figure 2 shows a snapshot of the main interface, handling three feeds.
The main application can handle a maximum of four live camera feeds simultaneously, and each feed can be assigned a chain of selected detection filters (which we call filter chains). The feed handlers, the filter chains, and the filters themselves were all written to be real time, and with SMP and heavy multithreading in mind; all filters and chains run separately and concurrently. Increasing the number of available processors can greatly increase the possibility of using more filters on more feeds. Currently all filters run real time, but a combination of multiple filters on multiple feeds can quickly run into processing and memory bottlenecks. Thus a careful selection of hardware and filter combinations is necessary.
Modules/filters can be added easily, either by coding them by using a provided class template as internal filters, or by a provided a library template, as a plugin. Either way, the coder needs only focus on developing the core algorithm, interfacing is seamless. In this section we describe some of the more important modules/filters currently deployed in the framework. The system also contains classical surveillance functions like image and video archiving, large display of the feeds on a secondary monitor and so on, which we will not detail here. The functions described here are all automatic and consist of a panorama image creator from panning camera feeds, maskable motion detector, camera jump detector for cameras that iterate among different stationary positions, unusual global and local motion detector, fight detector, left object detector, camera fail/tampering detector, annotation, search and review of events. Any order of filter combinations can be assigned to each camera feed separately, and they will run concurrently and independently of each other. All filters run automatically, in real time, and need no manual intervention.
The need of constructing and displaying a panorama image of the scene arose since
there have been a lot of panning cameras that cover a large field of view. This module
allows us to construct and display the full field of view of a camera for the operator,
and also to identify the actual camera position. The method continuously registers the
incoming frames and builds a mosaic. The properties of these cameras are totally
unknown and different on each camera source. There are some articles (e.g. [
To achieve this goal, in a learning phase long term global statistics are built from direction distributions and typical motion types based on extracted motion fields of the image sequence. Then, in the detection phase, we try to fit the actual motion data to one of the statistics, and raise alarms when no good fit can be found.
The motion field extracted from the image sequence is cut every t1 second into t 2 long segments, and we take the mean of such segments as a sample. If t1 < t 2 then we will get overlapping segments, which will help in smoothing the blockiness at segment borders. We keep collecting the samples for a t 3 time period, which will produce N samples. From the samples directional distributions are constructed, and directional histograms are built from the motion fields. The histograms are quantized into ε degree bins between 0 and 360. These directional histograms will represent the typical motion forms of the scene.
In the learning phase, the N samples are classified into k classes by K-means clustering. Distance between the samples is calculated by L2 norm. K-means needs a k to be given a priori, which we overcome by starting with a large class number, then performing a class consolidation step. If the means A0, B0 of classes A and B are closer than n0, then B is merged into A with a new A0* mean. In the end we will have k* classes, where k*<k.
During the detection phase, a decision step follows, where we try to fit the actual
sample to the ξj (j=1..k*) distributions with at most 3σj (jth deviation) discrepancy. If it
fits none of the distributions, an unusual motion form alarm is raised. Figure 5 shows
samples from the operation of this filter.
This filter (based on [
We collect eight-bin motion direction histograms for all image pixels. Larger
number of bins could enhance the adaptation, but would also increase the uncertainty.
We assume that the relative occurrence of motion vectors gives an effective estimate
of the empirical probability:
Dir ∈ {i ⋅ 45°, i = 0,7} . The probability that an observed vector belongs to an
unusually moving object is P (U ) Dir = 1 − PDir . The obtained local statistics are
smoothed by spatial averaging, with a Mean Shift [
PDir = Si PDir where
Si ∈ S = {S1, S2 ,....SN } and Si = [ x0 , y0 ,...xn , yn , PDir ] . where each segment Si is a connected component of the image labeled with a probability distribution PDir obtained by the classification step.
The method is based on statistical processing of raw data without object level understanding, and uses spatio-temporal information for the analysis of motion. We assume that the unusual event happens on at least two consecutive frames, suggesting a Markov Chain property: if we find an anomalously moving pixel and we estimated its motion direction at time t, then, projecting back onto the previous frame there should also be - with high probability - a corresponding anomalous pixel. This is formalized as:
P (U ,M ) x, y,t = P (U ) Dir ,x, y,t ⋅ max{P (U ) Dir ,x', y ',t −1}
x', y '∈R where R is the 5x5 pixel neighborhood of the motion compensated position (x’ and y’).
The algorithm of this filter detects fighting people (a few people or small group), and
raises an alarm when such disorderly motion patterns are detected in the video stream.
The tuning of this algorithm is easy and mostly invariant to the characteristics of
video (spatial resolution, refresh rate, view parameters, etc.). This filter runs in real
time, on live camera feeds. Trajectories are constructed from multiscale
LucasKanade-tracked [
Figure 7 shows two excerpts from real feeds where people were fighting, red overlay showing the frames where the filter raised an alarm signal. 3.6
In conventional video surveillance applications, the aims of background modeling and background subtraction modules are usually limited to moving object detection and analysis. However, relevant information can be exploited by following the changes in the background as well. We implemented a filter, which not only detects objects moving in front of the camera, but it detects changes in the static background and measure, • curved k = mean, ∑ ( mk − tk 2 )/ N is the deviation from the trajectory’s signals the appearance of new objects (i.e. objects that are brought into the field of view, then left there, see Figure 8, or objects that are taken from the field). The method can be used to observe abandoned or stolen objects, which is an important surveillance task.
a b
The proposed method (building on [
{x[
To model the recent history of the pixels, [
K P(x[t] (s)) = ∑ wk[t] (s) ⋅η (x[t] (s),μ k[t] (s),σ k[t] (s))
k=1 where k = 1,K, K are unique and in time static id’s of the mixture components, while η (.) is a Gaussian density function, with given μ mean and σ deviation. We ignore multi modal background processes, and consider the background Gaussian term to be equivalent to the Gaussian component in the mixture with the largest weight.
The mixture parameters are iteratively refreshed. The weight as updated as follows: w[t+1] (s) = (1 −α ) ⋅ wk[t] (s) +α ⋅ M [t] (k, x[t] (s))
k where the following matching operator is used: 1 if M [t] (k, x[t] (s)) = 0 x[t] (s) − μ k[t] (s) σ k[t] (s) otherwise < c
The mean and deviation parameters of the mixture components are similarly updated. Denote component with the maximal weight by: k m[ta]x (s) = arg max wk[t] (s)
k=1,...,K
Hereon, the background model can classify each pixel into three classes:
foreground, background and changed background as follows:
• If M [t] (k m[ta]x (s), x[t] (s)) = 0 : pixel s is foreground at time t .
• Else, if k m[ta]x (s) ≠ k m[ta−x1] (s) : pixel s is changed background
• Else: pixel s is background at time t .
This filter detects when a periodic camera movement (i.e. automatic panning camera)
is interrupted by an operator. This is an important case, since the system should run
on unsupervised feeds and stop when an operator is active. To detect the panning
period length, a method like [
Detection of the intervention is an online process and must be recognized
immediately. Building on the Lucas-Kanade [
After statistical analysis two hysteresis thresholds are calculated. If the value calculated from the current optical flow is outside the thresholds an intervention is signaled. This indicates that the operator e.g. stopped the panning, zoomed to a region of interest, or stopped the movement.
Sudden significant changes in illumination are also detected by another filter, which is useful to signal when e.g. the image gets too dark or too bright (relatively), lights are switched on or off in indoor views, the camera gets covered. 3.8
The application’s interface provides the possibility of annotating frames from different feeds. This is possible by selecting a certain feed and the annotation tool, and then putting in some text (Figure 10). This annotation will be stored in the database, as a viewable, searchable entry (Figure 11). This way it is possible to add comments to certain events/scenes to provide an easier way to review and search stored details. 3.9
Without going into details, it should be mentioned that the framework also has the traditional capabilities of archiving frames and videos besides events. Frame and video archiving can be done automatically in the background, for all or selected feeds, with custom frame rates. In the case of events at least a frame will be stored along the event in the database, which will become handy when running queries among the stored alarm data, showing not just textual and filter data, but also a browseable associated frame series. When a filter in any of the filter chains signals an alert, a message with the alert details and filter data at the time of the alert is sent to the main framework, which creates a database entry with the details of the alert and the filter data, also storing a captured frame. These data will be searchable, by location, time interval, type of alerts, annotation texts, and sample images. Also, the alarm also shows a visual alert on the main user interface, in multiple ways, to be easily noticeable. One way of visual alerting is shown on Figure 12: each feed has an associated event histogram graph, which shows the last events, color coded for different filters. Clicking on a graph point brings up a browseable view dialog, showing the alert details (Figure 13). Also, searching, viewing, and browsing through the stored events is possible in the search and query dialog (Figure 14).
Besides textual, time interval, alert type – and so on – searches, a basic content based search option is also available. It provides the possibility of selecting an area of interest on an alert frame, and search for similar frames in the database (Figure 17).
The search engine uses local maxima to calculate the distance between scale
invariant features (SIFT) [
D(i, j) = D(i, j) ⋅ (1 + w1(i) − w2( j) ) where w1 and w2 come from SIFT. The DTW works the classical way, only with a different input: instead of vectors, the compensated D matrix is used. The resulting distance is used to compare two locmax descriptors:
Dist = DTW (D) / k
Dimension reduction. The main problem with searching by SIFT descriptors is the high dimensionality of the feature space, which is also why fast tree structures (e.g. KD-Tree) cannot be used. Our proposed approach uses only the dominant values of locmax descriptors. In this case, the significant part of the distance between two features is estimated from the most important locmax values instead of the Euclidean distance of the whole feature vector. We have found empirically, that the optimal dimensionality is 7. The importance of these reduced descriptors is that there is no further need for supervised learning in a dimension reduction technique (e.g. covariance matrix). The proposed descriptor can be computed by a locmax search and a sort algorithm on locmax values. A sample matching using such values is shown in Figure 16.
We presented an automatic surveillance system, which is intended to be an aid for surveillance operators who handle hundreds of feeds, thus being physically unable to watch all feeds at once. We presented some of our current filters that are the base of the unusual event signaling and review framework. The actual version has been tested at local police surveillance stations, and we are working towards creating a finalized deployable version. The system is being developed to be as much modular and extendable as possible, with easy integration with existing systems.
Acknowledgments. This work has been supported by MUSCLE (FP6-507752) and JUMAS (FP7-214306) projects.