Analysing human behavior is a key step in smart home applications. Many reasoning approaches utilize information of location and posture of the occupant in qualitative assessment of the user's status and events. In this paper, we propose a vision-based framework to provide quantitative information of the user's posture which can be used to deduct qualitative representations for high-level reasoning. Furthermore, our approach is motivated by potentials introduced by interactions between the vision module and the high-level reasoning module. While quantitative knowledge from the vision network can either complement or provide specific qualitative distinctions for AI-based problems, these qualitative representations can offer clues to direct the vision network to adjust its processing operation according to the interpretation state. The paper outlines potentials for such interactions and describes two visionbased fusion mechanisms. The first employs an opportunistic approach to recover the full-parameterized human model by the vision network, while the second employs directed deductions from vision to address a particular smart home application in fall detection.
The increasing interest in understanding human behaviors and events in a
camera context has heightened the need for gesture analysis of image sequences.
Gesture recognition problems have been extensively studied in Human
Computer Interactions (HCI), where often a set of pre-defined gestures are used for
delivering instructions to machines [
In a multi-camera network, access to multiple sources of visual data often allows for making more comprehensive interpretations of events and gestures. It also creates a pervasive sensing environment for applications where it is impractical for the users to wear sensors. Having access to interpretations of posture and gesture elements obtained from visual data over time enables higher-level
o Vision processing o Wireless sensor networks o Embedded computing o Signal processing
Fig. 1. The relationship between vision networks and high-level AI reasoning, and a variety of novel applications enabled by both. reasoning modules to deduct the user’s actions, context, and behavior models, and decide upon suitable actions or responses to the situation.
Our notion of the role a vision network can play in enabling novel
intelligent applications derives from the potential interactions between the various
disciplines outlined in Fig. 1. The vision network offers access to quantitative
knowledge about the events of interest such as the location and other attributes
of a human subject. Such quantitative knowledge can either complement or
provide specific qualitative distinctions for AI-based problems. On the other hand,
we may not intend to extract all the detailed quantitative knowledge available
in visual data since often a coarse qualitative representation may be sufficient
in addressing the application [
In this paper we introduce a model-based data fusion framework for human posture analysis using opportunistic use of manifold sources of vision-based information obtained from the camera network in a principled way. The framework spans the three dimensions of time (each camera collecting data over time), space (different camera views), and feature levels (selecting and fusing different feature
subsets). Furthermore, the paper outlines potentials for interaction between the distributed vision network and the high-level reasoning system.
The structure of the vision-based processing operation has been designed in such a way that the lower-level functions as well as other in-node processing operations will utilize feedback from higher levels of processing. While feedback mechanisms have been studied in active vision areas, our approach aims to incorporate interactions between the vision and the AI operations as the source of active vision feedback. To facilitate such interactions, we introduce a human model as the convergence point and a bridge for the two sides, enabling both to incorporate the results of their deductions into a single merging entity. For the vision network, the human model acts as the embodiment of the fused visual data contributed by the multiple cameras over observation periods. For the AI-based functions, the human model acts as a carrier of all the sensed data from which gesture interpretations can be deducted over time through rule-based methods or mapping to training data sets of interesting gestures. Fig. 2 illustrates this concept in a concise way.
In Section 2 we outline the different interactions between the vision and AI modules as well as the temporal and spatial model-based feedback mechanisms employed in our vision analysis approach. Section 3 presents details and examples for our model-based and opportunistic feature fusion mecahnisms in human posture analysis. In Section 4 an example collaborative vision-based scheme for deriving qualitative assessment for fall detection is described. Section 5 offers some concluding remarks and the topics of current investigation. 2
The Framework
Queries Context Persistence Behavior attributes example enable the camera to examine the persistence of those features, or to avoid re-initialization of local parameters. In the network of cameras, spatial fusion of data in any of the forms of merged estimates or a collective decision, or in our model-based approach in the form of updates from body part tracking, can provide feedback information to each camera. The feedback can for example be in the form of indicating the features of interest that need to be tracked by the camera, or as initialization parameters for the local segmentation functions. Fig. 4 illustrates the different feedback paths within the vision processing unit. 3
Collaborative Vision Network We introduce a generic opportunistic fusion approach in multi-camera networks in order to both employ the rich visual information provided by cameras and incorporate learned knowledge of the subject into active vision analysis. The opportunistic fusion is composed of three dimensions, space, time and feature levels. For human gesture analysis in a multi-camera network, spatial collaboration between multi-view cameras naturally facilitates solving occlusions. It is especially advantageous for gesture analysis since human body is self-occlusive. Moreover, temporal and feature fusion help to gain subject-specific knowledge, such as the current gesture and subject appearance. This knowledge is in turn used for a more actively directed vision analysis. 3.1
The 3D Human Body Model
Fitting human models to images or videos has been an interesting topic for
which a variety of methods have been developed. Usually assuming a dynamic
model (such as walking)[
Early vision processing CAM 2
Early vision processing CAM N
Early vision processing
Temporal fusion Temporal fusion Temporal fusion
Features Features Features
Spatial fusion
Estimate fusion Decision fusion
Model-based fusion
Human
model
movements that differ much from the dynamic model. Therefore we always need
to be aware of the balance between the limited dynamics and the capability
to discover more diversified postures. For multi-view scenarios, a 3D model can
be reconstructed by combining observation from different views [
In our work, we aim to incorporate appearance attributes adaptively learned from the network for initialization of segmentation, because usually color or texture regions are easier to find than generic features such as edges. Another emphasis of our work is that images from a single camera are first reduced to short descriptions and then reconstruction of the 3D human model is based on time
local processing and spatial collaboration in the camera network
space
Description Layers Description Layer 4:
Gestures
G Description Layer 3: E1 Gesture Elements
E2
E3 DescriFpteiaotnurLeasyer 2: F1f11 f12 descriptions collected from multiple cameras. Therefore concise descriptions are the expected outputs from image segmentation.
In our approach a 3D human body model embodies up-to-date information from both current and historical observations of all cameras in a concise way. It has the following components: 1. Geometric configuration: body part lengths, angles. 2. Color or texture of body parts. 3. Motion of body parts. The three components are all updated from the three dimensions of space, time and features of the opportunistic fusion.
Apart from providing flexibility in gesture interpretations, the 3D human model also plays significant roles in the vision analysis process. First, the total size of parameters to reconstruct the model is very small compared to the raw images, and affordable through communication. For each camera, only segment descriptions are needed for collaboratively reconstructing the 3D model. Second, the model is a converging point of spatiotemporal and feature fusion. All the parameters it maintains are updated from the three dimensions of space, time and features of the opportunistic fusion. In sufficient confidence levels, parameters of the 3D human body model are again used as feedback to aid subsequent vision analysis. Third, although predefined appearance attributes are generally not reliable, adaptively learned appearance attributes can be used to identify the person Color segmentation and ellipse fitting in local processing Background subtraction
Rough segmentation
EM: refine color models
Watershed segmentation
Ellipse fitting Maintain current
model Update 3D model (color/texture, motion)
Previous color distribution 3D human body model
Previous geometric configuration and motion Y or body parts. Those attributes are usually more distinguishable than generic features such as edges once correctly discovered.
The 3D model maps to the Gesture Elements layer in the layered architecture
for gesture analysis (lower left part of Fig. 5) we proposed in [
The Opportunistic Fusion Mechanisms The opportunistic fusion framework for gesture analysis is shown in Fig. 5. On the top of Fig. 5 are spatial fusion modules. In parallel is the progression of the 3D human body model. Suppose now it is t0, and we have the model with the collection of parameters as M0. At the next instance t1, the current model M0 is input to the spatial fusion module for t1, and the output decisions are used to update M0 from which we get the new 3D model M1.
Now we look into a specific spatial fusion module (the lower part of Fig. 5) for the detailed process. In the bottom layer of the layered gesture analysis, image features are extracted from local processing. Distinct features (e.g. colors) specific for the subject are registered in the current model M0 and are used for analysis, which may be much easier than always looking for patterns of the generic features (arrow °1 in Fig. 5). After local processing, data is shared between cameras to derive for a new estimate of the model. Parameters in M0 specify a smaller space of possible M1’s. Then decisions from spatial fusion of cameras are used to update M0 to get the new model M1 (arrow °2 in Fig. 5). Therefore for every update of the model M , it combines space (spatial collaboration between cameras), time (the previous model M0) and feature levels (choice of image features in local processing from both new observations and subjectspecific attributes in M0). Finally the new model M1 is used for high-level gesture deductions in a certain scenario (arrow °2 in Fig. 5).
An implementation for the 3D human body posture estimation is illustrated in Fig. 6. Local processing in single cameras include segmentation and ellipse fitting for a concise parametrization of segments. For spatial collaboration, ellipses from all cameras are merged to find the geometric configuration of the 3D skeleton model. 3.3
In-Node Feature Extraction The goal of local processing in a single camera is to reduce raw images/videos to simple descriptions so that they can be efficiently transmitted between cameras. The output of the algorithm will be ellipses fitted from segments and the mean color of the segments. As shown in the upper part of Fig. 6, local processing includes image segmentation for the subject and ellipse fitting to the extracted segments.
We assume the subject is characterized by a distinct color distribution. Foreground area is obtained through background subtraction. Pixels with high or low illumination are also removed since for those pixels chrominance may not be reliable. Then a rough segmentation for the foreground is done either based on K-means on chrominance of the foreground pixels or color distributions from the known model. In the initialization stage when the model hasn’t been well established, or when we don’t have a high confidence in the model, we need to start from the image itself and use a method such as K-means to find color distribution of the subject. However, when a model with a reliable color distribution is available, we can directly assign pixels to different segments based on the existing color distribution. The color distribution maintained by the model may not be accurate for all cameras, since in different cameras illumination may change. Also the subject’s appearance may change due to the movement or lighting conditions. Therefore the color distribution of the model is only used for a rough segmentation in initialization of the segmentation scheme. Then an EM (expectation maximization) algorithm is used to refine the color distribution for the current image. The initial estimated color distribution plays an important role because it can prevent EM from being trapped in local minima.
Suppose the color distribution is a mixture of N Gaussian modes, with parameters Θ = {θ1, θ2, . . . , θ3}, where θl = {μl, Σl} are the mean and covariance matrix of the modes. Mixing weights of different modes are A = {α1, α2, . . . , α3}. The EM algorithms aims to find the probability of each pixel xi belonging to a certain mode θl: P r(yi = l|xi).
However, the basic EM algorithm takes each pixel independently, without
considering the fact that pixels belonging to the same mode are usually
spatially close to each other. In [
Vl(xi) =
X αl(xj )w(xi, xj ), where αl(xj)=P r(yj=l|xj) xj (1)
Then modifications are made to EM steps. In the E step, αl(k) is changed to α(k)(xi), which means that for every pixel xi, mixing weights for different modes l are different. This is partially due to the influence of neighbors. In the M step, mixing weights are updated by αl(k)(xi) = eηVl(xi) k=1 eηVk(xi) PN (2) η controls the “softness” of neighbors’ votes. If η is as small as 0, then mixing weights are always uniform. If η approaches infinity, the mixing weight for the mode with the largest vote will be 1.
After refinement of the color distribution with POEM, we set pixels with high probability (e.g., bigger than 99.9%) that belong to a certain mode as markers for that mode. Then watershed segmentation algorithm is implemented to assign labels for undecided pixels. Finally for every segment an ellipse is fitted to it in order to obtain a concise parameterization for the segment. 3.4
Posture Estimation Human posture estimation is essentially an optimization problem, in which we try to minimize the distance between the posture and ellipses from multi-view cameras. There can be several different ways to find the 3D skeleton model based on observations from multi-view images. One method is to directly solve for the unknown parameters through geometric calculation. In this method we need to first establish correspondence between points/segments in different cameras, which is itself a hard problem. Common observations for points are rare for human problems, and body parts may take on very different appearance from different views. Therefore it is difficult to resolve ambiguity in 3D space based on 2D observations. A second method would be to cast a standard optimization problem, in which we find optimal θi’s and φi’s to minimize an objective function (e.g., difference between projections due to a certain 3D model and the actual segments) based on properties of the objective function. However, if the problem is highly nonlinear or non-convex, it’ll be very difficult or time consuming to solve. Therefore searching strategies which do not explicitly depend on the objective function formulation are desired.
Motivated by [
PSO is suitable for posture estimation as an evolutionary optimization mechanism. It starts from a group of initial particles. During the evolution of the particles towards an optimal, they are directed to the good position while keep some randomness to explore the search space. Suppose there are N particles (test configurations) xi, each is a vector of θi’s and φi’s. vi is the velocity of xi. The best position of xi so far is xˆi, and the global best position of all xi’s so far is g. f (·) is the objective function that we wish to find the optimal position x to minimize f (x). The PSO algorithm is as follows: 1. Initialize xi and vi. vi is usually set to 0, and xˆi = xi. Evaluate f (xi) and set g = argminf (xi). 2. While the stop criterion is not satisfied, do for every xi – vi ← ωvi + c1r1(xˆi − xi) + c2r2(g − xi); – xi ← xi + vi; – If f (xi) < f (xˆi), xˆi = xi; If f (xi) < f (g), g = xi.
The stop criterion: after updating all N xi’s once, the increase in f (g) falls below
a threshold, then the algorithm exits. ω is the “inertial” coefficient, while c1 and
c2 are the “social” coefficients. r1 and r2 are random vectors with each element
uniformly distributed on [
Gestures Description Layer 3 :
Gesture Elements
E1
E2
E3 Description Layer 2 : f11 f12
Features F1 around a local optimal. While if c1 and c2 are big, the particles are more “social” with the other particles and go quickly to the best positions known by the group. In our experiment, N = 16, ω = 0.3 and c1 = c2 = 1.
Examples for in-node segmentation are shown in Fig. 7(a). Some examples showing images from 3 views and the posture estimates are in Fig. 7(b). 4
Towards Behavior Interpretation An appropriate classification is essential towards a better understanding of the variety of passive gestures. Therefore, we propose a categorization of the gestures as follows: – Static gestures, such as standing, sitting, lying; – Dynamic gestures, such as waving arms, jumping; – Interactions with other people, such as chatting; – Interactions with the environment, such as dropping or picking up objects. Silhouettebased Shape
Fitting
Body Orientation Aspect Ratio Goodness of Shape Fit
Alert Level
Weight
Camera 1
Camera 2 Camera 3
Logic to Combine States Multi-Camera Model Fitting
Event Interpretation -V I-n a i
t lid i
a a lz t i e e M S lode rceah
S p a c e fr o 3 D M o d e l
Posture Orientation
Head Position
Arms Positions
Legs Positions
Vertical Horizontal
?? Top Side ?? Left Right ?? ??
W a iftr o M o r e O b s e r v a it o n s to the silhouette and estimates the orientation and the aspect ratio of the fitted (e.g. elliptical) shape. The network’s objective at this stage is to decide on one of the branches in the top level of a tree structure (see Fig. 9) between the possible posture values of vertical, horizontal, or undetermined. To this end, each camera uses the orientation angle and the aspect ratio of the fitted ellipse to produce an alert level, which ranges from -1 (for safe) to 1 (for danger). Combining the angle and the aspect ratio is based on the assumption that nearly vertical or nearly horizontal ellipses with aspect ratios away from one provide a better basis for choosing one of the vertical and horizontal branches in the decision tree than when the aspect ratio is close to one or when the ellipse has for example, a 45-degree orientation.
Fig. 10 illustrates an example of the alert level function combining the orientation and aspect ratio attributes in each camera. The camera broadcasts the value of this function for the collaborative decision making process. Along with the alert level, the camera also produces a figure of merit value for the shape fitted to the human silhouette. The figure of merit is used as a weighting parameter when the alert level values declared by the cameras are combined.
Fig. 11 presents cases in which the user is walking, falling and lying down. The posture detection outcome is superimposed on the silhouette of the person for each camera. The resulting alert levels and their respective weights are shared by the cameras, from which the overall alert level shown in the figure is obtained. In this paper we explore the interactive framework between vision and AI. While vision is helpful to derive reasoning building blocks for higher levels, there is more in the framework. We claim that the feedback between the vision module and the reasoning module is able to benefit both.
A framework of data fusion in distributed vision networks is proposed. Motivated by the concept of opportunistic use of available information across the different processing and interpretation levels, the proposed framework has been designed to incorporate interactions between the vision module and the highlevel reasoning module. Such interactions allow the quantitative knowledge from the vision network to provide specific qualitative distinctions for AI-based problems, and in turn, allows the qualitative representations to offer clues to direct the vision network to adjust its processing operation according to the interpretation state. Two vision-based fusion algorithms were presented, one based on reconstructing the full-parameterized human model and the other based on a sequence of direct deductions about the posture elements in a fall detection application.
The current work includes incorporation of body part motion into the fullparameterized human body model allowing the model to carry the gesture elements in interactions between the vision network and the high-level reasoning module. Other extensions of interest include creating a link from the human model to the reduced qualitative description set for a specific application, and utilizing deductions made by the AI system as a basis for active vision in multicamera settings.
Safe -1
Alert Levels
Uncertain
Danger 0
1 Alert level = -0.9369 Confidence = 0.8391 Alert level = -0.1651 Confidence = 0.7346 Alert level = 0.6598 Confidence
= 0 are shown. After combining observations from the three cameras a final score is given indicating whether the person is standing (safe) or lying (danger).
Alert level = -0.9107 Confidence = 0.9282 Alert level = -0.7920 Confidence = 0.8153 combine Standing, safe ( -0.8075 ) combine Uncertain ( -0.3039 ) Alert level = 0.8370 Confidence = 0.7389
combine Lying down, danger
( 0.6201 )
Alert level = -0.9534 Confidence = 0.8298 Alert level = -0.5945 Confidence = 0.6517 Alert level = 0.8080 Confidence = 0.7695