In the context of Human-Robot interaction estimating gaze orientation brings useful information about human focus of attention. This is a contextual information : when you point something you usually look at it. Estimating gaze orientation requires head pose estimation. There are several techniques to estimate head pose from images, they are mainly based on training [3, 4] or on local face features tracking [6]. The approach described here is based on local face features tracking in image space using online learning, it is a mixed approach since we track face features using some learning at feature level. It uses SURF features [2] to guide detection and tracking. Such key features can be matched between images, used for object detection or object tracking [10]. Several approaches work on xed size images like training techniques which mainly work on low resolution images because of computation costs whereas approaches based on local features tracking work on high resolution images. Tracking face features such as eyes, nose and mouth is a common problem in many applications such as detection of facial expression or video conferencing [8] but most of those applications focus on front face images [9]. We developed an algorithm based on face features tracking using a parametric model. First we need face detection, then we detect face features in following order: eyes, mouth, nose. In order to achieve full pro le detection we use sets of SURF to learn what eyes, mouth and nose look like once tracking is initialized. Once those sets of SURF are known they are used to detect and track face features. SURF have a descriptor which is often used to identify a key point and here we add some global geometry information by using the relative position between key points. Then we use a particle lter to track face features using those SURF based detectors, we compute the head pose angles from features position and pass the results through a median lter. This paper is organized as follows. Section 2 describes our modeling of visual features, section 3 presents our tracking implementation. Section 4 presents results we get with our implementation and future works in section 5.
We use some basic properties of facial features to initialize our algorithm : eyes
are dark and circular, mouth is an horizontal dark line with a speci c color,...
Then we use sets of SURF to learn a better description of visual features. SURF
are extracted with their relative position in order to keep geometrical information
about features. We use a similarity measure to compare two sets of SURF (see
the probability measures in [
P 2S0 Q2S1 where S0 and S1 are two sets of SURF, k:k2D=desc are the Euclidean norms in image space (2D ) and in SURF descriptor space (desc). desc de nes the local tolerance (descriptor space) whereas 2D de nes the geometrical tolerance (image space). Learning a set of SURF is a matter of selecting SURF which are representative of the feature and putting them into the same set with their relative position (you can set the origin to the center of the feature). 3
We use a simple face model described by the position of eyes, center of mouth
and nose tip similar to the one in [
We have taken a video sequence of 1140 frames (Fig. 1) to compare the behavior of the algorithm with and without SURF and how the number of particles in the particle lter in uences the result. This sequence shows a single person making head moves : pan and tilt rotations as well as fast head movements, both rotations and translations and a simple \look at" movement. For each test, we ran the algorithm 21 times on the same sequence due to the stochastic nature of particle lters. We ran 3 tests with this sequence. The rst test with 250 particles without SURF and only 50 particles with SURF. The second test is with 50 particles for both versions of the algorithm. The last tests with 250 particles both with and without SURF.
It is slower to compute particles likelihood with SURF because of similarity calculations involved in each likelihood evaluation. This is the reason why the rst test does not compare the two versions of the algorithm with the same number of particles. Without optimizing likelihood calculations the algorithm runs 200 SURF particles as fast as 7000 particles without SURF. An interesting point to note is sets of SURF can learn the shape of a feature as it is varying, as a direct consequence if you close your eyes it can still track them, this is useful when the person is blinking.
The average standard deviation for the 3 tests with SURF is 10.9 for tilt, 14.9 for pan and 4.4 for roll, without SURF we get 10.5 for tilt, 15.8 for pan and 8.09 for roll. We get a smaller standard deviation over several runs with SURF and a small amount of particles than without SURF and with lots of particles. Nose tracking is not working properly yet and all the results are obtained with an ambiguity on the sign of angles which is not always solved properly (the model we use requires nose position to solve this sign ambiguity) which decreases greatly the quality of measures. We ran a few tests with a better nose detector (which uses SURF too) and already got much better results with SURF due to learning being based on the result of the particle lter.
We also ran the algorithm on the Yale B face database [
Our current implementation adds a noticeable overhead per particle but this can be optimized in order to have the same overhead per particle as the implementation without SURF. It would be a bit slower only because of the need to extract SURF from the image which takes approximately 8ms during our tests.
Using sets of SURF to detect and track face features makes the process more
robust and stable under various poses. It allows detecting and tracking features
with varying shapes. Our current implementation of SURF similarity
calculations is slow. With some optimization using SURF adds a small overhead and
we could use more SURF. Taking face geometry (features relative position) into
account could also improve detection robustness. The main idea is to use sets
of SURF to learn something to track and learning changes in its shape as it
moves. We are currently integrating our algorithm on a robot and have planned
to use both human focus of attention and gesture recognition [
This work was partially conducted within the EU STREP Project Commrob funded by the European Commission Division FP6 under Contract F P 6 045441 and the french ANR project AMORCES.