It is desirable to address accessibility issues within virtual worlds. Moreover, curbing criminal activities within virtual worlds is a major concern to the law enforcement agencies. Forensic investigators and accessibility researchers are gaining considerable interests in detecting and tracking avatars as well as describing their appearance within virtual worlds. Leinhart and Maydt have introduced a novel-set of Haar like features by extending the Viola Jones approach towards rapid object detection. We test this Haar cascade on human and avatar faces. Accuracy rates of 79% on human and 74% on avatar faces are obtained. The goal is to detect avatar faces in upright frontal face datasets and establish a baseline for future work in computer generated face recognition.
Virtual worlds are gaining widespread momentum. They have a potential to transform the way the society operates. They bring in a sense of a “personal” digital space (Trewin et al. 2008) for the user by providing real time interaction with other fellow users and mirroring the real world activities. Communities, social groups, enterprises and institutions are all present in these virtual worlds. These virtual environments are gaining popularity across the globe and moving towards being an integral part of not only the Internet but also the society. An “avatar” is the user’s virtual identity in these virtual worlds. The user can model its appearance to reflect either one’s own personality or pose as someone else. Avatars can navigate within this virtual space by moving around buildings, flying in the air, swimming in the water or teleporting to different locations.
Destructive behaviors like terrorist activities and
cybercrimes are reportedly on the rise within virtual
worlds. There are reports of Al-Qaeda terrorists
communicating and recruiting within the popular virtual
world Second Life (SL)
Making these worlds accessible to a broad set of users is being gradually addressed. They are not easy for anyone to use the first time. Controlling an avatar requires simultaneous visual, audio, cognitive and motor abilities (Trewin et al. 2008). This results in a number of challenges to users with disabilities. Disabled gamers are showing a strong desire to be a part of these virtual worlds and envision themselves in their own avatars. Visually impaired users wish to detect and identify fellow avatars as well as other objects in their surroundings.
Criminal activities in these worlds are becoming a major
problem for law enforcement agencies. Forensic experts
are expressing interest in accurately and automatically
tracking users and their avatars in these virtual
communities. Detecting and recognizing avatar faces
Profiling avatars based on their faces is a challenging and novel problem, contributing towards a new research direction in face recognition. Detecting avatar faces will address accessibility issues within virtual worlds, especially for visually impaired users, by describing the facial appearances of avatars in the vicinity by face detection.
Authenticating biological entities (human beings) is an
essential and well-developed science, utilized to determine
one’s identity in today’s modern society. However, avatar
authentication (non-biological entities) is an issue that
needs to be highlighted and addressed
Avatar and human faces are similar as well as different.
Both have consistent structure and facial component (eyes,
ears, nose, etc.) locations. These similarities motivate the
design of an avatar face detection framework based on
principles similar to the human face detection system.
Avatars have a wider range of colors than humans do that
helps distinguish the two entities (Yampolskiy, Klare &
Jain 2012). The goal is to detect avatar faces in the field of
view of the user’s avatar within virtual worlds that, along
with face recognition, will help build a complete biometric
authentication system for avatars. Currently available
biometric systems are not equipped to deal with the visual
and behavioral nature of artificial entities like avatars and
perform poorly under such circumstances. Concerns over
security and avatar identification are constantly voiced in
virtual worlds
Several challenges are involved in detecting avatar faces. They involve illumination, camera location, different skin color tones, pose, head rotation, etc. Certain preprocessing techniques such as geometric and color normalization may have to be applied (Yampolskiy, Klare & Jain 2012). In the context of investigating criminal activities in virtual worlds, we aim to examine some possibilities to authenticate avatars. These involve matching a human face to an avatar face when users upload their picture to model their avatars, matching the face of one avatar to another in a single as well as across multiple virtual worlds and matching a forensic sketch of an avatar to the avatar face (Yampolskiy, Klare & Jain 2012).
To the best of our knowledge, no prior research has been
reported in the area of avatar face detection. However,
there has been significant research in the domain of avatar
recognition. The current state of art in virtual reality
security, focusing specifically on emerging techniques for
avatar authentication has been examined
In this paper, we focus on applying the face detecting OpenCV Haar cascade as an appearance based face detection technique. The method is based on an extended novel set of rotated Haar-like features, efficiently calculated by enriching the basic set of simple Haar-like features. Our test set comprises of human and avatar face datasets with varying backgrounds, illumination, rotations and face occlusions. The goal here is to obtain accuracy estimations by simply applying the cascade on each of these varying datasets.
The paper is organized as follows. We begin with an
introduction to AdaBoost (Adaptive Boosting) learning
algorithm, extended Haar-like features
AdaBoost algorithm, invented by Schapire and Freund (Schapire 1999), helped in solving many practical difficulties faced by the earlier boosting algorithms. It initially distributes a set of equal weights over a training set. After each round, the weak learning algorithm increases the weights for the incorrectly classified examples. This helps in focusing on the hard examples in the training dataset. “Ada” is a short form for adaptive as it adapts to the error rates of the individual weak hypothesis of each stage of the boosting process. Here the basic classifier is used extensively with the concept of stacking or boosting to constitute a very strong classifier. Several of these strong classifiers are subsequently connected into a cascade classifier to achieve the detection. The cascading levels determine the response of the system and the error rate. AdaBoost face detection techniques are based on the expansion of the Haar-like features, image integration and the cascaded classifier. For each image feature, a corresponding simple classifier is generated and the error relative to the current initialization error weight is evaluated. The classifier with the smallest error is chosen and added to the stage classifier. The weights of the samples are appropriately updated. If the sample is correctly classified, then the error is 0 or else it is 1. Finally, a stage classifier is obtained by combining the individual simple classifiers into a cascade. The algorithm is fast, easy and simple to implement, has no tuning parameters and requires no prior knowledge about the weak learner. Thus, it can be combined flexibly with any method to evaluate the weak hypothesis.
These are a novel set of rotated Haar-like features, which
yields a 10% lower false alarm rate as a face detector
From Figure 1 we observe the 14 prototypes, which
include four edge features, eight line features and two
center-surround features. They are scaled independently in
vertical and horizontal direction to generate a rich,
complete set of features. The number of features obtained
from each prototype is large and differs for each prototype
One of the available features of Intel’s OpenCV (Intel) is face detection from images. Furthermore, it provides programs that are used to train classifiers for face detection systems, called Haar Training, to create custom object classifiers (Rhondasw 2009; Seo 2008).
From Figure 2 we observe that the process of Haar training consists of the following steps: Data preparation:
The process begins by gathering positive and negative datasets. The positive dataset contains images with the object of interest, i.e. the faces to be detected. The negative images are the ones that do not contain the object of interest, e.g. background images, non-face images etc. For real cascades there should be about 1000 positive and 2000 negative images. A general and acceptable positivenegative proportion is 1:2, but it is not a hard rule (Rhondasw 2009).
Creating the training dataset:
The prepared data now needs to be segregated into training and testing datasets. Here the training dataset is fed to a cascade of detection-boosted classifiers that yields the cascade xml file.
Cascade of Detection Classifiers:
Basic classifiers are put together to form stage classifiers which in turn are grouped together to form a cascade of stage classifiers. The series of such classifiers are applied to every sub-window of an image. A positive result from the first classifier stimulates the evaluation of the second classifier, which also has been adjusted to achieve high detection rates. A positive outcome from the second triggers the third and so on. Negative outcomes at any stage lead to rejection. Stages in the cascade are trained using AdaBoost and their thresholds are appropriately varied to minimize the false negatives. Thus, higher number of stages yields higher accuracies but leads to a decrease in performance time (Viola & Jones 2001). Cascade.xml:
It is comprised of the various stages built due to the Haar training with the appropriate thresholds for each stage.
The generated cascade xml file is evaluated on the testing dataset to get accuracy estimations of the results. The Haar cascade performs poorly on rotationally varying faces as well as occluded faces as it is based on object detection and it needs all the facial features (a pair of eyes, nose and mouth) to perform an accurate face detection.
For the purpose of analysis of face detection techniques, we needed datasets with
In our experiments, we used the following datasets: Set 1 - Caltech 450 samples of human frontal face images from the California Institute of Technology were used(Weber). All images have the dimensions of 896 x 592 pixels. The dataset contains images from 28 subjects and 3 sketches with complex backgrounds and varying illuminations. Set 2 – FERET 400 samples of human face images from the FERET ("The Color FERET Database") dataset were used. All images have the dimensions of 256 x 384 pixels. This dataset contains images for 52 subjects. Each subject is represented by 7-8 images with random head rotations varying from 67.5 degree to 15 degree rotations in both left and right directions, plain background and slightly varying illuminations.
Set 3 – Avatar
Avatar faces from the popular online virtual worlds,
Entropia Universe and Second Life, were used in the
avatar dataset. A scripting technique was designed and
implemented to automatically collect the avatar faces using
AutoIT as well as Second Life’s Linden Scripting
Language (LSL)
The dataset is subdivided into 3 parts with the number of samples indicated in the parenthesis next to it: Female avatars from Entropia Universe with complex backgrounds (150), Male avatars from Second Life with a regular or plain background (150) and Male avatars from Second Life with complex backgrounds (150). The dataset contains the avatar faces with a set of five images per avatar from different angles with complex backgrounds and varying illuminations. Figure 3 shows a sample of this dataset with different random head rotations and a complex background.
The experiment involved using a single OpenCV Haar cascade, haarcascade_frontalface_alt.xml, on these datasets and evaluating the algorithm’s performance on each of them. There is no training involved here. The experiment relies solely on the accuracy of this cascade on the human and avatar datasets. Basically, a classifier trained on human faces is used to detect avatar faces without any special preprocessing.
Initially, a code similar to the one found in (Hewitt 2007) was written in Microsoft Visual Studio 2008 using the OpenCV libraries (Intel). This code helps in detecting faces within images using the OpenCV Haar cascade. The cascade was applied on the dataset images. Promising results were obtained and a comparitive analysis was drawn.
A summary of the results, obtained by executing the above OpenCV Haar cascade on all three datasets, are shown in Table 1. These results include images that belong to multiple categories. A brief description of each category is given below.
Background: The nature of the background, which can be either plain or complex.
Positive: Signifies a positive detection with a face accurately detected.
Background Faces (BF): The background faces (if any) other than the main subject that are detected.
False Positives (FP): Non–face objects that are incorrectly detected as faces.
False Negatives (FN): Background faces that are not found by the detector.
Zero Detections (ZD): The face of the main subject as well as those in the background (if any) that are completely undetected by the detector.
Accuracy: The positive detections.
Error Rates: Comprises of false positives, false negatives and zero detections.
Average Accuracy and Error Rates: The average of Accuracy and Error Rates obtained.
E: Images with both ZD’s and FP’s F: Images with only FN’s G: Images with only ZD’s
On the 450 human face images, the OpenCV Haar cascade yielded an accuracy rate of 98.6% with an error rate of 6.4%.
cascade yielded an accuracy rate of 59% with an error rate of 41%. The error rate being high is mainly due to the poor performance of the cascade on rotationally varying faces as well as face occlusions.
Facial images from the three separate avatar datasets are fed to the OpenCV Haar cascade individually. Accuracies obtained are 76% (Female avatar-Entropia, complex background), 52% (Male avatar-Second Life, regular background), 93% (Male avatar-Second Life, complex background) and error rates of 24% (Female avatarEntropia, complex background), 48% (Male avatar-Second Life, regular background) as high as 7% (Male avatarSecond Life, complex background). From the results we observe that the cascade performs very well on the Male avatar-Second Life (complex background) but not so good on the Male avatar- Second Life (regular background).
This paper evaluates a standard OpenCV face detection algorithm, which utilizes a Haar cascade on a set of human facial images as well as virtual world entities: avatars, which are rapidly becoming part of the virtual society. Accuracy estimations reported from each dataset are compared. Good results are obtained, but poor performances are recorded for facial images involving head rotations, poor illumination and face occlusions. The average accuracy detection rates on human faces is 79% whereas for avatar faces it is 74%, which is quite good given the quality of the picture, appearance of the avatar as well as background and surrounding regions. The varying accuracy rates are due to the difference in the count of head rotations, face occlusions, illumination conditions and plain/complex backgrounds between the datasets.
Potential directions for future research involve improving the existing algorithm to achieve better accuracies by fusing the current algorithm with Local Binary Patterns (LBP). We will extend this approach over a larger dataset to yield more accurate evaluations and better results. Further, unifying this face detection technique with face recognition to build a complete face authentication system capable of authenticating biological (human beings) and non-biological (avatars) entities is the direction in which we have set our sights on. The Color FERET Database. Available from NIST The Color
FERET Database http://face.nist.gov/colorferet/ Hewitt, R. (2007). Seeing With OpenCV, Part 2: Finding Faces in Images. Cognotics: Resources for Cognitive Robotics, from http://www.cognotics.com/opencv/servo_2007_series/p art_2/index.html Intel. Open Computer Vison Library, from http://opencv.willowgarage.com/wiki/ http://sourceforge.net/projects/opencvlibrary/files/open cv-win/2.1/
Weber, M. A frontal face dataset collected by Markus Weber at the California Institute of Technology. Available from California Institute of Technology Bg-Caltech http://www.vision.caltech.edu/Image_Datasets/faces/RE ADME