This paper presents a user oriented evaluation methodology for comparing person search services on the Web. Many established system oriented methods from information retrieval cannot be applied to this domain. Our user oriented methodology is applied to a test comparing the person search engines yasni, pipl.com and 123people. The user study with over 30 participants led to relevant results. The coverage of data object types within the person search engine results is quite different. Especially the amount of pictures and social media network entries which are presented by the systems and which are perceived by the test users differ greatly. The results also revealed a tendency to judge people more positively when more information was found.
The evaluation of retrieval systems is central in information
retrieval research because the system performance cannot be
predicted. The most influential retrieval evaluation methodology
is called the Cranfield paradigm. Information retrieval research
has adopted an evaluation scheme which tries to ignore subjective
differences between users in order to be able to compare systems
and algorithms. The user is replaced by a prototypical and
constant user. Relevance judgments are provided by domain
experts [
Cranfield evaluations have often been criticised for several
reasons. The main objections come from advocates of user
oriented studies. The search situation of users depends on many
individual and contextual factors which can only be captured in
user experiments [
Person search engines have a higher chance to succeed than
general purpose search services. The retrieval with named entities
is known to be easier than searches without names entities [
The balance between control and realism is a challenge for each experiment. For the presented study, we chose a user experiment to test person search engines because an approach purely dedicated to retrieval power does not mirror the user experience for person search engines well. It is necessary to limit the realism in a user experiment in order to allow comparison across participants in the test. We selected a job applicant scenario in order to make the experiment interesting for the users. Applicant search is a very prominent usage type. The method was successful in making the experiment attractive. The test users liked the experiment very much and through word of mouth, more applicants wanted to register for the experiment than were needed. The selection of persons for the task defines the content for the test. It seemed necessary to identify people for whom much information can be found on the Web. If there were no videos, working results like presentations or social network entries, then the performance of the person search engine could not be tested with our experiment. So even if the persons selected are not representative in terms of amount of online information for the whole population or all persons who are indexed in a person search service it increases the validity of the test to select persons with a large amount of online information.
Three people were carefully selected who had similar
qualifications. For them, a job profile was developed which was
given to the participants together with the names of the people.
The users were asked to search for these people who would be
interviewed for the position and check if they were appropriate.
The job description and the name of each applicant were given to
the test persons. Each of the candidates was well qualified for the
job but had one negative aspect in his online data. One was an
advocate of nuclear power and the job was for offered by an
alternative energy company. The second applicant was a serial
entrepreneur who portrayed himself on Facebook in pictures with
attractive women and sports cars. The third applicant had party
photos online where he could be seen smoking cigarettes and he
considered himself as lazy in one social network while he had a
very business oriented self image in another social network.
Obviously, such a scenario has some limitations. Person search
engines need to disambiguate between people with the same
name. We decided to choose people who are not ambiguous in
order to have the same difficulty for each person. Such issues are
evaluated in the system oriented campaign WEPS [
Students of the University of Hildesheim were recruited through a mailing list of students. Participation was voluntarily and no gratification was given. None of the participants had a computer science background. They all were frequent Internet users and had searched for people before but only 10% had used a person search engine before. The others use Google or social networks to find information on people.
The issue of relevance is always a crucial one in information retrieval evaluation. In our study, any item could contribute to the full picture of the applicant. Despite the clearly defined scenario, it remains vague which information is needed and what type of information is useful. It is difficult to assign relevance to items or even weights to categories. The user interfaces of the person search engines present the items in categories like e.g. social network entries or videos.
A questionnaire study [
Contact information Profile on a social network Photo Information about professional accomplishments or interests The most frequently researched item, contact information does not apply for our scenario because the persons had sent a letter of application. The next two most frequent items are included. The fourth item is rather vague as some of the other items following as far as the categories of person search engines are concerned. As a consequence, the data available does not justify the assignment of weights to some items. In our study, all clicks on items were scored equally. The results will also show which of the items were most popular. The time per applicant was limited to 10 minutes. The entire experiment took 45 minutes on average including the pre- and post questionnaire.
One search service modified the interface after the first two tests. So it was necessary to eliminate three test sessions from the results and recruit further test users. This shows that not only the dynamics of the personal data presents a challenge for the test but also the ongoing modifications of the search engine. Overall, 34 took part in the experiment. Due to the problems of a relaunch of one service, we could consider the experiments of 10 users of 123people, 11 users of Pipl and 10 user of Yasni.
Each test person worked with one search engines on all three applicants. This between groups approach was applied was mainly applied to avoid a long learning phase for each of the person search engines. All tests were recorded with appropriate software.
The result description focuses on the information perceived by users and the performance of the test users in the application task. The information items clicked by the users were categorized. It can be seen that the services lead to a similar number of clicks when summed up over all users. Each of the services resulted in between 110 to 120 clicks for the ten test persons. In the case of Pipl, 11 test persons were considered. Each engine leads to a sufficient number of entries and has abundant information on the applicants in our scenario. This was a goal of the test design and was accomplished.
The type of information which was encountered was quite different. It can be easily seen, that 123.people facilitates access to photos whereas Pipl leads more users to social network entries. A comparative analysis for the services for the most popular item types is shown in Table 1.
In the post test questionnaire, users were asked about their
subjective impression of the service they had used. In the overall
satisfaction, 123people was rated highest. For the page structure,
pipl received the best grades and the coverage of different
business networks yasni was rated as most successful. In the latter
case, the finding from the objective click data was confirmed.
Further details on the results are provided in [
We presented a holistic evaluation methodology for person search engines. The performance of these search services is measured by observing the perception of test users. The test methodology is built on a realistic scenario and use case but it does not cover all the relevant quality aspects of person search engines. The important capability to resolve the ambiguity of names was not dealt with. In future work, it might be promising to develop a performance based test for this task only. The complete information seeking behaviour and its success is also not measured with our test. In a realistic scenario, people might access the social media networks through a person search engine and continue their search mainly there. This issue could be resolved by observing real behaviour.
In the test, the search engine 123people was the winner. It not
only led users to the highest number of items, but it was also
subjectively judged to be the best person search engine.
However, in several aspects other systems performed better and
were judged better. The evaluation showed that the different
tools are all based on the freely available data on the Web but
that they lead to different results. The most sought items in our
test were photos, entries and profiles in social and business
networks and personal homepages. Each of the engines
exhibited a strength in one of these items, e.g. 123people for
photos because they are shown as top results. This is also
confirmed by the questionnaire study among American
recruiters [
For the users who publish information about themselves and who become information providers by doing that the issue of information competence will become more and more important. Personal Online Identity Management is a growing field and several new companies are entering the market.