Nowadays there is a variety of TV channels and programs. This seems to be an advantage for the TV user, but in most cases the user is overwhelmed and not able to choose the most appropriate content though. Assistive systems are needed to support the user in selecting the most appropriate content regarding the user's interests. The research group Next Generation PVR faced the task to develop a user supporting Personal Video Recorder (PVR) in the form of a Bayesian classifier based recommendation system. The work on the prototype of the system is almost done. This paper focuses on the evaluation of the given system. We are presenting two types of evaluation scenarios as well as an approach for measuring user acceptance of a TV recommendation system. Within the evaluation, the acceptance will be questioned. In addition, the results of both scenarios and of the user acceptance survey are presented and discussed.
Kai-Christoph Hamborg University of Osnabrück
Seminarstr. 20 49069 Osnabrück +49 541 969 4703
The way of consuming media clearly changed in recent years.
Especially in times of high bandwidth and loads of appropriate
media delivering services, the Internet plays an important role in
cases of consuming audio/video content. However, live television
is still the most popular media. In Germany 97% of all households
posses a TV set with an average use of 220 minutes a day [1].
Therefore it is evident that the television market is still interesting
for broadcasters. The satellite operator ASTRA holds up to 1700
TV channels just for the region of Germany. Regardless of the
encrypted and shopping program, 53 is a realistic number of
receivable TV channels [2]. For the TV user it is quite difficult to
handle the enormous offer of content. In most cases extensive TV
guides list just a limited number of TV channels and often only
popular ones. The user will invest time to get an overview of all
the available content. Due to too much effort most of the users
focus on favored or popular TV channels and the most interesting
content regarding the user’s interests remains unnoticed. Because
of this, a user supporting Personal Video Recorder based on a
Bayesian classifier has been presented in [
Recommendation systems (RS) exist in several application areas
and for different types of content. One of the most common
examples is Amazon.com. Within the online shop, Amazon
recommends items like Books or CDs on the basis of already
purchased items. For this, Amazon is using an item-to-item based
approach, where relations between items get calculated [
The following section describes the system architecture of the
developed system. In section 2 several approaches have been
presented, where some ideas were also considered for the
following system architecture. In Figure 1 the components for the
recommendation process within the system are shown. The main
functionality of the Bayesian classifier is to classify given content
objects into the classes like or dislike. So the classifier is based on
a two-class decision model, where the conditional probability that
a certain content fits into one of the classes like or dislike gets
calculated. Within the calculation the classifier compares object
attributes and counts how often they occur in one of the pre
defined classes. For a detailed description of the classifier
calculation we refer to our early work [
Initial Adaptive
User Profiles Bayesian classifier
Class dislike
Class like
For the classification process a database is needed on which the
calculation depends. The system provides for this purpose User
Profiles. There are two types of user profiles: initial and adaptive
user profile. The initial profile is important in the beginning of the
system usage. In section 2 the cold start problem has already been
discussed. The user can fill the initial profile with TV content he
or she likes to get recommendation also in the beginning period of
the system. For this purpose, the system provides a web-based TV
guide. The initial profile is just created once, whereas the adaptive
profiles gets build up continuously during the system use. So it is
planned that the adaptive profile grows over the time and the
influence as a database for the recommendation process increases.
This means that, conversely, the influence of the initial profile
decreases. There is the assumption that the higher the adaptive
profile the better the generated recommendations. It remains an
open question how the adaptive profile is created. We are dividing
between implicit and explicit feedback, which is based on the
work of Das and Herman in [
In section 3 the main components with associated functionality of the system architecture has been described. We just focused on those parts of the system, which are important for the evaluation process in the following section. First we explain two evaluation scenarios. The first one is an online scenario to reach a high number of potential volunteers. The second one is a more realistic scenario, where the volunteers write down their TV habits in a TV diary. In section 4.2 we describe an approach to measure user acceptance for the given system. The approach is based on the Technology Acceptance Model (TAM), which characterizes the relationship between information systems and user acceptance.
An important question at the beginning of the scenario design is at what point of the recommendation process the quality can be measured. As pointed out before, the system uses a Bayesian classifier, which classifies TV content in one of the classes like or dislike. In the long term, the main database for generating recommendations is the adaptive user profile, which is gathered by the feedback of the user during the use of the system. The idea for the online scenario is to simulate the creation of an adaptive user profile on which basis the user gets personalized recommendations. We consider at least five steps where the user fills the adaptive profile and the system generates recommendations. After every step the user needs to rate each of the given recommendations. Every step represents one week of a user’s watching behavior, which means that the system generates recommendations from one week of EPG data. The rating of the recommendations is more a verification of the classification the system did. The system presents 20 TV items where ten of these items belong to the class like and ten to the class dislike. The items are displayed to the user without class affiliation. The user needs to perform a classification on his/her own. In the back of the process the system compares both classifications and stores the classification of the user within the adaptive profile for the next step. At the end of the scenario there is the sixth final classification step, where the system calculates the quality of the classification. To measure the quality, we are using the F1-score, which can be interpreted as the weighted average of precision and recall values. Precision and recall is a common metric for assessing the accuracy of classification systems. To meet the requirements of the evaluation, the relationship between temporal development of the adaptive user profile and the quality of the system classification needs to be respected. For this reason, every evaluation user needs to do overall six steps, so that all users need the same length for the scenario and have the same requirements. Internally, every volunteer gets randomly another number of steps. Thus, the completion of the adaptive profile varies between one and five weeks, so that just the data for that random count of steps gets stored. At the end of the evaluation we can compare the calculated quality between the varying lengths of the scenario. At the beginning of the scenario an initial profile by the user with ten objects will be created. 4.1.2 TV diary The measurement of quality is similar to the scenario described in section 4.1.1. Significant is the development of the adaptive user profile. Where the Online Scenario was more a simulation of data creation, whereas the TV diary deals with real watched TV content. The participants watch television as usual. They need to write down all the watched TV content in a TV diary with the name of a series, date and time so that the system can find the watched content within the given EPG data. For the participants it is also mandatory to write down TV content they began to watch, but switched the channel in case of dissatisfaction. Zapping behavior isn’t respected due to the fact that the system wouldn’t do either. The participants write down their viewing behavior for five weeks. After that, the evaluation supervisor collects all the diaries and transfers the diary data to the system. On the basis of the data, the system generates recommendations. Each participant needs to rate the system recommendations by dividing the displayed and unordered TV content into the classes like and dislike. The classification of the system and the participant are compared and on this basis the quality gets calculated. This is equally done within the sixth step in the Online Scenario (cp. section 4.1.1.). An initial profile won’t be created.
Measuring user acceptance for a software system is related to the
field of psychology. It is necessary to figure out the conditions for
a software system, which result in an actual use of the system.
Already in 1989 Davis presented the Technology Acceptance
Model (TAM). The TAM is an information-theoretic model,
which defines the Variables Perceived Usefulness (PU) and
Perceived Ease of Use (PEOU). These two variables are defined
as the main factors influencing the use of a system. Davis defines
the variables as follows:
Perceived Usefulness „the degree to which a person believes that
using a particular system would enhance his or her job
performance“ cp. [
There is a lot of research on User Acceptance Issues in
recommender systems cp. [
F2 F3 F6 F7 F8
The recommendations were as good as the recommendations from my friends
F10 Ifeceadnbaincfkluence the quality of the recommendations with my
F12 I know how the system generates recommendations after I’ve used it F13 The time the system needs to generate recommendations is appropriate F14 tFimorethe creation of the initial profile I needed to spend too much F15 The creation of the initial Profile was easy and comfortable F16 Irewcoomulmdecnredaatteiotnhse fiansittial profile again to get proper
F18 I(fe.tgh.ebreooisksa)n,oIthweorutledchunseoliotgy which recommends other things to me
Table 1 Questionnaire User Acceptance
In the following section we present the evaluation results. The section is again divided into two subsections. Within section 5.1 we present the results for the evaluation scenarios; in section 5.2 the evaluated user acceptance questionnaire is presented. The results will be discussed in section 6.
For the Online Scenario we had 51 participants, where nine of them didn’t finish the evaluation. So the number can be reduced to 42 with 34 male and eight female participants. A large part of the group can be classified as educated (higher education level or university degree) with an age between 18 and 39. Table 2 presents the results for the Online Scenario. The data were tested for independence using the chi square (χ2) test with a significance level of 5% (α=0.05). We’ve proved the calculated chi square value against the quantile of the chi square distribution p=3.84 respecting the significance level of α=0.05 and degree of freedom df=1.
Iterat.
Participants
Objects The table shows the iteration number and the associated number of participants to those the iteration number was randomly assigned. The column number Objects presents the size of the adaptive user profile e.g. for the scenario with one iteration the adaptive user profile stored 30 TV objects like films, series, etc.. In the fourth column the quality in form of the calculated F1 score is shown. As you can see at first the quality increases steadily. The system achieved a maximum quality with a database of 70 objects. With a more increasing data volume the quality decreases. 5.1.2 TV diary For the TV diary eight people attended, mostly between 18 and 39 years and with a high educational level. The number of participants decreases from ten to eight caused by incomplete TV diaries. Table 3 presents the results for the TV diary equally to the results in section 5.1.1. Due to a limited number of participants/data records we used Fisher’s exact test proving independence of the data set. We have no quality data for the 5th iteration because of insignificant test results. In contrast to the Online Scenario the quality increases continuously. The highest value in iteration four is similar to the highest quality in the Online Scenario.
Iterat.
Participants
F1 Score
p-Value 30 50 70 90 110
F1 Score
The following section presents the results of the proposed user
acceptance approach. In Figure 2 the median of the 20 items are
shown. Negative formulated questions are marked with a dot on
the top of the bar.
Figure 2 shows high results within the questionnaire, which is an
indication for general user acceptance. Negated questions were
overall answered with a low valence, nevertheless the results tend
into the right direction. For a closer look we refer to Table 1
including the question overview. The questions are divided into
three sections asking for Quality, Effort and Acceptance as
already mentioned in section 4.2. To analyze the results, we
interpret every subarea separately. Items one till twelve concern
the quality of the given recommendations. The questioned quality
within the questionnaire is a rather subjective interpretation of
each user, than an objective measurement. Especially the items
1,2,4 and 5 question the perceived quality by the user. All of the 4
items were rated with a minimum value of 4, which tends to be a
positive feedback. Item 6 questioned if the quality of the
recommendations correspond to the quality of TV
recommendations from friends. The middle value of 3 is a little
low, but to reach human interpretation level for the system wasn’t
a compound goal at all. Item 12 questioned the transparency of the
system, which is with a value of 3 also a little low. However items
8 to 11 were with a value of 4 approvingly answered. On the basis
of the given results with agreeing (4) and fully agreeing (5)
feedback we can say that a certain quality of the system
recommendations is given, which influences the user acceptance
positively. For the subarea of effort we questioned the items 13 to
17. Those items questioned the fact of needed effort for using the
system properly. We got a positive feedback for the items 14 to 17
with a value of 4 (agree) and 2 (not agree) for negative formed
questions. These items were questioned for the construction of the
initial feedback. The creation of the initial feedback is the only
direct interaction between user and system needed to get
recommendations presented. Just the time for creating new
recommendations by the system was rated with a neutral value
(3). So we can say that the effort for using the system is limited,
which is also a positive influence for user acceptance. Within the
subarea of acceptance for recommender systems in general and for
the tested system all questions were approvingly answered (4).
These results also tend to a positive user acceptance. These results
have proven that a general user acceptance for the system is given.
We have some particular shortcomings in comparison to
recommendations by friends. In this case the system could be
advanced by a connection to one of the big social networks like
Facebook or Twitter. A concrete approach for this was already
presented in one of our previous works cp. [
In the following section the results from the evaluation presented
in section 5 are discussed. We proposed two different approaches
for evaluating a Bayesian classifier based recommendation
system. Both approaches differ in the way data for the generating
process was collected. Within the Online Simulation data was
collected in a continuous manner. The adaptive profiles within the
TV diary were created more qualitatively with a real user
behavior. However, both scenarios were designed to examine the
quality of given recommendations and to examine if there is
connection between increasing database (adaptive user profile)
and generated TV recommendations. First of all we can say that
the system classifies objects with a quality between 81% and 83%
with a database between 56 and 70 already classified objects.
Because of this, it is clear that a sufficiently large database is
needed to reach a proper quality. Both result tables show that
there is an increasing quality with an increasing number of
classified data objects. Table 2 shows a decreasing quality at a
high number of data objects though. This effect has been observed
in other application areas, where also a Bayesian classifier has
been used for classification. In [
In this paper we presented an evaluation for a Bayesian classifier
based recommendation system for generating TV content on the
basis of user behavior. In addition to two evaluation scenarios,
which measure the quality of recommendations, we presented
results of user acceptance evaluation based on the work of [
[1] B. Engel, C.-M. Ridder, “Massenkommunikation 2010”,
Press Conference of ARD, ZDF, Sep. 2010. [2] SES Astra. Available at http://www.astra.de/2117/de