=Paper=
{{Paper
|id=Vol-490/paper-6
|storemode=property
|title=Playability as Extension of Quality in Use in Video Games
|pdfUrl=https://ceur-ws.org/Vol-490/paper_06.pdf
|volume=Vol-490
|dblpUrl=https://dblp.org/rec/conf/iused/GonzalezMZV09
}}
==Playability as Extension of Quality in Use in Video Games==
https://ceur-ws.org/Vol-490/paper_06.pdf
Playability as Extension of Quality in Use in Video Games
J. L. González Sánchez F. Montero Simarro N. Padilla Zea F. L. Gutiérrez Vela
Software Engineering Dept. Software Engineering Dept. Software Engineering Dept. Software Engineering Dept.
University of Granada University of Castilla - La University of Granada University of Granada
C/Periodista Daniel Saucedo Mancha C/Periodista Daniel Saucedo C/Periodista Daniel Saucedo
Aranda s/n E-18071 Campus Universitario s/n Aranda s/n E-18071 Aranda s/n E-18071
(Granada–Spain) E-02071 (Albacete–Spain) (Granada–Spain) (Granada–Spain)
+34 958 242 812 +34 967 599 200 Ext.: 2468 +34 958 240 849 +34 958 242 812
joseluisgs@ugr.es fmontero@dsi.uclm.es npadilla@ugr.es fgutierr@ugr.es
ABSTRACT games using playability to extend it for entertainment systems,
The quality of a software product is a main objective that every with different attributes, facets and metrics to characterize the
interactive system should aspire. There are many challenges to player experience with videogames.
achieve this quality that require a previous characterization to
ensure it. The International Standards Quality Models help to
characterize the quality of a software system. But, there are some 2. THE QUALITY IN A SOFTWARE
products that present „special‟ quality requirements. In this paper PRODUCT
we focus on special interactive systems: Video Games, whose When a Desktop System (DS) or Traditional Interactive System,
quality requirements are different than traditional software. This such as a word processor, is developed, the main objective is that
additional dimension is called „Playability‟. In this paper, an users can execute a set of tasks in a predetermined context, for
extension of Quality in use Model for Playability decomposition example working in an office. The quality of this kind of systems
(PQM) is introduced. In our playability quality model metrics are has two main components: The first covers the functional aspects
also considered and interpreted. Finally, we review different (functional utility) with two points of view: internally and
usability evaluation methods in order to identify what are the best externally. It has focused on disciplines such as Software
evaluation methods for supporting playability evaluation tasks. Engineering. Another component indicates the means by which
users can achieve this functionality. It is denominated Usability
Categories and Subject Descriptors which has a great importance in HCI discipline. Usability
H.1.2 [Information Systems]: User/Machine Systems - Human represents a measure of product use whereby users achieve
factors concrete objectives within a specific context of use.
General Terms Usability has been characterized in different international
Design, Experimentation, Human Factors. standards. ISO 9241-11:1998 [13] presents and define the
Usability only as a characteristic of the process of use. In ISO/IEC
Keywords 9226-1:2001[11] usability appears integrated in the properties of
Quality in Use, Interactive Systems, Video Games, Playability, any software product. But, it is important to remark that the means
Usability, User Experience. of usability in the different standards models is not the same. In
the first standard usability is: effectiveness, efficiency and
satisfaction. But, in the second it is the easy of learning,
understanding, operability and the attractiveness when use a
1. INTRODUCTION software system.
The Interactive Software Federation of Europe (ISFE) reveals new
research findings that video games and entertainment systems These discrepancies between the standards are present in the
collectively make up the biggest industry in terms of turnover, following standards models. In ISO/IEC TR 9126-4: 2004 [12]
more so than music and cinema. We can deduce that videogames appears the concept denominated Quality in Use whose definition
have become the preferred game of choice, exerting significant is the same as the usability, but add the attribute of security.
social and cultural influence over children, teens and adults [18]. Recently, ISO/IEC 25010:2009 [10] makes its contribution in this
As the quality of software has a direct bearing on product success direction. The quality of a software system is described in terms
and the User Experience, it should be taken into account of its elements and the interaction process. In this standard the
throughout product development (hardware or software), so as to Usability it is not one of the quality factor, it is an attribute of the
achieve the optimum experience for the player. The importance of Quality in Use with the flexibility and the security and they are
video games in the actual society justifies the need to ask if the associated to the interaction or process of use. Accepted
means of quality in this type of software is similar from the recommendation in user interfaces design to improve the user
definition of the interactive or desktops software quality definition experience can be found in [17, 22].
to guarantee an optimal User Experience.
In this work, we analyze how the game experience presents
characteristics that are not explicitly in the quality standards
models and why the usability or quality in use is not sufficient in
3. THE QUALITY IN VIDEO GAMES
The researches in HCI context have centred their objectives to
video games context. We present a quality in use model for video
study the user‟s abilities and cognitive process forgetting the
emotional dimension. A new concept, which is called User Table 1. Different objectives between UX and PX Design
Experience (UX) [9], appears with this dimension. In UX Usability Goals: PX Playability Goals:
entertainment systems it is only a partial vision of the reality, Productivity Entertainment
because it does not take into account all the quality attributes that 1. Task completion 1. Entertainment
influence the use of this „special‟ interactive systems. These 2. Eliminate errors 2. Fun to beat obstacles
attributes identify the Player Experience (PX). 3. External reward 3. Intrinsic reward
As we remarked previously, a videogame can be considered a 4. Outcome-based rewards 4. Process is its own reward
„special‟ interactive system, in that it is used for leisure purposes 5. Intuitive 5. New things to learn
by users seeking fun and entertainment. Whereas the purpose of a 6. Reduce workload 6. Increase workload
desktop system is to execute a task, determined by a clear 7. Assumes technology need 7. Assumes humans need to
functional objective, our objectives when playing a videogame are to be humanized be challenged
more likely to be diverse and subjective. A videogame is not
conceived for the user to deal with daily tasks, but rather it has a
very specific objective: to make the player feel good when playing 4. PLAYABILITY AS QUALITY OF GAME
it. This objective is more subjective and personal than traditional EXPERIENCE
software. Important recommendation for designing entertainment To characterize the quality of game experience we will make use
systems, based on this idea, can be found in [15, 21]. of a precise and complete analysis of Playability, attributes, and a
We propose that analyzing the quality of a videogame purely in conceptual framework to evaluate it in any video game, either
terms of its Usability or Quality in Use is not sufficient – we need from the viewpoint of the game as an interactive process or from
to consider not only functional values but also a set of specific the player who performed/plays with it [7, 8]. This
non-functional values, given the properties of videogames. characterization must be coherent with existed standard,
Additional factors to be considered might include, for example: especially the most recent because we understand that they are the
rules of play; goals; storytelling techniques; virtual world most consensual and complete.
recreation; character design, and so on. In other words, the PX As we have remarked, the quality of a software product has two
could be much more complex than the UX. Hence we need to main points to be analyzed: the quality of process and the quality
establish a set of attributes and properties to identify and measure of product. We need to consider additional aspects related to the
the experience of players playing a videogame. These properties user experience/player, which are related to the emotional aspects
indicate to us whether a game is „playable‟ or not – that is, they of interaction with video games.
will identify the Playability of the video game. Later, we can use
its properties to ensure the quality of a video game through a In [8] we defined Playability as:
process led by playability goals to improve experience when
players play the videogame, PX. In Table 1 we present the ‘a set of properties that describe the Player Experience using a
differences between some goal to achieve in the design of an specific game system whose main objective is to provide
optimal User Experience and Player Experience [16]. enjoyment and entertainment, by being credible and satisfying,
Playability is a live topic in the scientific community; it has been when the player plays alone or in company’.
studied from different points of view and with different objectives
without consensus on its definition or the elements that
characterise it. We have identified two specific strands of It is important to emphasise the „satisfying‟ and „credible‟
research: Playability as only Usability in video games context dimensions. The former is more difficult to measure in video
(understanding and control of the game system), and research games than in desktop systems due to the high degree of
based on particular elements of video games [5, 15]. In the second subjectivity of non-functional objectives. Similarly, the latter
line of research, we find references to: Playability in the quality of depends on the degree to which players assimilate and become
game elements [16, 20]. There are few studies focused on defining absorbed in the game during play – also difficult to measure
Playability formally, [4, 14], but without specific reference to objectively with traditional usability test. The Definition of
Playability attributes or properties to characterize it. Playability is Playability can be based on Quality in Use, but it should be added
based on Usability, but in the context of video games, goes much the above attributes. Also, the definition of particular properties or
further. Furthermore, Playability is not limited to the degree of Quality in Use must be rewriting. For example „Effectiveness‟ in
„fun‟ or „entertainment‟ experienced when playing a game. a video game is not related to the speed with which a task can be
Although these are primary objectives, they are concepts so completed, because typically a player will play for entertainment
subjective. It entails to extend and complete formally the User and relax, this being one of the game‟s main objective. With all of
Experience characteristics with players’ dimensions using a broad these considerations, Playability represents
set of attributes and properties in order to measure the Player ‘the degree in which specific player achieve specific game goals
Experience. with effectiveness, efficiency, flexibility, security and, especially,
In previous works, González Sánchez et al [6, 7, 8] proposed the satisfaction in a playable context of use.‟
characterization of the Player Experience with a video game based In Fig. 1 we present our Playability Quality Model (PQM) as an
on Playability (PM, Playability Model), showing which attributes extension of the Quality in Use model ([2, 10]). It is focus on
and examples of their properties are needed to analyze the „game video games software applications. Next each quality factor and
experience‟. They present a conceptual framework for analysis of attribute in our quality model will be defined following the
player experience and its relationship with the most common previously mentioned ISO standard.
elements that may form part of video game architecture.
these perspectives based on six Facets of Playability (PF). Each
facet allows us to identify the different attributes and properties of
Playability that are affected by the different elements of video
game architecture [7]. The six Facets of Playability are:
Intrinsic Playability: This is the Playability inherent in the
nature of the videogame itself and how it is presented to the
player. It is closely related to Game Core.
Mechanical Playability: This is related to the quality of the
videogame as a software system. It is associated to the Game
Engine
Interactive Playability: This is associated with player
interaction and videogame user interface development. It is
strongly connected to the Game Interface.
Artistic Playability: This facet relates to the quality of the
artistic and aesthetic rendering in the game elements (visual
graphics, melodies, storyline and storytelling).
Intrapersonal Playability or Personal Playability: This refers
to the individual outlook, perceptions and feelings that the
videogame produces in each placer and as such has a high
subjective value.
Interpersonal Playability or Social Playability: This refers to
the feelings and perceptions of users, and the group awareness
that arise when a game is played in company, be it in a
competitive, cooperative or collaborative way.
The overall Playability of a videogame, then, is the sum total of
values across all attributes in the different Facets of Playability. It
is crucial to optimize Playability across the different facets in
order to guarantee the best Player Experience.
5. PLAYABILITY AS MEASURE OF
QUALITY IN A VIDEO GAME
We complete Quality in Use model based on Playability with the
identification and association of metrics to the identified factors
and attributes. To approach this task we use the international
standards and we have adapted the different metrics and measures
Fig. 1. Quality model for playability to evaluate and test video games.
The metrics, Table 2, have as objective the estimation of the
Effectiveness: We define the degree to which specific users quality of Player Experience with video games. Each column
(players) can achieve the proposed goals with precision and reflects the characterization of the different identified metrics.
completeness in the context of use, the video game. These characteristics are: the name of the metric, the objective
Efficiency: It is the degree to which specific users (players) that we analyze with it, its formula, the interpretation of the
can achieve the goals proposed by investing an appropriate numerical value and the type of evaluation to estimate its value.
amount of resources in relation to the effectiveness achieved We must to remark all the indentified metrics are focused in the
in a context of use, the video game. This factor is determined use of the video game. Hence, the evaluation essentially requires
by the ease of learning and immersion. test with players, observation to players when are playing and in
Flexibility: It is the degree to which the video game can be players‟ satisfaction case the realization of questionnaires when
used in different contexts or by different player or game they complete the playtime.
profiles. Playability evaluation is related to evaluation of the user’s
Safety: It is acceptable level of risk to the player health or performance and satisfaction when using the game, product or
data in a context of use, the video game. system in a real or simulated entertainment environment.
Satisfaction: It is the degree to which users (players) are
satisfied in a context of use, the video game. In this factor we
consider various attributes such as fun, attractiveness,
motivation, emotion or sociable.
Playability analysis is a very complex process due to the different
perspectives that we can use to analyze the various parts of video
game architecture. In this work, we propose a classification of
In this paper, see Table 2, we identified many relationships The personalization is an advisable factor in video games because
between playability and quality in use metrics, and we think that in this software exists many design elements that try to distract,
quality in use metrics are useful for playability evaluation. But and to accompany the form of interaction. It should be flexible,
some metrics should be interpreted in a different manner. For for example supporting different interaction techniques: keys,
instance, if we have traditional software products, effectiveness pads, controls, menus, sounds and so on. The attribute of
metrics in international standards introduce tasks effectiveness or accessibility, however desirable and enforceable, traditionally has
task completion as metrics. But when a game and playability is not enjoyed much attention in the development of video games.
considered, we need to speak in terms of ‘goals’ in entertainment Nowadays this is changing and the presence of this attribute
game context, as the challenges that the game introduced. contributes to the use of it in the video game interface and
mechanics.
In a similar manner, error frequency metric in traditional software
has sense, and a value closer to 0 is the better, but in games we
propose attempt frequency as metric, and we can find values
closer to 0 if expert players are playing, and closer to 1 if novice
or clumsy players are considered. Normally, games introduce
difficulties to capture and suck new players; a very simple game is
not attractive, because it will be bored.
Table 2. Metrics associated to playability attributes
Evaluation
Metric name Purpose Formula Interpretation
method
What proportion of the goals is M1 = |1-ΣAi| M1 ϵ [0, 1], the closer to 1
Goal effectiveness User test
achieved correctly? Ai proportional value of each missing the better
What proportion of the goals X = A/B M1 ϵ [0, 1], the closer to 1
Effectiveness Goal completion A = n. of goals completed User test
are completed? the better
B = total number of attempted goals
What is the frequency of X=A Expert player closer to 0. At
Number of attempt User test
attempts? A = n. of attempts made by the player the beginning > 0
How long does it take to Novice players will have
Goal time X = Ta User test
complete a goal? more time
X ϵ [0, 1], closer to middle
Efficiency Goal efficiency How efficient are the users? X = M1/T User test
value
Relative user How efficient is a player X = A/B M1 ϵ [0, 1], the closer to 1
A = ordinary player’s goal efficiency User test
efficiency compared to an expert? the better
B = expert player’s goal efficiency
What proportion of the goals X = A/B M1 ϵ [0, 1], the closer to 1
Accessibility can be achieved by using A = goals with alternative interactions User test
the better
alternative ways of interaction? B = total number of goals
Flexibility
What proportion of the X = A/B M1 ϵ [0, 1], if closer to 1
Personalization personalization options are A = personalized elements original interaction way, User test
used by the players? B = elements in the game perhaps should be changed
X=1–A/B
What is the incidence of health
User health and A = number of players reporting M1 ϵ [0, 1], the closer to 1
problems among users of the User test
safety problems the better
product?
Safety B = total number of players
X=1–A/B
What is the incidence of M1 ϵ [0, 1], the closer to 1
Software damage A = n. occurrences of soft. corruption User test
software corruption? the better
B = total number of usage situations
X = A/B
A = questionnaire producing User test +
Satisfaction scale How satisfied is the player? X>0 the larger the better
psychometric scales questionnaires
B = population average
X = ΣAi /n Compare with previous
Satisfaction How satisfied is the user with User test +
A i= response to a question values, or with population
questionnaire specific software features? questionnaires
B = number of responses average
Satisfaction X = A/B
What proportion of potential A = number of times that specific
M1 ϵ [0, 1], the closer to 1 Observation of
Discretionary usage users choose to use the software functions are used
the better usage
system? B = number of times players are intended
to be used
X = A/B
What proportion of potential M1 ϵ [0, 1], the closer to 1
A = number of times that game is used in Observation of
Socialization users choose to use the collaborative game, closer to
a collaborative environment usage
system? 0 personal game
B = number of times that game is used
Accessibility is a quality attribute considered in the definition of [19] three categories of evaluation methods were gathered: Data
quality in use. In our playability model proposal, that attribute is gathering and modeling methods (DGMM), User Interactions
also considered. Accessibility problems can be considered to be evaluation methods (UIEM), Collaborative methods (CM) and
usability problems for particular group of players e.g. those with Mixed methodologies (MM),
disabilities. If a player cannot understand what is said in cut First group, DGMM, is used for gaining knowledge about users
scenes or cannot hear the footsteps of someone sneaking up and their activities. Two subcategories were distinguished: Data
behind him or her, because the player suffers from an auditory gathering methods (DGM) and Modeling methods (MM). These
disability or if the game does not support the use of specific input evaluation methods are useful for playability evaluation, but not
devices such as one handed controllers or sip and puff joysticks always. Surveys and questionnaires come from social sciences,
that allow severely physical disabled players to play the game. where surveys are commonly used and questionnaires are methods
The safety is an important factor nowadays in video games. The for recording and collecting information. In this context, games
game process is not only a static and mental activity. In some can be used by many kinds of user, for instance preschool
cases, it demands physical requirements, for example game children; 2 to 5 years old, surveys and questionnaires useful
controls that demands and important corporal or physical effort because it is also for them to verbalize their options. Think-aloud
and their effects can be sometimes potentially dangerous or not protocol is not a solution, because even school children ages 6 to
very recommendable to the player health if the player carries out 10 years may have difficulty with concurrent thinking aloud and
this activity for a long time. they cannot be left alone.
Satisfaction is the most important attribute in videogames due to Modeling methods (MM) are often associated with specific data
different aspects can be considered in it: cognitive, emotional, gathering methods or their combination. In this set of methods, an
physical, fun and social. The estimation of the degree of example is especially interesting, Personas [3]. It is a precise
satisfaction in a video game is realized using questionnaires and descriptive model of the user, what user whishes to achieve and
observing the player during the game process and analyzing the why. But this method is more a User-Centered Design
user preferences in the different game sessions with video games. complement. We think that other techniques associated, such as
Probably, when games are considered the more important or ConcurTaskTrees (CTT) or K-Made, are not useful when
determinant quality attribute is the achieved satisfaction rating. playability is considered. Normally, games need very complex
This attribute is subjective and in our playability quality model is models, because they have many interaction freedom degrees;
enriched by using additional quality attributes and sub-attributes. games and activities for entertainment are rich interactive
Thanks to proposed metrics, the quality model of the player applications, where users can do things in many different ways.
experience with videogames based on playability, (PQM) is Table 3. Heuristics and principles for game designing
complete as [1] recommend for quality models developing. (Korhonen and Koivisto, 2006) (Rouse, 2001)
In last column of Table 2 different playability evaluation methods 1. Don‟t waste the player‟s time. 1. Consistent World.
2. Prepare for interruptions. 2. Understand the Game-
are suggested for each metric. These evaluation methods are the World‟s Bounds.
3. Take other persons into
same that we use for usability evaluation. In the next section, we account. 3. Reasonable Solutions to
will discuss different evaluation methods; our main goal will be 4. Follow standard conventions. Work.
use these methods for playability evaluation purposes. 5. Provide gameplay help. 4. Direction.
6. Differentiation between 5. Accomplish a Task
device UI and the game UI Incrementally
6. PLAYABILITY EVALUATION should be evident. 6. Be Immersed.
METHODS 7. Use terms that are familiar to 7. Fail.
This section reviews usability evaluation methods (UEMs) the player. 8. A Fair Chance.
gathered in different reports from MAUSE project. MAUSE 8. Status of the characters and 9. Not Need to repeat
the game should be clearly themselves.
project was a COST Action, COST 294 from 2004 to 2009. The
visible. 10. Not Get Hopelessly Stuck.
ultimate goal of MAUSE was to bring more science to bear on
9. The Player should have clear 11. To Do, Not to Watch.
UEM development, evaluation, and comparison, aiming for goals.
results that can be transferred to industry and educators, thus 12. Do Not Know What They
10. Support a wide range of Want, But They Know It
leading to increased competitiveness of European industry and players and playing styles. When They See It.
benefit to the public. In this paper, we are focused on another 11. Don‟t encourage repetitive
quality factor; playability and we want to discuss if UEM are and boring tasks.
useful as playability evaluation.
In COST 294, four major research and development activities User Interaction Evaluation Methods (UIEM) are explicitly
were implemented by four working groups. Concretely, working targeted towards evaluation. Knowledge-based and empirical
group 1 did a critical review and analysis of individual UEMs. methods are considered in this group. In these methods experts
The primary goal of this activity was to build a refined, and experience is considered, but games are different from others
substantiated and consolidated knowledge-pool about usability kind of applications and heuristics or principles for them are not
evaluation, based on the expertise, experiences, and research the same than Shneiderman [22] or Nielsen‟s principles [7]. In
works of the participating project partners. Different reports were Table 3 some meaningful heuristics for game designing are shown
written and [19] were used in this paper as input. [9, 10].
In order to evaluate previous proposed metrics and quality model We think that user testing, observation and user testing (see Table
we need to specific playability evaluation methods (PEMs). In 2 – „Evaluation method‟ column) are the best manner in order to
playability evaluation. Many times these user testing are done [6] González Sánchez, J. L.; Gutiérrez, F. L.; Cabrera, M.;
with children and we must to know that tests cannot be done with Padilla Zea, N.: Design of adaptative videogame interfaces: a
children younger than 18 without the permission and supervision practical case of use in special education. Computer-Aided
of their parents. Questionnaires are useful tool for playability Design of User Interfaces VI. Lopez Jaquero, V.; Montero
evaluation too, but sometimes cannot be used, because children Simarro, F.; Molina Masso, J.P.; Vanderdonckt, J. (Eds.).
are too much young. Springer (2009),
[7] González Sánchez, J. L.; Padilla Zea, N.; Gutiérrez, F. L.:
From Usability to Playability: Itroduction to the Player-
7. CONCLUSIONS AND FUTURE WORK Centred Video Game Development Procces Proceedings of
The quality of a system is the result of the quality of the system HCI International 2009, San Diego, California, USA. (2009)
elements and their interaction. But every software applications are
[8] González Sánchez, J. L.; Padilla Zea, N.; Gutiérrez, F. L.;
not equal. In this paper, games and entertainment software are
Cabrera, C.: De la Usabilidad a la Jugabilidad: Diseño de
considered. In this context, playability is our main quality
Videojuegos Centrado en el Jugador, In Proceedings of
measure and we presented a playability quality model based on
INTERACCION-2008, pp. 99-109. (2008).
international standard and the interaction component of the quality
is especially taken into account. [9] Hassenzahl, M.; Tractinsky, N.: User Experience – A
Research Agenda. Behaviour and Information Technology,
We identified a direct connection between quality in use and
Vol. 25, N. 2. pp. 91-97 (2006).
playability. Quality in use is a useful concept when interaction
with traditional software is evaluated. But games are different in [10] ISO/IEC 25010-3: Systems and software engineering:
many aspects from others kinds of software. In this paper, Software product quality and system quality in use models.
meaningful differences between games and traditional software in (2009).
the quality model, metrics, and principles or heuristics were [11] ISO/IEC 9126-1: Software engineering – Product quality -
identified. In our proposal, the main contributions in playability Part 1: Quality model. (2001).
characterization are related with the player‟s satisfaction and
ISO/IEC 25010 [10, 19] was enriched in order to evaluate the [12] ISO/IEC TR 9126-4: Software engineering – Product quality
interaction with games. Our metrics are ISO 9126-4 [12] inspired, Part 4: Quality in use metrics. (2004).
but in this paper different interpretation and additional metrics are [13] ISO 9241-11: Ergonomic requirements for office work with
presented. visual display terminals (VDTs) Part 11: Guidance on
Nevertheless, these metrics need to be used and validated by using Usability. (1998).
real games and evaluations experiments, and, in this moment, we [14] Järvien, A., Heliö, S., Mäyrä, F.: Communication and
are doing several evaluations in order to validate the proposed Community in Digital Entertainment Services. Prestudy
metrics. Research Report. Hypermeda Lab. University of Tampere
(2002).
8. ACKNOWLEDGMENTS [15] Korhonen, H., Koivisto, E.: Playability Heuristic for Mobile
This research is financed by: the Spanish International
Games. MobileHCI‟06. ACM 1-59593-390-5/06/0009
Commission for Science and Technology (CICYT); the DESACO
(2006).
Project (TIN2008-06596-C02); and the F.P.U. Programme of the
Ministry of Science and Innovation, Spain. Thanks to MAUSE [16] Lazzaro, N.: The Four Fun Keys. Book Chapter in Game
project and COST nº. 294. Usability. Advancing from the Experts for Advancing the
Player Experience. Morgan Kaufmann, Burlington. (2008).
9. REFERENCES [17] Nielsen, J., Usability Engineering. Morgan Kaufmann, San
[1] Basili, Victor R. and Weiss, D., “A Methodology for Francisco. (1994).
Collecting Valid Software Engineering Data”, IEEE
Transactions On Software Engineering, pp 728-738. Nov. [18] Provenzo, E.: Video kids. Cambridge: Harvard University
(1984). Press (1991).
[2] Bevan, N. Extending Quality in use to provide a framework [19] R3UEMs: Review, Report and Refine Workshop (2007)
for usability measurement. Proceedings of HCI International Abstract: The current proceedings reflect the work conducted
2009, San Diego, California, USA. (2009) so far, and the contributions within WG1 of COST294-
MAUSE project, as it will be discussed in its 3rd.
[3] Cooper, A. The Inmates are Running the Asylum. SAMS, International Workshop, Athens, March 5, 2007: “Review,
(1999). Report and Refine Usability Evaluation Methods (R3
[4] Fabricatore, C., Nussbaum, M., Rosas, R.: Playability in UEMs)”. Dominique Scapin and Effie Law (Eds.)
Action Video Games: A Qualitative Design Model. Human- [20] Rollings, A., Morris, D.: Game Architecture and Design. Ed.
Computer Interaction, Vol 17, pp. 311-368 (2002). New Riders Games (2003).
[5] Federoff, M.: Heuristic and Usability Guidelines for the [21] Rouse III, R. Game Design: Theory and practice. Wordware
Creation and Evaluation of Fun Video Games. Master game developer‟s library. (2001).
Thesis, University (2002).
[22] Shneiderman, B. Designing the user interface. Addison-
Wesley Publishing. (1998).