<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Vanet-X: A Videogame to Evaluate Information Management in Vehicular Networks</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Sergio Ilarri</string-name>
          <email>silarri@unizar.es</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Eduardo Mena</string-name>
          <email>emena@unizar.es</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>V´ıctor R u´jula</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>IIS Department, University of Zaragoza</institution>
          ,
          <addr-line>Zaragoza</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Vehicular Ad Hoc Networks (VANETs) are attracting considerable research attention, as they are expected to play a major role for Intelligent Transportation Systems (ITS). Thus, according to a recent survey by ABI Research1, about 62% of new vehicles will be equipped with vehicle-to-vehicle (V2V) communications by 2027. Vehicular networks offer new opportunities for the development of interesting mobile applications for drivers, but at the same time they also bring challenges from the data management point of view. Thus, for example, techniques should be developed to estimate the relevance of the information exchanged among the vehicles and to propagate the relevant data in the network efficiently and effectively. As testing the proposals in a real large-scale scenario is impractical, simulators are often used. In this paper we present Vanet-X, an online multiplayer driving videogame that we have developed to help in the difficult evaluation task of data management strategies for VANETs. The idea behind the proposal is to exploit the potential of players around the world driving vehicles in the videogame to effortlessly collect data that can be used to extract some conclusions and fine-tune the proposed data management strategies. So, for example, the videogame allows to evaluate if a certain data management strategy is able to provide useful information to the driver/player (i.e., if the presented information represents an advantage for him/her). We argue that this videogame can be a good complement for existing simulators. As a proof of concept, we have performed some preliminary tests that show the potential interest of the proposal.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. INTRODUCTION</title>
      <p>
        The widespread availability of mobile devices and the
development of wireless communication technologies (such as
Wi-Fi, WAVE, etc.) have encouraged the development of
1http://www.abiresearch.com/press/
v2v-penetration-in-new-vehicles-to-reach-62-by-202.
services for drivers within the context of Intelligent
Transportation Systems (ITS). In particular, Vehicular Ad Hoc
Networks (VANETs) have become an attractive research
area [
        <xref ref-type="bibr" rid="ref1 ref14 ref15 ref20 ref24 ref26 ref30">1, 14, 15, 20, 24, 26, 30</xref>
        ]. In these vehicular
networks, the vehicles can exchange information directly by
using short-range wireless communication technologies. This
decentralized architecture provides some advantages over
other solutions such as the use of 3G communications: e.g.,
no need of an infrastructure, quicker transmission of
safetyrelated data in the vicinity, localized communications
without the need of a centralized server, and free of charges
(which also encourages the participation of peers in the
network). Numerous types of events can be relevant for drivers
(e.g., accidents, traffic congestions, an ambulance asking the
right of way, available parking spaces, etc.). These events
can be exchanged in the vehicular network and stored
locally by the vehicles. Then, a query processor can
periodically evaluate the interest of those events and decide if they
should be shown to the driver; there may be implicit queries
(e.g., information about an accident in the direction of travel
will be relevant for any driver) and explicit queries (e.g., a
driver may indicate his/her interest in finding an available
parking space or in receiving information about other
specific types of events).
      </p>
      <p>
        However, although VANETs offer interesting
opportunities for the development of data services for drivers, they
also bring new challenges. Thus, several difficulties arise
from the point of view of data management [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. As an
example, estimating the relevance of events in order to
disseminate them effectively and efficiently in the network is
a challenge [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. Similarly, disseminating information about
a scarce resource (e.g., an available parking space) to many
vehicles can lead to competition situations among them to
try to reach the resource [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. As a final example, the
relevance of events must also be considered in order to decide
if a specific event received by a vehicle should be shown to
the driver or not [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
      </p>
      <p>
        A big challenge is how to evaluate the data management
techniques proposed. Evaluating them in a real scenario
with a significant number of vehicles is simply impractical
and expensive. Therefore, simulations are frequently used
in this field. However, even with simulations the
evaluation task can be very time-consuming. For example, many
proposals depend on a number of parameters that can be
fine-tuned for a given scenario (e.g., see [
        <xref ref-type="bibr" rid="ref2 ref31">2, 31</xref>
        ]), and
determining a good choice of parameters for general
evaluation is quite challenging. On the other hand, crowdsourcing
strategies where users play the role of drivers could help to
introduce human behavior and facilitate new tests initiated
by the users themselves.
      </p>
      <p>So, in this paper we propose a complementary approach
that can be used in conjunction with the use of simulators.
In particular, we argue that we can benefit from players
having fun with a driving game to easily collect interesting data
that can be used to extract some conclusions and fine-tune
the proposed data management strategies. The videogame
is inspired by the classic videogame Rally-X (http://www.
klov.net/game_detail.php?game_id=9259, videogame
released in 1980) but it is a new development, with different
goals, game modes, and spirit. So, the basic idea is that
the vehicles can receive information through the vehicular
network and different data management techniques can be
plugged in the videogame (e.g., different data dissemination
strategies). Data received from other vehicles, if evaluated
as interesting by the local query processor in the car, are
shown on a radar and can provide a competitive advantage
to the player. During the game, a variety of data are
collected (e.g., number of messages received by the vehicles,
network overhead, time required by the vehicles to complete
their goals in the game, etc.), that can be analyzed later. So,
while playing, players contribute to collect data for a variety
of scenarios, and these data can be exploited to evaluate the
effects of particular data management strategies.</p>
      <p>The structure of the rest of this paper is as follows. In
Section 2 we describe the high-level architecture of the
videogame and its features. In Section 3 we summarize the main
behaviors implemented for the computer-managed vehicles.
In Section 4 we present some basic aspects about the way the
data are collected for later analysis. In Section 5 we present
the results of the first experiments that we have developed
as a proof of concept. In Section 6 we present some related
work. Finally, in Section 7 we present our conclusions and
some lines of future work.
2.</p>
    </sec>
    <sec id="sec-2">
      <title>ARCHITECTURE AND FEATURES</title>
      <p>Vanet-X is a car videogame that can be played by multiple
players connected to the Internet (see Figure 1 for a snapshot
showing parking spaces).
2.1</p>
    </sec>
    <sec id="sec-3">
      <title>Main Features</title>
      <p>We summarize some features of the game as follows:
• Both real (human) and computer-controlled players
can participate in the game. Human players can join
a game through the Internet.
• The game can be configured to execute on a server and
create new games when necessary. Alternatively, the
computer of any user can play the role of a server and
start a new game that other users can join.
• Any real map can be used in the game, by selecting
and downloading the data of the desired area from
OpenStreetMap (http://www.openstreetmap.org/).
• To increase the playability, real maps are combined
with some extra elements, such as enemy cars, smoke
emission devices to disturb enemies (see Figure 2),
evolution of events in game time rather than in real-world
time, higher maximum speeds for cars controlled by
humans, when the driver has a task to go to a certain
building he/she has to park nearby and then go by foot
to the destination (he/she will be a vulnerable target
for enemy cars, that will try to hit him/her, as shown
in Figure 3), the car can get damaged and be repaired
by paying a certain price (points accumulated during
the game), there is infinite or limited fuel depending on
the game mode (requiring refueling in a petrol station
when running out of fuel in the second case), etc.
games where the players have to complete a series of
goals in sequence (with tasks such as parking the
vehicle, going to a certain building/business or address,
etc.) as soon as possible to win the game. As shown
in the table, some game modes can be cooperative,
competitive, or both. For the tasks implying going
to a certain location, the task may require reaching
that location with the car, park and then get there by
foot, or just park as near as possible (in this last case,
the score for completing the task will be inversely
proportional to the distance between the parking location
and the final destination). Competitive games involves
from 1 to 4 teams in the game, being the winner the
team that obtains more points during the game.</p>
      <p>Game mode
Capture the flag
(capture 5 flags)
Capture the
enemy cars (1 or
more)
Solve tasks (1-3
tasks)
Survival (1 or 2
tasks)
Park (find one
available parking
spot)</p>
      <p>Possibility to
Multiplayer Inmortal get out of the
mode car and walk
ccoomoppeertaittiivvee no no
cooperative
competitive yes no
ccoomoppeertaittiivvee no yes
ccoomoppeertaittiivvee no yes
competitive
yes
yes</p>
      <p>
        Infinite
fuel
configurable
configurable
configurable
no
no
• Some default data management strategies, inspired by
the work performed in the VESPA project [
        <xref ref-type="bibr" rid="ref2 ref3 ref4 ref6 ref7">2, 3, 4, 6,
7</xref>
        ], have been implemented. Different tuning
parameters can be modified through the graphical user
interface of the videogame (see Figures 4 and 5). Moreover,
the design of the videogame allows an easy integration
of other data management alternatives.
• There is a “radar” (e.g., on the right part of Figure 2
we show a basic radar, and on the right part of
Figure 1 a radar in debug mode that shows some extra
elements about the scenario) that can provide some
information to the players. For example, a player can
see the following on the radar: his/her location, the
petrol stations, and the destination location (if any).
Besides, if the option to use a data sharing strategy for
that vehicle has been enabled, it will also show data
about interesting events received from other vehicles,
such as free parking spaces, enemy vehicles, items to
pick up (e.g., flags in Figure 6), priority vehicles like
ambulances, etc.
From a more technical point of view, we have used the
Java programming language to develop the video game.
Besides, some auxiliary libraries have been useful. For
example, we use Apache Xerces2 Java (http://xerces.apache.
org/#xerces2-j) to extract data from the XML files
obtained from OpenStreetMap, JLayer (http://www.javazoom.
net/javalayer/javalayer.html) to decode and reproduce
MP3 files for the game music, Guava-12.0 (https://code.
google.com/p/guava-libraries/), etc.
2.2
      </p>
    </sec>
    <sec id="sec-4">
      <title>Basic Architecture</title>
      <p>The basic architecture of the videogame is presented in
Figure 7 (the part concerning the collection of statistics
about the game is not shown here, as it will be described in
Section 4). At a high-level, we can briefly describe the main
components as follows:
• A client application receives commands from the player,
sends them to the server, and receives from the server
information about the objects that should be rendered
on the screen (see Figure 8).
• The server receives the input from the clients,
updates the current status of the game (e.g., by
considering the movements performed by the vehicles and the
tasks that they complete), and generates new goals and
events as needed (see Figure 9). The server is
multithreaded, with a thread per vehicle that performs a
basic cycle of “while a vehicle is alive, perform actions
and check for potential collisions”.
• An interface IDataManagementStrategy declares the
methods that should be implemented by a data
management strategy to allow its integration with the
videogame (e.g., a method to define the types of events
that are interesting for the driver, a method to
generate an event, etc.).
• Another interface IVehicle is implemented by the
vehicles to allow interacting with them (e.g., to obtain
a reference to the data manager in the vehicle or to
obtain information about the GPS location).</p>
      <p>Any data management strategy can potentially be
integrated in this framework, as long as it implements the
interface IDataManagementStrategy and calls the appropriate
methods to inform the vehicles (interface IVehicle). So, we
can easily plug in different alternative data management
techniques for testing.
3.</p>
    </sec>
    <sec id="sec-5">
      <title>BEHAVIORS OF THE VEHICLES</title>
      <p>
        We have implemented several behaviors for the vehicles
controlled by the computer, which adapt the steering
behaviors proposed in [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ]. In particular, we consider the following
basic behaviors:
• Seek implies directing the vehicle towards a certain
static target, by adjusting its direction and speed.
• Flee is the opposite behavior to Seek, as it implies
getting as much further as possible from the target.
• Pursuit is similar to Seek, but in this case the target
is a moving object. So, the expected movement of the
target is estimated, to try to catch it.
• Evasion is the opposite behavior to Pursuit (i.e., based
on Flee instead of Seek).
• Arrival implies the progressive reduction of speed as
the vehicle approaches the target.
• Obstacle avoidance provides vehicles with the ability
to dodge vehicles and other obstacles.
• Wander generates a random trajectory, to represent a
vehicle traveling around with no clear objective. This
is useful, for example, to represent a vehicle that is
searching for an available parking space in the vicinity.
• Path following allows a vehicle to circulate within the
boundaries of a certain path.
• Unaligned collision avoidance is a behavior that tries
to avoid the collision of vehicles moving in different
directions. Thanks to this behavior, vehicles can
estimate a potential collision risk with other vehicles in
the near future, to try to avoid it.
      </p>
      <p>Of course, all the vehicles exhibit the whole set of
behaviors at the same time, applying a priority ordering in case
several behaviors could be applied at the same time and are
in conflict to each other. Based on the previous basic
behaviors, we have defined the schema of a normal behavior
for different types of vehicles: enemy cars (that try to catch
the players or flee from the players, depending on the game
mode), ambulances (as representatives of emergency
vehicles which may ask the right of way), and traffic cars (that
represent neutral cars in the game). As an example, the
basic behavior of traffic cars is shown in Figure 10.
4.</p>
    </sec>
    <sec id="sec-6">
      <title>DATA COLLECTION AND EXPLOITA</title>
    </sec>
    <sec id="sec-7">
      <title>TION</title>
      <p>In this section, we summarize the strategy applied for data
collection during the game and the corresponding
exploitation of results. If a certain configuration option that
activates the collection of statistics during the game is enabled,
several data are collected: data about the scores obtained by
the players, the time needed by vehicles (the ones controlled
by humans as well as those managed by the computer) to
perform certain tasks (such as parking), and other measures
about the performance of the data management strategy
applied (e.g., events created, events that are considered
relevant by each vehicle, etc.). When the game ends, all these
data are stored in several files on the game server, along with
a file that contains information about all the configuration
parameters used in that game (e.g., game mode,
configuration parameters used for the data management strategy
considered, the wireless communication range simulated, etc.).</p>
      <p>To centralize the data collected, it is possible to set up a
Statistics Server, which is a process executing continuously
on a certain computer. In this way, the clients playing the
game automatically connect to the Statistics Server when a
game ends, in order to communicate the statistics collected
during the game. Besides, it is possible to connect to the
Master Server by using a terminal client (called Statistics
Client) that allows seeing and modifying the configuration
parameters as well as retrieving the statistics files generated.
Another option is to avoid the use of a Statistics Server and
collect the statistics in the computer that plays the role of
a server for a game. If we consider configuration settings
where there is a predefined game server and all the clients
connect to it to start a new game or join an existing game,
this option also keeps the statistics in a single location.
However, if there are several game servers then the statistics
would have to be centralized manually.</p>
      <p>Figure 11 provides an overview of the way the
different components of the game, and particularly the
Statistics Server, are distributed in a network. Notice that we
actually distinguish between a Master Server and a Game
Server. The Master Server is executing on the server
machine and a client first connects to it (so, it is the entry point
for clients); then the Master Server checks if a Game Server
is available and if not it creates one; finally, it returns the
port number of its Game Server to the client, as the client
will interact with the Game Server during the game.</p>
      <p>It should be noted that, as we collect information about
the performance of human players, the skills of those players
with the game will have an impact on the results and this has
to be taken into account when exploiting the results. Indeed,
directly comparing the achievements of several human
players without considering their game skills could lead to wrong
conclusions. For example, player1 without a data sharing
system could perform better than player2 with a data
sharing system, but we should not necessarily conclude that the
use of such a data sharing system is harmful. In other words,
we should always compare players with the same skills. For
this reason, each human player is assigned a certain skill
level (which may change along time, as the player improves
his/her performance) and the statistics about players are
tagged with the skill level corresponding to that player.
Besides, players that have a skill level below a certain threshold
are (by default) not allow to participate in games with
collection of statistics enabled, as performance data about them
are assumed to be unreliable and besides their clumsiness
could interfere with the normal development of the game.
The skill level of a player is computed based on his/her
ability to complete missions in the game (tasks per time unit).
5.</p>
    </sec>
    <sec id="sec-8">
      <title>EXPERIMENTAL EVALUATION</title>
      <p>We have performed a few preliminary experiments to
evaluate the interest of our proposal. As a use case for testing,
we focused on the case of available parking spaces, as these
are events that represent scarce resources, which implies
additional challenges for data management (i.e., the
competition among vehicles should be minimized).
5.1</p>
    </sec>
    <sec id="sec-9">
      <title>Data Management Strategies</title>
      <p>As a data sharing strategy for the vehicles, we considered
the following options.
5.1.1</p>
      <sec id="sec-9-1">
        <title>VESPA-P: VESPA With No Reservation</title>
        <p>
          First, we adapted the proposal in [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ], developed in the
context of the system VESPA (Vehicular Event Sharing with
a mobile P2P Architecture) [
          <xref ref-type="bibr" rid="ref4 ref6">4, 6</xref>
          ], which is based on the
computation of an Encounter Probability (EP).
        </p>
        <p>The EP between a vehicle and an event estimates the
likelihood that the vehicle will meet the event, based on
geographic computations that estimate the spatio-temporal
relevance of the event. For example, the relevance decreases
with the distance between the event and the vehicle, with the
time since the event was generated (e.g., consider the case
of information about an available parking space, which can
be unoccupied only for a limited amount of time), and the
direction of the vehicle (e.g., if it is approaching the event or
not). In particular, the directions of both the vehicle and the
event are estimated and several penalty coefficients (α, β, γ,
and ζ) are used to weigh the importance of four estimated
parameters: the minimum distance to the event over time
(Δd), the time until the closest position to the event (Δt),
the age of the event at the closest position (Δg), and the
angle between the vehicle and the event (c).</p>
        <p>So, when a vehicle receives an event it computes its EP
and disseminates the event again if the computed EP
exceeds a certain dissemination threshold (DT ). The intuition
is that vehicles should disseminate data that are relevant
for them (as those data are also probably relevant for the
neighboring vehicles). Two other thresholds are managed:
the storage threshold (ST ) and the relevance threshold (RT ).
The ST determines the minimum value of the EP for an
event to be stored locally in the vehicle, and the RT the
minimum value needed to show the event to the driver.</p>
        <p>
          Besides, the proposal in [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ] proposes a contention-based
approach for data dissemination in order to limit the
network overhead in the dissemination of messages (basically,
when there are several candidate vehicles to re-disseminate
an event, the message will be disseminated only by the
vehicle located further away from the vehicle that disseminated
the message previously). Several parameters are used in the
protocol, such as D (the maximum time to wait before
rediffusing) and D0 (time to wait for an acknowledgement that a
message sent previously was received by some other vehicle).
5.1.2
        </p>
      </sec>
      <sec id="sec-9-2">
        <title>VESPA+P: VESPA With Reservation Protocol</title>
        <p>
          Communicating the availability of a single parking space
to many vehicles could lead to an unfruitful competition
among the vehicles to try to reach the same parking space,
leading to dissatisfaction of the drivers and parking times
that could even exceed those that would be obtained if no
data sharing system were used. For this reason, the work
presented in [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ] proposed an enhancement to the previous
approach VESPA-P for the case of scarce resources such
as parking spaces. It provides an allocation protocol that
coordinates a procedure that ensures that the information
about an available parking space is communicated to a single
interested vehicle.
5.1.3
        </p>
      </sec>
      <sec id="sec-9-3">
        <title>Blind: No Data Sharing</title>
        <p>Finally, we also considered an approach where no data
sharing strategy is used. In this case, the vehicles receive no
information and the only data available for the drivers are
what they see with their own eyes. For vehicles trying to
find available parking spaces, this will lead to a blind search.
5.2</p>
      </sec>
    </sec>
    <sec id="sec-10">
      <title>Experimental Settings</title>
      <p>
        The basic configuration of the videogame for the
experimental evaluation is as follows. The communication range
considered for the vehicles is 200 meters and a maximum of
50% of the vehicles are assumed to be equipped with a data
sharing application. The penalty coefficients used to
compute the EP for VESPA are: α=1/1500 (Δd ≤ 500 meters),
β=1/180 (Δt ≤ 60 seconds), γ=1/360 (Δg ≤ 120 seconds),
and ζ=1/270 (c ≤ 90◦); these are parameters that can be
considered for a “medium” (not small, not large)
dissemination area, according to [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. The RT and the DT are both
set to 75%, and the ST is 60%. The query processor on each
vehicle re-evaluates the relevance of the events received with
a refreshment period of 2 seconds, showing on the radar the
events that are considered relevant. For the dissemination
protocol, D is set to 1 second and D0 to 2 seconds.
5.3
      </p>
    </sec>
    <sec id="sec-11">
      <title>Experimental Results</title>
      <p>We have simulated a varying number of vehicles moving
in an area of 1 squared kilometer around the street “Sophie
Oury” in the city of Valenciennes (France). In this scenario,
we measured the time needed by the vehicles to find free
parking spaces near certain destinations. In Figure 12 we
show the reduction on the average time needed by a human
player to find an available parking space near the target.
The experimental results show the interest of sharing data
among the vehicles (with both VESPA-P and VESPA+P),
as these data can later be shown on the radar to provide
interesting information to the drivers. Besides, according
to these results, using a reservation protocol to avoid the
competition problem (VESPA+P) is particularly beneficial.</p>
      <p>In Figure 13 we compare the performance of human
players (vehicles controlled by humans) and computer players
(vehicles controlled by the computer), by showing the
reduction on the average time needed to find an available
parking space near the target when using VESPA+P. According
to these experimental results, we can see that the human
players get more benefit from the use of the data sharing
strategy. The difference may be due to the way the artificial
intelligent behavior of the computer vehicles is implemented.
The experimental results obtained correspond to data
collected during a total of 14 hours playing the videogame
(about 400 parking actions by the human player during this
game time). The results are consistent with our intuition
and with other experimental results obtained previously by
using a simulator. Nevertheless, more tests are needed to
validate the results and evaluate other scenarios. For
example, we started to obtain some first preliminary results with
games played by more than one human player. It is also
interesting to perform experiments with other types of events
(e.g., accidents, obstacles on the roads, etc.); with
information about them, drivers could try to avoid those hazards
and so decrease the total travel time.
6.</p>
    </sec>
    <sec id="sec-12">
      <title>RELATED WORK</title>
      <p>As far as we know, this is the first attempt to develop a
videogame whose hidden purpose is to help with the
evaluation of information management strategies for vehicular
networks.</p>
      <p>
        Nevertheless, the idea of trying to benefit from human
actions to improve or evaluate a system is not new.
Exploiting the power of people to perform large-scale tasks that are
costly, time-expensive, or hard, is called crowdsourcing [
        <xref ref-type="bibr" rid="ref33">33</xref>
        ].
For example, mCrowd [
        <xref ref-type="bibr" rid="ref32">32</xref>
        ] benefits from sensors available on
iPhone devices to perform collaborative tasks such as image
tagging or road traffic monitoring. As another example,
reCAPTCHA [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ] exploits CAPTCHAs [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ] (Completely
Automated Public Turing test to tell Computers and Humans
Apart), as a security measure to avoid web access to
programs, in order to recognize words from scanned books that
are challenging for OCR (Optical Character Recognition)
systems. According to [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], “The practice of crowdsourcing
is transforming the Web and giving rise to a new field”.
      </p>
      <p>
        Particularly relevant for our work with Vanet-X are those
proposals that achieve the crowdsourcing results through the
use of a videogame. A notable example is the ESP game [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ],
where players implicitly help to label images while playing
the game. The use of videogames as learning tools is a clear
example of the benefits of using educative videogames; as
an example, CodeSpells [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] is a fantasy videogame where
players have to write spells in Java. Other games with a
hidden purpose exist, as commented in [
        <xref ref-type="bibr" rid="ref28">28</xref>
        ]. The multiplayer
online game Planet PI4 [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ] intends to serve as a testbed
environment for Peer-to-Peer (P2P) game architectures. It
is also interesting to mention that the term gamification has
appeared to denote a variety of software that is inspired
somehow by videogames [
        <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
        ].
      </p>
      <p>There exist some driving videogames that, as Vanet-X, are
based on the use of real road maps or city layouts, such as
Mini Maps (https://apps.facebook.com/minimaps/) and
Push-Cars 2: On Europe Streets (http://www.push-cars.
com). However, unlike in Vanet-X, in these games the
players do not contribute to any crowdsourcing task or data
management strategy evaluation.</p>
      <p>
        Finally, a good number of simulators of vehicular
networks and mobility generators have been developed, such as
TraNS [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ], SUMO [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ], Veins (Vehicles in Network
Simulation) [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ], GrooveNet [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ], or VanetMobiSim [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. Some
interesting surveys can be found in [
        <xref ref-type="bibr" rid="ref12 ref18">12, 18</xref>
        ]. As commented
along the paper, we argue that the videogame-based
approach can be an interesting complement (but not a
replacement) to the use of existing simulators to evaluate
information management strategies for vehicular networks. Besides,
mobility generators and vehicle simulators could potentially
be used to generate neutral traffic for Vanet-X.
7.
      </p>
    </sec>
    <sec id="sec-13">
      <title>CONCLUSIONS AND FUTURE WORK</title>
      <p>We have developed a videogame that can be used to
evaluate data management strategies for vehicular networks, as
a complement to existing simulators. Whereas the
opportunity of crowdsourcing through a videogame is attractive,
several challenges arise. Thus, the goal of developing a fun
videogame required the introduction of several elements that
would not appear in a real scenario (like enemy cars), which
could have an impact on the results, but on the other hand
this will attract people to play. Moreover, the results
obtained can depend not only on the benefits offered by the
data management strategy but also on the ability of the
specific player. So, whereas the videogame can provide an
ideal tool to collect many data for a variety of scenarios, the
experimental results obtained have to be judged with
caution (e.g., we label the collected data with the skill level of
the player). Even with these limitations, we argue that the
videogame helps to collect with less effort data that can be
used to fine-tune a protocol and/or obtain some initial
conclusions, prior to the evaluation in more realistic scenarios.</p>
      <p>Additional information regarding the videogame is
available at http://sid.cps.unizar.es/Vanet-X/, including a
playable version of the videogame, some videos, and
screenshots. This is a first step that shows the potential interest of
exploiting videogames to evaluate data management
strategies for vehicular networks. As future work, we would like
to optimize and improve the videogame, as well as to
develop a complete methodology and architecture to collect
the data, evaluating the interest of the results obtained in
other scenarios and in a larger scale.
8.</p>
    </sec>
    <sec id="sec-14">
      <title>ACKNOWLEDGMENTS</title>
      <p>This research work is currently supported by the CICYT
project TIN2010-21387-C02-02 and DGA-FSE. The data
management strategy adapted and used as an example in the
videogame has been proposed in the context of the VESPA
project, and we would like to warmly acknowledge the
collaboration with Dr. Thierry Delot in that project.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>J. J.</given-names>
            <surname>Blum</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Eskandarian</surname>
          </string-name>
          , and
          <string-name>
            <given-names>L. J.</given-names>
            <surname>Hoffman</surname>
          </string-name>
          .
          <article-title>Challenges of intervehicle ad hoc networks</article-title>
          .
          <source>IEEE Transactions on Intelligent Transportation Systems</source>
          ,
          <volume>5</volume>
          (
          <issue>4</issue>
          ):
          <fpage>347</fpage>
          -
          <lpage>351</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>N.</given-names>
            <surname>Cenerario</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          .
          <article-title>A content-based dissemination protocol for VANETs: Exploiting the encounter probability</article-title>
          .
          <source>IEEE Transactions on Intelligent Transportation Systems</source>
          ,
          <volume>12</volume>
          (
          <issue>3</issue>
          ):
          <fpage>771</fpage>
          -
          <lpage>782</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Cenerario</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          .
          <article-title>Vehicular event sharing with a mobile peer-to-peer architecture</article-title>
          .
          <source>Transportation Research Part C: Emerging Technologies</source>
          ,
          <volume>18</volume>
          (
          <issue>4</issue>
          ):
          <fpage>584</fpage>
          -
          <lpage>598</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          .
          <article-title>Data gathering in vehicular networks: The VESPA experience (invited paper)</article-title>
          .
          <source>In Fifth IEEE Workshop On User MObility and VEhicular Networks (LCN ON-MOVE 2011)</source>
          , pages
          <fpage>801</fpage>
          -
          <lpage>808</lpage>
          . IEEE Computer Society,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          .
          <article-title>Introduction to the Special Issue on Data Management in Vehicular Networks</article-title>
          . Transportation Research Part C: Emerging Technologies,
          <volume>23</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>2</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          . The VESPA Project:
          <article-title>Driving advances in data management for vehicular networks</article-title>
          .
          <source>ERCIM News</source>
          , (
          <volume>94</volume>
          ):
          <fpage>17</fpage>
          -
          <lpage>18</lpage>
          ,
          <year>July 2013</year>
          .
          <article-title>Special Theme on “Intelligent Vehicles as an Integral Part of Intelligent Transport Systems”</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>T.</given-names>
            <surname>Delot</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ilarri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Lecomte</surname>
          </string-name>
          , and
          <string-name>
            <given-names>N.</given-names>
            <surname>Cenerario</surname>
          </string-name>
          .
          <article-title>Sharing with caution: Managing parking spaces in vehicular networks</article-title>
          .
          <source>Mobile Information Systems</source>
          ,
          <volume>9</volume>
          (
          <issue>1</issue>
          ):
          <fpage>69</fpage>
          -
          <lpage>98</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>S.</given-names>
            <surname>Deterding</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Dixon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Khaled</surname>
          </string-name>
          , and
          <string-name>
            <given-names>L.</given-names>
            <surname>Nacke</surname>
          </string-name>
          .
          <article-title>From game design elements to gamefulness: Defining “gamification”</article-title>
          .
          <source>In 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek'11)</source>
          , pages
          <fpage>9</fpage>
          -
          <lpage>15</lpage>
          . ACM,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>S.</given-names>
            <surname>Deterding</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Sicart</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Nacke</surname>
          </string-name>
          ,
          <string-name>
            <surname>K. O'Hara</surname>
            ,
            <given-names>and D.</given-names>
          </string-name>
          <string-name>
            <surname>Dixon</surname>
          </string-name>
          . Gamification:
          <article-title>Using game-design elements in non-gaming contexts</article-title>
          .
          <source>In 2011 Annual Conference on Human factors in Computing Systems (CHI'11) - Extended Abstracts</source>
          , pages
          <fpage>2425</fpage>
          -
          <lpage>2428</lpage>
          . ACM,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Doan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Ramakrishnan</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. Y.</given-names>
            <surname>Halevy</surname>
          </string-name>
          .
          <source>Crowdsourcing systems on the World-Wide Web. Communications of the ACM</source>
          ,
          <volume>54</volume>
          (
          <issue>4</issue>
          ):
          <fpage>86</fpage>
          -
          <lpage>96</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>S.</given-names>
            <surname>Esper</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. R.</given-names>
            <surname>Foster</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W. G.</given-names>
            <surname>Griswold</surname>
          </string-name>
          .
          <article-title>On the nature of fires and how to spark them when you're not there</article-title>
          .
          <source>In 44th ACM Technical Symposium on Computer Science Education (SIGCSE'13)</source>
          , pages
          <fpage>305</fpage>
          -
          <lpage>310</lpage>
          . ACM,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>J.</given-names>
            <surname>Harri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Filali</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Bonnet</surname>
          </string-name>
          .
          <article-title>Mobility models for vehicular ad hoc networks: A survey and taxonomy</article-title>
          .
          <source>IEEE Communications Surveys &amp; Tutorials</source>
          ,
          <volume>11</volume>
          (
          <issue>4</issue>
          ):
          <fpage>19</fpage>
          -
          <lpage>41</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>J.</given-names>
            <surname>Ha</surname>
          </string-name>
          ¨rri,
          <string-name>
            <given-names>F.</given-names>
            <surname>Filali</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Bonnet</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Fiore</surname>
          </string-name>
          . VanetMobiSim:
          <article-title>Generating realistic mobility patterns for VANETs</article-title>
          .
          <source>In Third International Workshop on Vehicular Ad Hoc Networks (VANET'06)</source>
          , pages
          <fpage>96</fpage>
          -
          <lpage>97</lpage>
          . ACM,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>H.</given-names>
            <surname>Hartenstein</surname>
          </string-name>
          and
          <string-name>
            <given-names>K. P.</given-names>
            <surname>Laberteaux</surname>
          </string-name>
          .
          <article-title>A tutorial survey on vehicular ad hoc networks</article-title>
          .
          <source>IEEE Communications Magazine</source>
          ,
          <volume>46</volume>
          (
          <issue>6</issue>
          ):
          <fpage>164</fpage>
          -
          <lpage>171</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>G.</given-names>
            <surname>Karagiannis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Altintas</surname>
          </string-name>
          , E. Ekici,
          <string-name>
            <given-names>G. J.</given-names>
            <surname>Heijenk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Jarupan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Lin</surname>
          </string-name>
          , and
          <string-name>
            <given-names>T.</given-names>
            <surname>Weil</surname>
          </string-name>
          .
          <article-title>Vehicular networking: A survey and tutorial on requirements, architectures, challenges, standards and solutions</article-title>
          .
          <source>IEEE Communications Surveys &amp; Tutorials</source>
          ,
          <volume>13</volume>
          (
          <issue>4</issue>
          ):
          <fpage>584</fpage>
          -
          <lpage>616</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>M.</given-names>
            <surname>Lehn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Leng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rehner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Triebel</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Buchmann</surname>
          </string-name>
          .
          <article-title>An online gaming testbed for peer-to-peer architectures</article-title>
          .
          <source>ACM SIGCOMM Computer Communication Review</source>
          ,
          <volume>41</volume>
          (
          <issue>4</issue>
          ):
          <fpage>474</fpage>
          -
          <lpage>475</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>R.</given-names>
            <surname>Mangharam</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. S.</given-names>
            <surname>Weller</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Rajkumar</surname>
          </string-name>
          .
          <article-title>GrooveNet: A hybrid simulator for vehicle-to-vehicle networks</article-title>
          . In Second International Workshop Vehicle-to-
          <source>VehicleCommunications (V2VCOM'06)</source>
          , pages
          <fpage>1</fpage>
          -
          <lpage>8</lpage>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>F. J.</given-names>
            <surname>Martinez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. K.</given-names>
            <surname>Toh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.-C.</given-names>
            <surname>Cano</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. T.</given-names>
            <surname>Calafate</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Manzoni</surname>
          </string-name>
          .
          <article-title>A survey and comparative study of simulators for vehicular ad hoc networks (VANETs)</article-title>
          .
          <source>Wireless Communications &amp; Mobile Computing</source>
          ,
          <volume>11</volume>
          (
          <issue>7</issue>
          ):
          <fpage>813</fpage>
          -
          <lpage>828</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>J. E.</given-names>
            <surname>Michael</surname>
          </string-name>
          <string-name>
            <surname>Behrisch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Laura</given-names>
            <surname>Bieker</surname>
          </string-name>
          and
          <string-name>
            <given-names>D.</given-names>
            <surname>Krajzewicz</surname>
          </string-name>
          . SUMO -
          <article-title>Simulation of Urban MObility: An overview</article-title>
          .
          <source>In The Third International Conference on Advances in System Simulation (SIMUL'11)</source>
          , pages
          <fpage>63</fpage>
          -
          <lpage>68</lpage>
          . IARIA,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>S.</given-names>
            <surname>Olariu</surname>
          </string-name>
          and M. C. Weigle, editors.
          <source>Vehicular Networks: From Theory to Practice</source>
          . Chapman &amp; Hall/CRC,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>M.</given-names>
            <surname>Piorkowski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Raya</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. L.</given-names>
            <surname>Lugo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Papadimitratos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Grossglauser</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.-P.</given-names>
            <surname>Hubaux. TraNS:</surname>
          </string-name>
          <article-title>Realistic joint traffic and network simulator for VANETs</article-title>
          .
          <source>SIGMOBILE Mobile Computing and Communications Review</source>
          ,
          <volume>12</volume>
          (
          <issue>1</issue>
          ):
          <fpage>31</fpage>
          -
          <lpage>33</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>C.</given-names>
            <surname>Pope</surname>
          </string-name>
          and
          <string-name>
            <given-names>K.</given-names>
            <surname>Kaur</surname>
          </string-name>
          .
          <article-title>Is it human or computer? Defending e-commerce with Captchas</article-title>
          .
          <source>IT Professional</source>
          ,
          <volume>7</volume>
          (
          <issue>2</issue>
          ):
          <fpage>43</fpage>
          -
          <lpage>49</lpage>
          ,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>C. W.</given-names>
            <surname>Reynolds</surname>
          </string-name>
          .
          <article-title>Steering behaviors for autonomous characters</article-title>
          .
          <source>In Game Developers Conference</source>
          , pages
          <fpage>763</fpage>
          -
          <lpage>782</lpage>
          . Miller Freeman Game Group,
          <year>1999</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>M. L.</given-names>
            <surname>Sichitiu</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Kihl</surname>
          </string-name>
          .
          <article-title>Inter-vehicle communication systems: A survey</article-title>
          .
          <source>IEEE Communications Surveys &amp; Tutorials</source>
          ,
          <volume>10</volume>
          (
          <issue>1-4</issue>
          ):
          <fpage>88</fpage>
          -
          <lpage>105</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>C.</given-names>
            <surname>Sommer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>German</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Dressler</surname>
          </string-name>
          .
          <article-title>Bidirectionally coupled network and road traffic simulation for improved IVC analysis</article-title>
          .
          <source>IEEE Transactions on Mobile Computing</source>
          ,
          <volume>10</volume>
          (
          <issue>1</issue>
          ):
          <fpage>3</fpage>
          -
          <lpage>15</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Toor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mu</surname>
          </string-name>
          <article-title>¨hlethaler, A. Laouiti, and</article-title>
          <string-name>
            <given-names>A. de La</given-names>
            <surname>Fortelle</surname>
          </string-name>
          .
          <article-title>Vehicle ad hoc networks: Applications and related technical issues</article-title>
          .
          <source>IEEE Communications Surveys &amp; Tutorials</source>
          ,
          <volume>10</volume>
          (
          <issue>1-4</issue>
          ):
          <fpage>74</fpage>
          -
          <lpage>88</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>L. von</given-names>
            <surname>Ahn</surname>
          </string-name>
          and
          <string-name>
            <given-names>L.</given-names>
            <surname>Dabbish</surname>
          </string-name>
          .
          <article-title>Labeling images with a computer game</article-title>
          .
          <source>In SIGCHI Conference on Human Factors in Computing Systems (CHI'04)</source>
          , pages
          <fpage>319</fpage>
          -
          <lpage>326</lpage>
          . ACM,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>L. von</given-names>
            <surname>Ahn</surname>
          </string-name>
          and
          <string-name>
            <given-names>L.</given-names>
            <surname>Dabbish</surname>
          </string-name>
          .
          <article-title>Designing games with a purpose</article-title>
          .
          <source>Communications of the ACM</source>
          ,
          <volume>51</volume>
          (
          <issue>8</issue>
          ):
          <fpage>58</fpage>
          -
          <lpage>67</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29] L. von
          <string-name>
            <surname>Ahn</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Maurer</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>McMillen</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Abraham</surname>
            , and
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Blum</surname>
          </string-name>
          . reCAPTCHA:
          <article-title>Human-based character recognition via web security measures</article-title>
          .
          <source>Science</source>
          ,
          <volume>321</volume>
          (
          <issue>5895</issue>
          ):
          <fpage>1465</fpage>
          -
          <lpage>1468</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>T. L.</given-names>
            <surname>Willke</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Tientrakool</surname>
          </string-name>
          , and
          <string-name>
            <given-names>N. F.</given-names>
            <surname>Maxemchuk</surname>
          </string-name>
          .
          <article-title>A survey of inter-vehicle communication protocols and their applications</article-title>
          .
          <source>IEEE Communications Surveys &amp; Tutorials</source>
          ,
          <volume>11</volume>
          (
          <issue>2</issue>
          ):
          <fpage>3</fpage>
          -
          <lpage>20</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>B.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Ouksel</surname>
          </string-name>
          , and
          <string-name>
            <given-names>O.</given-names>
            <surname>Wolfson</surname>
          </string-name>
          .
          <article-title>Opportunistic resource exchange in inter-vehicle ad-hoc networks</article-title>
          .
          <source>In Fifth IEEE International Conference on Mobile Data Management (MDM'04)</source>
          , pages
          <fpage>4</fpage>
          -
          <lpage>12</lpage>
          . IEEE Computer Society,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>T.</given-names>
            <surname>Yan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Marzilli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Holmes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Ganesan</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Corner</surname>
          </string-name>
          .
          <article-title>mCrowd: A platform for mobile crowdsourcing</article-title>
          .
          <source>In Seventh ACM Conference on Embedded Networked Sensor Systems (SenSys'09)</source>
          , pages
          <fpage>347</fpage>
          -
          <lpage>348</lpage>
          . ACM,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [33]
          <string-name>
            <surname>M.-C. Yuen</surname>
            ,
            <given-names>I. King</given-names>
          </string-name>
          , and
          <string-name>
            <given-names>K.-S.</given-names>
            <surname>Leung</surname>
          </string-name>
          .
          <article-title>A survey of crowdsourcing systems</article-title>
          .
          <source>In Third International Conference on Privacy, Security, Risk and Trust (PASSAT 2011) and Third International Conference on Social Computing (SocialCom</source>
          <year>2011</year>
          ), pages
          <fpage>766</fpage>
          -
          <lpage>773</lpage>
          . IEEE,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>