Urban-related data and geographic information are becoming mainstream in the Linked Data community due also to the popularity of Location-based Services. In this paper, we introduce the UrbanMatch game, a mobile gaming application that joins data linkage and data quality/trustworthiness assessment in an urban environment. By putting together Linked Data and Human Computation, we create a new interaction paradigm to consume and produce location-speci c linked data by involving and engaging the nal user. The UrbanMatch game is also o ered as an example of value proposition and business model of a new family of linked data applications based on gaming in Smart Cities.
Urban environments are experiencing a progressive digitization that is leading to the creation and release of large amounts of data: information about interesting aspects of cities { ranging from street topology and tra c conditions to business activities, from points of interest (POI) to events, from environmental measures to people life-logs { are increasingly present on the Web and even proactively fed by the open community. The growing attention given to Smart Cities themes and problems and the ever increasing popularity of location-based applications make urban ecosystems at the center of the research and innovation agenda of public authorities and big industrial players such as IBM with its Smarter Planet initiative1, and CISCO with its Smart+Connected Communities initiative2.
Also the Linked Data world turned to urban data, especially in the area of spatial information. One of the most
connected_communities.html
curated dataset available in this eld is GeoNames3.
Recently, open Web APIs like Open Street Map4 and
geographic datasets from public administrations were (partially)
turned into a Linked Data form by e orts like
LinkedGeoData5 [
Still this massive bulk of urban data is largely unexplored
and poorly exploited [
Our research hypothesis consists in employing user-friendly
mobile gaming applications to engage people in mobility;
through those games, Linked Data related to urban
environments are consumed, created, improved and corrected.
While a similar approach was already partially explored for
Linked Data at large [
In this paper we present our rst experiment to prove our hypothesis: UrbanMatch is a mobile and location-aware Game with a Purpose that engages players to provide information related to the city of Milano. Speci cally, UrbanMatch is aimed at linking points of interests in the city with the most representative photos retrieved from Web sources.
The remainder of this paper is organized as follows. Section 2 introduces motivation and related work; Section 3 explains the mechanics and purpose of the UrbanMatch game, while the evaluation methodology is shown in Section 4; nally Section 5 draws some conclusions and future works. 2.
Our work focuses on link creation and quality assessment for Linked Data in urban scenarios. In this section, we il
lustrate the motivation problem, explain the speci city of Smart City solutions and introduce and discuss related work. 2.1
Data Quality [
A key point of Data Quality is that the quality
measurements are context-dependent: given a dataset, its quality
can be very high with respect to the ful lment of some tasks
but very bad for other ones. In other words, it is not
relevant (and not always possible) to de ne absolute quality
values [
For the last years, the Linked Data community has started
to follow this topic with growing interest: the birth and
growth of the Linked Open Data (LOD) cloud introduced a
huge amount of RDF data distributed across several datasets.
The Linked Data best practices alone [
Still, problems related to the available data quality soon arose in the LOD cloud. While at the beginning the number of statements and the number of links were used to estimate the relevance of the published data sets, more recently the de nition of more detailed and expressive metrics to describe the available data has become more and more important.
Flemming worked on the de nition of quality criteria for
linked data sources [
However, the assessment of data quality factors like accuracy, timeliness, completeness, relevance and comprehensiveness of data is intrinsically a hard task that Linked Data technologies do not make any easier. 2.2
In the last years we have been largely experimenting with
urban related linked (and non-linked) data in the
development of a number of Smart City demonstrators using Linked
Data related to urban environments. The Urban LarKC7
[
[
These experiences allow us to assert that: a) the quality of urban Linked Data unpredictably ranges from very good to very poor; b) it is possible to detect missing information or inconsistencies by cross-validating datasets that describe the same urban space from di erent points of view (e.g., if a road sign tells to turn right to reach a POI, and the POI is nearby, but no street reaches it, a street is missing in the topology dataset); c) when inconsistencies or missing data are detected, data quality can be easily increased by a small amount of manual work that does not require speci c skills, but often the physical presence in the urban environment.
Those three assertions may not be valid for Linked Data in general, but appear to be valid for urban Linked Data. The third point suggests us that a part of the manual work required to x urban Linked Data could be \crowd-sourced". The recent popularity of Location-based Services (LBS) like Foursquare demonstrates that people are willing to share small bits of information on-the-go by exploiting their mobile devices capabilities. \Unlock your city" { the slogan of Foursquare { reveals that LBS can be considered an effective means to collect useful information for Smart City applications. 2.3
Assessing data quality is a hard problem for computers.
We, as humans, are perfectly capable of it, but we are not
necessarily willing to. Human Computation [
The incentives to make people contribute to Human Computation can be of di erent kinds: they can give the participant an explicit and concrete reward (like in the popular Amazon Mechanical Turk { also named MTurk9 { in which people are paid to perform small and simple tasks) or they provide a di erent kind of implicit or more abstract return, for example by means of entertainment like in GWAPs.
In the Semantic Web community, Games with a Purpose
were already used to cover the complete Semantic Web
lifecycle [
UrbanMatch12 is a mobile location-based Game with a Purpose that aims at selecting the most representative photos related to the points of interest (POI) in an urban environment; more speci cally, UrbanMatch is oriented to link the monuments and relevant places of the city of Milano with their respective photos as retrieved from social media Web sites and to \rank" those links, so to identify the most characteristic ones and to discard the others, thus improving the quality. 3.1
Places & POIs from
OpenStreetMap
Manual Selection of linked photos Wikimedia Commons
Trusted source Uncertain sources
UrbanMatch server
UrbanMatch clients
Trusted links: <POI> foaf:depiction <photo>
The input data come from available Web sources (cf. Figure 1). Points of interest in Milano were collected and chosen among those available from OpenStreetMap; an RDF description of those POIs is also available in LinkedGeoData, the linked data version of OpenStreetMap. For each of the 34 POIs of this set, we manually selected 5-6 photos depicting them (some chosen on the Web, some taken by ourselves). In this way we built a \trusted set " of 196 links that relate the POIs with their respective images; those links are expressed in the form: <POI> foaf:depiction <photo> . 6
A much higher number of photos of Milano POIs was collected from Wikimedia Commons13 { the media collection of Wikipedia { and from Flickr14, probably the most popular social media sharing site dedicated to photos. The images were collected either by keyword/concept search (i.e., photos explicitly related to Milano POIs) or via location-based queries (e.g., search by geographical coordinates). Among the collected photos, we considered only those released with an open license, allowing for a free reuse of the image (like CreativeCommons \Attribution" license).
This second set of information is considered { for the game purpose { an \uncertain source": it consists of more than 37,000 \candidate" links that relate the POIs with the images that potentially depict them. This link-set is uncertain or untrusted because the retrieved photos can be incorrect (i.e. they are not related to Milano POIs even if returned by the search), their metadata can be wrong or incomplete or they 12http://bit.ly/urbanmatch 13http://commons.wikimedia.org/ 14http://www.flickr.com/ cannot be considered as representative for the game purpose (e.g., a photo is actually taken in the proximity of a POI, but it does not depict it or it is focused on an irrelevant detail).
Those candidate links are expressed as RDF links using the foaf:depiction predicate as explained before; however, those links are further annotated with a con dence value that expresses the lack of certainty about their trustworthiness (e.g., the initial con dence of links to Wikimedia images is set to 60%, links to Flickr to 40%). 3.2
The UrbanMatch game is a photo coupling game. The
game mechanics respects the best practice of casual games
and Games with a Purpose [
The links between the surrounding POIs and the presented photos is not the same for all the presented 8 photos: some links are certain, because they come from the trusted source; some are uncertain, because they are taken from the set of candidate links; nally, some are distractors, i.e. they are not related to the surrounding POIs and are used to check the reliability of players. Each game is organized in multiple levels of increasing di culty, i.e. with a varying number of certain/uncertain/distracting links (the higher the level, the greater the uncertainty degree).
The user geographic position taken from the mobile device sensors computes the user proximity to the \playable places", i.e. the game locations; even if the game allows for playing from any place, the user location is used to distinguish between the choices operated \on site" { based on the player's experience knowledge { and couples selected on the basis of the player's domain knowledge.
The UrbanMatch game is available at http://bit.ly/ um-itunes on the iTunes store. Being a research prototype, the game will be available only for a few months.
Through the UrbanMatch game we aim at identifying and selecting the POI-photo \correct" links among the candidate links in the uncertain input source. Through a Human Computation approach, we aim at collecting evidences of players decisions to correlate images.
The approach we follow to post-process the collected data
is similar to that of other Games with a Purpose [
Trusted
link POI Candidate links
POI
UrbanMatch players
PostProcessing
POI
New trusted
link
POI Incorrect link discarded
Intuitively, the game purpose processing is represented in Figure 3: in each game level, trusted POI-photo links (the green one on the left with Milano's Duomo picture) are presented together with a number of candidate links related to the same POI (the yellow ones on the left, with uncertain photos retrieved from Wikimedia Commons and Flickr as being related to Milano's Duomo) and with a number of distractors (for simplicity not drawn in the gure).
If players couple the same two photos, those photos reasonably belong to the same POI: thus, if a player associates a trusted photo with an uncertain photo, the candidate link related to the latter is given a sign of \trust" and its con dence value is increased.
On the contrary, if there is no evidence of the association between an uncertain photo and any other one, the candidate link to that photo is not validated: the lack of coupling actions is considered as a sign of \distrust" and decreases the con dence value of the candidate link.
The same POI-photo candidate link is given as input to multiple users; each player action modi es the link con dence value, by increasing or decreasing it. After a variable number of played games, this con dence value crosses some thresholds, thus leaving its uncertainty status and becoming either a trusted link or an incorrect one.
When the con dence value of a link becomes greater than a given upper threshold (e.g., 70%), the POI-photo link becomes \trustable" and is inserted in the trusted link-set; in the gure, the green link on the right is associated to Milano's Duomo and it can be used to validate other candidate links. Similarly, if the con dence value of a POI-photo link becomes smaller than a lower limit (e.g., 20%), the link is discarded and is no more given as input to other players, like for the red one on the right of Figure 3 (in which the photo evidently depicts Roma's Colosseum).
The evaluation of the UrbanMatch game is currently being performed and it is aimed at two di erent results: the assessment of the game \purpose" { the ability of our Human Computation approach to actually derive meaningful and quality links between Milano POIs and their related photos { and the appraisal of the game \playability" { the intrinsic fun or entertaining characteristic of the game.
Regarding the assessment of the game purpose, we
identi ed a number of metrics to measure the UrbanMatch
capability to improve the \ tness-for-use" quality [
We de ne completeness as the capability of the game to assess all the input candidate links, deciding if they are either trustable or incorrect. The completeness is calculated by dividing the number of assessed links (i.e. the links that became either trusted or incorrect after the gameplay) by the total number of input uncertain links.
We de ne accuracy as the capability of the game to make correct assessments about the input links, minimizing the \false positive" outcomes (i.e., POI-photo links considered trustable but actually incorrect) and \false negative" outcomes (i.e., POI-photo links considered incorrect but actually trustable). To measure the game accuracy, we need to know the ground truth, thus we manually check the assessed links to identify the false positive/negative items. The accuracy is then calculated by dividing the number of correct assessments (true positive and true negative items) by the total number of input uncertain links.
Our preliminary evaluation is based on an early set of played games: we collected evidences from 54 unique players (not including the development team), who played 290 games for a total of 781 levels, in which they tested 2,006 uncertain links. Setting the thresholds on the con dence value to 70% and 20% for the upper and lower limits respectively, the game assessed the correctness/incorrectness of 1,284 uncertain links, getting to an improvement of the global completeness from 1.54% to 4.98% with a nal accuracy of 99.4% (4 false positive and 8 false negative links).
On the other hand, it is clear that an important success factor for Games with a Purpose lies in the gaming feature: the more engaging and entertaining the game, the higher the number of participant players and thus the collected data. For those reasons, we believe that an important part of the UrbanMatch evaluation consists in assessing the \playability" of the game itself.
As suggested by several studies about traditional games [
At writing time, we collected the feedbacks of 12 players whose opinion was quite positive: UrbanMatch was evaluated to be easy (91%) and clear (61%); most players \spread the word" suggesting their friends to play (64%) and supported our hypothesis that the physical presence in the urban environment makes the gameplay easier (55%). 5.
While Linked Data research is continuously evolving and improving, new ways to interlink information from di erent independent sources are being formulated and explored. In this paper, we presented our UrbanMatch application, a mobile and location-based Game with a Purpose, oriented to create high-quality links between existing datasets, namely OpenStreetMap, LinkedGeoData and Flickr. UrbanMatch is our rst experiment to prove that mobile gaming applications can be successfully employed to consume, create and improve urban-related Linked Data; our early experience seems to con rm our research hypothesis, even if further evaluation is needed.
It is also worth noting that the data gathered via UrbanMatch have a clear business value: linked and ranked photos of places represent a valuable dataset which can be used to improve a number of services ranging from image search to geo-marketing. More generally, Games with a Purpose aimed at linking, collecting or correcting Linked Data related to urban spaces and Smart Cities constitute a potential business for a number of di erent stakeholders: local public authorities, local businesses (shops, transportation companies, tourism actors, etc.) and citizens.
Di erent datasets can be created or improved via Human Computation approaches for local services, trade, tourism, tra c optimization, environmental sustainability or simple information. This includes both urban related Linked Data and non-linked data such as municipality datasets or other private datasets related to Smart Cities. We believe that applications like UrbanMatch show a clear business model to be exploited with special regards to location-aware Linked Data.
A nal note on the \gaming" approach: data linkage and content creation or assessment tasks need an active and productive engagement of users, but not all crowdsourcing campaigns are successful and e ective. Shaping the Human Computation missions as gaming activities (like in Games with a Purpose) is a potential solution to this issue. A further research question could be oriented to prove that a funny or entertaining avour can accomplish more than an extrinsic reward: nding a good balance between the game rules and the purpose achievement is often a hard task.
This work was partially supported by Siemens Corporate Research and Technologies and by the PlanetData EU project (FP7-257641).