Evaluating Reputation in VGI-enabled Applications

Abstract

Volunteered Geographic Information (VGI) is an approach to crowdsource information about geospatial objects around us, as implemented in Open Street Map, Google Map Maker and WikiMapia projects. The value of this content has been recognized by both researchers and organizations for acquiring free, timely and detailed spatial data versus standard spatial data warehouses where objects are created by professionals with variable updating time. However, evaluating its quality and handling its heterogeneity remain challenging concerns. For instance, VGI data sources have been compared to authoritative geospatial ones on speci c regions/areas in order to determine an average overall quality level. In user-oriented VGI-based applications, it can be more relevant to assess the quality of particular contents, like speci c Points of Interest. In this case, evaluation can be performed indirectly by reputation scores associated with the speci c content. This paper focuses on this last aspect. Our contribution primarily provides a comprehensive model and architecture for reputation evaluation aimed to assess quality of VGI content. On the other hand, we also focus on applications by discussing two motivating scenarios for reputation-enhanced VGI data in the context of geospatial decision support systems and in recommending tourist itineraries.

1. INTRODUCTION

Following the Web 2.0 trend, more and more content published on the Web is generated by end users. Due to the growing availability of Geographic Positioning System (GPS)enabled mobile devices, this user-generated content is commonly associated with geographical coordinates. This feature is more and more recognized as relevant by researchers and organizations [ 14, 11 ]. Recent platforms, including Open Street Map (OSM) and Wikimapia, permit users to contextualize the content of these maps. These platforms make it easy interacting with the map content through some editing functions, such as add and edit. These activities are known as volunteered geographical information (VGI). VGI could provide more timely, updated and detailed content than authoritative sources. Moreover, unlike commercial geospatial sources, VGI contents have a special value because they are able to capture multiple perspectives of the same location as perceived by di erent users from di erent cultural backgrounds (e.g., a Buddhist temple could be depicted as a touristic attraction or a religious location; a street with speed bumps can be a described a potential danger by a motorbike user and as a safe street by a pedestrian one).

The process of VGI generation is intrinsically subjective and loosely controlled; it relies very often on devices having variable level of precision and on untrained volunteers. As a consequence, VGI data are highly heterogeneous in coverage, density and quality. Techniques to estimate and improve the quality of these data are therefore needed [ 2, 9, 15 ].

A classi cation of approaches for assessing the quality of VGI have been discussed in [ 10 ]. Other approaches, as in [ 9 ], aimed to establish the average quality of VGI data on speci c regions/areas by comparing it with authoritative data and measuring the existing discrepancies in terms of standard quality metrics such as completeness, consistency and both positional and temporal accuracy. However, knowing the quality of speci c VGI data, like description and location of a speci c point of interest (PoI), is more relevant for user-oriented applications focusing on the description of particular geospatial objects. In these cases, there is the need to know whether to trust or not the VGI description. Examples are applications providing information on location of architectural barriers [ 7 ] or potability of water wells [ 3 ].

Therefore, de ning quality indicators for VGI data and, speci cally, PoIs is relevant and various ones have been proposed [ 15 ]. Indicators focus on aspects in uencing the quality without measuring it directly. Some of them are: lineage (which relies on the history and evolution of the dataset describing, for example, a PoI), quality of textual descriptions [ 4 ], experience [ 4 ], trustworthiness and reputation [ 8 ].

In this paper, we provide a contribution to the VGI research which is twofold. Firstly, we discuss two motivating applicative scenarios for reputation-enhanced georeferenced data in the context of geospatial decision support systems and in recommending tourist itineraries. Then, we present and evaluate a comprehensive model and some preliminary experiments for evaluating reputation in user-generated georeferenced data. This contribution extends and re nes our previous work [ 12 ] by including time and other features in the model. The paper connects its contribution to the challenges arising in linked data management, by leveraging OpenStreet Map, Wikimapia, Panoramio, or Googlemaps datasets. The described approach could be implemented with Linked Data and, as future direction, it can become a use-case for the recent work on Open Annotation1.

The remainder of this paper is organized as follows. In Section 2, the applicative scenarios are presented to show the potentiality of integrating reputation scores and usergenerated georeferenced data in speci c applications. Some related work is discussed in Section 3. In Section 4, we present our multi-layer architecture to enhance VGI reputation. In Section 5, the Reputation Model is proposed. The paper concludes in Section 6 with some suggestions for a future model extension and our research roadmap.

MOTIVATING SCENARIOS

VGI in Spatial Decision Support Systems

Spatial Decision Support Systems (SDSSs) are being created to allow several stakeholders to collaboratively plan their actions. In such systems, Spatial Data Warehouses (SDWs) are extensively deployed as common repositories where stakeholders' data sources are integrated and stored [ 5 ]. In order to load data into a SDW, a complex Extract-LoadTransform (ETL) process is typically used. Because of the growing complexity in several real-life scenarios, decisionmakers are relying more on advanced technologies to collect data anytime, anywhere, about events and objects of interest. As these technologies became widely available as wearable and portable devices (e.g., smartphones), decisionmakers are increasingly soliciting individuals to actively report on ongoing events. Several tools are therefore being created to enable VGI datasets to be acquired within the context of SDSSs. The creation of such tools is motivated by the important role of VGI techniques in collecting on-they valuable data, improving the understanding of ongoing events, discovering behavioural patterns that would improve decision-making processes, and implementing proper mechanisms to provide individuals with customized services. Although some progress is being achieved, several obstacles are still challenging the e orts of integrating VGI capabilities within SDWs. For instance, the VGI datasets are commonly unstructured, with varying qualities, formats, and granularities. In order to meet the DW requirements, ex1http://www.openannotation.org/.

Module. As this investigation is expected to happen frequently, assigning reputation values to VGI participants and contributions is a relevant option. These values are used to prioritize data loading into the SDW and relieve this process from unnecessary costly processing activities.

Spatial Data Warehouse (SDW) Creation Platform

Archive Data Sources (DS) Spatial

Data Warehouse Integration Module

Extract-Transform-Load (ETL) Module …..

ETL1 ETLp

ETL2 Data Transformation

Module ….. DS 1 DS n

DS 2 Data Loading Scheduler Reputation Database

Reputation Module

VGI Repository

Volunteered Geographic Information (VGI) Platform tensive transformation e orts must be performed prior to sending the VGI datasets to the ETL process. These e orts are supported in our architecture (See Fig. 1) by the Integration Module. Furthermore, since VGI datasets are voluntary collected by experts, non-experts, and even malicious individuals, their values, credibility, veracity, and trustworthiness will be investigated by our Reputation Module. As this investigation is expected to happen frequently, assigning reputation values to VGI participants and contributions is a relevant option. These values are used to prioritize data loading into the SDW and relieve this process from unnecessary costly processing activities. 2.2

Recommending sigthseeing tours with Tripbuilder

Tourists approaching their destination for the rst time deal with the problem of planning a sightseeing itinerary that covers the most subjectively interesting attractions while tting the time available for their visit. TripBuilder2 is an unsupervised system helping tourists to build their own personalized sightseeing tour [ 6 ]. Given the target destination, the time available for the visit, and the tourist's pro le, TripBuilder recommends a time-budgeted tour that maximizes tourist's interests and takes into account both the time needed to enjoy the attractions and to move from one PoI to the next one. A distinctive feature of TripBuilder is that the knowledge base feeding the recommendation model is entirely and automatically extracted from publicy available crowdsourced data like Flickr3 and Wikipedia4.

TripBuilder extracts from Wikipedia the multilingual name of the PoI, its geographic coordinates, the categories of the PoI according to a category. By clustering and spatially matching tourists' photo albums from Flickr on the relevant PoIs extracted from Wikipedia pages, we can thus derive a knowledge base that represents the behavior of people visiting a given city. The Wikipedia categories of the PoIs visited by a given tourist are used to build her pro le and to characterize the trajectories across the PoIs.

However, Wikipedia as a source of PoIs for tourism recom2http://tripbuilder.isti.cnr.it 3http://www. ickr.com 4http://www.wikipedia.com mendations has some limitations. For example, several areas of the world, like latin-america or asian countries, are not covered by a su cient number of Wikipedia pages describing individual tourist attractions inside a city like museums or monuments. This means that in these areas the PoIs are too sparse to describe itineraries and thus we need to rely on other, possibly local, sources of crowd data. This calls for the integration of local-based, up-to-date, user-generated and reliable Volunteer Geographical data source capable to describe the Points of Interests, crucial for expanding Tripbuilder in these areas. The integration of these data requires a reputation evaluation mechanism to ensure that the user-generated data are reliable enough for recommending tourists.

RELATED WORK

Several research and development works have attempted to estimate reputation in VGI applications. For example, Bishr and Khun [ 3 ] described a reputation model based on coherence between volunteers' reports on potability of water wells in developing countries. The potability status has only two possible values: good or bad. Time is explicitly included in the model. For instance, trustworthiness in the reports about potability is reduced proportionally to the elapsed time since the creation time of reports. Our model considers more general VGI scenarios allowing for complex descriptions of objects. Another approach is given in Zhao et al. [ 17 ]. The approach estimates the trustworthiness of VGI data based on contributor's reputation as well as on analyzing several versions of VGI data. This is similar to the approaches proposed by Ke ler et al. [ 13 ] and D'Antonio et al. [ 8 ]. Each version is created by a contributor and describes the current status of a geospatial object. The level of trust for a speci c version depends on: (i) contributor's reputation; (ii) similarity distance between this version and the previous one for the same object; and (iii) level of trust in the previous version. This approach looks actually inspired by D'Antonio et al. [ 8 ], but additionally it provides a detailed data model. As per our model, these authors distinguish between implicit and explicit assessment of contributors.

Trustworthiness and reputation have been studied in other contexts, for example crowdsourcing [ 1 ] and multi-agent systems [ 16 ]. In particular, the work [ 1 ] is about trustworthiness of semantic annotations of textual contents. The discussion is interesting because it considers as not avoidable some levels of disagreement in users' annotations for the same text. A particular annotation is considered acceptable when the disagreement with other annotations is not exceeding a given level. In fact, in some cases, this situation re ects the presence of semantic ambiguity in the described object, which can be perceived and annotated by di erent users according to di erent points of view.

Our approach aims to provide a comprehensive evaluation model for reputation in VGI. It extends our previous work by including temporal aspects in uencing reputation and the asymmetry of feedbacks, as proposed too in [ 3 ]. It distinguishes as well between direct and indirect user feedbacks. Another bene t of our contribution is to be resistant to manipulation attempts done by users with a malicious attitude. Some preliminary experimentation provides encouraging results in this direction.

Layer of Users Layer of Evaluations Layer of Versions VGI Sources Real world objects

Reputation User a score a = ...

Reputation User b score b = ...

Reputation

User c score c = ...

Modification Feedback (+/-)

Completion

Modification VeVrserIsDI,DO_bi,jOUbRjIURI <attr, value> <<aattttrr,_iv1a,luveal>ue_i1> <<aattttrr,_iv2a,luveal>ue_i2> <attr_…i3, value_i3> … Reputation score i = ...

Vers ID, Obj URI Vers ID_j, Obj URI <attr_j3, value_j1> <a<tatrt_trj2_j,3v,avlualeu_ej2_j>1> <a<tatrt_trj3_j,2v,avlualeu_ej3_j>2> <attr_j3…, value_j3> … Reputation

score j = ...

...

VGI Repository

VGI Repository

Reputations

AN ARCHITECTURE FOR REPUTATION

ENHANCED-VGI

In this section, we present a multi-layer architecture (see Fig. 2) for VGI enhanced with reputation scores. This architecture brie y introduces the main concepts of our model for reputation evaluation. Main focus of this paper is the layer of evaluations and, secondly, the layer of versions. According to [ 17 ] a VGI version is the description of a state of a geospatial real world object at a certain time and authored by a user. For the same state more versions can be produced, as well as the state of an object can change during the object lifespan (e.g., its address can change) making out of date previous versions describing it. We model a version (e.g., the description of a georeferenced PoI or a element on a map, as a street) as set of pairs < attribute; value > with a URI identifying the described object and can be completed or modi ed many times. Versions are stored in VGI repositories, like OSM and WikiMapia. In order to improve an existing version, a user shall modify or complete it to originate another version for the same object. In Fig. 2, versions stored in the same repository and describing the status of an object are grouped together. A user can also give a direct feedback, positive or a negative, to versions in order to express agreement or disagreement on the content.

We consider four user activities as below: A user can: (i) create a new version, (ii) modify an existing version, (iii) complete an existing version, or (iv) give a direct feedback to a version.

A completion is a set of pairs < attribute; value > added to an existing version (e.g., adding <phone2, [phone]> to the version). A modi cation consists of new values for pairs < attribute; value > already existing in the version (e.g., the pair <opening, Monday-Friday> corrects <opening, Monday-Saturday>). Every time a user makes a modi cation (respectively, a completion) to a version A, a new version A' is generated from A by substituting the original with the modi ed content (respectively, by adding content). For each version (e.g., A, A'), its creation time is stored with the version. A reputation value in [0; 1] is associated with each user and each version. Modi cations, completions and direct feedbacks on A modify both the reputations of the version A and of the author of A. In Fig. 2, reputation scores of authors and versions are stored and managed by a separated platform from the VGI source repositories.

REPUTATION EVALUATION MODEL

In this section, we describe some requirements we adopted for reputation evaluation and we provide the metrics. Our purpose is to formalize an evaluation model based on our multi-layer architecture by taking into account a set of requirements we consider important for quantifying reputation. According to the architecture in Fig. 2, presented in the previous section, a user can perform activities to produce georeferenced content (i.e., versions) and to evaluate other peers' content. In our approach, these activities are subject to constraints. For example, a user is not allowed to give directly a feedback to another user and he cannot evaluate more than once a version. This is set in order to prohibit a malicious user from deliberately increasing reputation of other speci c peers or their contents. Reputation of versions does not depend only on feedbacks, considered as direct evaluations, but also on modi cations and completions, which we consider kind of indirect feedbacks. In fact, we assume that a modi cation reduces the reputation because the user producing it considers the original version as including errors or requiring updates. On the contrary, a user making a completion to a version recognizes it as correct, but incomplete. Therefore, a completion activity has to increase the reputation of the original version. In the following, we formalize the metrics.

User reputation.

Every time a feedback, completion or modi cation are made on a version, the User reputation score of its author is updated as follows: uRu = ( (1

uR0 e n) P OSu+NEGu + e n uR0

P OSu (1) where uRu 2 [0; 1]. The rst case of Eq. (1) applies when no activities have been made yet to any versions authored by user u, i.e., P OSu = 0 and N EGu = 0. The term uR0 is the initial user reputation, usually set to a low value (e.g., 0.3) equal for every user. Alternatively, this value could be set proportionally to the completeness in lling the user registration pro le. In the second case of Eq. (1), parameter n is the total number of versions produced by user u. This includes new versions and versions produced due to the modi cations or completions of other versions. The role of coe cient e n is to reduce the amplitude of the initial oscillations of the uRu value when n is low (e.g., n < 3). This avoids that a single positive or negative feedback can increase or decrease considerably the reputation uRu when a user has produced only few versions. Moreover, as n increases the initial reputation uR0 counts less and the user reputation score depends more on the feedbacks provided by other users.

De nition of terms P OSu and N EGu are as below: P OSu =

P OSv h(tv)

N EGu = v2V (u) v2V (u) (2) where V (u) is the set of contributions produced by u and v is one version. P OSc and N EGc are terms de ned in the following and summarizing, respectively, positive and negative feeedbacks expressed on v. h(tv) is a coe cient weighting the contribution of P OSv and N EGv and depends on the creation time tv of v.

Aging of versions.

The coe cient h() assigns a higher weight to feedbacks expressed on recent versions. Its value decreases linearly with the di erence between the time t, when the P OSu and N EGu expressions are evaluated, and the creation time tv of version v. In particular, h(tv) is equal to 0 when this di erence is larger than value . For example, in our experiments we have chosen equivalent to 180 days so the feedbacks on a version are no more changing the author's reputation when the version is older than 180 days. This coe cient is de ned as: h(tc) = ( (t tv) 0 if t tv < if t tv where h(tc) 2 [0; 1]. Because P OSv and N EGv are always positive then P OSu and N EGu are positive as well.

Reputation of versions.

The reputation score vRv of a version v is based on (implicit and explicit) feedbacks given to v by other peers. vRv = ( (1

vR0 e k) P OSv+NEGv + e k vR0

P OSv where vRv 2 [0; 1] and k is the number of feedbacks on v. The rst case in Eq. (4) applies when k = 0. In the second case, when k 6= 0, reputation score vRv is de ned according to an expression similar to the one for user reputation score, as in Eq. 1.

The initial reputation vR0 2 [0; 1] depends on the type of activity (new version, modi cation, completion) to create v and on the reputation of its author u, as follows: vR0 = 8 > < (1

uRu

Sim(v; vP rev)) uRu+ >: Sim(v; vP rev) min(vRvP rev; uRu) if vers(v) otherwise (5) where vers(v) is true when v is a new version and, in this case, the initial reputation vR0 is set to be equal to the reputation uRu of the author. Otherwise, vR0 is a weighted mean of: (i) the author reputation uRu, and (ii) the minimum between reputation of the parent contribution vP rev, which v is obtained from by modi cation or completion, and uRu. The rationale for selecting the minimum is the following. If reputation of vP rev is high and uRu is low, a user u could maliciously modify a version with high reputation in order to produce a version v with some wrong content (e.g., a di erent web site address) but having initially a high reputation. By taking the minimum, this cannot happen since (3) (4)

Correct Identi cations (U) 90% 70% 50% (a) the user reputation uRu is the upper bound of the initial reputation of v. Sim(v; vP rev) 2 [0; 1] is a similarity coefcient between the previous and the current version. More these versions are dissimilar (i.e., Sim(v; vP rev) 0), more the reputation score is near to the author's reputation score uRu and therefore independent of the reputation of vP rev. This is because the version v modi es signi cantly vP rev so the reputation of v should be more focused on the author than on the previous version vP rev. We will not detail here how to calculate Sim(v; vP rev). It is proportional to the number of pairs < attribute; value > equal in both the versions v and vP rev.

Feedback evaluation.

The values of P OSv and N EGv, summarizing the positive and negative amount of direct and indirect feedbacks on version v, are updated every time a new feedback on v is produced. The impact of a feedback by a user uf is proportional to reputation of uf . In particular, the equation for updating the current P OSv when either a positive feedback or a completion f is submitted by uf with reputation uRf , is the following:

P OSv0 = P OSv+ ! ( uRf

if f is pos. feedback (6)

Sim(v0; v) uRf if f is completion In case of completion, v0 denotes the upgraded version of v after this operation. More the versions v and v0 are similar, higher the increasing of P OSv due to the presence of the similarity coe cient Sim(v0; v). The rationale is that if v0 is very di erent from v, it is not considered a completely positive evaluation of v, i.e., v was evaluated as incomplete. In this case, P OSv0 has not to di er a lot from P OSv.

We de ne as well the equation for updating N EGv when either a negative feedback or a modi cation f is submitted by uf with reputation uRf : N EG0v = N EGv+ (1 !) ( uRf

if f is neg. feedback (1

Sim(v0; v)) uRf if f is modi cation (7) Finally, to note that the coe cient ! 2 [0; 1] in Eq. (6) and in Eq. (7) permits to make asymmetric the way P OSc and N EGc are increased by a certain feedback. By assigning a low value to !, a negative feedback increases more N EGv than a positive one does on P OSv. Here we include in the model a notion of asymmetry between positive and negative feedback as Bishr and Khun [ 3 ], who observe that negative feedbacks in real life have a stronger absolute impact on reputation than positive ones and therefore should be weighted di erently.

Experimental evaluation.

We discuss some preliminary experimental results to test our approach. Even if these experiments are limited the obtained results are encouraging. We implemented the reputation model as described in the previous section and tested its accuracy on a simulated dataset in a Matlab environment.

In our simulation, a user is labeled either cooperative or non-cooperative and versions are labeled either correct (i.e., supposed to represent correctly the reality) or incorrect. During the simulation, cooperative users can: (i) create correct versions, and (ii) perform activities on existing versions that are coherent with their nature; which means if a version they evaluate is correct then they can give either a positive feedback or complete it. Non-cooperative users, instead, behave according to a malicious attitude. They can: (i) create incorrect versions, (ii) give either negative feedbacks or modify correct versions, to reduce the reputations of versions and authors. Moreover, a user can be active by performing any type of action, or non active, by giving direct feedbacks only.

By running some simulations, we measure the accuracy of the reputation model in identifying correctly, i.e., according to the labels, cooperative/non-cooperative users and correct/incorrect versions. We recall that the objective of the proposed reputation model is to assign high reputation scores to cooperative users and to correct versions, or low reputation scores otherwise. At the end of a simulation, a cooperative user is identi ed correctly if the reputation score is in the interval [0:5; 1]; a non-cooperative user is identi ed correctly if the reputation score is in the interval [0; 0:5]. We use the same criteria for identi cation of versions. In the experiment, we want to observe this phenomenon by varying the number of cooperative users. We also disregard some features like the asymmetry of feedbacks (i.e., we set ! = 0:5) and the aging of versions (i.e., we set h(tc) = 1). The result permits to show good levels of accuracy even in a simpli ed version of the model. All the numerical results presented in the following are obtained as averages of ten running simulations. We set the percentage of active users to 10%. Therefore, the remaining 90% of users are non active ones. Approximately, this proportion re ects the one between active users, contributing as reviewers, and users giving only feedbacks, with reference to the Amazon community5. In our simulations, active users are selected randomly among others. The type of an activity performed during the simulation is chosen randomly based on the probability distribution: creation of a new version, 1%; modi cation, 1%; completion, 1%; direct feedback, 97%. The results concerning the amount of users identi ed correctly is shown in Fig. 3(a). The result concerning the amount of identi ed versions is in Fig. 3(b).

We can notice the high identi cation performances with 5https://www.quora.com/What-percentage-of-buyerswrite-reviews-on-Amazon percentages of cooperative users of 90% and 70%. The results are even better if we consider the correct identi cation of only versions having at least 5 feedbacks, where with 70% of cooperative users, we got 92:2% of correct identi cations (not reported in the gure). On the contrary, with only 50% of cooperative users the performances drop down. In this case, the number of cooperative and non-cooperative users are the same and every user has the same initial reputation score uR0, so the model is not able to identify who is cooperative and who is not.

CONCLUSIONS AND PERSPECTIVES

In this work, we discussed a novel comprehensive model and architecture for reputation evaluation of Volunteered Geographic Information content. An initial evaluation based on simulations has been presented, as well. Our approach de nes metrics to score a composite reputation of VGI data coming from unknown contributors. We promote the usefulness of our model by discussing the integration of VGI data with reputation scores in two di erent application scenarios. Future work includes dealing with inconsistency that may arise due to repeated updates and completions for the same PoI. We are also planning to perform a deeper evaluation of the model, studying its possible joint use with other techniques for quality assessment in VGI data, and improving the architecture and the model in order to better integrate them with the discussed applicative scenarios.

References

[1] Lora Aroyo and Chris Welty . Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard . WebSci2013. ACM , 2013 , 2013 .

[2] Jamal Jokar Arsanjani , Peter Mooney , Alexander Zipf, and Anne Schauss . Quality assessment of the contributed land use information from openstreetmap versus authoritative datasets . In OpenStreetMap in GIScience, pages 37 { 58 . Springer, 2015 .

[3] Mohamed Bishr and Werner Kuhn . Trust and reputation models for quality assessment of human sensor observations . In Thora Tenbrink , John Stell, Antony Galton, and Zena Wood, editors, Spatial Information Theory: 11th International Conference, COSIT 2013 , Scarborough , UK, September 2- 6 , 2013 . Proceedings, pages 53 { 73 . Springer International Publishing, Cham, 2013 .

[4] Gloria Bordogna , Paola Carrara, Laura Criscuolo, Monica Pepe, and Anna Rampini . A linguistic decision making approach to assess the quality of volunteer geographic information for citizen science . Information Sciences , 258 : 312 { 327 , 2014 .

[5] Boubaker Boulekrouche , Nafaa^ Jabeur, and Zaia Alimazighi . Toward integrating grid and cloud-based concepts for an enhanced deployment of spatial data warehouses in cyber-physical system applications . Journal of Ambient Intelligence and Humanized Computing , 7 ( 4 ): 475 { 487 , 2016 .

[6] Igo Ramalho Brilhante , Jose Antonio Macedo, Franco Maria Nardini, Ra aele Perego, and Chiara Renso . On planning sightseeing tours with tripbuilder . Information Processing & Management , 51 ( 2 ):1 { 15 , 2015 .

[7] Maria Antonia Brovelli , Marco Minghini, and Giorgio Zamboni . Public participation in fGISg via mobile applications . fISPRSg Journal of Photogrammetry and Remote Sensing , 114 : 306 { 315 , 2016 .

[8] Fausto D 'Antonio, Paolo Fogliaroni, and Tomi Kauppinen . Vgi edit history reveals data trustworthiness and user reputation . In 17th AGILE International Conference on Geographic Information Science (Short Paper) , 2014 .

[9] Helen Dorn , Tobias Tornros, and Alexander Zipf. Quality Evaluation of VGI Using Authoritative Data - A Comparison with Land Use Data in Southern Germany . ISPRS International Journal of Geo-Information , 4 ( 3 ): 1657 { 1671 , 2015 .

[10] Michael F. Goodchild and Linna Li . Assuring the quality of volunteered geographic information . Spatial Statistics , 1 : 110 { 120 , 2012 .

[11] Denis Havlik , Javier Soriano, Carolos Granell, Stuart E. Middleton, Hylke van der Schaaf, Arne Berre, and Jasmin Pielorz . Future internet enablers for VGI applications . In EnviroInfo 2013. September 2013 .

[12] Roula Karam and Michele Melchiori . A crowdsourcing-based framework for improving geo-spatial open data . In 2013 IEEE International Conference on Systems, Man, and Cybernetics , pages 468 { 473 . IEEE, 2013 .

[13] Carsten Ke ler , Johannes Trame, and Tomi Kauppinen . Tracking editing processes in volunteered geographic information: The case of openstreetmap. Identifying objects, processes and events in spatio-temporally distributed data (IOPE ), workshop at conference on spatial information theory , 12 , 2011 .

[14] Peter Mooney , Padraig Corcoran, and Blazej Ciepluch . The potential for using volunteered geographic information in pervasive health computing applications . Journal of Ambient Intelligence and Humanized Computing , 4 ( 6 ): 731 { 745 , 2013 .

[15] Hansi Senaratne , Amin Mobasheri, Ahmed Loai Ali, Cristina Capineri, and Muki Haklay . A review of volunteered geographic information quality assessment methods . International Journal of Geographical Information Science , May 2016 .

[16] H. Yu , Z. Shen , C. Leung , C. Miao , and V. R. Lesser . A survey of multi-agent trust management systems . IEEE Access , 1 : 35 { 50 , 2013 .

[17] Yijiang Zhao , Xiaoguang Zhou , Guangqiang Li , and Hanfa Xing . A Spatio-Temporal VGI Model Considering Trust-Related Information . ISPRS International Journal of Geo-Information , 5 ( 2 ): 10 , 2016 .