Evaluating Reputation in VGI-enabled Applications Marco Gusmini Nafaâ Jabeur Dept. of Inf. Engineering German University of Università degli Studi di Technology in Oman (GUtech) Brescia Athaibah, Sultanate of Oman Brescia, Italy nafaa.jabeur@gutech.edu.om Roula Karam Michele Melchiori Chiara Renso Dept. of Inf. Engineering Dept. of Inf. Engineering ISTI, CNR Università degli Studi di Università degli Studi di Pisa, Italy Brescia Brescia chiara.renso@isti.cnr.it Brescia, Italy Brescia, Italy michele.melchiori@unibs.it ABSTRACT 1. INTRODUCTION Volunteered Geographic Information (VGI) is an approach Following the Web 2.0 trend, more and more content pub- to crowdsource information about geospatial objects around lished on the Web is generated by end users. Due to the us, as implemented in Open Street Map, Google Map Maker growing availability of Geographic Positioning System (GPS)- and WikiMapia projects. The value of this content has been enabled mobile devices, this user-generated content is com- recognized by both researchers and organizations for acquir- monly associated with geographical coordinates. This fea- ing free, timely and detailed spatial data versus standard ture is more and more recognized as relevant by researchers spatial data warehouses where objects are created by pro- and organizations [14, 11]. Recent platforms, including Open fessionals with variable updating time. However, evaluating Street Map (OSM) and Wikimapia, permit users to contex- its quality and handling its heterogeneity remain challeng- tualize the content of these maps. These platforms make it ing concerns. For instance, VGI data sources have been easy interacting with the map content through some editing compared to authoritative geospatial ones on specific re- functions, such as add and edit. These activities are known gions/areas in order to determine an average overall quality as volunteered geographical information (VGI). VGI could level. In user-oriented VGI-based applications, it can be provide more timely, updated and detailed content than au- more relevant to assess the quality of particular contents, thoritative sources. Moreover, unlike commercial geospa- like specific Points of Interest. In this case, evaluation can tial sources, VGI contents have a special value because they be performed indirectly by reputation scores associated with are able to capture multiple perspectives of the same loca- the specific content. This paper focuses on this last aspect. tion as perceived by different users from different cultural Our contribution primarily provides a comprehensive model backgrounds (e.g., a Buddhist temple could be depicted as and architecture for reputation evaluation aimed to assess a touristic attraction or a religious location; a street with quality of VGI content. On the other hand, we also fo- speed bumps can be a described a potential danger by a cus on applications by discussing two motivating scenarios motorbike user and as a safe street by a pedestrian one). for reputation-enhanced VGI data in the context of geospa- The process of VGI generation is intrinsically subjective tial decision support systems and in recommending tourist and loosely controlled; it relies very often on devices having itineraries. variable level of precision and on untrained volunteers. As a consequence, VGI data are highly heterogeneous in coverage, Keywords density and quality. Techniques to estimate and improve the quality of these data are therefore needed [2, 9, 15]. Reputation Evaluation, Volunteered Geographic Informa- A classification of approaches for assessing the quality of tion, e-Tourism, Spatial DSS, User-Generated Content. VGI have been discussed in [10]. Other approaches, as in [9], aimed to establish the average quality of VGI data on spe- cific regions/areas by comparing it with authoritative data and measuring the existing discrepancies in terms of stan- dard quality metrics such as completeness, consistency and both positional and temporal accuracy. However, knowing the quality of specific VGI data, like description and loca- tion of a specific point of interest (PoI), is more relevant for user-oriented applications focusing on the description of par- ticular geospatial objects. In these cases, there is the need 2017, Copyright is with the authors. Published in the Workshop Proceed- to know whether to trust or not the VGI description. Ex- ings of the EDBT/ICDT 2017 Joint Conference (March 21, 2017, Venice, Italy) on CEUR-WS.org (ISSN 1613-0073). Distribution of this paper is amples are applications providing information on location of permitted under the terms of the Creative Commons license CC-by-nc-nd architectural barriers [7] or potability of water wells [3]. 4.0 Module. As this investigation is expected to happen frequently, assigning reputation values to VGI participants and contributions is a relevant option. These values are used to prioritize data loading into the SDW and relieve this process from unnecessary costly processing activities. Therefore, defining quality indicators for VGI data and, Spatial Data Warehouse (SDW) Creation Platform specifically, PoIs is relevant and various ones have been pro- Extract-Transform-Load Archive Data Sources (DS) posed [15]. Indicators focus on aspects influencing the qual- (ETL) Module Spatial ….. ity without measuring it directly. Some of them are: lineage Data ETL1 ….. DS 1 (which relies on the history and evolution of the dataset Warehouse ETLp describing, for example, a PoI), quality of textual descrip- ETL2 DS n DS 2 tions [4], experience [4], trustworthiness and reputation [8]. In this paper, we provide a contribution to the VGI re- Integration Module Data Loading Scheduler search which is twofold. Firstly, we discuss two motivating Data Transformation applicative scenarios for reputation-enhanced georeferenced Module Reputation Reputation Module data in the context of geospatial decision support systems Database and in recommending tourist itineraries. Then, we present and evaluate a comprehensive model and some preliminary experiments for evaluating reputation in user-generated geo- VGI Repository referenced data. This contribution extends and refines our previous work [12] by including time and other features in Volunteered Geographic Information (VGI) Platform the model. The paper connects its contribution to the chal- lenges arising in linked data management, by leveraging Figure 1: VGI Integration Module in a Spatial DW OpenStreet Map, Wikimapia, Panoramio, or Googlemaps architecture. datasets. The described approach could be implemented with Linked Data and, as future direction, it can become a tensive transformation efforts must be performed prior to use-case for the recent work on Open Annotation1 . sending the VGI datasets to the ETL process. These efforts The remainder of this paper is organized as follows. In are supported in our architecture (See Fig. 1) by the Inte- Section 2, the applicative scenarios are presented to show gration Module. Furthermore, since VGI datasets are volun- the potentiality of integrating reputation scores and user- tary collected by experts, non-experts, and even malicious generated georeferenced data in specific applications. Some individuals, their values, credibility, veracity, and trustwor- related work is discussed in Section 3. In Section 4, we thiness will be investigated by our Reputation Module. As present our multi-layer architecture to enhance VGI reputa- this investigation is expected to happen frequently, assign- tion. In Section 5, the Reputation Model is proposed. The ing reputation values to VGI participants and contributions paper concludes in Section 6 with some suggestions for a is a relevant option. These values are used to prioritize data future model extension and our research roadmap. loading into the SDW and relieve this process from unnec- essary costly processing activities. 2. MOTIVATING SCENARIOS 2.2 Recommending sigthseeing tours with Trip- 2.1 VGI in Spatial Decision Support Systems builder Spatial Decision Support Systems (SDSSs) are being cre- Tourists approaching their destination for the first time ated to allow several stakeholders to collaboratively plan deal with the problem of planning a sightseeing itinerary their actions. In such systems, Spatial Data Warehouses that covers the most subjectively interesting attractions while (SDWs) are extensively deployed as common repositories fitting the time available for their visit. TripBuilder2 is where stakeholders’ data sources are integrated and stored [5]. an unsupervised system helping tourists to build their own In order to load data into a SDW, a complex Extract-Load- personalized sightseeing tour [6]. Given the target destina- Transform (ETL) process is typically used. Because of the tion, the time available for the visit, and the tourist’s profile, growing complexity in several real-life scenarios, decision- TripBuilder recommends a time-budgeted tour that maxi- makers are relying more on advanced technologies to col- mizes tourist’s interests and takes into account both the time lect data anytime, anywhere, about events and objects of needed to enjoy the attractions and to move from one PoI to interest. As these technologies became widely available as the next one. A distinctive feature of TripBuilder is that wearable and portable devices (e.g., smartphones), decision- the knowledge base feeding the recommendation model is makers are increasingly soliciting individuals to actively re- entirely and automatically extracted from publicy available port on ongoing events. Several tools are therefore being crowdsourced data like Flickr3 and Wikipedia4 . created to enable VGI datasets to be acquired within the TripBuilder extracts from Wikipedia the multilingual context of SDSSs. The creation of such tools is motivated name of the PoI, its geographic coordinates, the categories by the important role of VGI techniques in collecting on-the- of the PoI according to a category. By clustering and spa- fly valuable data, improving the understanding of ongoing tially matching tourists’ photo albums from Flickr on the events, discovering behavioural patterns that would improve relevant PoIs extracted from Wikipedia pages, we can thus decision-making processes, and implementing proper mech- derive a knowledge base that represents the behavior of peo- anisms to provide individuals with customized services. Al- ple visiting a given city. The Wikipedia categories of the though some progress is being achieved, several obstacles PoIs visited by a given tourist are used to build her profile are still challenging the efforts of integrating VGI capabili- and to characterize the trajectories across the PoIs. ties within SDWs. For instance, the VGI datasets are com- However, Wikipedia as a source of PoIs for tourism recom- monly unstructured, with varying qualities, formats, and 2 granularities. In order to meet the DW requirements, ex- http://tripbuilder.isti.cnr.it 3 http://www.flickr.com 1 4 http://www.openannotation.org/. http://www.wikipedia.com mendations has some limitations. For example, several areas Layer of Users of the world, like latin-america or asian countries, are not Reputation Reputation score a = ... Reputation covered by a sufficient number of Wikipedia pages describ- User a User b score b = ... User c score c = ... ing individual tourist attractions inside a city like museums or monuments. This means that in these areas the PoIs are Layer of Modification Evaluations Feedback (+/-) Completion too sparse to describe itineraries and thus we need to rely Modification on other, possibly local, sources of crowd data. This calls for the integration of local-based, up-to-date, user-generated and reliable Volunteer Geographical data source capable to Layer of Versions Vers ID, Obj URI Vers ID, Obj URI Vers ID_j, Obj URI Vers ID_i, Obj URI value_j2> describe the Points of Interests, crucial for expanding Trip- value_j3> value> … value_j3> builder in these areas. The integration of these data re- … Reputation … Reputation score j = ... quires a reputation evaluation mechanism to ensure that the score i = ... user-generated data are reliable enough for recommending VGI Sources tourists. ... VGI Repository VGI Repository Reputations 3. RELATED WORK Real world objects Several research and development works have attempted to estimate reputation in VGI applications. For example, Bishr and Khun [3] described a reputation model based on Figure 2: Reference multi-layer architecture. coherence between volunteers’ reports on potability of wa- ter wells in developing countries. The potability status has only two possible values: good or bad. Time is explicitly in- 4. AN ARCHITECTURE FOR REPUTATION cluded in the model. For instance, trustworthiness in the reports about potability is reduced proportionally to the ENHANCED-VGI elapsed time since the creation time of reports. Our model In this section, we present a multi-layer architecture (see considers more general VGI scenarios allowing for complex Fig. 2) for VGI enhanced with reputation scores. This ar- descriptions of objects. Another approach is given in Zhao chitecture briefly introduces the main concepts of our model et al. [17]. The approach estimates the trustworthiness of for reputation evaluation. Main focus of this paper is the VGI data based on contributor’s reputation as well as on an- layer of evaluations and, secondly, the layer of versions. Ac- alyzing several versions of VGI data. This is similar to the cording to [17] a VGI version is the description of a state approaches proposed by Keßler et al. [13] and D’Antonio of a geospatial real world object at a certain time and au- et al. [8]. Each version is created by a contributor and de- thored by a user. For the same state more versions can be scribes the current status of a geospatial object. The level of produced, as well as the state of an object can change during trust for a specific version depends on: (i) contributor’s rep- the object lifespan (e.g., its address can change) making out utation; (ii) similarity distance between this version and the of date previous versions describing it. We model a version previous one for the same object; and (iii) level of trust in the (e.g., the description of a georeferenced PoI or a element on previous version. This approach looks actually inspired by a map, as a street) as set of pairs < attribute, value > with D’Antonio et al. [8], but additionally it provides a detailed a URI identifying the described object and can be completed data model. As per our model, these authors distinguish or modified many times. Versions are stored in VGI repos- between implicit and explicit assessment of contributors. itories, like OSM and WikiMapia. In order to improve an Trustworthiness and reputation have been studied in other existing version, a user shall modify or complete it to origi- contexts, for example crowdsourcing [1] and multi-agent sys- nate another version for the same object. In Fig. 2, versions tems [16]. In particular, the work [1] is about trustworthi- stored in the same repository and describing the status of ness of semantic annotations of textual contents. The dis- an object are grouped together. A user can also give a di- cussion is interesting because it considers as not avoidable rect feedback, positive or a negative, to versions in order to some levels of disagreement in users’ annotations for the express agreement or disagreement on the content. same text. A particular annotation is considered accept- We consider four user activities as below: A user can: (i) able when the disagreement with other annotations is not create a new version, (ii) modify an existing version, (iii) exceeding a given level. In fact, in some cases, this situation complete an existing version, or (iv) give a direct feedback reflects the presence of semantic ambiguity in the described to a version. object, which can be perceived and annotated by different A completion is a set of pairs < attribute, value > added users according to different points of view. to an existing version (e.g., adding Our approach aims to provide a comprehensive evalua- to the version). A modification consists of new values for tion model for reputation in VGI. It extends our previous pairs < attribute, value > already existing in the version work by including temporal aspects influencing reputation (e.g., the pair corrects ). Every time a user makes a modifica- distinguishes as well between direct and indirect user feed- tion (respectively, a completion) to a version A, a new ver- backs. Another benefit of our contribution is to be resistant sion A’ is generated from A by substituting the original with to manipulation attempts done by users with a malicious the modified content (respectively, by adding content). For attitude. Some preliminary experimentation provides en- each version (e.g., A, A’), its creation time is stored with the couraging results in this direction. version. A reputation value in [0, 1] is associated with each user and each version. Modifications, completions and direct feedbacks on A modify both the reputations of the version X X A and of the author of A. In Fig. 2, reputation scores of au- P OSu = P OSv · h(tv ) N EGu = N EGv · h(tv ) v∈V (u) v∈V (u) thors and versions are stored and managed by a separated platform from the VGI source repositories. (2) where V (u) is the set of contributions produced by u and v is one version. P OSc and N EGc are terms defined in the following and summarizing, respectively, positive and negative feeedbacks expressed on v. h(tv ) is a coefficient 5. REPUTATION EVALUATION MODEL weighting the contribution of P OSv and N EGv and depends In this section, we describe some requirements we adopted on the creation time tv of v. for reputation evaluation and we provide the metrics. Our purpose is to formalize an evaluation model based on our Aging of versions. multi-layer architecture by taking into account a set of re- The coefficient h() assigns a higher weight to feedbacks quirements we consider important for quantifying reputa- expressed on recent versions. Its value decreases linearly tion. According to the architecture in Fig. 2, presented in with the difference between the time t, when the P OSu and the previous section, a user can perform activities to pro- N EGu expressions are evaluated, and the creation time tv duce georeferenced content (i.e., versions) and to evaluate of version v. In particular, h(tv ) is equal to 0 when this other peers’ content. In our approach, these activities are difference is larger than value α. For example, in our ex- subject to constraints. For example, a user is not allowed periments we have chosen α equivalent to 180 days so the to give directly a feedback to another user and he cannot feedbacks on a version are no more changing the author’s evaluate more than once a version. This is set in order to reputation when the version is older than 180 days. This prohibit a malicious user from deliberately increasing repu- coefficient is defined as: tation of other specific peers or their contents. Reputation ( α−(t−tv ) of versions does not depend only on feedbacks, considered α if t − tv < α h(tc ) = (3) as direct evaluations, but also on modifications and com- 0 if t − tv ≥ α pletions, which we consider kind of indirect feedbacks. In fact, we assume that a modification reduces the reputation where h(tc ) ∈ [0, 1]. Because P OSv and N EGv are always because the user producing it considers the original version positive then P OSu and N EGu are positive as well. as including errors or requiring updates. On the contrary, a user making a completion to a version recognizes it as cor- Reputation of versions. rect, but incomplete. Therefore, a completion activity has The reputation score vRv of a version v is based on (im- to increase the reputation of the original version. plicit and explicit) feedbacks given to v by other peers. In the following, we formalize the metrics. ( User reputation. vR0 vRv = (4) Every time a feedback, completion or modification are (1 − e−k ) · P OSPvOS v +N EGv + e−k · vR0 made on a version, the User reputation score of its author is updated as follows: where vRv ∈ [0, 1] and k is the number of feedbacks on v. The first case in Eq. (4) applies when k = 0. In the second ( case, when k 6= 0, reputation score vRv is defined according uR0 to an expression similar to the one for user reputation score, uRu = (1) as in Eq. 1. (1 − e−n ) · P OSPuOS u +N EGu + e−n · uR0 The initial reputation vR0 ∈ [0, 1] depends on the type of activity (new version, modification, completion) to create v where uRu ∈ [0, 1]. The first case of Eq. (1) applies when and on the reputation of its author u, as follows: no activities have been made yet to any versions authored by user u, i.e., P OSu = 0 and N EGu = 0. The term uR0 is  uRu if vers(v) the initial user reputation, usually set to a low value (e.g.,   0.3) equal for every user. Alternatively, this value could vR0 = (1 − Sim(v, vP rev)) · uRu +   Sim(v, vP rev) · min(vR otherwise be set proportionally to the completeness in filling the user vP rev , uRu ) registration profile. In the second case of Eq. (1), param- (5) eter n is the total number of versions produced by user u. where vers(v) is true when v is a new version and, in this This includes new versions and versions produced due to case, the initial reputation vR0 is set to be equal to the rep- the modifications or completions of other versions. The role utation uRu of the author. Otherwise, vR0 is a weighted of coefficient e−n is to reduce the amplitude of the initial mean of: (i) the author reputation uRu , and (ii) the mini- oscillations of the uRu value when n is low (e.g., n < 3). mum between reputation of the parent contribution vP rev, This avoids that a single positive or negative feedback can which v is obtained from by modification or completion, and increase or decrease considerably the reputation uRu when uRu . The rationale for selecting the minimum is the follow- a user has produced only few versions. Moreover, as n in- ing. If reputation of vP rev is high and uRu is low, a user u creases the initial reputation uR0 counts less and the user could maliciously modify a version with high reputation in reputation score depends more on the feedbacks provided by order to produce a version v with some wrong content (e.g., other users. a different web site address) but having initially a high rep- Definition of terms P OSu and N EGu are as below: utation. By taking the minimum, this cannot happen since Cooperative Users Correct Identifications (U) Cooperative Users Correct Identifications (V) 90% 99.80% 90% 99.07% 70% 99.60% 70% 93.44% 50% 49.40% 50% 49.80% (a) (b) Figure 3: Identification of: (a) users, (b) versions, using the simplified model. the user reputation uRu is the upper bound of the initial Experimental evaluation. reputation of v. Sim(v, vP rev) ∈ [0, 1] is a similarity coef- We discuss some preliminary experimental results to test ficient between the previous and the current version. More our approach. Even if these experiments are limited the ob- these versions are dissimilar (i.e., Sim(v, vP rev) ∼ 0), more tained results are encouraging. We implemented the reputa- the reputation score is near to the author’s reputation score tion model as described in the previous section and tested its uRu and therefore independent of the reputation of vP rev. accuracy on a simulated dataset in a Matlab environment. This is because the version v modifies significantly vP rev In our simulation, a user is labeled either cooperative so the reputation of v should be more focused on the au- or non-cooperative and versions are labeled either correct thor than on the previous version vP rev. We will not detail (i.e., supposed to represent correctly the reality) or incor- here how to calculate Sim(v, vP rev). It is proportional to rect. During the simulation, cooperative users can: (i) cre- the number of pairs < attribute, value > equal in both the ate correct versions, and (ii) perform activities on existing versions v and vP rev. versions that are coherent with their nature; which means if a version they evaluate is correct then they can give either a Feedback evaluation. positive feedback or complete it. Non-cooperative users, in- The values of P OSv and N EGv , summarizing the posi- stead, behave according to a malicious attitude. They can: tive and negative amount of direct and indirect feedbacks (i) create incorrect versions, (ii) give either negative feed- on version v, are updated every time a new feedback on v is backs or modify correct versions, to reduce the reputations produced. The impact of a feedback by a user uf is propor- of versions and authors. Moreover, a user can be active by tional to reputation of uf . In particular, the equation for performing any type of action, or non active, by giving direct updating the current P OSv when either a positive feedback feedbacks only. or a completion f is submitted by uf with reputation uRf , By running some simulations, we measure the accuracy is the following: of the reputation model in identifying correctly, i.e., ac- cording to the labels, cooperative/non-cooperative users and correct/incorrect versions. We recall that the objective of P OSv0 = P OSv + the proposed reputation model is to assign high reputation scores to cooperative users and to correct versions, or low ( uRf if f is pos. feedback (6) ω· reputation scores otherwise. At the end of a simulation, a Sim(v 0 , v) · uRf if f is completion cooperative user is identified correctly if the reputation score is in the interval [0.5, 1]; a non-cooperative user is identified In case of completion, v 0 denotes the upgraded version of v correctly if the reputation score is in the interval [0, 0.5]. after this operation. More the versions v and v 0 are similar, We use the same criteria for identification of versions. In higher the increasing of P OSv due to the presence of the the experiment, we want to observe this phenomenon by similarity coefficient Sim(v 0 , v). The rationale is that if v 0 varying the number of cooperative users. We also disregard is very different from v, it is not considered a completely some features like the asymmetry of feedbacks (i.e., we set positive evaluation of v, i.e., v was evaluated as incomplete. ω = 0.5) and the aging of versions (i.e., we set h(tc ) = 1). In this case, P OSv0 has not to differ a lot from P OSv . The result permits to show good levels of accuracy even in We define as well the equation for updating N EGv when a simplified version of the model. All the numerical results either a negative feedback or a modification f is submitted presented in the following are obtained as averages of ten by uf with reputation uRf : running simulations. We set the percentage of active users N EG0v = N EGv + to 10%. Therefore, the remaining 90% of users are non ac- ( tive ones. Approximately, this proportion reflects the one uRf if f is neg. feedback between active users, contributing as reviewers, and users (1 − ω) · (1 − Sim(v 0 , v)) · uRf if f is modification giving only feedbacks, with reference to the Amazon commu- (7) nity5 . In our simulations, active users are selected randomly among others. The type of an activity performed during the Finally, to note that the coefficient ω ∈ [0, 1] in Eq. (6) and simulation is chosen randomly based on the probability dis- in Eq. (7) permits to make asymmetric the way P OSc and tribution: creation of a new version, 1%; modification, 1%; N EGc are increased by a certain feedback. By assigning a completion, 1%; direct feedback, 97%. The results concerning low value to ω, a negative feedback increases more N EGv the amount of users identified correctly is shown in Fig. 3(a). than a positive one does on P OSv . Here we include in the The result concerning the amount of identified versions is in model a notion of asymmetry between positive and negative Fig. 3(b). feedback as Bishr and Khun [3], who observe that negative We can notice the high identification performances with feedbacks in real life have a stronger absolute impact on rep- utation than positive ones and therefore should be weighted 5 https://www.quora.com/What-percentage-of-buyers- differently. write-reviews-on-Amazon percentages of cooperative users of 90% and 70%. The re- [7] Maria Antonia Brovelli, Marco Minghini, and Giorgio sults are even better if we consider the correct identification Zamboni. Public participation in {GIS} via mobile of only versions having at least 5 feedbacks, where with 70% applications. {ISPRS} Journal of Photogrammetry of cooperative users, we got 92.2% of correct identifications and Remote Sensing, 114:306 – 315, 2016. (not reported in the figure). On the contrary, with only [8] Fausto D’Antonio, Paolo Fogliaroni, and Tomi 50% of cooperative users the performances drop down. In Kauppinen. Vgi edit history reveals data this case, the number of cooperative and non-cooperative trustworthiness and user reputation. In 17th AGILE users are the same and every user has the same initial rep- International Conference on Geographic Information utation score uR0 , so the model is not able to identify who Science (Short Paper), 2014. is cooperative and who is not. [9] Helen Dorn, Tobias Törnros, and Alexander Zipf. Quality Evaluation of VGI Using Authoritative Data - 6. CONCLUSIONS AND PERSPECTIVES A Comparison with Land Use Data in Southern Germany. ISPRS International Journal of In this work, we discussed a novel comprehensive model Geo-Information, 4(3):1657–1671, 2015. and architecture for reputation evaluation of Volunteered [10] Michael F. Goodchild and Linna Li. Assuring the Geographic Information content. An initial evaluation based quality of volunteered geographic information. Spatial on simulations has been presented, as well. Our approach Statistics, 1:110 – 120, 2012. defines metrics to score a composite reputation of VGI data [11] Denis Havlik, Javier Soriano, Carolos Granell, coming from unknown contributors. We promote the useful- Stuart E. Middleton, Hylke van der Schaaf, Arne ness of our model by discussing the integration of VGI data Berre, and Jasmin Pielorz. Future internet enablers for with reputation scores in two different application scenarios. VGI applications. In EnviroInfo 2013. September Future work includes dealing with inconsistency that may 2013. arise due to repeated updates and completions for the same PoI. We are also planning to perform a deeper evaluation of [12] Roula Karam and Michele Melchiori. A the model, studying its possible joint use with other tech- crowdsourcing-based framework for improving niques for quality assessment in VGI data, and improving geo-spatial open data. In 2013 IEEE International the architecture and the model in order to better integrate Conference on Systems, Man, and Cybernetics, pages them with the discussed applicative scenarios. 468–473. IEEE, 2013. [13] Carsten Keßler, Johannes Trame, and Tomi Kauppinen. Tracking editing processes in volunteered 7. REFERENCES geographic information: The case of openstreetmap. [1] Lora Aroyo and Chris Welty. Crowd truth: Harnessing Identifying objects, processes and events in disagreement in crowdsourcing a relation extraction spatio-temporally distributed data (IOPE), workshop gold standard. WebSci2013. ACM, 2013, 2013. at conference on spatial information theory, 12, 2011. [2] Jamal Jokar Arsanjani, Peter Mooney, Alexander Zipf, [14] Peter Mooney, Padraig Corcoran, and Blazej Ciepluch. and Anne Schauss. Quality assessment of the The potential for using volunteered geographic contributed land use information from openstreetmap information in pervasive health computing versus authoritative datasets. In OpenStreetMap in applications. Journal of Ambient Intelligence and GIScience, pages 37–58. Springer, 2015. Humanized Computing, 4(6):731–745, 2013. [3] Mohamed Bishr and Werner Kuhn. Trust and [15] Hansi Senaratne, Amin Mobasheri, Ahmed Loai Ali, reputation models for quality assessment of human Cristina Capineri, and Muki Haklay. A review of sensor observations. In Thora Tenbrink, John Stell, volunteered geographic information quality assessment Antony Galton, and Zena Wood, editors, Spatial methods. International Journal of Geographical Information Theory: 11th International Conference, Information Science, May 2016. COSIT 2013, Scarborough, UK, September 2-6, 2013. [16] H. Yu, Z. Shen, C. Leung, C. Miao, and V. R. Lesser. Proceedings, pages 53–73. Springer International A survey of multi-agent trust management systems. Publishing, Cham, 2013. IEEE Access, 1:35–50, 2013. [4] Gloria Bordogna, Paola Carrara, Laura Criscuolo, [17] Yijiang Zhao, Xiaoguang Zhou, Guangqiang Li, and Monica Pepe, and Anna Rampini. A linguistic decision Hanfa Xing. A Spatio-Temporal VGI Model making approach to assess the quality of volunteer Considering Trust-Related Information. ISPRS geographic information for citizen science. Information International Journal of Geo-Information, 5(2):10, Sciences, 258:312 – 327, 2014. 2016. [5] Boubaker Boulekrouche, Nafaâ Jabeur, and Zaia Alimazighi. Toward integrating grid and cloud-based concepts for an enhanced deployment of spatial data warehouses in cyber-physical system applications. Journal of Ambient Intelligence and Humanized Computing, 7(4):475–487, 2016. [6] Igo Ramalho Brilhante, Jose Antonio Macedo, Franco Maria Nardini, Raffaele Perego, and Chiara Renso. On planning sightseeing tours with tripbuilder. Information Processing & Management, 51(2):1 – 15, 2015.