The tools of citizen science: An evaluation of map-based crowdsourcing platforms

1. Introduction

There seems to be a persistent yet inaccurate sentiment that collecting vast amounts of data via citizen science is virtually free, especially compared to the cost of privatized scientific endeavors (Bonney et al., 2009; Cooper, Hochachka & Dhondt, 2011) . However, performing scientific procedures with the assistance of the public is often far more complex than traditional scientific enquiry (Bonter & Cooper, 2012) .

Citizen science promotes the participation of the public in scientific endeavors (Hecker et al., 2018) . While citizen science is not synonymous with volunteered geographic information (VGI)— broadly defined as the creation of geographic information by citizens (Goodchild, 2007) —it often produces geographic information. Similar to VGI, citizen science projects tend to follow specific protocols to ensure the crowdsourced geographic data serves as an input for (scientific) research (Haklay, 2013) . Also similar to VGI, citizen science projects often require software applications and specialized training to facilitate citizen data collection. Notably, citizen science projects are increasingly requiring a webbased participatory mapping platform—i.e., Geoweb (Leszczynski & Wilson, 2013) —to coordinate the proliferation of citizen contributions.

Many scientists, researchers, and especially geographers, have developed their own platforms to facilitate the capture, management, and sharing of citizen science contributions. Notable examples include Geolive (Corbett & Cochrane, 2017) , GeoKey (Roick, Haklay, & Ellul, 2016) , and Argumap (now Cartoforum) (Rinner, Kumari, & Mavedati, 2011; Sani & Rinner, 2011) . These platforms “provide an online space for collective place-based knowledge, experience and wisdom to be captured, shared, exchanged, contested and negotiated” (Corbett & Cochrane, 2017, p.2) . However, these map-based data platforms are often developed with significant financial expense, time investment and risk tolerance (Bonney et al., 2009) . It is unrealistic to assume that the developers of every geographic citizen science project will have the time, capital, and technical skill necessary to develop their own platform.

As an alternative to in-house developed tools, there are several commercial and open-source online mapping platforms dedicated to crowdsourcing data that could be useful to individuals starting citizen science projects. These tools have been developed with two key features: a front-end user interface, and a “sign up and go” back-end to enable the creation of a project or deployment by anyone. No app development or programming is required for the person initiating a project.

In this research, we investigate publicly available commercial and opensource map-based tools that enable citizen science projects. Building on a comprehensive comparative framework, we conduct a systematic evaluation and overview of five map-based crowdsourcing platforms: Ushahidi, Maptionnaire, Survey123 (ArcGIS Online), Open Data Kit, and GIS Cloud. These tools have additional uses that extend beyond the field of citizen science; however, the scope of the investigation was narrowed to focus on aspects most suitable for citizen science endeavors, such as the collection, management, visualization and dissemination of crowdsourced data. It is our intention to provide information on how these publicly available crowdsourcing platforms suit generic geographic citizen science crowdsourcing needs.

2. Methods and Data

2.1 Platform Selection

Three of the selected platforms (Maptionnaire, Survey123 and GIS Cloud) are commercial tools, while Open Data Kit is an open-source application. Ushahidi is commercially available through their website, however it can also be found as an open-source application on GitHub. The commercial version of Ushahidi hosted on the web was explored in this investigation. Our five platforms were chosen based on meeting the following conditions: the platform enabled crowdsourced contributions, has a map-based interface, is web-ready (i.e., no server install required), and has some free functionality.

Citizen science has been performed by birders for over a century (Silvertown, 2009) , so we chose to run a mock ornithology project to streamline the comparison, ensuring each individual platform evaluation was consistent and impartial. However, all the functionality from each platform was fully explored; the thoroughness of the comparison was not limited by the mock ornithology project.

For each platform, we signed up and started a project called “Calgary Ornithology Survey”. Once the back-end was set up, we explored the site’s data collection tools (web surveys and mobile applications). The final step was submitting the ornithology dataset

Robin

Robin

Robin

Eagle

Eagle

Chickadee

Chickadee

Chickadee

Chickadee

Blue Jay

Blue Jay

Blue Jay

Blue Jay

King Fisher

King Fisher

Parkland

Southcentre

Chinook Centre

Trico Centre for

Family Wellness

University of

Calgary

Rocky Ridge YMCA

Saddledome

Calgary Tower

Southern Alberta

Institute of

Technology

Mount Royal

University

Southland Leisure

Centre

Market Mall

Fairview Arena

Cranston Sobeys

YYC Airport

2.2 Evaluation Metric

These investigations followed a specific evaluation metric that tested the same features and functionality of each platform. This evaluation metric was based on a similar one established by Fast and Hossain (under review) in their evaluation of WebGIS platforms and includes the categories introduced in Table 2.

Citizen science terminology has been a topic of debate in recent years, with different groups of individuals deriving different meanings from the same terms (Eitzel et al., 2017) . This study considers users and participants as non-professionals participating in a citizen science program. Authors and administrators are defined as from the public interface (the front-end). The fifteen entries consisted of four types of birds (Table 1). the individuals that create and operate the citizen science project.

Options to import pre-existing datasets.

The ability to edit the collected data.

The ability to export the collected data.

Specific tools that allow the author to buffer,

clip and aggregate, amongst others.

How the data is displayed on the platform.

How a map can be shared and made publicly available.

The number of available unique base maps.

Default base map options Symbology and labelling Map customization options. options

Widget options

Availability of customizable software applications.

Incurred costs of operation and software downloads.

Paid options/Upgraded versions

3. Results

The results of the evaluation of the crowdsourcing platforms are organized by data input, management, analysis, visualization, and associated costs. The components are summarized in Table 3.

3.1 Data Input

Each platform collects data via surveys, and only Ushahidi offers data collection via SMS and email. The data input types differ for each platform and Maptionnaire is the only platform that does not offer a date or time field. This is a significant downside to the platform since the date and time fields could be crucial to a project.

Ushahidi data can be collected via the web and Android devices. Maptionnaire is entirely web-based while Open Data Kit and GIS Cloud are based on mobile applications for data collection (although GIS Cloud apparently does offer web data collection for their true crowdsourcing services). The Open Data Kit Collector application is available for download solely on Android devices, which severely limits public access to the survey and its data. Survey123 is the only platform to offer web surveys and mobile applications that support both Android and iOS devices.

3.2 Data Management

Survey123 and GIS Cloud offer builtin databases, the others do not. Every platform save for Open Data Kit allows data imports (paid option in Ushahidi), with each platform supporting different file formats. Data edit and delete options are available in each platform, however in Maptionnaire Table 3: Summary of the options provided by each platform. only the editing of single data points is free. Survey123 is the only platform that Evaluation Criteria Ushahidi

Maptionnaire

Survey 123

Open Data Kit

GIS Cloud supports the removal of single data entries, mass deletions, sorting and filtering, and the editing of single entries. Ushahidi, Open Data Kit and GIS Cloud have no mass delete option, and Open Data Kit is the only platform that does not allow submissions to be edited. The data export options are limited. The largest number of supported formats is five, available in Survey123 and GIS Cloud.

3.3 Data Analysis

Data analysis options are available in Maptionnaire, Survey123 and GIS Cloud, but are essentially all paid-for services. Maptionnaire does not allow access to the survey data without a payment plan, Survey123 is only accessible with an ESRI organizational account, and the premium trial of GIS Cloud’s Map Editor expires after 31 days. Therefore, if the author wishes to perform any significant analysis on their data they will likely have to pay. Ushahidi and Open Data Kit do not offer geoprocessing tools.

3.4 Data Visualization

Survey123 offers the most data visualization options, while Maptionnaire has no options without payment. Open Data Kit offers lists, graphs and a map view (however the map view is watermarked). Survey 123 (ArcGIS online) and GIS Cloud have the most symbology options; the author can make new feature layers, which allows each feature to be represented by its own symbol. The option to customize symbols is available in Maptionnaire, Survey123 and GIS Cloud. GIS Cloud offers labeling options as well.

3.5 Associated Costs

Ushahidi, Maptionnaire and GIS Cloud require payment for full access to all their features, although Ushahidi is free for some endeavors upon discretion of the platform. Survey123 is not free and requires an ESRI organizational account with publisher capabilities, restricting its accessibility to citizen science projects. Open Data Kit is advantageous for low or no budget projects since it has no incurred costs or upgraded versions requiring a payment plan. However, Open Data Kit is the only platform that requires a software download. This may restrict its use if an author does not have admin privileges on the computer they start their project on. 4. Discussion and Conclusion

We evaluated and compared five map-based crowdsourcing platforms based on their ability to facilitate geographic citizen science endeavors. It is difficult to highlight a single platform as the best option, since citizen science projects are not equally funded and are operated by people of varying technological expertise. Most platforms do have incurred costs of operation, with Survey123 being the most functional but also expensive option. Ushahidi, Maptionnaire, GIS Cloud and Open Data Kit offer alternatives to investing time and capital into developing an in-house platform. In particular, the Open Data Kit open-source software provides a free-ofcharge option for projects, however its usage is not as intuitive and user-friendly as the commercial and paid alternatives. While these platforms do not offer much public engagement beyond the crowdsourcing of data, they are viable options for the collection, management, and visualization of citizen generated data.

Citizen science is helping to break barriers between scientific processes and the public; to make science and scientific enquiry openly available. The tools of citizen science, specifically web-based participatory mapping platforms, are enabling the public to become more involved, not just in the collection of scientific data, but in the planning and execution of data collection. The platforms, discussed here and the others like them, are excellent options for those lacking the capital and technical knowledge to build a data collection, management and visualization platform from the ground up. These crowdsourcing platforms make citizen science project planning more accessible to the masses by eliminating the advanced technical skills required to develop a map-based collection program. These crowdsourcing platforms facilitate the front-end and back-end of data collection allowing individuals to focus on their data and analyze and share their results.

Overall, these platforms make citizen science more accessible to nonprofessionals by eliminating the need for the advanced technical skills and capital necessary to develop a map-based crowdsourcing platform. We anticipate a growing market for these platforms as niche “amateur” citizen science projects grow.

References

Bonney , R. , Cooper , C.B. , Dickinson , J. , Kelling , S. , Phillips , T. , Rosenberg , K.V. , & Shirk , J. ( 2009 ). Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy . BioScience , 59 ( 11 ), 977 - 984 .

Bonter , D.N. , & Cooper , C.B. ( 2012 ). Data validation in citizen science: a case study from Project FeederWatch . Frontiers in Ecology and the Environment , 10 ( 6 ), 305 - 307 .

Cooper , C.B. , Hochachka , W.M. , & Dhondt , A.A. ( 2011 ). The opportunities and challenges of citizen science as a tool for ecological research . In Dickinson, J.L. , Bonney , R. , (Eds.), Citizen science: public collaboration in environmental research (pp. 99 - 113 ). New York, New York: Cornell University Press.

Corbett , J.M. , & Cochrane , L. ( 2017 ). Engaging with the Participatory Geoweb: Experiential Learning from Practice . In Campelo, C.E.C. , Bertolotto , M. , Corcoran , P. (Eds.) Volunteered Geographic Information and the Future of Geospatial Data , (pp. 1 - 18 ). IGI Global.

Eitzel , M.V. , Cappadonna , J.L. , SantosLang , C. , Duerr , R.E. , West , S.E. , Kyba , C.C.M. , … Jiang , Q. ( 2017 ). Citizen science terminology matters: Exploring key terms . Citizen Science: Theory and Practice . 2 ( 1 ), 1 , 1 - 20 .

Goodchild , M.F. ( 2007 ). Citizens as sensors: the world of volunteered geography . Geojournal , 69 ( 4 ), 211 - 221 .

Haklay , M. ( 2013 ). Citizen science and volunteered geographic information: overview and typology of participation . In: Sui, D.Z. , Elwood , S. , & Goodchild , M.F . (Eds.) Crowdsourcing geographic knowledge: volunteered geographic information (VGI) in theory and practice , (pp. 105 - 122 ). Dordrecht: Springer.

Hecker , S. , Haklay , M. , Bowser , A. , Makuch , Z. , Vogel , J. , & Bonn , A. ( 2018 ). Innovation in open science, society and policy - setting the agenda for citizen science . In Hecker, S., Haklay , M. , Bowser , A. , Makuch , Z. , Vogel , J. , & Bonn , A . (Eds.) Citizen Science: Innovation in Open Science, Society and Policy, (pp. 1 - 23 ). London: UCL Press.

Leszczynski , A. , & Wilson, M.W. ( 2013 ). Guest editorial: theorizing the geoweb . GeoJournal , 78 , 915 - 919 . doi: 10 .1007/s10708-013-9489-7

Rinner , C. , Kumari , J. , & Mavedati , S. ( 2011 ) A geospatial web application to map observations and opinions in environmental planning . In Li, S. , Dragicevic , S. , & Veenendall , B . (Eds.) Advances in Web-based GIS , Mapping Services and Applications , (pp. 277 - 291 ), London, England: Taylor and Francis.

Roick , O. , Haklay , M. , & Ellul , C. ( 2016 ). GeoKey - open infrastructure for community mapping and science . Hum Comput , 3 , 143 - 159 . doi: 10 .15346/hc.v3i1. 8

Sani , A.P. , & Rinner , C. ( 2011 ). A Scalable Geoweb Tool for Argumentation Mapping . Geomatica, 65 , 145 - 156 .

Silvertown , J. ( 2009 ). A new dawn for citizen science . Trends in ecology & evolution, 24 ( 9 ), 467 - 471 . doi: https://doi.org/10.1016/j.tree. 2009 . 03 .017