NeurIPS 2020 Crowd Science Workshop:
Remoteness, Fairness, and Mechanisms as Challenges
    of Data Supply by Humans for Automation


                       Dmitry Ustalov                              Fabio Casati∗
                           Yandex                                   ServiceNow
                   Saint Petersburg, Russia                    Santa Clara, CA, USA
                 dustalov@yandex-team.ru                     fabio.casati@unitn.it

                     Alexey Drutsa                           Daria Baidakova
                        Yandex                                   Yandex
                    Moscow, Russia                           Moscow, Russia
                  adrutsa@yandex.ru                    dbaidakova@yandex-team.ru




Preface
The first workshop on Crowd Science was organized in conjunction with the 34th Conference on
Neural Information Processing Systems (NeurIPS 2020). This workshop is the first in a series of
events that has the goal of helping crowdsourcing “transition” from art to science, and tackles the
research challenges that we face to make crowdsourcing a technology that users can easily access and
leverage, and produces results that researchers and businesses can rely on.
This workshop features five invited talks, seven paper presentations, and a panel discussion. We
received 13 submissions, out of which 7 were accepted as the talks at the workshop after peer review.
Only 6 papers were included in the proceedings as the authors of one accepted paper expressed
their interest in non-archival talk. We thank all the authors for their contributions and the effort they
put into it, and we are very grateful to the excellent work of our reviewers and program committee
members.
Last, but not least, we would like to thank NeurIPS workshop chairs for their assistance that made the
workshop organization process a pleasure. The 2020 edition of the workshop took place remotely,
and we look forward to the forthcoming (in-person) editions of the Crowd Science workshop.

December 11, 2020                                                                      Dmitry Ustalov
Vancouver, BC, Canada (Online)                                                            Fabio Casati
                                                                                        Alexey Drutsa
                                                                                      Daria Baidakova




   ∗
       On leave from University of Trento, Italy.


NeurIPS 2020 Crowd Science Workshop: Remoteness, Fairness, and Mechanisms as Challenges of Data Supply
by Humans for Automation. Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Table of Contents
Conversational Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben
What Can Crowd Computing Do for the Next Generation of AI Systems? . . . . . . . . . . . . . . . . . . . . . . 7
Ujwal Gadiraju and Jie Yang
Real-Time Crowdsourcing of Health Data in a Low-Income Country: A Case Study of Human Data
Supply on Malaria First-Line Treatment Policy Tracking in Nigeria . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo
Active Learning from Crowd in Document Screening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik
Human Computation Requires and Enables a New Approach to Ethics . . . . . . . . . . . . . . . . . . . . . . . . 26
Libuše H. Vepřek, Patricia Seymour and Pietro Michelucci
A Gamified Crowdsourcing Framework for Data-Driven Co-Creation of Policy Making and Social
Foresight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Andrea Tocchetti and Marco Brambilla
Program Committee

Marcos Baez          University of Trento
Boualem Benatallah   University of New South Wales
Alessandro Bozzon    Delft University of Technology
Alessandro Checco    University of Sheffield
Anna Lisa Gentile    IBM
Gleb Gusev           Sberbank
Evgeny Krivosheev    University of Trento
Alexey Kushnir       Carnegie Mellon University
Anna Lioznova        Yandex
Lucas Maystre        Spotify
Svetlana Nikitina    University of Trento
Maria Sagaidak       Yandex
Ivan Stelmakh        Carnegie Mellon University
Jie Yang             Delft University of Technology
Fedor Zhdanov        Amazon
Xiong Zhou           Amazon