=Paper=
{{Paper
|id=Vol-2390/PaperB1
|storemode=property
|title=A Platform for Commonsense Knowledge Acquisition Using Crowdsourcing
|pdfUrl=https://ceur-ws.org/Vol-2390/PaperB1.pdf
|volume=Vol-2390
|authors=Christos Rodosthenous,Loizos Michael
}}
==A Platform for Commonsense Knowledge Acquisition Using Crowdsourcing==
https://ceur-ws.org/Vol-2390/PaperB1.pdf
A Platform for Commonsense Knowledge Acquisition Using Crowdsourcing
Christos T. Rodosthenous1 , Loizos Michael1,2
1
Open University of Cyprus
2
Research Center on Interactive Media, Smart Systems, and Emerging Technologies
P.O Box 12794, 2252, Nicosia, Cyprus
{christos.rodosthenous, loizos}@ouc.ac.cy
Abstract
In this article, we present our work on developing and using a crowdsourcing platform for acquiring commonsense knowledge aiming
to create machines that are able to understand stories. More specifically, we present a platform that has been used in the development
of a crowdsourcing application and two Games With A Purpose. The platform’s specifications and features are presented along
with examples of applying them in developing the aforementioned applications. The article concludes with pointers on how the
crowdsourcing platform can be utilized for language learning, referencing relevant work on developing a prototype application for a
vocabulary trainer.
Keywords: Games With A Purpose, Crowdsourcing, cloze tests, commonsense knowledge
1. Introduction selecting the missing word and finally they verify the ap-
plicability of the contributed knowledge on filling a gap in
Human computation (Law and von Ahn, 2011) or crowd- a story where similar words are present. The process is
sourcing (von Ahn and Dabbish, 2008) is applied in cases repeated using a story which contains the previously identi-
where machines are not able to perform as good as hu- fied words but with the missing word not explicitly present
mans can. In this paper, we focus on our work for develop- in the text. This application can also find use in language
ing a platform which utilizes crowdsourcing for acquiring learning, since generated cloze tests can be delivered to lan-
knowledge about our world, i.e, commonsense knowledge. guage learners, while crowdsourcing the answers.
This platform was used to develop crowdsourcing applica- In the following sections, we present the developed crowd-
tions, including Games With A Purpose (GWAPs) for ac- sourcing platform and its features, along with examples of
quiring commonsense knowledge suitable for understand- how the platform was used in real scenarios for acquir-
ing stories. More specifically, we present how the various ing commonsense knowledge. In the penultimate section,
platform features were used for the creation of two GWAPs: we present related work in using crowdsourcing and dis-
“Knowledge Coder” (Rodosthenous and Michael, 2014) cuss the differences with our approach for acquiring com-
and “Robot Trainer” (Rodosthenous and Michael, 2016) monsense knowledge. In the final section, we give an
and a crowdsourcing application for acquiring knowledge overview of our work, provide insights on future directions
that can be used in solving cloze tests, i.e., an exercise and present a relevant extension of the crowdsourcing ap-
where a word from a passage or a sentence is removed and plication in developing a vocabulary trainer for language
readers are asked to fill the gap. learning.
The two games were designed to help the acquisition of
commonsense knowledge in the form of rules. The first 2. The Crowdsourcing Platform
game implements a four-step methodology, i.e, acquiring, Following our vision for acquiring commonsense knowl-
encoding, generalizing knowledge and verifying its appli- edge using crowdsourcing, we designed a platform which
cability in other domains than the one used to generate offers features and services that can be used to facili-
it. The second game uses a hybrid methodology, where tate commonsense knowledge gathering from a number of
both human players and an automated reasoning system, paradigms, such as games, crowdsourcing tasks and mini
based on the STory comprehension through ARgumenta- applications. Most of the platform’s specifications are ap-
tion (STAR) system (Diakidoy et al., 2015), are combined plied in the majority of crowdsourcing platforms and appli-
to identify and verify the contributed knowledge. Knowl- cations and some of them are specific for the task of acquir-
edge gathered is tested on answering questions on new un- ing commonsense knowledge.
seen stories using the STAR system. Both games use a
number of ready-made gamification elements from the plat- 2.1. Platform Specifications
form to increase player contribution and interest to the task. For developing the platform, we considered the following
Furthermore, the crowdsourcing platform’s back-end inter- key design options: 1. the selection of a suitable technology
face was employed for real-time monitoring of the acquisi- for delivering task-based applications and GWAPs, 2. the
tion process and presentation of metrics and statistics in an handling of contributors’ profiles, and 3. the representation
intuitive dashboard. of knowledge in a structured form that can be reused and
For the crowdsourcing application, a three-step method- verified. The platform should also allow monitoring of the
ology was used, where contributors first find the missing acquisition process both in terms of contributors and ac-
word in a story, then they identify the words that lead to quired knowledge.
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 25
Furthermore, the platform should be able to offer a number Developers need to prepare the main functionality of their
of design elements needed in games and educational ap- system by coding it in PHP, or any other language and en-
plications. These include but are not limited to: 1. leader capsulate its executable in the platform and deliver the re-
boards, 2. contributors’ ranking, 3. medals and awards, sult using HTML, CSS and JavaScript. During this process,
4. progress-bars, 5. live feedback with notifications (both they need to prepare a list of parameters that can be used
synchronous and asynchronous) for the events, and other in the experiments and code it in XML format. These pa-
gamification elements needed to provide the user with a rameters can be incorporated in the code and control how
pleasant experience while contributing. various elements are displayed (e.g., display/hide web tour
On the back-end, the platform should be able to provide and guidance, choose what knowledge is presented for ver-
tools for designing a crowdsourcing application and man- ification, etc.).
aging contributors. These tools should provide developers The next steps involve the selection of knowledge acquisi-
the ability to easily change parameters of the application, tion tasks. Developers can select among acquisition, veri-
e.g., number of raters for acquired knowledge to be valid, fication and knowledge preference identification tasks and
dynamic loading and changing of datasets (testing and val- map the methodology steps to application screens or game
idation) and export statistics on the system usage. missions (depending on the chosen paradigm). The knowl-
We chose to develop a web-based system using the Joomla1 edge preference selection task involves the ability of a hu-
content management system (CMS) framework. The spe- man contributor to choose pieces of knowledge that are
cific CMS inherently covers a lot of the aforementioned used in a given situation and discard the ones that are not.
features in its core and it has a plethora of extensions for For example, when reading a story about birds, readers can
users to install, such as a community building component infer that birds can fly. From a similar story, where it is ex-
for creating multi-user sites with blogs, forums and social plicitly mentioned that birds are penguins, readers can infer
network connectivity. Additionally, the CMS provides a that penguins cannot fly.
very powerful component development engine, for devel- For each task, a data stream is required. The data stream
opers to deploy additional elements that can be reused in can be anything from text inserted directly from contribu-
multi-domain applications. tors, i.e, a dedicated task in the application, a pre-selected
There are many cases where crowdsourcing applications re- dataset such as Triangle-COPA (Maslan et al., 2015) or
quire functionality from other systems or knowledge bases, ROCStories (Mostafazadeh et al., 2016), or the outcome
e.g., automated reasoning engines, datasets and natural lan- of another task.
guage processing systems. For the crowdsourcing plat- Developers are free to design and code the logic behind
form we constructed an Application Programming Inter- each task as they see fit to achieve their goals. The platform
face (API) to the Web-STAR system (Rodosthenous and has a number of pre-defined functions for storing common-
Michael, 2018) for story understanding related processing sense knowledge in the form of rules or facts, both in natu-
and we offer a direct integration to the Stanford CoreNLP ral language and in a logic-based format, e.g., hug(X,Y)
(Manning et al., 2014) system. It is also able to retrieve and implies like(X,Y) where X and Y are arguments and in-
process factual knowledge, from ConceptNet (Speer et al., tuitively means if a person X hugs a person Y then person X
2016), YAGO (Suchanek et al., 2007) and WordNet (Fell- likes person Y.
baum, 2010). Developers can integrate other SPARQL- Moreover, the platform incorporates a number of visualiza-
based (Quilitz and Leser, 2008) knowledge bases since the tion libraries (e.g., d3.js2 , Cytoscape.js3 , chart.js4 ) to pro-
methodology used is generic. vide live feedback to the contributor.
The crowdsourcing platform offers a number of features for For each application, developers need to choose how con-
promoting the application to groups of users, either in so- tributed knowledge is selected and what are the criteria for
cial media or user forums. Contributors can share their con- storing this knowledge in the accepted knowledge pool.
tribution status/points/awards to social media groups. This Developers can choose among a number of strategies or
tactic can increase user retention to the application. More- a combination of them, such as selecting knowledge that
over, developers can enable the “invitations” functionality, was contributed by at least n number of persons, knowl-
where contributors gain extra points when they invite other edge that is simple (e.g., rules with at most n predicates in
people to contribute. their body), knowledge that is evaluated/rated by at least n
raters and knowledge that is evaluated by an automatic rea-
2.2. Steps for Designing a Crowdsourcing soning engine. Depending on the type of application, de-
Application Using the Platform velopers also need to choose a marking scheme that fits the
In this section, we showcase the steps needed for a de- logic behind the application and reward contributors, e.g.,
veloper to design and deploy a crowdsourcing application. points and medals for games.
These steps are also depicted in Figure 1. First, a template When the design of the various tasks is completed, the de-
must be selected to match the application domain. There veloper needs to choose how contributors will have access
are a number of templates available to match a number of to the platform (e.g., anonymously, through registration or
crowdsourcing paradigms (e.g., GWAPs, language learn- social networks) and what details need to be filled in their
ing applications) which can be customized according to the profiles.
specific needs of the task. 2
https://d3js.org/
3
1 http://js.cytoscape.org/
https://www.joomla.org/ 4
https://www.chartjs.org/
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 26
Figure 2: In this figure, a screenshot of a data visualization
Figure 1: The architectural diagram of the Crowdsourcing diagram is depicted where readers can follow the data flow
platform, presenting the main components of the platform in the system for both players (top stream) and common-
and the data flow. sense knowledge (bottom stream).
2.3. Technological Infrastructure by novice users in conjunction with using a text-based edi-
In terms of technological infrastructure, the platform re- tor for the same task.
lies on a web-server with Linux-Apache-MariaDB-PHP
(LAMP) stack and on the Joomla framework. The platform 3. An Example of Developing a
also utilizes the JQuery5 and the bootstrap frameworks both Crowdsourcing Application
for designing elements and for application functionality.
The platform employs the Joomla Model-View-Controller In its current state, the platform was used to develop two
(MVC)6 framework that allows the development of compo- GWAPs and a crowdsourcing application. There is an ex-
nents by separating the data manipulation functions from tensive presentation of the two GWAPs in our previous
the view controls. The controller is responsible for exam- work (Rodosthenous and Michael, 2014; Rodosthenous
ining the request and determining which processes will be and Michael, 2016) and readers are directed there to learn
needed to satisfy the request and which view should be used more about the design, the various elements employed
to return the results back to the user. This architecture al- and the experiments performed to acquire commonsense
lows the usage of both internal (e.g., database) and external knowledge.
data sources (e.g., APIs, files) and of course deliver these In this section, we focus on how the platform was used for
services in an abstraction layer that can be used by other the task of acquiring knowledge in the form of natural lan-
applications. guage rules for solving cloze tests. For running this task,
For user authentication, both the Joomla internal mecha- first we retrieved stories from the ROCStories dataset in a
nisms are used and the Oauth7 authentication methods that tabular format and loaded them in the platform’s database
permit the seamless integration of social network authenti- table. Then we parsed each story sentence through the
cation with the platform. CoreNLP system and got the Parts-Of-Speech (POS) for
each word and its base form (lemma). The aforementioned,
2.4. Data Visualization were stored in a database table. For each story, a noun word
It is important for application developers to be able to visu- was obscured and more than 1000 cloze tests were created.
alize acquired knowledge for better understanding what and For each test at least 5 possible answers were generated and
how users behaved during the crowdsourcing experiment. stored, including synonyms and antonyms retrieved from
In Figure 2 an example of a Sankey type graph is presented Wordnet. This workflow was developed in the back-end by
for the Robot Trainer game where results for both the con- reusing components from the two GWAPs and by adding
tributors and the acquired knowledge are depicted on the new functionality specifically used for that workflow.
same diagram. This type of functionality is possible by us- The task was separated in three subtasks and for the front-
ing the D3.JS library with data feed from the database and end design, each one of these tasks is presented on a sep-
a graph theory (network) library for visualization and anal- arate screen (see Figure 3). Each screen comprises an in-
ysis called Cytoscape.js. The latter was also used for repre- struction area on top, the active task area below that on the
senting and contributing commonsense knowledge rules in left and the visual representation area on the right. The vi-
a graphical manner in WebStar and was evaluated positively sual representation area is dynamically updated with every
user action. Directly below these two are the task controls.
5
https://jquery.com/ This template, based on bootstrap, was chosen for its sim-
6
https://docs.joomla.org/ plicity, since we wanted to avoid users paying attention to
Model-View-Controller unnecessary elements.
7
https://oauth.net/2/ To start contributing, users need to create an account using
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 27
either the registration form or one of the social media con- ing Commonsense knowledge facts, Common Consensus
nected account methods inherently present in the platform. (Lieberman et al., 2007), i.e., a GWAP for gathering com-
After entering their credentials, contributors are firstly pre- monsense goals, GECKA (Cambria et al., 2015), i.e., a
sented with a test (see Figure 2a) which they solve and state game engine for commonsense knowledge acquisition, the
how confident they are in solving it, in a scale of 0 to 100%. Concept Game (Herdagdelen and Baroni, 2010), i.e, a
Secondly, the contributors are asked to highlight the words GWAP for verifying commonsense knowledge assertions,
in the text that helped them decide the missing word (see the FACTory Game (Lenat and Guha, 1990) where players
Figure 2b), and thirdly, they are presented with a new test are asked to verify facts from Cyc, the Virtual Pet and the
where both the correct answer selected in the 1st step and Rapport games (Kuo et al., 2009) for commonsense data
the highlighted words selected in the 2nd step are present collection and many other.
(see Figure 2c). The contributor is asked to verify if the There are also approaches where contributors are moti-
highlighted words are used to find the missing word. Fi- vated by money such as the Amazon Mechanical Turk
nally, a new test appears which includes the highlighted (Buhrmester et al., 2011) and Figure Eight8 and others,
words from the 2nd step but not the selected missing word where motivation is geared towards contributing to science
from the 1st step. Contributors are asked if the missing or other noble causes. These approaches rely on citizen sci-
word is implied in the story. Each contributor is also pre- ence frameworks for crafting crowdsourcing tasks, such as
sented with a task to verify if the chosen words selected by Pybossa9 and psiTurk10 .
another contributor are useful for solving the cloze test (see The aforementioned systems and games are very interest-
Figure 2d). ing and provide a lot of features, but their design is focused
Each test is retrieved randomly from the database and for on targeting a single task, rather than a series of tasks that
the verification task, tests are selected randomly at first, and are chained. Furthermore, the majority of systems is lim-
by prioritizing selection of tests that have at least one con- ited to the templates and standard workflow processes of-
tribution. That way, we give priority to verifying contribu- fered in order to accommodate the most common and most
tions. This is set before running the experiment in the pa- popular crowdsourcing tasks. The task of commonsense
rameters screen on the back-end. All user contributions are knowledge acquisition is more complex and requires more
recorded and stored in a database table recording both the complex workflows to be used, e.g, contribution and then
task data (e.g., missing word, highlighted words, verifica- verification.
tion response) and metadata (e.g., response time). Record- There are cases where crowdsourcing only is not the best
ing is possible using the JQuery AJAX libraries and APIs, option for acquiring a specific type of knowledge and hy-
which allow dynamic update of the content without refresh- brid solutions, i.e., solutions that employ both human con-
ing the browser webpage and make the contributor to loose tributors and machine processing, should be used towards
focus on the task. that direction. Such an example is the acquisition of com-
Through these tasks, we are able to acquire knowledge both monsense knowledge in the form of rules, where we com-
for cases where the word is explicitly stated in the text and pared a pure crowdsourcing approach (“Knowledge Coder”
for cases that it is implied. The crowdsourcing applica- game) with a hybrid one (“Robot Trainer” game). The re-
tion was tested with a small crowd and initial experiments sults suggest, that the hybrid approach is more appropriate
showed that acquired rules can be used both for solving for gathering general commonsense knowledge rules, that
cloze tests and for generating inferences from a story. For can be used for question-answering. This is one of the rea-
example the following two rules were generated and veri- sons we chose to develop a custom made platform in order
fied on unseen stories: to have more flexibility in developing such tasks. Ready-
made templates offered by the mainstream platforms can-
• when words (or their lemmas) friends and high not give you this flexibility, since they aim in a broader set
exist in a story then the missing word is probably of experimenters. Of course, this comes at the cost that
school some development should be made from the experimenter.
• when words (or their lemmas) player and scored The crowdsourcing platform has internal mechanisms for
exist in a story then the missing word is probably knowledge representation in the form of rules, which can
team be reused in many different applications that serve a similar
purpose. Using one of the mainstream platforms requires
4. Related Work handling the knowledge rule representation using external
Currently, there are many attempts to harness the power of tools that need to be developed beforehand. The crowd-
the crowd for several tasks such as image tagging, knowl- sourcing platform also engulfs natural language processing
edge gathering, text recognition, etc. The motives for peo- tools for treating datasets, before requesting crowd workers
ple contributing, are categorized between intrinsic and ex- to process them. There are also modules for direct inte-
trinsic (Kaufmann et al., 2011). Intrinsic motivation in- gration with knowledge bases (e.g., ConceptNet, YAGO)
cludes enjoyment and community based contributions and that can be used in conjunction with the crowd tasks, either
extrinsic includes immediate and delayed payoffs and so- for knowledge verification or to reduce the ambiguities in
cial motivation. language. The aforementioned features cannot be found in
For the purpose of acquiring commonsense knowledge 8
https://www.figure-eight.com/
there are examples of games and frameworks such as Ver- 9
https://pybossa.com/
bosity (von Ahn et al., 2006), i.e., a GWAP for Collect- 10
https://psiturk.org/
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 28
(a) Screenshot of the 1st step where contributors first select a (b) Screenshot of the 2nd step where contributors highlight the
word to fill the gap from one of the possible answers. words that led in selecting the missing word.
(c) Screenshot of the 3rd step where contributors verify if the (d) Screenshot of the verification step where contributors verify
highlighted words from the 2nd step can be used to identify the other contributors highlighted words used for solving a cloze test.
same missing word with that of the 1st step on a new cloze test.
Figure 3: Screenshots of the online environment for gathering knowledge and delivering cloze tests. The three steps of the
process are depicted followed by the verification step.
platforms such as PYBOSSA or psiTurk which concentrate symbolic rule hit(X,Y) IMPLIES angry(X) mean-
on designing crowdsourcing experiments nor in GECKA ing that a person X hits a person Y implies that person X
which is focused in designing GWAPs. is angry. Through the game, 1501 commonsense knowl-
edge rule evaluations were gathered and the interesting part
5. Discussion and Future Work is that players added a “Positive” evaluation to simple rules
instead of more complex ones.
In this article we presented an overview of the crowdsourc-
ing platform developed to facilitate the development of We are currently investigating how this work can be used
crowdsourcing applications and GWAPs focused on acquir- in the context of language learning by using commonsense
ing commonsense knowledge. Examples of how the plat- knowledge databases for creating exercises such as cloze
form was used to acquire commonsense knowledge were tests, “find synonyms, antonyms, etc.” and delivering them
depicted along with how the various platform elements to students. The platform can be used to deliver vocab-
were used to achieve the goal of the application. ulary exercises, generated from commonsense knowledge
The key features of the crowdsourcing platform include the databases and ontologies such as ConceptNet. The re-
ability to design complex workflows for acquiring com- sponses can be used to expand the knowledge bases that
monsense knowledge, a storage and handling mechanism the exercises originated from. A prototype implementation
for acquired knowledge and numerous tools for dataset of this (Rodosthenous et al., 2019), was developed during
processing and integration with large semantic knowledge the CrowFest organized by the European Network for Com-
bases and reasoning engines. Moreover the platform offers bining Language Learning with Crowdsourcing Techniques
a wide range of visualizations and analytics to the experi- (EnetCollect) COST Action (Lyding et al., 2018).
menters that can be customized to facilitate the monitoring The crowdsourcing platform can also be used on our re-
and reporting needed during crowd experiments. search for identifying the geographic focus of a story. We
In terms of results, from the first GWAP we implemented, have developed a system called GeoMantis (Rodosthenous
i.e., “Knowledge Coder” game, we gathered 93 knowledge and Michael, 2019) that reads a story and returns the pos-
rules from 5 contributors. These rules were too specific on sible countries of focus for that story. GeoMantis uses
the story that was used to generate them and did not of- commonsense knowledge from ConceptNet and YAGO to
fer any value for understanding or answering questions on perform this task. We plan to launch a crowdsourcing
other stories. When the crowdsourcing platform was used task where users will be presented with knowledge about a
for the “Robot Trainer” game we were able to recruit 800 country, e.g., parthenon atLocation Greece and
players from Facebook and some popular game forums in will be asked to evaluate if it is a good argument to identify
a period of 153 days. These players contributed 1847 com- the geographic focus of a story to that specific country, aim-
monsense knowledge rules (893 unique). Contributed rules ing to add weights on each argument and test if the system
were general enough to be used in other domains, e.g., the yields better results.
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 29
6. Bibliographical References tra, D., Vanderwende, L., Kohli, P., and Allen, J. (2016).
A Corpus and Evaluation Framework for Deeper Under-
Buhrmester, M., Kwang, T., and Gosling, S. D. (2011).
standing of Commonsense Stories. In Proceedings of
Amazon’s Mechanical Turk. Perspectives on Psycholog-
the 2016 North American Chapter of the ACL (NAACL
ical Science, 6(1):3–5.
HLT).
Cambria, E., Rajagopal, D., Kwok, K., and Sepulveda,
Quilitz, B. and Leser, U., (2008). Querying Distributed
J. (2015). GECKA: Game Engine for Commonsense
RDF Data Sources with SPARQL, pages 524–538.
Knowledge Acquisition. In Proceedings of the 28th In-
Springer Berlin Heidelberg, Berlin, Heidelberg.
ternational Flairs Conference, pages 282–287.
Rodosthenous, C. T. and Michael, L. (2014). Gath-
Diakidoy, I.-A., Kakas, A., Michael, L., and Miller, R.
ering Background Knowledge for Story Understand-
(2015). STAR: A System of Argumentation for Story
ing through Crowdsourcing. In Proceedings of the 5th
Comprehension and Beyond. In Working Notes of the
Workshop on Computational Models of Narrative (CMN
12th International Symposium on Logical Formaliza-
2014), volume 41, pages 154–163, Quebec, Canada.
tions of Commonsense Reasoning (Commonsense 2015),
Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik.
pages 64–70.
Rodosthenous, C. and Michael, L. (2016). A Hybrid Ap-
Fellbaum, C. (2010). Theory and Applications of On-
proach to Commonsense Knowledge Acquisition. In
tology: Computer Applications. Springer Netherlands,
Proceedings of the 8th European Starting AI Researcher
Dordrecht.
Symposium, pages 111–122.
Herdagdelen, A. and Baroni, M. (2010). The Concept
Rodosthenous, C. T. and Michael, L. (2018). Web-STAR:
Game: Better Commonsense Knowledge Extraction by
A Visual Web-based IDE for a Story Comprehension
Combining Text Mining and a Game with a Purpose.
System. Theory and Practice of Logic Programming,
AAAI Fall Symposium on Commonsense Knowledge, Ar-
pages 1–43.
lington, (2006):52–57.
Rodosthenous, C. and Michael, L. (2019). Using Generic
Kaufmann, N., Schulze, T., and Veit, D. (2011). More than Ontologies to Infer the Geographic Focus of Text. In
fun and money. Worker Motivation in Crowdsourcing-A Jaap van den Herik et al., editors, Agents and Artificial
Study on Mechanical Turk. In AMCIS, volume 11, pages Intelligence, pages 223–246, Cham. Springer Interna-
1–11. tional Publishing.
Kuo, Y., Lee, J., Chiang, K., and Wang, R. (2009). Rodosthenous, C., Lyding, V., König, A., Horbacauskiene,
Community-based Game Design: Experiments on Social J., Katinskaia, A., ul Hassan, U., Isaak, N., Sangati,
Games for Commonsense Data Collection. In Proceed- F., and Nicolas, L. (2019). Designing a Prototype Ar-
ings of the 1st ACM SIGKDD Workshop on Human Com- chitecture for Crowdsourcing Language Resources. In
putation (HCOMP 2009), pages 15–22, Paris, France. Proceedings of the 2nd Language, Data and Knowledge
Law, E. and von Ahn, L. (2011). Human Computation. (LDK) Conference (to appear). CEUR-WS.
Morgan & Claypool Publishers, 1st edition. Speer, R., Chin, J., and Havasi, C. (2016). ConceptNet 5.5:
Lenat, D. B. and Guha, R. V. (1990). Building Large An Open Multilingual Graph of General Knowledge. In
Knowledge-based Systems: Representation and Infer- Proceedings of the 31st AAAI Conference on Artificial
ence in the Cyc Project. Addison-Wesley Longman Pub- Intelligence, pages 4444–4451.
lishing Co., Inc., Boston, MA, USA, 1st edition. Suchanek, F. M., Kasneci, G., and Weikum, G. (2007).
Lieberman, H., Smith, D. A., and Teeters, A. (2007). Com- Yago: A Core of Semantic Knowledge. In Proceedings
mon Consensus: A Web-Based Game for Collecting of the 16th International Conference on World Wide Web,
Commonsense Goals. In Proceedings of the Workshop pages 697–706.
on Common Sense and Intelligent User Interfaces, Hon- von Ahn, L. and Dabbish, L. (2008). Designing Games
olulu, Hawaii, USA. With a Purpose. Communications of the ACM, 51(8):57.
Lyding, V., Nicolas, L., Bédi, B., and Fort, K., (2018). In- von Ahn, L., Kedia, M., and Blum, M. (2006). Ver-
troducing the European NETwork for COmbining Lan- bosity: A Game for Collecting Common-Sense Facts.
guage LEarning and Crowdsourcing Techniques (enet- In Proceedings of the 25th SIGCHI Conference on Hu-
Collect), pages 176–181. Research-publishing.net. man Factors in Computing Systems (CHI 2006), page 75,
Manning, C. D., Bauer, J., Finkel, J., Bethard, S. J., Sur- Montréal, Québec. ACM.
deanu, M., and McClosky, D. (2014). The Stanford
CoreNLP Natural Language Processing Toolkit. In Pro-
ceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics: System Demonstrations,
pages 55–60.
Maslan, N., Roemmele, M., and Gordon, A. S. (2015).
One Hundred Challenge Problems for Logical Formal-
izations of Commonsense Psychology. In Proceedings of
the 12th International Symposium on Logical Formaliza-
tions of Commonsense Reasoning, Stanford, California,
USA.
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Ba-
EnetCollect WG3&WG5 Meeting, 24-25 October 2018, Leiden, Netherlands 30