In this article, we present our work on developing and using a crowdsourcing platform for acquiring commonsense knowledge aiming to create machines that are able to understand stories. More specifically, we present a platform that has been used in the development of a crowdsourcing application and two Games With A Purpose. The platform's specifications and features are presented along with examples of applying them in developing the aforementioned applications. The article concludes with pointers on how the crowdsourcing platform can be utilized for language learning, referencing relevant work on developing a prototype application for a vocabulary trainer.
Human computation
The two games were designed to help the acquisition of
commonsense knowledge in the form of rules. The first
game implements a four-step methodology, i.e, acquiring,
encoding, generalizing knowledge and verifying its
applicability in other domains than the one used to generate
it. The second game uses a hybrid methodology, where
both human players and an automated reasoning system,
based on the STory comprehension through
ARgumentation (STAR) system
For the crowdsourcing application, a three-step methodology was used, where contributors first find the missing word in a story, then they identify the words that lead to selecting the missing word and finally they verify the applicability of the contributed knowledge on filling a gap in a story where similar words are present. The process is repeated using a story which contains the previously identified words but with the missing word not explicitly present in the text. This application can also find use in language learning, since generated cloze tests can be delivered to language learners, while crowdsourcing the answers. In the following sections, we present the developed crowdsourcing platform and its features, along with examples of how the platform was used in real scenarios for acquiring commonsense knowledge. In the penultimate section, we present related work in using crowdsourcing and discuss the differences with our approach for acquiring commonsense knowledge. In the final section, we give an overview of our work, provide insights on future directions and present a relevant extension of the crowdsourcing application in developing a vocabulary trainer for language learning.
2.
Following our vision for acquiring commonsense knowledge using crowdsourcing, we designed a platform which offers features and services that can be used to facilitate commonsense knowledge gathering from a number of paradigms, such as games, crowdsourcing tasks and mini applications. Most of the platform’s specifications are applied in the majority of crowdsourcing platforms and applications and some of them are specific for the task of acquiring commonsense knowledge.
For developing the platform, we considered the following key design options: 1. the selection of a suitable technology for delivering task-based applications and GWAPs, 2. the handling of contributors’ profiles, and 3. the representation of knowledge in a structured form that can be reused and verified. The platform should also allow monitoring of the acquisition process both in terms of contributors and acquired knowledge. Furthermore, the platform should be able to offer a number of design elements needed in games and educational applications. These include but are not limited to: 1. leader boards, 2. contributors’ ranking, 3. medals and awards, 4. progress-bars, 5. live feedback with notifications (both synchronous and asynchronous) for the events, and other gamification elements needed to provide the user with a pleasant experience while contributing.
On the back-end, the platform should be able to provide tools for designing a crowdsourcing application and managing contributors. These tools should provide developers the ability to easily change parameters of the application, e.g., number of raters for acquired knowledge to be valid, dynamic loading and changing of datasets (testing and validation) and export statistics on the system usage. We chose to develop a web-based system using the Joomla1 content management system (CMS) framework. The specific CMS inherently covers a lot of the aforementioned features in its core and it has a plethora of extensions for users to install, such as a community building component for creating multi-user sites with blogs, forums and social network connectivity. Additionally, the CMS provides a very powerful component development engine, for developers to deploy additional elements that can be reused in multi-domain applications.
There are many cases where crowdsourcing applications
require functionality from other systems or knowledge bases,
e.g., automated reasoning engines, datasets and natural
language processing systems. For the crowdsourcing
platform we constructed an Application Programming
Interface (API) to the Web-STAR system
The crowdsourcing platform offers a number of features for promoting the application to groups of users, either in social media or user forums. Contributors can share their contribution status/points/awards to social media groups. This tactic can increase user retention to the application. Moreover, developers can enable the “invitations” functionality, where contributors gain extra points when they invite other people to contribute. 2.2.
In this section, we showcase the steps needed for a developer to design and deploy a crowdsourcing application. These steps are also depicted in Figure 1. First, a template must be selected to match the application domain. There are a number of templates available to match a number of crowdsourcing paradigms (e.g., GWAPs, language learning applications) which can be customized according to the specific needs of the task. Developers need to prepare the main functionality of their system by coding it in PHP, or any other language and encapsulate its executable in the platform and deliver the result using HTML, CSS and JavaScript. During this process, they need to prepare a list of parameters that can be used in the experiments and code it in XML format. These parameters can be incorporated in the code and control how various elements are displayed (e.g., display/hide web tour and guidance, choose what knowledge is presented for verification, etc.).
The next steps involve the selection of knowledge acquisition tasks. Developers can select among acquisition, verification and knowledge preference identification tasks and map the methodology steps to application screens or game missions (depending on the chosen paradigm). The knowledge preference selection task involves the ability of a human contributor to choose pieces of knowledge that are used in a given situation and discard the ones that are not. For example, when reading a story about birds, readers can infer that birds can fly. From a similar story, where it is explicitly mentioned that birds are penguins, readers can infer that penguins cannot fly.
For each task, a data stream is required. The data stream
can be anything from text inserted directly from
contributors, i.e, a dedicated task in the application, a pre-selected
dataset such as Triangle-COPA
Developers are free to design and code the logic behind each task as they see fit to achieve their goals. The platform has a number of pre-defined functions for storing commonsense knowledge in the form of rules or facts, both in natural language and in a logic-based format, e.g., hug(X,Y) implies like(X,Y) where X and Y are arguments and intuitively means if a person X hugs a person Y then person X likes person Y.
Moreover, the platform incorporates a number of visualization libraries (e.g., d3.js2, Cytoscape.js3, chart.js4) to provide live feedback to the contributor.
For each application, developers need to choose how contributed knowledge is selected and what are the criteria for storing this knowledge in the accepted knowledge pool. Developers can choose among a number of strategies or a combination of them, such as selecting knowledge that was contributed by at least n number of persons, knowledge that is simple (e.g., rules with at most n predicates in their body), knowledge that is evaluated/rated by at least n raters and knowledge that is evaluated by an automatic reasoning engine. Depending on the type of application, developers also need to choose a marking scheme that fits the logic behind the application and reward contributors, e.g., points and medals for games.
When the design of the various tasks is completed, the developer needs to choose how contributors will have access to the platform (e.g., anonymously, through registration or social networks) and what details need to be filled in their profiles.
In terms of technological infrastructure, the platform relies on a web-server with Linux-Apache-MariaDB-PHP (LAMP) stack and on the Joomla framework. The platform also utilizes the JQuery5 and the bootstrap frameworks both for designing elements and for application functionality. The platform employs the Joomla Model-View-Controller (MVC)6 framework that allows the development of components by separating the data manipulation functions from the view controls. The controller is responsible for examining the request and determining which processes will be needed to satisfy the request and which view should be used to return the results back to the user. This architecture allows the usage of both internal (e.g., database) and external data sources (e.g., APIs, files) and of course deliver these services in an abstraction layer that can be used by other applications.
For user authentication, both the Joomla internal mechanisms are used and the Oauth7 authentication methods that permit the seamless integration of social network authentication with the platform.
It is important for application developers to be able to visualize acquired knowledge for better understanding what and how users behaved during the crowdsourcing experiment. In Figure 2 an example of a Sankey type graph is presented for the Robot Trainer game where results for both the contributors and the acquired knowledge are depicted on the same diagram. This type of functionality is possible by using the D3.JS library with data feed from the database and a graph theory (network) library for visualization and analysis called Cytoscape.js. The latter was also used for representing and contributing commonsense knowledge rules in a graphical manner in WebStar and was evaluated positively
Model-View-Controller
by novice users in conjunction with using a text-based editor for the same task.
3.
In its current state, the platform was used to develop two
GWAPs and a crowdsourcing application. There is an
extensive presentation of the two GWAPs in our previous
work
In this section, we focus on how the platform was used for the task of acquiring knowledge in the form of natural language rules for solving cloze tests. For running this task, first we retrieved stories from the ROCStories dataset in a tabular format and loaded them in the platform’s database table. Then we parsed each story sentence through the CoreNLP system and got the Parts-Of-Speech (POS) for each word and its base form (lemma). The aforementioned, were stored in a database table. For each story, a noun word was obscured and more than 1000 cloze tests were created. For each test at least 5 possible answers were generated and stored, including synonyms and antonyms retrieved from Wordnet. This workflow was developed in the back-end by reusing components from the two GWAPs and by adding new functionality specifically used for that workflow. The task was separated in three subtasks and for the frontend design, each one of these tasks is presented on a separate screen (see Figure 3). Each screen comprises an instruction area on top, the active task area below that on the left and the visual representation area on the right. The visual representation area is dynamically updated with every user action. Directly below these two are the task controls. This template, based on bootstrap, was chosen for its simplicity, since we wanted to avoid users paying attention to unnecessary elements.
To start contributing, users need to create an account using either the registration form or one of the social media connected account methods inherently present in the platform. After entering their credentials, contributors are firstly presented with a test (see Figure 2a) which they solve and state how confident they are in solving it, in a scale of 0 to 100%. Secondly, the contributors are asked to highlight the words in the text that helped them decide the missing word (see Figure 2b), and thirdly, they are presented with a new test where both the correct answer selected in the 1st step and the highlighted words selected in the 2nd step are present (see Figure 2c). The contributor is asked to verify if the highlighted words are used to find the missing word. Finally, a new test appears which includes the highlighted words from the 2nd step but not the selected missing word from the 1st step. Contributors are asked if the missing word is implied in the story. Each contributor is also presented with a task to verify if the chosen words selected by another contributor are useful for solving the cloze test (see Figure 2d).
Each test is retrieved randomly from the database and for the verification task, tests are selected randomly at first, and by prioritizing selection of tests that have at least one contribution. That way, we give priority to verifying contributions. This is set before running the experiment in the parameters screen on the back-end. All user contributions are recorded and stored in a database table recording both the task data (e.g., missing word, highlighted words, verification response) and metadata (e.g., response time). Recording is possible using the JQuery AJAX libraries and APIs, which allow dynamic update of the content without refreshing the browser webpage and make the contributor to loose focus on the task.
Through these tasks, we are able to acquire knowledge both for cases where the word is explicitly stated in the text and for cases that it is implied. The crowdsourcing application was tested with a small crowd and initial experiments showed that acquired rules can be used both for solving cloze tests and for generating inferences from a story. For example the following two rules were generated and verified on unseen stories: when words (or their lemmas) friends and high exist in a story then the missing word is probably school when words (or their lemmas) player and scored exist in a story then the missing word is probably team
Currently, there are many attempts to harness the power of
the crowd for several tasks such as image tagging,
knowledge gathering, text recognition, etc. The motives for
people contributing, are categorized between intrinsic and
extrinsic
For the purpose of acquiring commonsense knowledge
there are examples of games and frameworks such as
Verbosity
There are also approaches where contributors are
motivated by money such as the Amazon Mechanical Turk
The aforementioned systems and games are very interesting and provide a lot of features, but their design is focused on targeting a single task, rather than a series of tasks that are chained. Furthermore, the majority of systems is limited to the templates and standard workflow processes offered in order to accommodate the most common and most popular crowdsourcing tasks. The task of commonsense knowledge acquisition is more complex and requires more complex workflows to be used, e.g, contribution and then verification.
There are cases where crowdsourcing only is not the best option for acquiring a specific type of knowledge and hybrid solutions, i.e., solutions that employ both human contributors and machine processing, should be used towards that direction. Such an example is the acquisition of commonsense knowledge in the form of rules, where we compared a pure crowdsourcing approach (“Knowledge Coder” game) with a hybrid one (“Robot Trainer” game). The results suggest, that the hybrid approach is more appropriate for gathering general commonsense knowledge rules, that can be used for question-answering. This is one of the reasons we chose to develop a custom made platform in order to have more flexibility in developing such tasks. Readymade templates offered by the mainstream platforms cannot give you this flexibility, since they aim in a broader set of experimenters. Of course, this comes at the cost that some development should be made from the experimenter. The crowdsourcing platform has internal mechanisms for knowledge representation in the form of rules, which can be reused in many different applications that serve a similar purpose. Using one of the mainstream platforms requires handling the knowledge rule representation using external tools that need to be developed beforehand. The crowdsourcing platform also engulfs natural language processing tools for treating datasets, before requesting crowd workers to process them. There are also modules for direct integration with knowledge bases (e.g., ConceptNet, YAGO) that can be used in conjunction with the crowd tasks, either for knowledge verification or to reduce the ambiguities in language. The aforementioned features cannot be found in
(a) Screenshot of the 1st step where contributors first select a word to fill the gap from one of the possible answers. (b) Screenshot of the 2nd step where contributors highlight the words that led in selecting the missing word. (c) Screenshot of the 3rd step where contributors verify if the highlighted words from the 2nd step can be used to identify the same missing word with that of the 1st step on a new cloze test. (d) Screenshot of the verification step where contributors verify other contributors highlighted words used for solving a cloze test. platforms such as PYBOSSA or psiTurk which concentrate on designing crowdsourcing experiments nor in GECKA which is focused in designing GWAPs.
In this article we presented an overview of the crowdsourcing platform developed to facilitate the development of crowdsourcing applications and GWAPs focused on acquiring commonsense knowledge. Examples of how the platform was used to acquire commonsense knowledge were depicted along with how the various platform elements were used to achieve the goal of the application. The key features of the crowdsourcing platform include the ability to design complex workflows for acquiring commonsense knowledge, a storage and handling mechanism for acquired knowledge and numerous tools for dataset processing and integration with large semantic knowledge bases and reasoning engines. Moreover the platform offers a wide range of visualizations and analytics to the experimenters that can be customized to facilitate the monitoring and reporting needed during crowd experiments. In terms of results, from the first GWAP we implemented, i.e., “Knowledge Coder” game, we gathered 93 knowledge rules from 5 contributors. These rules were too specific on the story that was used to generate them and did not offer any value for understanding or answering questions on other stories. When the crowdsourcing platform was used for the “Robot Trainer” game we were able to recruit 800 players from Facebook and some popular game forums in a period of 153 days. These players contributed 1847 commonsense knowledge rules (893 unique). Contributed rules were general enough to be used in other domains, e.g., the symbolic rule hit(X,Y) IMPLIES angry(X) meaning that a person X hits a person Y implies that person X is angry. Through the game, 1501 commonsense knowledge rule evaluations were gathered and the interesting part is that players added a “Positive” evaluation to simple rules instead of more complex ones.
We are currently investigating how this work can be used
in the context of language learning by using commonsense
knowledge databases for creating exercises such as cloze
tests, “find synonyms, antonyms, etc.” and delivering them
to students. The platform can be used to deliver
vocabulary exercises, generated from commonsense knowledge
databases and ontologies such as ConceptNet. The
responses can be used to expand the knowledge bases that
the exercises originated from. A prototype implementation
of this