We present an architecture for crowdsourcing language resources from language learners and a prototype implementation of it as a vocabulary trainer. The vocabulary trainer relies on lexical resources from the ConceptNet semantic network to generate exercises while using the learners' answers to improve the resources used for the exercise generation. 2012 ACM Subject Classification Information systems → Web services; Information systems → Ontologies Acknowledgements This paper is based upon work from the European Network for Combining Language Learning with Crowdsourcing Techniques (EnetCollect) COST Action, supported by COST (European Cooperation in Science and Technology)
1
We present a prototype architecture for crowdsourcing language resources from language learners and a first implementation of it for creating interactive vocabulary exercises which crowdsource [3] the learners’ answers, aiming to improve the language resources used to generate the content of the exercises.
The current architecture is designed to accommodate various language resources, such as mono- and bilingual corpora or lexicons as well as content from commonsense knowledge bases and ontologies. The architecture foresees that exercises can be delivered via several user interfaces thanks to the implementation of a RESTful API approach, allowing the logical separation between computation and presentation layers.
Work presented here is similar to that of Duolingo, a platform [14] which is used to crowdsource translations from learners. Other related work includes initiatives using explicit crowdsourcing, which have primarily employed Amazon Mechanical Turk for data collection. For instance (among many others), in [1] the authors created a Turk Bootstrap Word Sense Inventory of frequently used nouns in English.
Also, approaches of implicit crowdsourcing, which mostly rely on Games With A Purpose (GWAPs), relate to the logic underlying the architecture presented here. For example, in [9] a platform that combines automated reasoning with games for acquisition of knowledge rules was developed. Moreover, in [6], a web based game called Common Consensus is described, based on the popular TV game show ‘Family Feud’. That game is used to collect and validate commonsense knowledge about everyday goals.
The proposed architecture as well as a vocabulary trainer prototype and its features are presented in the following sections. The code of the project resides on GitLab1 for interested readers to test it or even more, help in expanding it. 2
The presented prototypical architecture builds on an implicit crowdsourcing paradigm which follows the idea that: IF a language resource can be used to automatically generate language learning exercises, THEN learners’ answers to these exercises can also be used to improve the resource.
This paradigm thus exploits a win-win strategy [7]) between people in need of high quality language resources and people in need of online language learning material. It bootstraps a virtuous circle between both parties, where the answers of the learners allow the enhancement of the language resources, which in turn will result in higher quality learning content, due to the fact that it is generated from the improved language resources.
Such a paradigm can be applied to any scenario in which a language resource (e.g., treebanks, wordnets, corpora) can be paired with a specific language learning exercise, in the sense that the exercise content can be automatically generated from the LR.
There is a somewhat counter-intuitive aspect to this paradigm: the assumption that a crowd of learners, with their natural deficiencies regarding their knowledge of the language, can be of use for improving language resources–a task that is usually performed by expert
linguists.2
However, the lack of expertise of the crowd can be compensated in two ways: (
Regarding (
Regarding (2), cross-matching judgments of learners to deduce the correct answers can be addressed by an aggregation approach which relies on both the classic trade-off between quantity and quality (a low quality of answers is made up for by a higher quantity of answers), and the possibility to decompose any complex question in smaller grained elements that can be asked to learners through a set of boolean questions (e.g., ‘Does the learner believe that the French word “manger” is a verb?’). Indeed, provided that the crowdsourced answers allow to directly or indirectly deduce a boolean opinion, then all answers from learners with performances superior to 50% to such a task allow to progress towards statistical certainty. Therefore, one only needs to keep on asking the same question to different learners until the set of answers allows to reach a statistical threshold ensuring good quality (e.g., a reliability score above 98%). 3
The proposed architecture is based on a modular schema and comprises four modules: (
The exercise generation module is responsible for content retrieval from any type of language resource (LR) like corpora, knowledge bases and lexica, which contain language data 2 Readers can picture it as asking a group of tourists for a route in the city they are visiting. in a structured form. It handles the retrieval of specific data from a LR, e.g., all collocates of the word ‘challenge’ from a collocation lexicon, and automatically processes them in order to create exercises. The processing could include grouping the collocates by word class, normalize singular and plural forms of substantives, etc. The exercise generation module delivers the exercises to the data dispatcher which can deliver valid answers to the exercises, back into the LR. Furthermore, it can also use natural language processing techniques to convert data, e.g., extracting the lemma of a word.
The data dispatcher module handles all transactions between the various modules. It caters for receiving and passing on data in a generic exchange format (such as JSON3). For example it may receive generated exercises of different types and passes them on to multiple user interfaces. In return it receives back the response from the completed exercises from the user interfaces and passes them on to the evaluation module. After receiving the processed results, it can return the crowdsourced data to the original language resource that the exercises were generated from. The whole communication is done through secure web-service transactions between the various modules.
The evaluation module processes the results retrieved from learners when completing exercises. Different types of aggregation methods can be applied to determine correct and wrong answers. This validation information is used for two purposes in the presented architecture: firstly to update or enhance the LR with new generated (crowdsourced) information and secondly to provide feedback to the learners about their performance while completing the exercises.
The user interaction module can handle integrations of the data dispatcher with different user interfaces such as chatbots (e.g., Telegram4) and web-based applications. The architecture can be utilized by any user interface that is able to consume the exercises structure, data and incentive mechanism through its API, while preserving the same logic behind the exercise. 4
As a first implementation of the prototype architecture for crowdsourcing language resources, we present an interactive vocabulary trainer, which is built using data from ConceptNet, a commonsense semantic network [13]. It offers vocabulary exercises to practice semantic relations between words.
In language learning, vocabulary enhancing exercises based on words semantic relations are considered to be effective activities. In [11], the aspects of background knowledge, context and morphology to learn words more effective and clarify word meaning as essential to vocabulary instruction are presented. The richness of acquired vocabulary depends not only on the number of learned lexical items but also from the ability to connect and share semantic networks of similar concepts. Authors of [2] argue that ‘word learning is not simply the process by which isolated object– label associations are added to the mental lexicon one by one but also involves the learning of interrelated clusters of concepts, in which the knowledge of one concept supports the learning of another’ (p. 42).
ConceptNet is a large semantic network that describes general human knowledge and how it is expressed in natural language. Facts in ConceptNet originate from sources like DBPedia
[5], Wiktionary5 and popular GWAPS and crowdsourcing projects, such as Verbosity [15], the Open Mind Common Sense project [12], etc.
The exercise generation module is responsible for retrieving content from ConceptNet and for creating language learning exercises from the retrieved content. This is done by quering directly the conceptnet.io APIs for relevant content.
ConceptNet provides a large set of background knowledge about different facts connected with other facts using relations such as relatedTo, AtLocation, PartOf, IsA, etc.
For instance, if a search for knowledge that relatesTo the term ‘cat’ is initiated, ConceptNet will return results such as ‘feline’, ‘pet’, ‘dog’, etc. Afterwards, the exercise generation module processes the results using a natural language processing application to remove stopwords and duplicates, retrieve lemmas and store them in a database.
An example of a generated exercise is ‘Name one thing that is related to cat’, where the learner is expected to provide a word that exists in the results retrieved from the knowledge base. In cases where new words are added, the evaluation module checks whether they should be added to the knowledge base or not, while a specific user feedback strategy is used to account for the unknown correctness of the answer.
The data dispatcher module of the vocabulary trainer is handling transactions between
the various modules by using secure web-services, where requests are received and the
outcome is presented in JSON6 format. Detailed specification of the API is available at the
project repository. The architecture provides web-services for: (
Within the vocabulary trainer, the evaluation module processes the learners’ answers in order to both update the knowledge base with new words and to assign points and badges to the learners and make the whole process interesting.
Whenever a learner completes an exercise, the evaluation module validates the provided answer against the knowledge base. If the answer is already part of the knowledge base then the learner receives points. If the answer is not part of the knowledge base then it is put on hold until a certain number of new words (i.e., candidate words for the knowledge base) have been accumulated for that specific exercise. In the second case, the learner receives a feedback message explaining that there are additional points pending to be approved.
Once the pre-defined threshold of new words is met the list of candidate words is ranked according to the frequency and feedback is sent to each learner who provided an answer. For the highest-ranked word among the list of candidates the pending points are turned into actual points and the knowledge base is updated with the word. All learners who had provided that answer are awarded points, and the learners who provided that answer first, receive also an additional badge.
The user interaction module is currently populated by two user interfaces in the
vocabulary trainer: (
In this paper, we presented an architecture to crowdsource language resources from language learning exercises delivered via several user interfaces. The proposed architecture is versatile and expandable and it is not restricted to a specific paradigm or dataset. Different language resources can be used for generating learning content and several types of exercises can be added. Also different evaluation strategies to cross-match learners’ answers can be incorporated to accept or reject an answer and update the corresponding language resource.
Furthermore, we presented the first prototype implementation on top of the architecture, i.e., a vocabulary trainer that relies on ConceptNet to deliver exercises. Early tests with both the Telegram chatbot and the Bootstrap web application show promising results in terms of acquisition of knowledge facts and usefulness of the architecture for that purpose.
We are currently designing an experiment to formally evaluate all components of the architecture. We also plan to deliver the exercises via the language learning platform Revita [4] and existing knowledge-based GWAPs [8, 9]. Future directions of our research could also include exercises related to geography, which can be used to populate a knowledge base for identifying the geographic focus of a text [10], using words that are related to a specific geographic location, e.g., feta RelatedTo greece.