In this paper, we evaluate PYBOSSA, an open-source crowdsourcing framework, by performing case studies on blends, neologisms and language variation. We describe the procedural aspects of crowdsourcing, such as working with a crowdsourcing platform and reaching the desired audience. Furthermore, we analyze the results, and show that crowdsourcing can shed new light on how language is used by speakers.
Crowdsourcing (or: citizen science) has shown to be a
quick and cost-efficient way to perform tasks by a large
number of lay people, which normally have to be
performed by a small number of experts
For the Dutch language, crowdsourcing has been valuable
in the past. We distinguish two types of approaches. On
one hand, there are fixed tasks, where more or less one
answer is correct. As fixed tasks, crowdsourcing has been
applied to the transcription of letters from 17th and
18thcentury Dutch sailors
On the other hand, there are open tasks, referred to in
empirical sciences as elicitation tasks, where different answers
by different users are welcomed, in order to capture
variation. Examples of open tasks are Palabras
1Homepage: http://pybossa.com. DOI: https:// doi.org/10.5281/zenodo.1485460
2http://www.meertens.knaw.nl/
vertrokken-nederlands/
Open-task crowdsourcing has been applied to lexicography
for other languages, such as Slovene, where crowdsourcing
was integrated in the thesaurus and collocation dictionary
applications
2.
As the basis for our experiments, we hosted an instance of PYBOSSA at our institute. We named our crowdsourcing platform Taalradar (‘language radar’): this signifies both the ‘radar’ (overview) we would like to gain over the entire language through crowdsourcing, and the personal ‘language radar’ or linguistic intuition of contributors, which we would like to exploit. We ran two crowdsourcing rounds: in september 2018 and in novemberdecember 2018. 2.1.
We designed four tasks, which are well-suited to reach our
goals: documentation of the Dutch language and
developing material for language learning. Since we would like to
get a picture of the speakers of the language, we ask for
user details (gender, age and city of residence) in all tasks.
The tasks were created as Javascript/HTML files inside PB.
Tasks 1 and 2: Blends analysis and recognition Blends
are compound words, formed “by fusing parts of at least
two other source words of which either one is shortened in
the fusion and/or where there is some form of phonemic or
graphemic overlap of the source words”
Task 3: Neologisms In this task, contributors were asked
to judge neologisms (new words) in a citation, on two
criteria: endurance of the concept (“This word will be used
for long time.”) and diversity of users and situations (“This
word will be used by different people [eg. young, old] in
different situations [eg. conversation, newspaper].”). We
selected these two criteria from the FUDGE test, a test
to rate the sustainability of a neologism, which normally
consists of 5 criteria
Task 4: Language variation In this task, contributors are asked how they call a certain concept or how they would express a certain sentence. The goal is to chart dialectal variation, but also other kinds of language variation. We used a list of questions from Taalverhalen.be, a website which tries to chart language variation using questionnaires3. The list contains 16 questions: 9 questions on words for sweets, and 6 questions about the general vocabulary. An example of a question is: “How do you call VINEGAR?”. On top of the user details we ask in other tasks (gender, age, city of residence), we also ask for province, mother tongue and educational level.
Our experiments were advertised via our institutional newsletter, which reaches 3891 subscribers with an interest in language. We assume this was the channel with the largest reach: in the first round, the newsletter article received 519 clicks, and in the second round, 65 clicks. In both rounds, we observed an increase in contributions after the release of the newsletter. Additionally, we attended two linguistics events, where we offered visitors the possibility to engage in our crowdsourcing experiments: the meeting 3http://taalverhalen.be, maintained by Miet Ooms. of the international society of Dutch linguistics and Drongo festival, an event for the language sector in The Netherlands. Finally, we advertised our experiments via social media (Twitter, LinkedIn) and a Dutch linguistics blog. 3.
The results section consists of two parts. We will first describe our experiences with PYBOSSA as a crowdsourcing platform. Then, we will report on the linguistic findings on the language phenomena we performed experiments on.
Table 1 shows the number of contributors for each of the tasks. It can be observed that only a small number of visitors did not finish the whole task. This could be due to the small number of questions we offered per task. The tasks in the second round (november-december 2018) received less contributors than in the first round (september 2018), this could be due to a less prominent place of the announcement in our newsletter in the second round. In all experiments, more women than men participated. Also, participants with ages above 50 were well represented. More participants came from The Netherlands than from Flanders. # started # completed period Task Blends analysis Blends recognition Neologisms Language variation
We will now discuss our experiences with the PYBOSSA platform. A strength of PB is the freedom it offers when designing a task: the whole interface can be written in HTML and Javascript and can be customized. This also makes it easy to share tasks with other researchers4. The account system and saving/loading of tasks is handled by PB, so this does not have to be implemented by the task developer. Responses of the PB authors on the bug tracker are quick and concise. It is clear that PB is mainly designed for fixedtask crowdsourcing, not focusing on variation and the details of the contributor. For open-task, linguistic purposes, some points require attention (at time of writing). Firstly, there is no built-in support for asking contributor details. We handled this by asking contributor details via a normal question. However, since all given answers are visible publicly in PB, this also applies to the details, which may not be ideal from a privacy perspective. Secondly, contributors cannot go back to a previous task and change their answers. Thirdly, multiple anonymous logins from the same computer are not allowed, making it harder to use PB on e.g. a trade fair. A workaround is possible, but not built in PB by default. Also, anonymous users are identified by IP address: this can cause problems when multiple anonymous contributors connect via a shared internet connection, such 4Our tasks can be downloaded from: https://github. com/INL/taalradar. Endurant
Diverse
Status as in classroom use. Finally, there is no built-in possibility for a contributor to stop answering after a subset of the total number of questions available, and show an end screen. All in all, PYBOSSA, is a convenient crowdsourcing tool, but has its limitations with regard to open-task crowdsourcing.
Blends For the blends analysis task, we compared the contributor answers to the attested analyses from the ANW. Contributors showed an average accuracy of 42%, with average accuracies per word ranging between 2-83%. Table 2 shows the given answers for the analysis of the blend preferendum. This shows that there is not always one correct analysis of a blend, when a related noun and verb can both be filled in as source word: while prefereren ‘to prefer’ + referendum is the attested analysis, preferentie ‘preference’ + referendum may also be an option. It is even more interesting to see that a number of contributors analyze this blend entirely differently than the attested analysis: they analyze the blend as pre ‘before’ + referendum.
Answer referendum, prefereren referendum, preferentie pre, referendum do not know preferent, referendum
Frequency
For the blends recognition task, the contributor answers were compared to the ANW entry in which the citation occurs. Contributors had an average recognition accuracy of 87%, with average accuracies per word ranging between 54-97%. The accuracies are high: most blends are recognized correctly. Table 3 shows the given answers for the recognition of one specific blend: twittie. twittie ‘twitter fight’ is a blend of twitter and fittie ‘fight’ (slang). Most contributors correctly recognize this as blend. Many people however also perceive fittie (which does also occur in this citation) and tweet as blends, possibly because these words appear new or unknown.
Neologisms Table 4 shows the endurance and diversity judgments for the 15 words in the neologisms task. These results show that in general, neologisms rejected by lexicographers also receive lower crowd endurance scores. For diversity, this pattern is not as clear. Woord gendertransformatie insectenafname dreigingsmonitor belevenisstad vluchtelingenpraktijk multimediamerk zonnepriesteres seniorenmodebranche moeilijkheidsparadox afradertje tijdstrends nachtnanny lighttaks korttheater dieetopenbaring
Language variation In the language variation task, we found that most people used the standard Dutch term to signify a word, only a minority of the given forms was a dialectal form. However, it is interesting to investigate the differences between Dutch and Flemish contributors. The number of contributors from The Netherlands (around 100 per question) is larger than the number of Flemish contributors (around 15 per question). Table 5 shows the relative frequencies of given answers for the concept TAKE A SEAT, split per language area. ga lekker zitten is very popular in The Netherlands, while zet u is only used in Flanders.
Flanders
The Netherlands Utterance ga zitten ga lekker zitten neem plaats zet u pak een stoel Total answers 31% 0% 6% 31% 6% 16 38% 18% 7% 0% 4% 115
These differences per area are observed for more questions. For example, a SWEET ON A STICK is referred to by many Flemish contributors as lekstok, whereas contributors from the Netherlands mainly use the form lolly. And WISHING A GOOD NIGHT is done by saying slaap wel in Flanders, while welterusten is used more in The Netherlands.
Integrating crowdsourcing into a lexicographic workflow Our case study on neologisms shows the potential of crowdsourcing for lexicography. Crowdsourcing becomes even more useful, if it becomes fully integrated into the lexicographic workflow. Currently, at the INT, newspaper material is fed in and sentences with unknown words are automatically extracted. Lexicographers then manually decide on inclusion in the dictionary. In an ideal workflow, the extracted sentences are automatically imported into a crowdsourcing application and shown to the public. Contributor judgments can help lexicographers in deciding on dictionary inclusion. A challenge will be to motivate a crowd to contribute over a long period of time. To maintain worflow stability, also in case of a temporary drop in crowd participation, crowd consultation will be an optional step in the workflow.
Language learning We have not yet performed
crowdsourcing experiments for language learning, but we are
looking into future directions which seem promising.
Crowdsourcing can be used to cluster word senses, which
could help people with language or speech disabilities.
Crowdsourcing has been used for word sense
disambiguation before
5.
Our experiments have shown that crowdsourcing proves useful for documenting the Dutch language, and can be valuable for developing Dutch language learning material in the future. We used the PYBOSSA framework for our crowdsourcing experiments, which is very powerful, but also has its limitations when using it for linguistic purposes.
This work was supported by EU COST action CA160105 enetCollect, which is gratefully acknowledged. We thank Miet Ooms for supplying the questions for the language variation task. We thank our colleagues at the INT for valuable advices.