Case Study on Intelligent Crowdworker Selection via Deep Learning

Abstract

from the specific error causes to jointly estimate crowdworkers' performances and instances' true annotations. Commonly, these techniques are employed after completing a crowdworking campaign. However, leveraging the crowdworkers' performance estimates to optimize an ongoing campaign appears beneficial. Therefore, this article studies whether crowd-learning techniques can answer the question “Who knows best?” to select crowdworkers intelligently. In a case study with classification data, we show that such a crowdworker selection reduces the number of erroneous annotations and allows us to train DNNs with lower misclassification rates than a random selection of crowdworkers at the same annotation costs.

1. Introduction

CEUR

2. Problem Setting

Let there be ∈ ℕ >0 instances = ( 1, ..., )T ∈ ℝ × , ∈ ℕ >0 drawn independently from an unknown probability density function Pr( ). The true class labels = ( 1, … , )T ∈ {1, … , } , ∈ ℕ >1, drawn independently from an unknown categorical distribution Pr( ∣ ), are unobserved due to the lack of an omniscient annotation source. Rather, there are ∈ ℕ >0 error-prone crowdworkers = ( 1, ..., )

T ∈ ℝ× , ∈ ℕ >0, where represents crowdworker metadata [ 13 ], e.g., educational background, interests. If such data is unavailable, each crowdworker is identified via a one-hot encoded vector, i.e., annotation of crowdworker for instance as an unobserved annotation. Each observed annotation = ∈ {0, 1} . We refer to the ∈ {1, … , } ∪ {⊗} , where

= ⊗ indicates is drawn independently from an = ( 1 , ..., unknown categorical distribution Pr( ∣ , , ). We denote annotations per instance as )T and annotations of all instances as the matrix = ( 1, ..., ) . We define a T crowdworking campaign as a process with ∈ ℕ >0 iterations. Iteration ∈ {1, … , } starts with ∈ ℕ >0 selected instances = { 1, … ,

∣ 1, … , ∈ {1, … , }} . Subsequently, we select a crowdworker for each instance by specifying instance assignments ℎ ∶ → { 1, … , }. At the end of iteration , we update the annotations { ∣ ∈ , ℎ ( ) = } to obtain the matrix . 0, … , with 0 as initial and as final annotation matrix. Given these prerequisites, we investigate two objectives for optimizing the crowdworker selection.

Objective 1: The crowdworking campaign produces a final annotation matrix minimizing the number of erroneous annotations: =1 =1 = arg min (∑ ∑ ( ≠ )⋅ ( ≠ ⊗)) , where ∶ { false, true} → {0, 1} is an indicator function with ( false) = 0and ( true) = 1. Objective 2: The crowdworking campaign produces a final annotation matrix to learn a classification function ∶̂ ℝ → {1, … , } rate:

minimizing the expected misclassification = arg min ( , [ ( ̂ ( ∣ , , ) ≠ )]). (1) (2)

3. Intelligent Crowdworker Selection

We aim to select crowdworkers based on their respective performances per instance. Concretely, we interpret crowdworker performance as the probability Pr( = ∣ , )of obtaining a correct annotation. This leads to the following assignments of instances to crowdworkers: ℎ ( ) = arg max (Pr( = ∣ , )).

(3) The true probabilities of correct annotations are unknown in practice. Therefore, we estimate them via multi-annotator deep learning (MaDL) [ 8 ], which is a state-of-the-art crowd-learning technique. MaDL uses the annotated data obtained in each successive crowdworking iteration to estimate the class probabilities of each instance and a probabilistic confusion matrix for each instance-crowdworker pair. By combining both estimates, it is then possible to approximate the annotation correctness probability Pr( = ∣ , )in Eq. 3.

4. Case Study

In this case study, we investigate the potential to optimize crowdworker selection during crowdworking campaigns. Publicly available crowdworking datasets are sparsely annotated [ 5 ], so the selection of crowdworkers is highly limited. Therefore, we rely on LETTER [ 14 ] and CIFAR10 [ 15 ] as common benchmark datasets and simulate = 10 error-prone crowdworkers for each. We use standard simulation methods from literature [ 8 ] and generate varying types of crowdworkers, e.g., one adversarial crowdworker, crowdworkers specialized in certain classes, and crowdworkers specialized in certain clusters of instances. The simulated crowdworking campaign is organized into = 25 iterations. Initially, each crowdworker annotates 16 randomly selected instances to obtain the initial annotation matrix 0. In each subsequent iteration, = 256 randomly selected instances are assigned to the crowdworkers for annotation. After each iteration, we train a simple multi-layer perception for the LETTER dataset and a ResNet-18 [ 1 ] for the CIFAR10 dataset. We evaluate each crowdworking campaign by quantifying the rate of obtained erroneous annotations (cf. Objective 1) and the DNN’s misclassification rate on a separate test set (cf. Objective 2). For evaluation, we compare the following approaches: • Random-DL is the baseline approach. A standard DNN is trained on the annotated instances, and the selected instances are randomly assigned to the crowdworkers. • Random-MaDL is a more advanced approach. MaDL is trained on the annotated instances, and the selected instances are randomly assigned to the crowdworkers. • Intelligent-MaDL is the most advanced approach. MaDL is trained on the annotated instances, and the selected instances are assigned to the crowdworkers according to Eq. 3. Our repository at https://github.com/ies-research/intelligent-crowdworker-selection provides the approaches’ hyperparameters and code. A crowdworking campaign is replicated five times for each approach and dataset. Figure 2 reports the results’ means and standard deviations.

For both datasets, the approach Random-DL performs worst, indicated by the highest misclassification rate of its DNN across almost all iterations. In contrast, its erroneous annotation rate is identical to Random-MaDL (the green curve hides the blue curve) because both approaches assign the instances randomly to the crowdworkers. Intelligent-MaDL consistently outperforms the other two approaches. These results confirm that MaDL improves not only the training of DNNs (lower misclassification rate of Intelligent-MaDL and Random-MaDL than Random-DL) but also the selection of crowdworkers (lowest erroneous annotation rate of Intelligent-MaDL).

5. Conclusion and Outlook

This article demonstrated the potential gains of employing a state-of-the-art crowd-learning technique during an ongoing crowdworking campaign. Our takeaways are that intelligently selecting crowdworkers reduces the number of erroneous annotations (cf. Objective 1) and improves the training of DNNs on the resulting annotated data (cf. Objective 2). Still, there are multiple future research directions to enhance the crowdworker selection further: • Collecting metadata [ 13 ] about the crowdworkers may allow flexible and efective integration of new crowdworkers into an ongoing crowdworking campaign. • Transferring knowledge about crowdworkers between crowdworking campaigns may improve the selection of crowdworkers for subsequent campaigns. • Improving the uncertainty estimation [ 16 ] of crowd-learning techniques may enhance the exploration of crowdworkers’ performances. • Leveraging active learning strategies [ 11 ] to select instances intelligently may further improve the eficiency of training DNNs from crowdworking data. • Assigning an instance to multiple crowdworkers (instead of only one crowdworker as done in Fig. 1 and Eq. 3) may better identify erroneous annotations or ambiguous instances. For a successful deployment of intelligent crowdworker selections into actual crowdworking campaigns, we need to consider the following aspects: • In certain settings, crowdworkers are only occasionally available, which may hinder the selection of the best crowdworker. • Typically, the sets of instances assigned to a single crowdworker must be larger [ 17 ]. • Experiments with real-world crowdworking datasets, a larger number of annotators, and a larger number of selected instances per crowdworking iteration are necessary to validate the efectiveness of intelligent crowdworker selections.

References

[1] K. He , X. Zhang , S. Ren, J. Sun , Deep Residual Learning for Image Recognition , in: Conf. Comput. Vis. Pattern Recognit ., Las Vegas , NV , 2016 , pp. 770 - 778 .

[2] I. Blohm , S. Zogaj , U. Bretschneider , J. M. Leimeister , How to manage crowdsourcing platforms efectively? , Calif. Manage. Rev . 60 ( 2018 ) 122 - 149 .

[3] F. Daniel , P. Kucherbaev , C. Cappiello , B. Benatallah , M. Allahbakhsh , Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions , ACM Comput. Surv . 51 ( 2018 ) 1 - 40 .

[4] M. Herde , D. Huseljic , B. Sick , A. Calma , A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classiifcation, IEEE Access 9 ( 2021 ) 166970 - 166989 .

[5] F. Rodrigues , F. Pereira , Deep Learning from Crowds , in: AAAI Conf. Artif. Intell., New Orleans, LA, 2018 , pp. 1611 - 1618 .

[6] H. Wei , R. Xie , L. Feng , B. Han , B. An , Deep Learning From Multiple Noisy Annotators as A Union , IEEE Trans. Neural Netw. Learn. Syst . ( 2022 ).

[7] S. Rühling Cachay , B. Boecking , A. Dubrawski , End-to-End Weak Supervision , in: Adv. Neural. Inf. Process. Syst., Virtual Conf ., 2021 .

[8] M. Herde , D. Huseljic , B. Sick , Multi-annotator Deep Learning: A Probabilistic Framework for Classification , arXiv:2304.02539 ( 2023 ).

[9] V. S. Sheng , J. Zhang, Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions , in: AAAI Conf. Artif. Intell, Honolulu, HI , 2019 , pp. 9837 - 9843 .

[10] B. Settles , Active Learning Literature Survey , Computer Sciences Technical Report 1648 , University of Wisconsin-Madison, 2009 .

[11] J. Yang , T. Drake , A. Damianou , Y. Maarek , Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa , in: World Wide Web Conf., 2018 , pp. 23 - 32 .

[12] K. Li , G. Li , Y. Wang , Y. Huang , Z. Liu , Z. Wu, CrowdRL: An End-to-End Reinforcement Learning Framework for Data Labelling , in: Int. Conf. Data Engineering , Chania, Greece, 2021 , pp. 289 - 300 .

[13] L. Zhang , R. Tanno , M. Xu , Y. Huang , K. Bronik , C. Jin , J. Jacob , Y. Zheng , L. Shao , O. Ciccarelli , et al., Learning from Multiple Annotators for Medical Image Segmentation, Pattern Recognit . ( 2023 ) 109400 .

[14] P. W. Frey , D. J. Slate , Letter recognition using Holland-style adaptive classifiers , Machine Learn . 6 ( 1991 ) 161 - 182 .

[15] A. Krizhevsky , Learning Multiple Layers of Features from Tiny Images , Master's thesis , University of Toronto, 2009 .

[16] D. Huseljic , B. Sick , M. Herde , D. Kottke , Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks , in: Int. Conf. Pattern Recognit ., Virtual Conf ., 2021 , pp. 9172 - 9179 .

[17] D. E. Difallah , M. Catasta , G. Demartini, P. G. Ipeirotis , P. Cudré-Mauroux , The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , in: World Wide Web Conf., Florence Italy , 2015 , pp. 238 - 247 .