We deal with the problem of localized in-video taxonomic human annotation in the video content moderation domain, where the goal is to identify video segments that violate granular policies, e.g., community guidelines on an online video platform. High quality human labeling is critical for enforcement in content moderation. This is challenging due to the problem of information overload - raters need to apply a large taxonomy of granular policy violations with ambiguous definitions, within a limited review duration to relatively long videos. Our key contribution is a novel human-machine learning (ML) collaboration framework aimed at maximizing the quality and eficiency of human decisions in this setting - human labels are used to train segment-level models, the predictions of which are displayed as “hints” to human raters, indicating probable regions of the video with specific policy violations. The human verified/corrected segment labels can help refine the model further, hence creating a human-ML positive feedback loop. Experiments show improved human video moderation decision quality, and eficiency through more granular annotations submitted within a similar review duration, which enable a 5-8% AUC improvement in the hint generation models.
categories such as Profanity, Violence, Nudity, etc., each
of which contains several granular violations. For
inThe importance of content moderation on online video stance, Violence could include a range of granular classes
platforms such as TikTok, YouTube or Instagram is grow- such as animal abuse or graphic violence in video games.
ing [
For content moderation decisions there is a growing cision. We achieve this by tackling the key issue of
"inneed for transparency in detected policy violations to pro- formation overload" faced by raters in providing high
vide feedback to content creators [
tion of human annotations to perform a variety of tasks
such as text processing [
Although there is existing literature on video annotation,
it is primarily focused on identifying actions or labeling Figure 1: Human-ML collaboration set up.
entities easily distinguishable by humans using visual
information only, e.g., high jump, thunderstorms. The
primary goal of these tasks is to create large datasets for ing. The novelty of our approach is that the models
facilitating Machine Learning/Perception applications are re-trained continuously on the output of the human
[
Human-ML collaboration is an emerging area of re- that humans find it easier to verify or correct suggessearch with two main categories of work: tions rather than produce new annotations from scratch
ML-Assisted Human Labeling: ML-assistance [
language question answering model. ML-assistance has
been shown to also improve the eficiency of the human Collaboration Framework
labeling task [
Improving ML-models Through Human Annota- collaboration framework visualized in Figure 1. We use tions: Human annotations are useful in constructing the predictions of ML models to provide assistance to hybrid human-ML systems that leverage the complemen- human raters and evaluate the efectiveness of diferent tary strengths of both to improve the performance of ML user interfaces for the ML-assistance. Since the policy models [22, 23]. Existing work on active learning [24, 25] violation prediction task is hard for ML models, the feedshows that strategically sampling data points can reduce back from human raters is useful to improve the models. human workload, but the purpose is to improve machine ML-assistance enables raters to provide segment level anlearning models instead of assisting raters. Recent work notations more eficiently leading to more ground truth on explainable active learning (XAL) [26] has called for to train/update the ML models. Additionally, we can enbetter designing for the human experience in the human- able raters to interact with the ML hints (accept/reject), AI interface. providing direct feedback to refine the model, establish
Our work shows that it is possible to achieve model ing a positive human-ML feedback loop. improvements and assist human raters, bridging the gap between ML-assisted human labeling and active learn
tion overload problem due to a combination of granular, complex policy definitions and limited time per video. Often, raters need to use their judgement to decide what parts of the video to watch to identify violating segments, resulting in fleeting violations being missed, or inconsistencies across raters watching diferent sections of the same video.
It is intuitive that raters would benefit from pointers to likely unsafe regions within a video, labeled with the exact policies being violated. Even if not completely precise, this will enable them to optimize their review bandwidth by focusing on potentially more relevant regions, making them less likely to miss violations. We achieve this by training per policy ML models and transforming their predictions into "hints" provided to raters, described in more detail in Section 3.1.2. We tune the models to be high recall to minimize uncaught violations, while relying on human judgement to improve the precision of the labeled violations. 3.1.1. Segment Level Model Training 3.1.2. Techniques to Provide ML-Assistance
the following modeling problem - given multi-modal fea- We proceed to develop ways to use model predictions tures for a fixed length video segment, predict whether to most efectively assist human reviews, and provide the segment contains specific policy violations. We gen- details on two diferent designs (V1 and V2). erate training datasets by extracting per-frame visual and V1 Hints: Continuous Line Graphs. For video anaudio features from the human labeled violation region. notations, it is standard to display the video itself with The visual and audio features are the dense embedding playback controls and additional information in the form layers of standard convolutional network image seman- of a timeline [37]. For V1, we display the ML predictions tic similarity [34] and audio classification models [ 35] as a line graph across the entire timeline of the video. respectively. Based on empirical evidence, we select flat The user interface is demonstrated in Figure 2. Raters concatenation to aggregate the frame features over a seg- can examine the line graphs and jump to the point in the ment, versus average/max pooling. The final model we video where a peak (policy violation) occurs. train is a multi-label DNN model with the aggregated While we have model predictions for hundreds of granframe-level visual and audio features as input, where ular policies, due to visual clutter, we only display plots each label corresponds to a fine grained policy violation. for a small subset of the most frequent policies. Raters We use MultiModal Versatile Networks [36] during model also don’t have the ability to provide feedback to improve training to learn a better representation for audio and model predictions. visual features for our classification task, which further V2 Hints: Towards a Scalable and Interactive-ML improves model performance. We use a sliding window Assistance UI. In V2, we borrow elements from recapproach to utilize the trained model to generate predic- ommender systems [38, 39] to develop a more scalable tion scores per policy violation for a fixed length segment and interactive interface (see Figure 3), where we prestarting at each frame of the video. Using a window of populate video segments that may contain policy violaframe size n and stride of 1 frame, we produce model tions detected by machine learning models. scores for segments with start and end frames [0 to n-1], To generate video segments, as shown in Figure 3, [1 to n], and so on until the end of the video, padding we introduce an algorithm to binarize continuous model with empty features to fit the segment length for the last scores per policy into discrete segments and use a rankn frames. ing algorithm to recommend the most useful segments to raters. For simplicity, we used a threshold-based algorithm. The threshold selection constitutes a tradeof between the precision and recall, where precision captures the utility of the predicted segments to raters and recall captures the comprehensive coverage of all video violations. The algorithm chooses a threshold maximizing recall, while maintaining a minimum precision (40% based on user studies). The regions of the video where the ML scores are above this chosen threshold are displayed as predicted policy violating segments to raters.
Several heuristics are then applied to maximize segment quality, e.g., we merge segments that are close to avoid visual clutter (<3% of the whole video length apart).
Finally, to reduce information overload, we borrow from the learning to rank concept [33] to rank candidate segments and limit the number of displayed segments.
The ranking algorithm prioritizes segments based on the max score across segment frames, and egregiousness of the predicted policies. We then select the top segments to display to raters, with selected through user studies.
We pre-populate each ML suggested segment in the video timeline UI as seen in Figure 3. Raters can choose to accept or reject the suggested segments. These logged interactions can be used to provide feedback to improve the ML models.
productivity targets as generalists do. They can hence spend more time reviewing each video comprehensively, leading to higher decision quality. In our experiment setup, each video in an evaluation dataset is independently reviewed by 1 expert and 2 generalist raters. We use the labels from expert raters on the dataset as the ground truth to evaluate the 2 sets of generalist rater decisions.
4.1.1. Datasets
a large online video platform: (1) "Live trafic" dataset: Sampled from live trafic, hence containing a very low proportion of policy violating videos; (2) "Afected slice" dataset, sampled from live trafic and filtered to only videos with ML-hints present, containing 13-20% policy violating videos.
The ambiguous and fluid policy definitions along with a changing distribution of videos on online platforms poses a challenge for building robust models to accurately predict policy violations for providing ML-assistance. We 4.1.2. Metrics hence continuously need more data and human feedback Our human ratings quality metric is calculated at the to improve the models. We show that ML-assistance in- video-level and conveys the correctness of the final, bicreases the total number of segment labels submitted nary content moderation decision, e.g., take down or not. vs. no assistance. The labels in turn can serve as new We compute the precision (P), recall (R), and disagreeground truth for continuously re-training the models and ment rate for each of the 2 sets of generalist’s video-level improving performance. Additionally, with the reject but- decisions. We consider the expert decision as ground ton shown in V2 we collect clean negative labels; earlier, truth and report the averaged values across the 2 sets. On we had only "weak" negatives from videos where no vio- live trafic datasets, we use P/R over standard inter-rater lations were annotated 1. Further exploiting the human disagreement metrics due to the high class imbalance. feedback in combination with active learning strategies For rater eficiency, we measure: (1) percentage of is an area of future work. policy violating videos where raters provide segment annotations. (2) number of segment annotations submitted by raters per video (3) average review duration per video.
Our proposed methodology is evaluated using raters from the author’s organization that regularly perform video content moderation reviews on live queues. Raters are separated into two pools: experts and generalists, with 150 and 400 raters respectively. The expert pool is a group of more experienced quality assurance (QA) raters with demonstrably better decision quality over a long period of time. Since their focus is QA, they don’t have fixed
video segments the rater did not watch, hence the negative labels inferred are weak/noisy.
4.2.1. Rater Quality Improvements We conduct experiments on both "live trafic" and "affected slice" datasets, with the baseline as a review process without ML-hints. Tables 1 and 2 compare the ratings quality metrics of our proposed V1 (line plot of model scores) ML-assistance treatment relative to the baseline, and evaluate the incremental benefit of the V2 (pre-populated segments) treatment over V1, in the V1 + V2 vs. V1 row.
We also see large rater quality gains of V2 over V1 on both live trafic and afected slice datasets. The segmentation and ranking algorithms in V2 allows us to overcome the scalability limitation of V1 and expand the number of granular policies covered by model hints from 7 to 18. Specifically for violence related violations, we see a 35% relative recall gain over V1 by expanding policies with ML-hints from 2 to 9. The V2 design can be scaled to cover hundreds of policies in future versions by dynamically surfacing the most relevant violating segments, further improving recall. 4.2.2. Rater Eficiency Improvements We observe reduced review duration on policy violating videos with V1 hints vs. without, with more eficiency benefits on longer videos as expected; -14% on videos longer than 10 minutes and -20% on videos longer than 30 minutes. With V2, relative to V1, we see a 3% increase in review duration, but it is traded of by a 9% increase in the percentage of policy violating videos with exact segments annotated, and a 24% increase in the number of segment annotations submitted per video. 4.2.3. Interactive ML-Assistance Metrics
4.2.4. Model Quality Improvements
Policy Area Sexually Suggestive
Nudity Illegal Acts
AUCPR +5.9% +4.9% +8.6% # Positive Labels +12.7% +12.7% +12.1% raters accepting model hints without verification. Our video-level ratings quality evaluation metrics are robust to this since the ground truth comes from expert (QA) raters who review videos comprehensively, looking beyond ML-hints. In practice, we observe little evidence of both (i) and (ii). 56% of the violation segments submitted by raters in the V2 setup are organically created, i.e., don’t overlap with pre-populated hint segments. The segment acceptance rate is 35%, aligned with our segmentation model precision tuning point of 40%, indicating that raters are verifying and rejecting false positive hints at the expected rate. We could mitigate the risk of (ii) further by enforcing that at least some percentage of hint segments/video is actually watched or by surfacing the model’s confidence in the predicted hint to raters. To ensure robust evaluation of model quality, the AUC improvements in Section 4.2.4 are evaluated on a set of labels collected without model generated segments.
For content moderation to scale to the size of online platforms, it is necessary to take model-based enforcement action. We would like to explore the relation between improved ground truth and improvement of automated, model based enforcement. Leveraging active learning strategies in combination with utilizing rater feedback on model generated segments to show further quality improvements in the models is another open area of research. Finally, we will explore multi-armed bandits to balance active learning based exploration for model improvement with model exploitation for providing high quality ML-assistance [41].
This paper used content moderation as the test bed for our human-ML collaboration proposal. However, it is a more generalized framework that applies to the problem of granular, localized video annotation encountered in various other industry applications such as identifying products/brands in videos to inform the placement of relevant ads, which we would like to explore further. New York, NY, USA, 2019, p. 29–38. URL: https: formation overload on consumers’ subjective state //doi.org/10.1145/3287560.3287590. doi:10.1145/ towards buying decision in the internet shopping 3287560.3287590. environment, Electronic Commerce Research and [18] J. Park, R. Krishna, P. Khadpe, L. Fei-Fei, Applications 8 (2009) 48–58.
M. Bernstein, Ai-based request augmentation [29] H. Hu, L. Xie, Z. Du, R. Hong, Q. Tian, One-bit to increase crowdsourcing participation, Pro- supervision for image classification (2020). URL: ceedings of the AAAI Conference on Human https://arxiv.org/abs/2009.06168. doi:10.48550/ Computation and Crowdsourcing 7 (2019) 115– ARXIV.2009.06168. 124. URL: https://ojs.aaai.org/index.php/HCOMP/ [30] R. M. Losee Jr, Minimizing information overload: article/view/5282. the ranking of electronic messages, Journal of In[19] M. Bartolo, T. Thrush, S. Riedel, P. Stenetorp, R. Jia, formation Science 15 (1989) 179–189.
D. Kiela, Models in the loop: Aiding crowdwork- [31] K. Koroleva, A. J. Bolufé Röhler, Reducing inforers with generative annotation assistants, CoRR mation overload: Design and evaluation of lfitering abs/2112.09062 (2021). URL: https://arxiv.org/abs/ & ranking algorithms for social networking sites 2112.09062. arXiv:2112.09062. (2012). [20] Z. Ashktorab, M. Desmond, J. Andres, M. Muller, [32] J. Chen, R. Nairn, L. Nelson, M. Bernstein, E. Chi, N. N. Joshi, M. Brachman, A. Sharma, K. Brimi- Short and tweet: experiments on recommending join, Q. Pan, C. T. Wolf, E. Duesterwald, C. Dugan, content from information streams, in: ProceedW. Geyer, D. Reimer, Ai-assisted human labeling: ings of the SIGCHI conference on human factors in Batching for eficiency without overreliance, Proc. computing systems, 2010, pp. 1185–1194. ACM Hum.-Comput. Interact. 5 (2021). URL: https: [33] A. Karatzoglou, L. Baltrunas, Y. Shi, Learning to //doi.org/10.1145/3449163. doi:10.1145/3449163. rank for recommender systems, in: Proceedings of [21] S. Anjum, A. Verma, B. Dang, D. Gurari, Exploring the 7th ACM Conference on Recommender Systems, the use of deep learning with crowdsourcing to 2013, pp. 493–494. annotate images, Human Computation 8 (2021) [34] C. Szegedy, V. Vanhoucke, S. Iofe, J. Shlens, Z. Wo76–106. doi:10.15346/hc.v8i2.121. jna, Rethinking the inception architecture for [22] I. Arous, J. Yang, M. Khayati, P. Cudré-Mauroux, computer vision, 2015. URL: https://arxiv.org/abs/ OpenCrowd: A Human-AI Collaborative Approach 1512.00567. doi:10.48550/ARXIV.1512.00567. for Finding Social Influencers via Open-Ended An- [35] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. swers Aggregation, Association for Computing Ma- Gemmeke, A. Jansen, R. C. Moore, M. Plakal, chinery, New York, NY, USA, 2020, p. 1851–1862. D. Platt, R. A. Saurous, B. Seybold, M. Slaney, URL: https://doi.org/10.1145/3366423.3380254. R. J. Weiss, K. W. Wilson, CNN architec[23] J. W. Vaughan, Making better use of the crowd: tures for large-scale audio classification, CoRR How crowdsourcing can advance machine learning abs/1609.09430 (2016). URL: http://arxiv.org/abs/ research, Journal of Machine Learning Research 1609.09430. arXiv:1609.09430. 18 (2018) 1–46. URL: http://jmlr.org/papers/v18/17- [36] J.-B. Alayrac, A. Recasens, R. Schneider, 234.html. R. Arandjelović, J. Ramapuram, J. De Fauw, [24] B. Settles, Active Learning Literature Survey, Com- L. Smaira, S. Dieleman, A. Zisserman, Selfputer Sciences Technical Report 1648, University of supervised multimodal versatile networks, Wisconsin–Madison, 2009. 2020. URL: https://arxiv.org/abs/2006.16228. [25] Y. Yang, Z. Ma, F. Nie, X. Chang, A. G. Hauptmann, doi:10.48550/ARXIV.2006.16228.
Multi-class active learning by uncertainty sampling [37] C. Vondrick, D. Ramanan, D. Patterson, Eficiently with diversity maximization, International Journal scaling up video annotation with crowdsourced of Computer Vision 113 (2015) 113–127. marketplaces, in: European Conference on Com[26] B. Ghai, Q. V. Liao, Y. Zhang, R. Bellamy, K. Mueller, puter Vision, Springer, 2010, pp. 610–623.
Explainable active learning (xal): An empirical [38] I. Portugal, P. Alencar, D. Cowan, The use of mastudy of how local explanations impact annota- chine learning algorithms in recommender systems: tor experience, 2020. URL: https://arxiv.org/abs/ A systematic review, Expert Systems with Applica2001.09219. doi:10.48550/ARXIV.2001.09219. tions 97 (2018) 205–227. [27] M. J. Eppler, J. Mengis, The concept of information [39] K. Swearingen, R. Sinha, Interaction design for overload-a review of literature from organization recommender systems, in: Designing Interactive science, accounting, marketing, mis, and related Systems, volume 6, Citeseer, 2002, pp. 312–334. disciplines (2004), Kommunikationsmanagement [40] L. J. Skitka, K. L. Mosier, M. Burdick, Does automaim Wandel (2008) 271–305. tion bias decision-making?, International Journal [28] Y.-C. Chen, R.-A. Shang, C.-Y. Kao, The efects of in- of Human-Computer Studies 51 (1999) 991–1006. [41] J. McInerney, B. Lacker, S. Hansen, K. Higley,
H. Bouchard, A. Gruson, R. Mehrotra, Explore, exploit, and explain: personalizing explainable recommendations with bandits, in: Proceedings of the 12th ACM conference on recommender systems, 2018, pp. 31–39.