This research introduces novel methodologies and tools to combat content evasion in multilingual Natural Language Processing on social networks. A unique Python package, ”pyleetspeak”, is developed, ofering a customizable system for simulating multilingual content evasion through word camouflage techniques. The study also presents a synthetic multilingual dataset of camouflaged words, facilitating the training of models for camouflage detection. In a comparative analysis of various models, the multilingual MPNET-ideal model, pre-trained on an extended mSTSb dataset, outperforms other models in detecting camouflaged content across languages. The research underscores the utility of the tool in improving content moderation, enhancing online security, and serving as a potential data augmentation tool for AI systems. This work constitutes a significant contribution towards combating information disorders on social networks and sets the stage for further research in this field.
CEUR Workshop Proceedings dataset of camouflaged words 2, and a multilingual Transformer-based model3 to identify various word camouflage techniques and prevent content evasion over 20 languages 4. The eficacy of multilingual pre-training in semantic similarity for enhancing such models is also explored.
A novel system for simulating multilingual content evasion through word camouflage
techniques is developed based on literature references [
The “pyleetspeak”1 package also showcases its utility as a data generator, which uses KeyBERT to extract semantically relevant words, apply camouflage methods, and generate data annotated in Spacy format. The data is tagged with four entities representing diferent camouflage methods, and a dictionary detailing parameters applied to each instance ensures process interpretability.
An experimental protocol was designed to address the problem of word camouflage in
multilingual content. The protocol starts with the creation of a synthetic multilingual dataset
from non-camouflaged text data. This dataset, curated from various sources (OPUS
NewsCommentary [
To handle the task of word camouflage detection, a variety of models is employed. These
include paraphrase-multilingual-mpnet-base-v2 (MPNET-base) [
The developed model and the curated dataset are made publicly available for broader research and application. The open accessibility of these resources promotes transparency, encourages reproducibility, and potentially enables further advancements in the field of content evasion detection.
In a research efort to develop the best multilingual NER model for word camouflage detection, the study conducted various experiments and presented impressive findings. The most striking result was the performance of the MPNET-ideal model, a version of the MPNET that was pre-trained using the semantic textual similarity task with a multilingual extended mSTSb dataset. The MPNET-ideal outperformed all other trained multilingual models across most datasets, demonstrating its superiority in word camouflage detection. Specifically, the model exhibited improved performance over the monolingual baseline models, with the most substantial enhancement in Italian language detection where the F1 score went from 0.7061 to 2https://github.com/Huertas97/XX_NER_WordCamouflage 3https://huggingface.co/Huertas97/xx_LeetSpeakNER_mstsb_mpnet 4ar, az, da, de, el, en, es, fi, fr, hu, id, it, kk, nb, ne, nl, pt, ro, ru, sl, sv, tg, tr 0.8913.
The models were also evaluated across diferent camouflage techniques, revealing that detection of inversion camouflage was more challenging compared to punctuation or leetspeak camouflage. The results suggested that the MPNET-ideal multilingual model could accurately detect camouflaged entities across multiple languages and diferent types of text with high precision and recall. It was further demonstrated that the model could efectively diferentiate between diferent camouflage techniques and handle a variety of languages. For instance, the confusion matrices revealed the dificulty of diferentiating “MIX” entities from “LEETSPEAK” or “PUNCT_CAMO” entities due to the mixed elements, but the MPNET-ideal model still performed admirably.
Finally, the research validated the model’s performance using an external tool, AugLy [17]. Though designed for monolingual data augmentation, AugLy could apply transformations that resembled camouflage techniques, making it an apt tool for external validation. The study discovered that the model could accurately detect new camouflage strategies, such as upsidedown letters or emoticons in place of letters. However, it struggled to detect modifications in less semantically meaningful words like articles and pronouns. This shortcoming highlighted the importance of focusing on semantically meaningful words when dealing with camouflage detection. Overall, the MPNET-ideal model’s validation results underlined its impressive capabilities in detecting various camouflage techniques, cementing its position as an efective tool for multilingual word camouflage detection.
To conclude, this research ofers significant insights and practical solutions for addressing content evasion in multilingual Natural Language Processing. The novel tool “pyleetspeak” and the robust multilingual NER camouflage detection model efectively enhance content moderation and improve online security. The tool’s utility extends beyond its immediate application, indicating its potential in data augmentation for AI systems and future expansion to other languages and evasion strategies.
This summary encapsulates the key findings from [ 18] research paper, highlighting the development and utilization of a synthetic multilingual dataset and the Python package “pyleetspeak” for addressing the issue of content evasion in social networks. The original article presents more in-depth insights and discusses the broader impacts of word camouflage on content moderation. This research represents a significant stride towards combating information disorders on social networks and provides a solid foundation for future research in this crucial area. Acknowledgments This research has been supported by the Spanish Ministry of Science and Education under FightDIS (PID2020-117263GB-I00) and XAI-Disinfodemics (PLEC2021-007681) grants, by Comunidad Autónoma de Madrid under S2018/ TCS-4566 (CYNAMON), by BBVA Foundation grants for scientific research teams SARS-CoV-2 and COVID-19 under the grant: ” CIVIC: Intelligent characterisation of the veracity of the information related to COVID-19”, and by IBERIFIER (Iberian Digital Media Research and Fact-Checking Hub), funded by the European Commission under the call CEF-TC-2020-2, grant number 2020-EU-IA-0252. Finally, David Camacho has been supported by the Comunidad Autónoma de Madrid under ”Convenio Plurianual with the Universidad Politécnica de Madrid in the actuation line of Programa de Excelencia para el Profesorado Universitario” M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised Cross-lingual Representation Learning at Scale, 2019. doi:10.48550/ARXIV.1911.02116. [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423. [17] Z. Papakipos, J. Bitton, Augly: Data augmentations for robustness, 2022.
arXiv:2201.06494. [18] Álvaro Huertas-García, A. Martín, J. Huertas-Tato, D. Camacho, Countering malicious content moderation evasion in online social networks: Simulation and detection of word camouflage, Applied Soft Computing 145 (2023) 110552. doi: https://doi.org/10.1016/ j.asoc.2023.110552.