Bullying and cyberbullying are phenomena which, due to their growing difusion, have become a real social emergency. In this context, artificial intelligence can be a powerful weapon to identify episodes of violence and fight bullying both in the virtual and in the real world. Through machine learning, it is possible to detect the language patterns used by bullies and their victims and develop rules to detect cyberbullying content automatically. The BullyBuster project merges the know-how of four interdisciplinary research groups to develop a framework useful for maintaining psycho-physical well-being in educational contexts.
addressing bullying and cyberbullying. The BB approach integrates cutting-edge AI techniques with psychological Bullying and cyberbullying are pervasive social issues models to create tools that can efectively assess the risk that adversely afect the well-being and mental health of of perpetuating, assisting, or sufering from bullying and millions of children adolescents and adults worldwide. violence in both physical and digital environments. The These harmful behaviors impact the immediate victims BB research group has gained significant experience durand contribute to a toxic environment in educational ing the project, and their preliminary results have earned institutions, workplaces, and online spaces. With the them recognition, including selection for inclusion in the growing reliance on digital platforms for communica- “Maker Faire European Edition 10th Anniversary Book” tion and socialization, the need to develop innovative and selection as a promising project by the “Research solutions to identify, prevent, and address bullying and Centre on Artificial Intelligence under the auspices of cyberbullying has become increasingly critical. UNESCO” (IRCAI) in the Global Top 100 list of AI projects
In this paper, we present the “BullyBuster - A frame- addressing the 17 United Nations Strategic Development work for bullying and cyberbullying action detection Goals1. by computer vision and artificial intelligence methods and algorithms” project (BB), funded under the tender relating to Projects of Relevant National Interest (PRIN) 2. The BullyBuster framework in 2017, which involves four multidisciplinary research groups belonging to four universities in Southern Italy (University of Bari Aldo Moro, University of Cagliari, University of Foggia, University of Naples Federico II).
This interdisciplinary project combines artificial intelligence, technology, law, and psychology expertise to develop a comprehensive framework for detecting and
the BullyBuster architectural framework, as depicted in Figure 2. The BullyBuster project’s (Figure 1) purpose is to integrate behavioural biometrics, content analysis and crowd analysis into computer vision systems in order to detect all of the behaviours and indications that might define a bullying occurrence. The system, in particular, may be separated into four modules: (1) a module for crowd analysis for identifying through video-surveillance
bullying-and-cyberbullying-action-detection-by-computer-visionand-artificial-intelligence-methods-and-algorithms/ camera potential bullying incidents; (2) a text analysis module, that analyze textual content for the detection of signs of bullying or cyberbullying, (3) a third module of keystroke dynamics analysis to assess a potential victim’s emotional state; and (4) a module of deepfake detection, to prevent the spread of malicious multimedia content on social networks and chatlines. These modules have been designed based on the psychological modeling of the behavior of bullies, victims, and spectators. The proposed solutions have been studied from a legal and juridical point of view to resolve the crucial aspects of privacy and data protection.
Bullying is a repeated, aggressive and anti-social
behaviour of one or more individuals against a specific
victim. It can have severe and long-term consequences
for the victim’s mental, emotional, and physical
wellbeing [
and social environment that discourages such behaviour and difuses a mental and cultural attitude that emphasizes diversity and tolerance as a mean of mutual enrichment thanks to the eforts of the Ministry of Education, University and Research. The cyberbullying detection is in charge of Ministry of Justice as it was recently acknowledged as a crime by the Italian Law no. 71 from June, 18th, 20172. This law oficially described the criminal action of “cyberbullying” and the way (rules, investigative tools, burden of proof) to prevent and counter it. Moreover, it has been promoted the National Cyberbullying Observatory (http://www.cyberbullismo.com/), by which it is possible to report these criminal acts. In particular, the smartphone application named “Youpol” was implemented thanks to the eforts of Ministry of Internal Afairs and released for public use. This app allows reporting several criminal actions, including bullying and cyberbullying. The legal basis of the research is the art. 9, lett. j) GDPR: 1. [. . . ] processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, shall be prohibited. 2. Paragraph 1 shall not apply if one of the following applies: [. . . ] j. processing is necessary for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1) based on Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject.
From the privacy point of view, the Garante per la protezione dei dati personali3 , the italian authority responsible that people’s privacy is not violated, recently allowed the installation of video surveillance cameras in the Istituto Galileo Ferraris in Verona. Furthermore, in Italy, Figure 3: Anomaly detection pipeline. (1) A subset of the with Law No. 71, May, 29th, 2017, the use of “special in- total frames is selected from the whole sequence of frames; vestigative tools”, without identifying people, is admitted. (2) Low-level features are extracted to obtain the number of It represents a considerable motivation to elaborate by groups in each scene; (3) High-level features compute statistics technicians new forms of electronic evidence to meet as of dynamics patterns; (4) Anomalies are obtained through soon as possible the legislative needs. thresholding a specific pattern.
Understanding human behaviours and actions through images and videos can be highly beneficial in the fight against bullying. Physical violence, isolation, or other physical patterns such as encirclement are among the behavioral “indicators” that can signify the presence of a problem, both as a victim and as a bully. Based on these psychological models, designing a computer vision system to identify and report anomalous occurrences attributed to bullying is possible. With this purpose, we designed a novel descriptor for crowd behavior analysis and anomaly detection [4]. It enables the observation of groups of persons that are not individually identifiable but can provide enough information to detect violence Figure 4: Demonstrator of the crowd analyzer built into the or panic. The created technique, inspired by the concept BullyBuster framework. The system detects suspicious beof the one-dimensional Local Binary Pattern algorithm haviour and reports it to a human operator. (1D-LBP) [5], aims to evaluate the speed of creation and disintegration of crowd groups using appropriate patterns. The number of groups observed in a time interval images, videos, etc.) to cause harm to another person, can discriminate an ordinary scene from an abnormal referred to as a victim. The following definition is very one. We hypothesize that abrupt changes in the number close to cyberbullying, which is “an intentional aggressive of groups are caused by an anomalous occurrence, which act by an individual or group of individuals, using elecmay be recognized by translating these variations onto tronic forms of contact, repeated over time against a victim a temporal sequence of strings, which will be consider- who cannot easily defend himself or herself” [6]. Unlike ably diferent from those corresponding to a condition standard forms of bullying, Cyberbullying is very danwithout anomalies (3). Through these algorithms, we gerous because it can be perpetrated anywhere, anytime, have developed a detector that analyzes a video sequence and victims cannot often stop or reduce the spread of and returns the probability of anomaly in the scene. The these activities [7]. system alerts the human operator when this anomaly This project branch focuses on cyber aggression (texexceeds a certain threshold, as shown in Figure 4. tual or verbal), starting from textual comments of various post-Italians on Twitter through innovative steps of Fea2.4. Verbal abuse detection ture Engineering, including slang and modern and closeto-aggressive speaking methods. According to the preCyberbullying is a widespread phenomenon that gener- viously reported definition, Cyberbullying is a repeated ally includes discrimination, aggression, and harassment. cyber-attack by one or more users against one or more This phenomenon has become more widespread due to specific victims, repeated in multiple posts over time. new communication channels allowing people to send An example of victims of these attacks are “celebrities” and receive messages worldwide. The definition of cyber- who are constantly attacked on their public profiles. In bullying has, among its consequences, cyber aggression. addition, actual conflicts arise from these aggressive comCyber aggression is defined as aggressive behavior exer- ments between users with diferent thoughts. The goal cised on the Internet that uses digital media content (text, was to search for recurring patterns in the wording of sentences or types of attacks. First, we focused on the type of language used by the attackers, noting that in 3http://www.garanteprivacy.it/web/guest/home/docweb/-/docwebdisplay/docweb/1651744 • Presence of bullying terms (KW). A Boolean value indicates cyberbullying keywords (e.g., idiot, stupid, jerk, clown, whale, trash, ...). In this case, a vocabulary containing 359 terms identified as insults and possible insults was created. • Comment length (L). This feature represents the length of the comment in terms of words. It has been noted that most negative comments consist of only a few words, usually no more than three.
The innovation related to this branch is Feature Engineering, dictionary creation, and a Dataset of aggressive and non-aggressive comments. The models used and tested are basic Shallow learning approaches to testing features[13]. 2.5. Stress detection almost all cases, it turns out to be approximate, full of expressions and words belonging to a vulgar jargon that leaves little room for misunderstanding; this led to thinking about the possibility of working with a list of these words, to recognize them in a comment. Next, it was noted how to measure a word’s weight in a sentence, both negative and positive. Again, it was noticed that some negative comments were written in capitals as if to simulate a higher tone of voice. This led to thinking of a way to keep track of this peculiarity. Finally, another feature highlighted was the presence of the word “no/not” (“not” in English) in the aggressive comments; in many cases, it was noted that the aggression sessions began with this word to contradict the victim. All these observations were the basis for determining the right features to extract in the preprocessing phase. This study was conducted through Twitter comments in Italian, a language little used in the current state-of-the-art studies.
The Feature Engineering created is considered because they have been studied and analyzed. In contrast, others were considered because they have been exploited in some state-of-the-art studies, namely: Number Negative words (BW), Number of “not/not” (NN), Uppercase (U), Positive/Negative weight of the comment (PW/NW), Use of the second person (SP), Presence of threats (TR), Presence of bullying terms (KW), Comment length (L).
More specifically, based on the feature, it was deemed appropriate to create a dictionary as well: In the context of (cyber)bullying detection and prevention, there is a growing need to identify and prevent negative emotional states of users while they engage in online communication. Afective computing, the ability to recognize users’ emotional states, has been an ambition in this field for some time. However, current technologies are either expensive or obtrusive, making them impractical for widespread use. An alternative solution is Keystroke Dynamics (KD), which uses behavioural biometrics to identify or verify a person’s identity by analyzing habitual rhythm patterns they use while typing • Number of negative words (BW). Through a “Bad- on a keyboard.
Word” vocabulary containing 540 extremely vul- On this line, we focused on the application of KD to
gar negative words used for aggressive purposes, continuously anticipate users’ emotional states during
insults, and humiliation [8]. message-composing sessions [14]. In particular, in the
• Number of “not/not” (NN). Using “no/not” within study we introduced a time-windowing approach that
ala sentence completely changes the sentence’s lows for the analysis of users’ writing sessions in various
meaning from positive to negative or vice versa. batches, even when the writing window under
consideration is quite brief. This approach is particularly relevant
• Uppercase (U). This is a Boolean value that indi- in the world of social media, where communications are
cates whether the comment is capitalized. It can frequently shared quickly and sparingly. The results
be interpreted as an attack on someone [9]. of the study suggest that even extremely brief writing
• Positive/negative comment weight (PW/NW). This windows (on the order of 30 seconds) are suficient to
recfeature includes two values: the positive and neg- ognize the subject’s emotional state with the same level
ative weight of the comment within the range of accuracy as systems based on the analysis of longer
[
data and determines the presence of deepfakes, violent comments, and stress levels. The outcome is a report that the instructor can consult; it provides the percentage risk of cyberbullying actions for the specific modules and the overall risk for the class. introduction of deep fake technology, which uses deep learning technologies to change the identity of people in images or videos. In the context of cyberbullying, the seriousness of the problem is indicated by the fact that even a single person’s act can spread extensively and be repeated by others, resulting in repetition and an imbalance of power. The problem is worsened by the ability of current mobile devices to generate counterfeit content with a simple application. For these reasons, developing trustworthy algorithms for appropriately classifying videos as real or fake, i.e., deepfake detectors, is critical. Although many deepfake detectors have achieved excellent accuracy levels, generalization remains a significant challenge. Deepfake detectors, in particular, can identify just the manipulations on which they have been trained. For the BullyBuster deepfake detector, we exploited the complementarity of diferent individual classifiers with appropriate fusion rules to increase the generalization capacity of modern deepfake detection systems [15, 16]. Moreover, we designed a novel deepfake detection approach based on the Discrete Cosine Transform (DCT) representation of manipulated and original images at diferent scaling and compressing levels [17].
ferent use cases: Bullying and cyberbullying are social issues that afect individuals and communities on various levels. They have • The BB questionnaire aims to collect data from broader implications for the overall well-being and safety specializing the BullyBuster automatic tools. The of the social environment because they afect the individstudent watches animated videos illustrating bul- ual’s mental health and disrupt social relationships. The lying and cyberbullying before completing the BullyBuster project presents a comprehensive, multidisquestionnaire. We chose questions based on psy- ciplinary approach to addressing the pervasive of these chological analyses to assess how much a per- issues. The framework addresses many aspects of these son’s actions in real life and on the internet put phenomena in both physical and digital environments by them at risk of perpetuating, helping, or sufer- combining cutting-edge artificial intelligence algorithms, ing bullying and violence. The development of computer vision, and psychological models. The partthe questionnaire and data collection was car- nership of four Southern Italian universities resulted in ried out in compliance with the aspects of pri- a comprehensive system that includes crowd analysis, vacy and ethics. text analysis, keyboard dynamics, and deepfake detection • The “Teacher Tool” enables the teacher to trans- modules. The BullyBuster framework has demonstrated fer data obtained from the class chat and videos efectiveness in recognizing and preventing bullying and retrieved from the video surveillance system to cyberbullying events. Its modular design enables ongoa desktop application and evaluate them using a ing improvement and adaptation to new challenges and deepfake detector, a text analyzer, and a crowd technical advances. Furthermore, the legal and juridical analyzer. The system generates a report that in- component of the project assures that the developed socludes the risk percentages of bullying/cyberbul- lutions comply with privacy and data protection rules, lying behaviors for each module and the overall making the BullyBuster framework a feasible and ethirisk percentage for the class. cal solution for educational institutions, workplaces, and • The “Guided discussion tool” requires students to online platforms.
utilize the Bullybuster chat software on the desktop machines in the school’s computer room to discuss an assigned topic (environment, politics, current events, etc.). The system analyzes chat
ucation, University and Research (MIUR) within the PRIN2017 - BullyBuster - A framework for bullying and cyberbullying action detection by computer vision and artificial intelligence methods and algorithms (CUP: F74I19000370001). The project has been included in the Global Top 100 list of AI projects addressing the 17 UNSDGs (United Nations Strategic Development Goals) by the International Research Center for Artificial Intelligence under the auspices of UNESCO.