Introduction

The widespread use of social media in recent years has helped to spread many types of dangerous cybercrimes. Fraud, swearing, harassment, terrorism, bullying, and drug dealing are examples of many cybercrimes that people have been suffering from in recent years. Cyberbullying is common among adolescents and has negative effects on the psychological and even physical state of the victim. Therefore, focus has recently turned to developing solutions capable of automatically detecting and predicting cybercrimes based mainly on artificial intelligence and natural language processing (NLP).

It is obvious that the domain of NLP is developing very well for some languages, such as English. Yet, for the Arabic language, there are only some experimental attempts that are issued here and there, which remain insufficient. In addition, the Arabic language poses another kind of difficulty related to the way it is written. Also, there is no unified language among the Arab countries, as each country uses its own dialect.

In recent years, researchers have proposed very effective machine learning-based solutions to detect cyberbullying by processing published texts, images, and videos. In this study, we are only interested in methods based on NLP. Arabic cyberbullying detection using sentiment analysis was proposed by Almutiry and Abdel Fattah [1]. Abusive language detection on Arabic tweets based on matching texts with a list of obscene words was presented by Mubarak et al. [2]. Aldjanabi et al. [3] developed a classification system for determining offensive and hate speech using a multi-task learning (MTL) model built on top of a pre-trained Arabic language model. Fatemah Husain [4] applied a single learner machine learning approach and an ensemble machine learning approach for offensive language detection in the Arabic language.

This article presents an approach to solving the problem of detecting cyberbullying through text analysis using a dynamic graph neural network (DGNN). The proposed method is a graphlevel classification to classify an entire comment session, whether it contains cyberbullying or not. These sessions are a set of comments where each relationship between two comments is a response relationship between the two. Each node in the graph is a comment, and an edge between them is a reply link. The temporal aspect of the data must be taken into consideration because texts are time-series data. The size of comments varies and is not fixed, so the type of time graph that will be used is dynamic time graphs. According to our modest research, the majority of methods on time graphs assume that graphs are static and often treat them as static [28]. We propose a method that takes into account the dynamism of time graphs with a shift method for processing temporal and spatial aspects.

The remainder of this article is organised as follows. Section 2 summarises the state of the art of the proposed solutions to detect cyberbullying using artificial intelligence methods. Section 3 explains in detail the approach proposed in this article. Section 4 summarises the conducted experiments and the obtained results. This article is finished with conclusions on the most relevant observations and results.

Related works

Several researchers have tried to handle cyberbullying detection using machine learning algorithms. To detect aggressive behaviours, the work by [5] employs three deep learning models: Closed Bi-directional Recurrent Unit (BiGRU), Transformers, and Convolutional Neural Networks (CNN). Experimental studies were conducted to examine how effective the models are at classifying well-known hate speech Twitter data-sets into aggressive and non-aggressive. An accuracy of about 88% was reached.

The authors proposed by [6] another method based on supervised machine learning approaches. Several classifiers have been used to train and detect acts of bullying. The data set shows that neural networks perform better and achieve an accuracy of 92.8%, whereas simple vector machines (SVM) achieve 90.3%. Reynolds et al. [7] used textual features and constructed several metrics to assess swear words through a swear word dictionary. What makes cyberbullying detection particularly challenging compared to offline bullying is that language can pose difficulties like ambiguity and sarcasm. Role prediction can also be a difficult task if we look at a conversation thread between the individuals involved in the conversation (victim, bully, bully type, advocate, etc.). The predictions made by the previous automatic methods for the detection of cyberbullying do not reflect the complexity of the task described above. Recent efforts include combined word normalisation, named entity recognition to detect person-specific references [8]. In their work, [9] collected a large data set from ask.fm and used BOW (Bag Of Words) features as a first test, then extended it with term lists, subjectivity lexicons, and subject model features.

Recently, neural word embedding and neural network techniques have been applied. Convolutional neural networks (CNN) on phonetic features have been applied by [10] and [11] to study, among others, the same architecture on textual features in combination with long-short-term memory (LSTM) networks.

People tend to describe their experience of bullying in messages called bullying traces. The researchers by [12] presented a method to examine bullying traces. They identify several key issues in using social media to study bullying: text categorisation, role labelling, and topic modelling.

Using graphs to visualise potential cyberbullies and their connections is a recent method. The researchers by [13] concretely implemented techniques from graph theory: Network functionalities (degree of centrality: number of incident links from a node; close centrality: average distance the shorter from each vertex to the other vertex.), Content-based features (length, sentiment, offensive words, pronouns).

The focus of most existing work done on cyberbullying is the independent content analysis of comments within social media sessions. Suyu et al. [14] cited three different key limitations for detection of cyberbullying, 1) only consider the content within a single comment rather than the topic coherence across comments 2) remain generic and exploit limited interactions between social media users, 3) overlook the temporal correlations among different comments. They showed that interaction within the same session may evolve. Modelling topic coherence between comments and temporal interactions is critical to capturing repetitive bullying characteristics. leading to better predictive performance. To achieve this goal they constructed a unified temporal graph for each social media session and then proposed a principled graph-based approach for modelling the temporal dynamics and topic coherence throughout user interactions.

Research in psychology and social science has shown that cyberbullying is carried out repeatedly against victims [15]. Studies of user interactions allow us to characterise their repetition by both content and temporal analysis. However, modelling interaction in social media sessions presents some challenges: sparsity, repetition, and user characteristics.

The proposed approach

Our approach is based on dynamic graph neural networks. This part explains every step of the procedure. It is composed of: data collection, graph construction and how constraints are handled, coordination matrix creation, tokenisation and training the model.

Graphs generation

Data generation was the hardest task and the process that took us the most time. Understanding the data structure is also an important factor for data generation. In our case, data generation is a process based mainly on dynamic temporal graph creation.

Data collect

With some research and exploitation, we were able to collect 11268 comments, of which 8417 are negative instances that do not contain cyberbullying behaviour and 2851 positive instances that do contain cyberbullying behaviour. The links to the databases used were shared by [16].

Construction of graphs

The construction of temporal graphs consists of understanding our data structure as well as the two types of temporal graphs.

Structure of a session

The sessions will be represented by temporal graphs where each node represents a comment and the edges between them a response relationship. At the beginning (step 𝑡 = 0), the graph has nodes representing tokens corresponding to the 𝑖 𝑡ℎ comment. Each session is a dynamic temporal graph, dynamic temporal graphs are records over time. The records are graphs at each time step, with the 𝑖 𝑡ℎ node representing the token of the 𝑖𝑡ℎ comment at time step 𝑡. Except the size of comments is not fixed, so some nodes and edges will be removed over time.

There are two constraints that describe the structure we want to use for the sessions we are going to process:

• edge loop. Each node in our graph is a comment and not a user. If a comment 𝑗 is a reply to a comment 𝑖, and we add a cycle edge (i.e) comment 𝑖 is also a reply to 𝑗, this just doesn't make sense. • session owner. The session owner or the owner of a post is often the person bombarded with comments and insults.

Constraint of dynamic temporal graphs creation

• Node Deletion. nodes must be deleted only if the comment that corresponds to the node is less than the current time step, if the comment contains 10 tokens then the node will be deleted from the graph in the 11 𝑡ℎ time step.

• Edge removal. Edges should be removed only if one of the nodes linked to the edge is removed.

Removing and adding nodes is important for processing the temporal aspect of the graph. The deletion of the edges is done if a node linked to it is deleted, because no spatial data is propagated in the network. • Number of nodes equal to the number of nodes in the COO matrix. The COO matrix (CO-Ordination matrix) is a an optimised representation of the edges in the graph that consumes less space and reduces the number of units of calculation.

The matrix contains two rows. In each column the first row contains the sender nodes and the second row the receiver nodes. The union of the two lines gives us all the nodes of the matrix. If the set of nodes in the COO matrix is greater than the set of nodes in the node embedding matrix, it means that the embedding matrix is missing information about some nodes. This will make message passing impossible. • Isolated node. Isolated nodes cause problems because they do not belong to the set of nodes in the COO matrix. The absence of a node in the COO matrix puts us in a case where we do not know if the node has been deleted or is isolated. Deleting a node causes an edge to be deleted and deleting an edge causes a node to be isolated. • Shifted node embedding. A node is added to the graph at time 𝑡 if and only if the length of the comment which corresponds to the node is greater than 𝑡. If between [0 − 𝑡 − 1], some nodes have been deleted. Deleting these nodes will cause a shift in the order of the nodes in the node embedding matrix. This means that the embedding of node 14 can actually be the embedding of node 18.

Graph creation

• tokenisation: The Transformers library [17] offers an entire hub where multiple developers can put their algorithm to use. For the segmentation of Arabic words, we use the segmenter [18] and for the tokenisation algorithm we use a WordPiece algorithm developed by [19] for the Arabic language. WordPiece was originally designed by Google when designing the famous BERT model. The tokenisation is performed on the entire data-set. When creating temporal graphs, for each node (𝑖 for example) that we add to a graph in a time 𝑡, we add the identifier of the 𝑤 𝑖 token of the comment 𝑖 in the embedding matrix. These identifiers will then be passed to an embedding layer which as output gives us an embedding vector (the dimension of the vector is a hyper-parameter). • Coordination matrix (COO-matrix). We start by picking a random number to determine the length of the session (number of comments or node in the session).

In the first time step 𝑡 = 0 we build the first graph where each node 𝑖 will pick a random number 𝑒 ∈ [1, 𝑠𝑖𝑧𝑒 𝑠 𝑒𝑠𝑠𝑖𝑜𝑛], 𝑒 is the node that will be connected to node𝑖. This will build the COO matrix which represents the edges and structure of the graph. We are still in the first step in time. To solve the isolated node problem, we add a self loop connection to the isolated node in the CO-Ordination matrix. Both the sender and the receiver are the isolated node and the weight of the edge is null.

To solve all the rest of the constraints for the creation of the dynamic temporal graph mentioned above, we proposed shifting the indices of nodes. For each step in time, we save a COO matrix with the new structure (the nodes not deleted) and another COO matrix where the indexes of the nodes are shifted backwards.

In the case of spatial data processing, we can use the nodes shifted edge indices, because the indices will be reset and this will prevent us from falling in the case where a node index is greater than the number of nodes in the embedding matrix.

In the case of time series data processing, we use the original indices to check if the node is not deleted and the shifted edge indices matrix to select the embedding vector of nodes for processing their temporal information. Finally, we were able to generate 753 dynamic temporal graphs from which 191 contain cyberbullying behaviour and 562 do not.

Model training

Spacial data processing

Generally, message passing is done through a dot product between the adjacency matrix and the node embedding. We have a CO-Ordination matrix which consists of two rows and in each column the two connected nodes. How can the COO matrix be used for message passing?

The embedding matrix is created using an embedding layer that takes the token identifiers of the nodes at time step 𝑡 as input. The embedding dimension we chose is 𝑑𝑖𝑚 = 128. We create a copy of the embedding matrix and we update the vectors of the nodes which receive information from their neighbours in the embedding matrix. This can be achieved as follows:

For each column in the shifted edge indices CO-Ordination matrix, we update the embedding vector of the receiver node in the new embedding matrix. Example the first column is: 𝑠ℎ𝑖𝑓 𝑡𝑒𝑑 𝑒 𝑑𝑔𝑒 𝑖 𝑛𝑑𝑖𝑐𝑒𝑠 ..,1 = [1, 3], in the new node embedding we will use the vectors in the position 1 and 3 to update the vector in the position 3. And if the vector 3 has already been updated, we use the vectors in position 1 and the new vector in position 3. This procedure must be done for all the columns in the shifted edge indices COO matrix. The result are weighted by parameterised readable weight matrix to learn the interactions (the number of layers we have chosen is three.). After applying these transformations to the graph in the first time step, we will do the same for all the graphs in the future time steps individually. This way, all spatial information about neighbouring nodes having been propagated through the graph in all time steps, the resulting node embedding can be used as input data for processing the temporal aspect.

Handling temporal aspect of graphs

After applying message passing for each node in each graph over time, our node embedding all contain information about their neighbouring nodes. It was previously mentioned that deletion leads to an embedding lag which can cause problems for handling the dynamic graph temporal aspect. If the nodes will be shifted then the time series data processing will learn corrupted and incorrect sequences. To solve this problem, we shifted the indices of the CO-Ordination matrix so that Deep Learning architectures used for processing time-series data learn the correct sequences. This is achieved as follows:

We start by concatenating all the records of a node through time. This will prepare the input data for the chosen Deep Learning architecture and will make processing temporal data very easy. To concatenate the vectors of the node through time, we have to loop over the time steps and check if the node has not been deleted from the COO matrix (this will indicate that the number of comment tokens is greater than 𝑡) and we have the shifted index of the node so we can extract the node embedding vector from the embedding matrix at time 𝑡, concatenate it with the node embedding vector at time 𝑡 − 1. We will have a representation of all the records over time in a matrix that will be used as input data for an LSTM layer. This concatenation operation will be done for all the nodes of the session.

An LSTM architecture will be used to process the temporal aspect of the data, the result of each node that passes through the LSTM will also be concatenated. Then we calculate the transpose the resulting matrix and get the average on the last dimension for shape reduction. A finale output layer is added to predict if the session contains cyberbullying or not.

Experimental results

In order to verify the effectiveness of the proposed approach, we conducted experimentation using the dataset. The input data contains 11268 Arabic comments (tweets), where 8417 do not contain cyberbullying and 2851 do contain cyberbullying. From these comments, 753 dynamic temporal graphs were generated, of which 191 contained cyberbullying and 562 did not.

The training results of the first epoch were 49% but we saw improvements in the second epoch with an accuracy of 74% which means that the learning took place and the proposed techniques applied to the CO-Ordination matrix worked. We were not able to continue the training for more than 2 epochs due to OOM (Out Of Memory) reasons. The temporal graphs are very memory-consuming data structures. We believed that the imbalance and the size of the database reduced the performance. The model that we have chosen is a prototype model to test the efficiency and profitability of the shifted edge indices COO matrix that we proposed.

We could have used regularisation technique for generalisation or scheduling the learning rate for better optimisation, but performance of the model was not our goal.

We tested on the same set that we used for training and we noticed that there was an extreme bias in the model predictions. This is not only because of the imbalance in the dataset, where almost 75% of the examples are negative examples, but also because of the small size of the dataset. From this confusion matrix, we calculated the following micro metrics (positive predictive rate, recall, specificity, negative predictive rate, precision) and the f1 score as a macro metric. The results are summarised in Table 2. Knowing that comments are text in nature and responses between one comment and another can be structured with a temporal graph gives us the ability to harness the power of dynamic temporal graphs. With different message-passing algorithms available, GNNs capture the relationships between nodes and learn to properly represent a graph in node embedding that represents each node and the information acquired from their neighbours. And with different deep learning architectures that process time-series data, we can learn how these node interactions change over time. We leverage the dynamism of the dynamic temporal graphs with a bunch of techniques that prove the learning has taken place despite the complex structure of the dynamic temporal graph being kept.

Conclusion

Cyberbullying is a dangerous cybercrime that is lightly taken by governments despite the psychological damage it causes to its victims. It has spread very rapidly due to the spread of social media, which has caused an increase in the number of bullies that has led to a permanent cycle of violence.

Session classification seems more applicable for cyberbullying detection. Graph neural networks (GNN) make it possible to capture the relationships between entities, and by adding an architecture that deals with the temporal aspect of graphs, we have the ability to model relationships between entities over time. We decided to leverage dynamic temporal graphs because there is little research on this data structure, and hence the majority of research considers these graphs to be static temporal graphs. When turning a dynamic temporal graph into a static temporal graph, we think that the algorithm will waste a lot of training steps learning insignificant alternatives that we added to make the structure static. Therefore, we have proposed a technique that uses the dynamism of this structure. The method proved that learning took place. Unfortunately, we did not have the necessary means to properly test the performance of the method.