These days we face more and more often internet bullying. Our goal was to develop software, which would recognize emotions from the bare text. Our project is based on Twitter posts, but it could be also used in every single platform, in which users communicate via text messages. We use a few solutions to make our program as accurate as it possibly could get. Firstly we picked a large database to get the biggest context, We used World2Vec to represent words as vectors, and lastly, we used a neural network to predict output from sentences beyond our database. Our article is mostly about diferent versions of the algorithm and comparison to choose the best approach to the problem. As we learned the biggest diferencemaker was the amount of hidden layers and number of neurons inside each one of them, type of activation function, and training algorithm. We attach a big amount of plots to visualize each of our tries. In our article, we will try to show our approaches and data which is connected to those approaches. We created functions to monitor our error, the accuracy function to sum up our algorithm - how eficient it is, precision function - to diagnose what proportion of identifications was correct, recall - a fraction of relevant instances that were retrieved and f1 which combines precision and recall to make an average of it valued from 0 to 1.
or data which possibly could make our algorithm less
reliable. Therefore we cleared it out of stuf like Names,
The assumption of our project was to create an algo- links, and mentions.
rithm based on an artificial neural network. Its main Next algorithm used in our program is Word2Vec
goal was to recognize whether the entry is neutral, neg- which is responsible for translating our sentences and
ative, or positive. The algorithm is learning on a base words into numbers. Every word is represented by a
that contains 1.6m tweets using backpropagation algo- 10-dimensional vector. The algorithm which stands
berithm. We decided to use neural network as classifiers as hind word2Vec is nothing else than an artificial neural
they have been reported in various interesting applica- network, which would be explained later. Because of the
tions [
In [
We got our database from Kaggle but it was full of run and addlayer, which are responsible for adding layunnecessary data like date or user. We cleared it and left ers and running the whole algorithm. The run function only 2 columns - target and text, we got rid of columns returns 2 output neurons, which represent, by using softthat contained information like date, the user, or tweet max, for the probability of label. The first neuron is the id. possibility of positive output and the second one for nega
To make our algorithm work we needed to divide it tive. We also add a function that check the absolute value into few subsections. The first of them is a section con- of its diference, if it’s small enough then the output is nected to a database. Firstly, as mentioned before, we equal to neutral. The artificial network includes the input dropped most of the columns, but secondly, we needed layer with 2800 neurons, a first hidden layer with 600 to make sure that our data is not containing unused data neurons, a second hidden layer with 200 neurons, third hidden layer with 20 neurons, and output layer which consists of 2 neurons. Our database consists of 1 600 000 tweets, each item is represented by 5 columns. One record includes the date of the tweet, posting users nickname, the id of a tweet, neural network and it allows us to make mathematical the content of the tweet, and the label - if it’s positive or operations on words. Gensims Wor2Vec implements two negative. We needed to modify our database to contain functions - Continuous Bag Of Words(CBOW) and Skiponly 2 of 5 columns because only text and labels will be Gram. used. Except that we needed to adapt our database to our In the CBOW model surrounding words is combined to use and clear it of meaningless text. predict the word they surround, while in the Skip-Gram
So firstly we loaded the database using ’pandas’ and we use a word to predict the context. provided our data frame with labels to make the access easier. Also, we’ve implemented a function whose main task was to clear any irrelevant text, such as pronouns, 5. Mathematical representation of conjunctions, links, and mentions, to make sure our al- Skip-Gram model gorithm will learn properly.
Our algorithms inputs are sentences that in the next steps are converted into words. Using Word2Vec each word in the sentence is converted into ten-dimensional vectors and those vectors are inserted into the array, each vector as a separate element of our array. Those words are easily available because of two mechanisms, word to id and id to word. Now using an artificial network, weighted sum and activation function the algorithm is filling every single neuron with proper values. In the output layer, we have 2 neurons that, at the end of the algorithm, return two values between 0 and 1. Because of softmax, those values can be identified as the probability of each label. When learning those two values are collated with expected outcomes. That way we get the distance between our result and the real label and we used that values in the algorithm of backward propagation to change all of the weights so that our program is getting more and more precise with each iteration.
Mathematically, we can describe n-word sentence 1, . . . , , using skip-gram as following formula: = 1 , 2 , . . . , | ∑︁ − − 1 < =1 (1) where k is max skip-distance and n to subsequence length.
For example, when we have the sentence ”I love to write scripts” and k is equal to 1 and n to 2, that means we will connect 2 words, which have a maximum of one word between them. Those connections would be: {I,love},{I, to},{love, to},{love, write},{to, write},{to, scripts},{write,scripts}.
We are using the gensim library to implement the algorithm of Word2Vec. This part of our code lets us change words in the database to vectors, so they can be used in our calculations. The algorithm as input data takes the whole data frame with all sentences, each row represented as one sentence. Firstly sentences need to be divided into words. Nextly we count how many times each word occurs in the text and based on that information we create 2 dictionaries - word to id and id to word, which would make the conversion from text to id and inversely easier. In the built-in function, we need to specify the size 7. Activation function of the vector, minimal number of occurrences, window, and source of words. In our example we set minimum Activation function is inseparable element of every aroccurrence to 1, size of vector to 10 and window to 7, tificial neural network. We have lots of them available to make sure our dictionary would be big, to connect but each of them is diferent. We use s-shaped function big amounts word witch each other and also we needed - hiperbolic tangent as our activation function. It deter10-dimensional vector for every word so it would fit our mines the output of artificial neural network. All of the input layer. Word2Vec is nothing else than an artificial output values are between 0 and -1. The advantage of A backward propagation algorithm is used in our program to modify the weights of each neuron to get the best results. Our neurons have pregenerated weights from 0 to 1. To make our algorithm more precise by analyzing the errors backward propagation algorithm correct native weights starting from the end of our artificial neural network. As an input, it takes the probability of each label and the expected label. It calculates the error of each of the output neurons and those errors are propagated to previous layers. Each weight in our network is being modified based on the value of the error. This algorithm has its limit so you need to be careful while setting its iterations. After a few runs values are being modified to a lesser extent, so when those changes are minor, that’s the sign to stop the algorithm. our activation function is mapping, all positive and negative values will be presented as strong values and those which are close to 0 would be close to 0 on the tanh graph.
We also have chosen tanh function because is strongly advised when neural network has only 2 outputs.
Our activation function - Hyperbolic tangent might be represented as: tanh = sinh cosh
Our neural network algorithm is divided into to main classes. NeuralNetwork class and Neuron class.
NeuralNetwork has 2 variables - layers and weight. Which store respectively arrays of Neurons and theirs weights. First funtion is addlayer which was written to allow creating layers with specific number of neurons inside it given as an argument. When used it adds elements of Neuron class into the array of layers. Get size function is used to return number of neurons in whole artificial neural network, thanks to that function we are able to properly use generate weights. With the result of the previous function we use generate weights to create array with randomly generated numbers from 0 to 1. Load weights function is responsible for assigning weights to neurons. The last of them is run, it firstly checks if algorithm has the same amount of input neurons and inputs given by a user. If it returns true it starts to assign values to neurons.
The next class - Neuron, is responsible for calculating
weighted sum and using activation function to assign
values to each neuron.
40–47
(4)
10. Inference
Inference in our algorithm is simply choosing an option
with a higher probability. Thanks to softmax we get on
our output layer two neurons with probabilities for each
label. Firstly we need to convert output as it is in a form
that cannot be compared to label from our database. The
output is a two-dimensional array, where first we got
the probability of positive tweet and the second as the
negative one. So we need to make a variable ’expected’,
so it would be represented as a two-dimensional array.
Next we check if the absolute value of the subtraction is
bigger than 0.1, if it’s not we make our entry neutral. If
one of the values is big enough we assign respectively
the label. As we compare these two values we also
calculate the accuracy of our algorithm. Below we
represent the pseudocode of inference. [H] Input Data:
sentence label k, array of vectors j (sentence represented
as an array of vectors), Choosing the label of sentence
== 0 : = [
absolute = Absolute value of subtraction both output values
absolute < 0.1: Neutral neo [0] < neo [
Additionaly apart from calculating softmax, we need to get its derivative. Its used by backpropagation function during calculating diference between expected values and outputs from our net. We start by separately computing derivatives. First for the first neuron. (1) 1 = 1 (1 + 2 ) · 1 1 · (1 + 2 ) − 1 1
Now for the second one. (2) 1
= so we have: (1) = (1) × (1 − (1)) (8) 2 (1 + 2 ) · 2 1 · (1 + 2 ) − 1 {︃() × (1 − ()) if = − () × ( ) if ̸=
If we are computing
() the output is always () × (1 − ()). However when we are computing
() the output changes to − () × ( ) 12. Precision function Precision is a function which shows the proportion of true positive identifications. If we analyse retrieval of information, precision is fraction of correct results divided by all returned results. We calculate precision from two variables: TP and FP. Which respectively stands for true positive and false positive. By the term true positive we mean outcome where the model correctly predicted positive class and false positive as incorrect prediction of positive class.
Precision is given by the following formula: = (13)
+
When precision rate is equal to 1.0, that means that model produces no false positives.
Example of calculating Precision: = =
+
13. Recall function (10) (11) (12)
Recall function is very similar to precision function. The only diference is that we compare true positive values to false negatives (incorrect predictions - model predict (9) incorrectly negative class). Our model is most eficient when recall factor is 1.0 - that means there are no false negatives.
The equation is also very similar:
= (17)
+
And when we calculate Recall one the same set of data as Precision that’s our outcome:
And when we do the opposite thing, we decrease number of FN and increase number of FP: We got:
Precision ≈ 0.71 Recall ≈ 0.91 When is equal to 2 recall weights are higher than
So we got to conclusion that they are not quite com- precision, however when its equal to 0.5 weights of preparable, but there is another method which uses both of cision are higher than recall. Example of calculating them in the calculations and it’s named F1 score. score 15. F1 score Firstly the name of F1 score, also known as F-measure, is believed to refer to diferent F funcion, which was concluded in Van Rijsbegens Book, when introduced to the Fourth Message Understanding Conference.
F1 score is measurement of test’s accuracy. It is calculated from the recall and precision. F-meause is the harmonic mean (the reciprocal of the arithmetic mean of the reciprocals of the given set of observations) of Precision and Recall. It can be modified by additional weights, valuing precision or recall more than other.
The highest value of F1 score is 1.0 which indicates the best precision and recall and 0 indicates that one of precision or recall is equal to 0.
F-measue is also known as Sørensen–Dice coeficient or Dice similarity coeficient (DSC).
Example of calculating F1 score: 1 = 1 = 2 1 =
2 − 1 + − 1
× × +
+ 12 ( + )
1 = 1 = 10 + 12 (3 + 2) 10 16. Experiments We’ve experimented with • Activation Function • Artificial neural network learning algorithms • Structure of artificial neural network 40–47 (26) (27) (28) (29) read articles that proves that this type of function is the best to neural network with 2 neuron in output layer. Thing that outweigh the decision was its shape. Because of tanh function we are able to easily spot negative values and those which are close to 0.
We also have used softmax function to represent values on our output neurons as probability of each label. Apart from that we have used derivative of soft max in algorithm of back propagation to decrease the error.
We tried Particle Swarm Optimization and backward propagation as our learning algorithms. After reading articles and running some test, we decided to use backward propagation as it was easier to use with softmax and also was more eficient than PSO.
After many tries we ended our tests with 3 hidden layers: first - 600, second 200, third 10, as input and output layer number of neurons is constant - 2800 inputs and 2 outputs. 17. Conclusions In our work we have tested application of neural networks to word processing purposes. We have used special library to work with tweets. Our idea was tested and results show we have good model which is able to work with tweets. in future works we will try to develop furhter our project to make it also compare tweets between various authors. We will also work to apply other models and ideas to compar them to this presented neural network.