Artificial neural networks, such as feed-forward networks (FFN), convolutional neural networks (CNN), recursive neural network (RNN) are becoming powerful tools that are starting to replace many classical algorithms. It is known that, for image recognition, CNNs are often the best choice in terms of accuracy. In this paper we show that feed forward networks are capable of achieving comparable performance, with less complicated architecture in comparison to CNNs. After presentation of underlying theory of Feed Forward networks, we present diferent methods, that allowed us to get past network local minima, then we show experiments and conclusions that followed.
be done using neural networks [7]. We can also find applications of vision support in virtual reality enterArtificial neural networks such as feed-forward net- tainment systems [8], where neural networks are used works (FFN), convolutional neural networks (CNN), to improve perception. There are also many examples recursive neural network (RNN) are becoming pow- in which neural networks are used for understanding erful tools that are starting to replace many classical context and emotions from movies. In [9] adaptive atalgorithms. It is known that, for image recognition, tention model was used to recognize patterns, while CNNs are often the best choice in terms of accuracy. emotions from clips were detected by neural networks We went with FFN because they are simpler in their [10] or complex neuro-fuzzy systems [11]. structure and with suficient amount of data and heuris- This paper addresses both of those issues. We show tic approach they can achieve comparable performance, that combining heuristic and backpropagation algowhile being easier to implement and manage. rithm allows, for eficient overcoming of ”minima traps”
One of the most efective learning method is back- and methods that help networks to generalize their propagation (BP) algorithm. BP uses gradient calcula- knowledge. First we introduce basic theory of Feedtion to determine ”direction” in which network should Forward Networks, alongside with explanation of back”go”. The downside of strict mathematical approach is propagation algorithm. Then we describe our example that gradient-based methods often ”get stuck” in local model and series of experiments that were performed minima of a function. Another common problem in on the model. At the end of our paper, we present conmany kinds of networks is knowledge generalization; clusion that we have gathered and give performance we often train our models on large data sets, to avoid metrics of our model. ifxation of a network on small set of examples.
We can find many applications of neural networks in image processing. In [1] CNN was adopted to rec- 2. Feed-Forward Networks ognize archaeological sites. We can also find many Theory applications in medicine where such systems extract bacteria [2] or detect respiratory malfunctions or other pathologies [3, 4, 5]. In movie and advertisements field there are also many applications of such ideas. Movie scenes can be segmented by using background information [6] or even prediction about such content can Artificial Neural Network (ANN) is a mathematical model of biological neural networks that compose a human brain; similarly to one, the ANN is built of neurons which are parts of layers. Each neuron from each layer is connected to all neurons from the previous layer and all neurons of the next layer are connected by synapses. These connections have randomly initialized weights, which are being modified in the learning process.
The first layer, responsible for receiving input data, is called input layer. Similarly the last one, which returns output data, is called output layer. There can be by adjusting synaptic weights in strictly defined way.
There are many methods of learning, but the most popular training technique that is used in Feed-forward Neural Network is back-propagation in which modifying weights of the connections goes from output layer to input layer sequentially. This direction is opposite to the way inserted information moves in all FFNs. The goal of training process is to minimize the value of loss function, for all elements included in training set (T).
The training set consists of input vectors, which are inserted to first layer by input synapses, and expected output vectors, which are compared with gained outputs every time neural network is fed. The loss function shows the distance between predicted value and the actual one. To calculate it we can use Mean Square Error Loss or Cross-Entropy Loss (also known as Log Loss). Using MSE, total error of training set can be described with equation: most commonly used is bipolar linear function. It’s equation is:
( ) =
2 1 + − − 1 = 1 − − 1 + − vergence rate) of activation function. where is a coeficient that influences ”width” (conit is time to teach it object recognition. ANN learns
After artificial neural network has been properly built, tial of ℎ layer is: zero, one or more hidden layers between them. The goal of neural network architect is to find optimal sizes of layers, to make learning process much more eficient. Input neurons’ count depends on number of features the analyzing object has and output neurons’ count on how many classes it can be classified to.
Every neuron receives value on input, transforms it using activation function and sends the output signal to the next layer. The input signal of ℎ neuron of ℎ =1 = ∑
−1, value. The output signal of ℎ neuron is: ron of ℎ layer and ℎ neuron of previous layer and −1 is ℎ neuron of previous layer’s output signal = ( ) = (∑
−1). where is a weight of connection between ℎ neu- ing Rate’ and ∇ is a value of gradient of synapse’s where is correcting coeficient commonly called ’Learnweight’s error:
=1 = ∑ ∑( − ) ,
2 ℎ position of output vector and where is a dimension of output vector (and also count of neurons on output layer), is predicted value on
is actual value on ℎ position of output vector. Correction of synaptic weights starts in last layer and goes backwards through all hidden layers until it reaches input layer. The weight is changed according to equation: = + ∇ , (1) (2) transfer functions or threshold functions). One of the
There are many activation functions (also known as layer’s ℎ neuron’s input signal and −1 is previous 1 2 ⋅ ⋅ 2 ⋅ = 2 −1,
(3) where is value of a change of error function, for ℎ
layer equals: layer’s ℎ neuron’s output signal value. On last, ℎ = 1 2 ⋅ = 2 ⋅ 1 ( − )2 = ′( )⋅(
− ), (4) where ′ ( ) is value of activation function’s diferenoutput neuron’s input signal. On this layer the value depends mainly on distance between preuse numbers calculated in previous steps: dicted and actual values - on error. Other layers’ s =
1 +1 1 2 ⋅ = ⋅ ∑ 2 =1 +1 +1 = ′( ) +1 ∑ =1 +1 +1, (5) where +1 is count of neurons on ( + 1)ℎ layer.
In Algorithm 1 you can see the full process of training Feed-Forward Neural Network using back-propagation algorithm. It uses array called , which consists of values. This jagged array has rows and as many columns, in a row, as many neurons the layer has, it corresponds to. All elements of arrays are written like ,
which is just alternative form, for writing [ ][ ]. What’s more, numerical intervals that are used in the for-loops are half-open (the numbers after to don’t count to these intervals). There are also symbols like , , and . They mean respectively: Learn
ing Rate ( ), ℎ layer’s ℎ neuron’s input and output values and weight of synapse between ℎ neuron of ℎ layer and ℎ neuron of previous layer. Moreover there is ′(⋅) symbol that signifies the diferential of the activation function - for bipolar linear function the diferential is: ( ) = 1 − − 1 + − ′( ) =
2 − (1 + − )2 .
(6) (7)
Our experimental FFN takes in an 50x25x3 image and outputs six dimensional vector of decision values. Each decision value represents one movie that is associated with given frame. 3.1. Data preparation 3.1.1. Image preparation After loading data set into the memory, we cut each image to meet 2:1 aspect ratio, this value was chosen because our input vector is an image that has an aspect ratio 2:1. If we took an image with aspect ratio that does not meet input aspect ratio, we would need to stretch or shrink an image; that would in turn add
end end end , (
) for = 0 to ℎ Result: Higher precision of neural network ∶= count of network’s layers; ∶= empty jagged array, for values;
do for = 0 to Training Set’s Length do
Insert ℎ vector from to first layer’s input synapses; for = 0 to L do layer weights layer); layer Calculate on all neurons on ℎ by summing products of synaptic and output values of previous layer neurons (or synapses if it is input Calculate on all neurons on ℎ by using activation function; end end
( output values
) ∶= vector made of values from output neurons; for = 0 to output neurons count do ( −1), = ( , − ) ⋅ ( ′
−1); for = − 2 to 0 step −1 do for = 0 to ℎ layer’s neurons count do , = 0; for , end , count do for end
end neurons count do = 0 to ( + 1)ℎ layer’s = , + ( +1), ⋅
+1; = , ⋅ ′( )
; end end for = − 2 to 0 step −1 do for = 0 to ℎ layer’s neurons neurons count do = 0 to ( − 1)ℎ layer’s = 2 ⋅ ⋅ ,
⋅ −1; Algorithm 1: FFN training algorithm. some pixels and thus make results more inaccurate. was not capacious enough to accommodate amount of After cropping to target aspect ratio, we crop another knowledge that could recognize four or more movies. 1 of width and height, from each side, to eliminate Eventually, model that contained 3 hidden layers with p12ossible letterboxes. Finally we shrink an image to get the following count of neurons: 800, 200, 50 came out it to 50x25 dimensions. Such dimensions have been to be an optimal solution. chosen to cut down on training times. Obviously to The back-propagation algorithm has one significant eliminate possible artifacts introduced by the prepro- defect. It does not use any heuristics that could help cessing steps odescribed above the image will be fil- it to deal with local minima. Solution to this issue tered [12]. is quite simple and easy to implement. In this case, adding a random number from range of [−0.002; 0.002] 3.1.2. Data labeling to weights of all connections turned out to be extraordinarily eficient. However it’s hard to say about exEach image is given label, signifying movie it belongs act values because, the method was crossed with graditno.gTshete.n 110 of image samples are redirected into test- aumaloiunnctreoafseclaosfscelsasrseessulcteodunint.Sluerappriosfinqgulayl,itiyncirneadseincigsions made by the network. 3.2. Training network This phenomenon was observed, after few epochs of learning, after class addition. Sometimes network After preparation of training and testing datasets, all needed randomization mentioned above for the phe225 per movie images, converted to the vectors, are in- nomenon to occur. serted to the neural network. As it was said in ”Feed- At the time of writing this article absolute accuracy Forward Networks Theory” section, the information of predictions made by the network, for six classes exgoes through all layers to the output layer with us- ceeded maximal accuracy given on four classes. We ing activation function in all encountered neurons and suspect that this could be explained by growing generthen goes back in back-propagation process, modify- ality of classifier contained in the model with increasing synaptic weights. This sequence is repeated mul- ing amount of known classes. tiple times and results in improvement of network’s Methodology of learning was following. Model was ability to recognizing objects. trained until it’s accuracy hadn’t increased, for few 1 hidden layer with 50 neurons epochs. In next stage networks’ weights was randomMovies count Highest precision ized to get three diferent child networks. All four in4 76.0 % stances was learned in parallel. At the end, the best 5 71.2 % one of them was picked, reproduced by randomizing 6 52.7 % their weights and then all the process were repeated.
When the network had stalled with it’s advance3 hidden layers: 800, 200 and 50 neurons ment, one class was added to its possible outputs and Movies count Highest precision learning was continued in the same way. One of the 4 82.0 % most interesting issues that we have experienced in 5 88.8 % the initial part of the research was a significant drop in accuracy when class set contained ”Shrek 2” movie. 6 84.6 %
After preliminary learning on dataset that hadn’t contained this movie, this problem disappeared. 4. Experiments We were also experimenting with diferent weight initialization techniques, described in [4, 13]. We have tried the following methods: Main problem that was dealt with in order to provide optimal learning accuracy and time, was selecting appropriate architecture that grants relatively good accuracy, but on the other hand, contains as few neurons as required. Minimal neuron count results in minimal computational time.
Initially, when tests were carried out on 4 classes of data, network that contained only one hidden layer which consisted of as few as 50 neurons seemed to be the most eficient and accurate (76%). Nevertheless it [−1.0, 1.0] with vious layer • He weight initialization - multiplies random value √
is a size of pre 2 where tialization, but with • Xavier’s weight initialization - similar to He ini√ 1 • We also used the following technique:
The tables below contain example predictions made by the network. Provided images are screenshots from the Netflix platform that were taken independently from the learning and testing datasets.
As it can be seen, the accuracy is satisfactory high. It makes a few mistakes when it comes to dark and distant frames. Fixing this issue will be our next goal.
After all experiments done with this network, we can state the following conclusions: √
Unfortunately none of those methods worked correctly. In testing, we used networks with significant number of neurons in nearly every layer (3750 neurons in input layer) that in turn set initial weight values to minute values.
Results of our research are shown in the tables below.
First table presents the most satisfactory accuracies of the neural network (with 1 hidden layer consisting of 50 neurons) before the experiment. The second one shows the efects of randomization. Percents situated in right columns are calculated by dividing amount of correctly recognized movies by size of testing dataset (which is 25 frames per movie). As it shows, the results are surprisingly high. [1] M. Woźniak, D. Połap, Soft trees with neural components as image-processing technique for archeological excavations, Personal and Ubiquitous Computing 24 (2020) 363–375.
Origin Indiana Jones Indiana Jones Indiana Jones Indiana Jones The Lego Movie The Lego Movie The Lego Movie The Lego Movie Mada- gascar Mada- gascar Mada- gascar Mada- gascar
Matches yes yes yes no yes yes no yes yes no yes yes
Matches yes yes yes yes yes yes yes yes yes yes yes yes