Computer-based image analysis and various predictive applications are commonly used in different fields of human activity, one of which is agriculture. The number of farms in Ukraine has increased significantly and their productivity is growing, so the importance of computer technology in the automation of agricultural processes is gradually increasing. When raising cows, the relationship between live weight (LW), milk yield and feed intake can be taken as criteria for organizing the care and nutrition of animals in modern keeping conditions. These parameters are quite important and must be strictly controlled. When they go beyond permissible limits, this significantly affects the immune system of cows, and, accordingly, the economic efficiency of the farm. Negative changes in live weight may indicate animal's health problems, inappropriate environmental conditions, and nutritional errors. Therefore, a parameter such as live weight (LW) is certainly important for dairy cows [1]. It should be noted that at present the process of measuring and servicing cattle is still carried out manually and is very expensive.
The aim of this study is the estimation of cows LW using several neural network models: the convolution artificial neural network for recognition of a cow at the pictures and its breed identification with subsequent determination of the size of its body by the stereopsis method with subsequent utilization of multilayer perceptron for LW estimation of a cow on the basis of information regarding its breed and size.
For more accurate estimation of the animal's physical parameters 3D camera (Intel RealSense D435i) was additionally used. It should be noted that the use of a 3D camera alone does not yield to good results due to its low resolution. Thus, cows images taken at different angles are used to determine parameters of bodies cows using photogrammetric method. Parameters such as the withers height (WH), hip height (HH), the body length (BL) and the hip width (HW) of cows were obtained via photogrammetry.
Model estimation based on the ANN has been developed using those parameters (input parameters WH, HH, BL, HW, and the output parameter -LW).
Cow's body dimensions are determined from the analysis of animal images taken synchronized cameras from different perspectives. Initially, a cow is identified at the image and its breed is determined by using Mask-rcnn convolution neural network. Then withers height, hip height, length and width of a cow determined via stereopsis method which allows to obtain geometric parameters of the objects at digital images and perform their measurements. Digital imaging and photogrammetric processing include several completely certain steps that can generate three-dimensional or twodimensional digital model of the animal's body. Then obtained data about the species and its size are fed to the predictive model, which determines the estimated weight of the animal.
In the literature different estimation of animals LW for various purposes is typically performed by using regression equations. However, in recent years the methods and means of computational intelligence based on ANN are increasingly used.
Prediction of LW of bulls' slaughter value from growth data by using ANN carried out in [2], an artificial intelligence technology in dairy industry [3], a comparison of an artificial neural network and a method of linear regression for prediction of the LW of hair goats [4], comparison of ANN and decision tree algorithms used for prediction of LW at post weaning period [5], weight prediction of broiler chickens with the help of 3D-computer vision is performed in [6], prediction of the goats masses [7], artificial neural network to predict rabbits body weight [8], prediction of carcass meat percentage in young pigs using linear regression models and artificial neural networks [9], weighing pigs using machine vision and artificial neural networks [10].
The purpose of this study is to estimate cows LW by using artificial neural networks. Body dimensions (BD) were determined by the photogrammetric method on images in which the cow is identified and classified by the convolution neural network.
The traditional methods of measuring the mass of cows
This method is used to determine LW of the adult cattle by the following formula , 100
where A -chest girth behind the shoulders, cm; B -direct length of the trunk, measured with a stick, cm; K -a correction factor (2 -for dairy cattle and 2.5 -for milk-meat and beef breeds).
To determine the approximate body weight special tables are used. For these tables the initial data is measurements of animals taken at the correct animal's position (feet must stand upright, a head at the level of the back).
There are some other methods of LW and cow's body size estimation that could be applied, in particular Pin Bone method [11] , 300
where BW -Body Weight estimation (pounds); HG -heart girth (inch); BL -Body Length (inch).
Thus, in estimating of cattle LW the important task of determining its size measure arises. This problem is complicated especially when measures are made in real conditions. The traditional animal measuring process is shown in Fig. 2. The first step in the proposed algorithm is detection of a cow. This step is based on the use of convolution neural network for the cows detection at the picture and stereopsis method, which allows the system to obtain measurements of the real world objects, located at different distances from the cameras.
The stereopsis principle
This principle can be explained as follows: Suppose there are two cameras, defined by their matrices P and P' in some coordinate system. In this case we say that there is a pair of calibrated cameras. If the cameras centers do not match, the pair of cameras can be used to determine the threedimensional coordinates of the observed points.
Often, the coordinate system is chosen in such a way that the cameras matrix are of the form ] 0
(this is always possible, if we choose the origin coinciding with the first camera's center, and direct a Z-axis along the optical axis).
Consider a point P in the real three-dimensional space projected simultaneously in two image points p and p' through the two camera projection center (C and C'). The points P, p, p', C and C' lie in a plane, called the epipolar plane. Epipolar plane inter-sects each image forming the intersection lines. These lines (l and l') correspond to the projection ray through the p and P, and p' and P, and are called epipolar lines. This projection epipolar geometry is described in [12] (Fig. 3).
Epipolar geometry is used to search for stereo pairs and for verifying that a pair of points can be a stereo pair (i.e. the projection of a point in space).
Epipolar geometry has a very simple description in the coordinates. Suppose there is a pair of calibrated cameras, and let p is homogeneous coordinates of the point at the image from the first camera, and p' -from the second camera. A 3 × 3 matrix F exists, such that the pair of points p, p' is a stereo pair if and only if:
The matrix F is called a fundamental matrix. Its rank is 2 and it is determined up to a nonzero factor that only depends on the original matrix cameras P and P'.
In the case where the matrix cameras are of the following form ] 0
the fundamental matrix may be calculated with the formula:
where the e vector notation
In [13] the various calculation algorithms of F with using a set of points are considered. In particular, gradient descent algorithm, Newton's method and the Levenberg-Marquardt algorithm are described.
Epipolar line equations are calculated with the help of the fundamental matrix. For the point x, the vector that defines the epipolar line is of the form Fp l , and the equation of the epipolar line itself is:
. Similarly, for a point p', the vector defining the epipolar line is of the form p F l T .
Depth map -is an image in which each pixel, instead of color, keeps a distance from the camera. To this end, for each point in one image its pair is searched on the other image. A pair of corresponding points can be used for triangulation and determination of their prototype coordinates in three dimensions. Knowing the three-dimensional coordinates of the prototype image, the depth is calculated as the distance to the camera's plane [13].
Thus it is possible to determine the size of the object.
CNN was first proposed in [14] as the development of the neocognitron model [15], intended for effective image recognition. This network uses pattern recognition technology based not on hardcoded by developers algorithms, but on system training, which involves a consistent identification of a huge number of images. Subsequently, the R-CNN (Regions With CNNs) networks were built based on CNN. R-CNN were used for detecting all objects of the given classes and determining the bounding box for each of them (object detection). R-CNN creates bounding boxes for each object in the image or suggestions regions using selective search process. Fast R-CNN increases productivity of R-CNN and classifies objects of each region together with tighter bounding boxes. Next network, Faster R-CNN, improved generation mechanism of used therein candidate regions by computing the regions not on the original image but on the features map derived from CNN. For this purpose the module called Region Proposal Network (RPN) was added.
Finally, the Mask R-CNN network [16] improves Faster R-CNN architecture by adding one more sub-module, which predicts the position of the detected object covering mask, and thus solves the task of instance segmentation. After image processing, the network outputs objects bounding boxes (bbox), their classes (class) and masks (mask). It worth to note, that Mask R-CNN is one of the fastest network at the moment. The structure of this network is shown in Fig. 4. In accordance with these arrangements, three main layers are used to build a convolutional neural network: convolution, pooling (otherwise subsampling or downsampling), fully connected layer.
Convolution equation for the l -th ) ,..., 1 ( L l network layer has the following form [17]
It reflects movement of the l w core along the image or map of input attributes for the layer
-indices of the elements in the matrices, S -the value of the convolution step.
As seen from ( 3) convolution are performed for each element j i, of image matrix l x .
Convolution preserves spatial relationships between the pixels. Each convolutional layer is followed by subsampling or computational layer that is serving to reduce the dimension of the image by averaging the values of the local output neurons.
Subsampling layer zooms planes by local averaging of the output neurons values . Thus, the hierarchical organization is achieved. Subsequent layers are extracted more common characteristics that less depend on the image distortion.
The difference between a subsampling layer and a convolution layer is that in the convolution layer regions of the neighboring neurons overlap, which does not occur in the subsampling layer.
Pooling layer operates independently of the input data depth and scales the spatial volume by using a maximum function.
The architecture of the convolution network assumes that the presence of a sign is more important than information about its location. Therefore, the maximum one is selected from several neighboring neurons in the feature map and its value is considered as a single neuron in the feature map of lower dimension.
In addition to maximum subsampling, pooling layers can perform other functions, such as averaging subsampling or even L2-normalized subsampling.
On these layers within the network, a nonlinear activation function ) ( f is applied to all input values and the result is sent to the output. Thus, the activation layer does not change the input resolution.
Usually due to significant positive properties for hidden layers, ReLu(ReLU (x) = max (0; x)) and its various modifications (Leaky ReLU, Parametric ReLU, Randomized ReLU) [18] are used. SoftMax function
(for solving classification problems) or linear function (for regression tasks) are used for fully connected layer.
Different regularization techniques are used to avoid network retraining. Dropout is a simple and effective regularization method and consists in the fact that in the process of training a network, a subnet is randomly allocated from its aggregate topology, i.e. part of the neurons is turned off from the process, and the next update of the scales occurs only within the allocated subnet. Thus, only weights of remaining neurons are changed. Each neuron is excluded from the total network with a certain probability, which is called the dropout rate.
This layer reduces the time of one training epoch due to the smaller number of optimized parameters, and also allows to better deal with retraining of the network compared to standard regularization methods.
The standard normalization of inputs occurs on this layer (the sample average of their values is subtracted, and the result is divided by the root of the sample variance). Sampled values are calculated taking into account the values at the inputs of this layer at previous training iterations. This approach allows to increase the speed of learning the network and improve the final result.
This layer is an ordinary multilayer perceptron, the purpose of which is classification. It models a complex nonlinear function, the optimization of which improves the quality of recognition. The neurons of each map of the previous subsample layer are associated with one neuron of the hidden layer. Thus, the number of neurons in the hidden layer is equal to the number of maps in the subsample layer.
As in ordinary neural networks, in a fully connected layer, neurons are connected to all activations in the previous layer. Their activations can be calculated by multiplying matrices and applying bias.
The differences between fully connected and convolutional layers are that the neurons of the convolutional layer 1) connected only with the local area of the input; 2) can share parameters. In practice, a quadratic function, cross entropy, or some combined functional are used as a criterion for the error function.
Backpropagation neural network (BPNN) is a multilayer feedforward neural network that uses a supervised learning algorithm known as error back-propagation algorithm. Errors accumulated at the output layer are propagated back into the network for the adjustment of weights. Multilayer perceptron (MLP) is a neural network with several layers, each of which consists of computing nodes (neurons). The topology of the MLP is shown in Fig. 5 [19]. Network inputs are connected to each neuron in the first layer. The outputs of the neurons of the first layer then become inputs to the neurons of the second layer and so on. The last layer is the output layer, all other layers between the input and output layers are called hidden layers. The architecture of the multi-layer perceptron can be conveniently written as
where 0 n is the network's input vector dimension, and i n ,
denotes the number of nodes in the respective layers.
where i W -vector of weight parameters of neurons in the i-th layer of the network;
Since in practical applications it is necessary to perform various operations not only with the activation function itself, but also with its first derivative, it is necessary to use a monotonous, differentiable and limited function as an activation function. A particularly important role is played by such functions in modeling nonlinear relationships between input and output variables. These are the so-called logistic or sigmoidal (S-shaped) functions.
In general, the representation of the input-output can be expressed as
Approximate capacity of MLP have been studied and described by many authors [19][20][21][22][23]. The basic idea is that every continuous function
can be uniformly approximated with arbitrary precision function f ˆ by f D , where f D is a compact subset of l n R , provided that there is a sufficient number of hidden layers in the network. This is true even for networks with only one hidden layer. A typical assumption for the activation functions in the hidden layer is the following: ) ( f is a continuous, non-constant and limited.
MLP is in general function approximator and it ensures that the network with one hidden layer will always be enough to represent any arbitrary continuous function. But this statement says nothing about the number of neurons in the hidden layer, which provide a given accuracy of approximation. It is also important that the proof of the possibility of approximation by means of MLP suggests that weights are set correctly. However, the question of a learning algorithm choice remains open.
The pseudocode algorithm for BPNN is given below [24]. (VI) Periodically evaluate the network performance. Repeat Forward and Backward computations until the network converges on the target output.
The structure of the developed system Fig. 6 shows the structure of the developed intellectual breed recognition and evaluation of DA cattle. The problem of recognizing various breeds of cows and assessing their linear sizes with the subsequent determination of mass using a neural network predictive model based on a multilayer perceptron was solved. To recognize the breed of a cow, the Mask-RCNN network trained on the COCO sample was used, the source code of which is freely available on the Internet [25]. The network weights were fine-tuned on 250 images of representatives of each of the considered breeds of cows (Ayrshire, Holstein, Jersey, Red Steppe). To increase the size of the training and test samples, the augmentation method was used [26], which made it possible to obtain additional images from the original ones. The network was retrained over 5000 epoches using the SGD algorithm. To train the network, we used the Nvidia GeForce 1060 graphics card, which allowed us to speed up the learning process many times as compared to learning on the CPU. The results of recognition and evaluation of the size and weight of cows are given in Table 1. Examples of correct recognition are presented in fig. 7.
It should be noted that, despite the rather extensive training sample and the achieved high recognition accuracy (92%), as a result of the network, false recognition occurs.
An example of such recognition is shown in Fig. 8. The obtained masks of recognized objects were used to determine their linear dimensions using the triangulation method (epipolar geometry). As additional information that allows us to adjust the obtained sizes, we used images from a 3D camera, which allow us to estimate the distance to the recognized object. Examples of such images are presented in Fig. 9.
This project has been funded with support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
The proposed method for measuring cattle using neural network image processing algorithms allows modern farmers to quickly and accurately measure the weight of the animal, as well as recognize its breed, saving time and reducing effort without compromising the habitat and disturbing livestock growth. This is possible because the method allows studying cows both in corrals and pastures, without interfering with their normal behavior. The proposed approach can be used to replace traditional methods that use direct observation and measurement, which adversely affect the behavior of animals. The system can be widely used in the management of modern farming. The accuracy and performance of the proposed methodology were tested with the participation of the experts. The same measurements were carried out by the farm staff who confirmed the effectiveness of the proposed system.