On the Problem of Development of Methods and Algorithms Based on the Ob ject-Oriented Logic Programming for Intelligent Video Monitoring of Laboratory Rats

Abstract

The problem of the video monitoring the laboratory rats by the meansof the object-oriented logic programming is considered. The main task of the video monitoring is the analysis of the behavior of the animals in cognitivt esting. An essential feature of the video records is in that the experiments are conducted in the same cage where the animal lives, that is, the background of the cage isawdust. The color of the animals is about the same as the color of the sawdust; thus the detection of the animals is not a simtpalsek. An additional difficulty is in that the videos were recorded simultaneously with electroencephalograms (EEG) in the animals; thus the head of the rat is connected with EEG cable that moves and causes false detections of recognition algorithms. In the paper, development of low-level algorithms for video analysis as well as logical methods for the analysis of the animal behavior idsiscussed. The methods and algorithms are implemented in the Actor Prolog object-oriented logiclanguage.

1. Introduction

In recent years, automation of neurophysiological experiments on laboratory animals was recognized as an important direction in the computer vision and intelligent video monitoring [ 1, 2 ]. The methods of computer vision enable automation of routine of animal behavior analysis and, that is more important, make the results of the analysis independent from the human factor. Usually, a neurophysiological research is based on the relative change of the quantity of given events (for instance, actions performed by the laboratory animal) under given experimental conditions, but not on the absolute quantity of these events. Moreover, often it is not possible to estimate the exact quantity of the events because the behavior of the animal is not clearly expressed; in this case, the recognition of the events depends on the experience and subjective opinion of the experimenter. The state of the experimenter including his fatigue and his current ideas about the importance of various experimental events can influence the results of the recognition too. Therefore it is important to provide the constancy and uniformity of the recognition of required elements of animal behavior. This uniformity can be provided by the methods of automatic video analysis (intelligent video monitoring).

Another important problem that can be solved using intelligent video monitoring is standardization of neurophysiological experiments and providing reproducibility [ 3 ] of the experimental results by independent researchers in various laboratories.

There are free and commercial software available for automation of video processing and laboratory animal behavior analysis in biomedical experiments [ 4–8 ]. Nevertheless, new problems arise constantly in the neurophysiological experiments that require video processing that is out of capabilities of the existed software. It is expedient to use high-level programming languages specialized for intelligent video monitoring for solving these problems. In this paper, a video monitoring problem of such kind and video analysis methods used for solving this problem are considered.

The videos considered below are produced in neurophysiological experiments on the study of a convulsive electrical activity of the brain cortex. In the experiments, videos of a behavior of laboratory rats were recorded simultaneously with EEG signals. A comparison of EEG data with the behavior of the animals is necessary because sharp motions of the animals can result in EEG artifacts that are very similar to the epileptic discharges. Thus, the first task of the video monitoring is recognition of the sharp motions of the animals and using this information for proper interpretation of the results of the experiments. The second task of the video monitoring is the analysis of the behavior of animals in cognitive testing (in the tests on social recognition and recognition of a new object). An essential feature of the video records is in that the experiments are conducted in the same cage where the animal lives, that is, the background of the cage is sawdust. The color of the animals is about the same as the color of the sawdust; thus the detection of the animals is not a simple task.

Initial experiments with the video analysis have demonstrated that the methods of object detection implemented in commercial software based on the analysis of brightness, analysis of color, and background subtraction cannot provide stable recognition of the laboratory rats on the sawdust background. Thus, we have applied more sophisticated texture-oriented methods implemented in the Actor Prolog logic programming system for the recognition of the animals. The texture-oriented methods provide stable detection of the animals at the expense of a decrease of spatial resolution of the detection. In action, it implies a loss of information about the coordinates of the contour of the animal and impossibility of using modern model-based tracking methods for recognition of postures and actions of the animals. We have developed logical methods for analysis of the behavior of laboratory rats based on the information about the coordinates and velocity of objects that can be obtained using the low-level video processing means implemented in the Actor Prolog system [ 9–16 ]: (i) The coordinates and velocity of the centroid of the animal are computed using the texturebased methods of image processing. (ii) The coordinates of the EEG cap that connects the animal with the EEG cable are computed using the color-based methods. (iii) The exact coordinates of auxiliary objects placed in the cage are computed using the colorbased methods too.

We have developed a logical definition ( a s et o f l ogical r ules) o f t he e xploratory b ehavior o f the laboratory rats that provides an acceptable quality of recognition of the required behavior in the cognitive tests.

The logical approach to the definition and analysis of laboratory animal behavior is described in the first section of the paper. The experimental conditions and peculiarities of the video data to be processed are described in the second section. A description of an experimental program for the video analysis implemented in the Actor Prolog logic language and the results of the experiment are discussed in the third section. 2. The Logic Programming Approach to the Intelligent Video Monitoring The idea of using mathematical logic and logic programming for intelligent video surveillance was developed in research projects W4 [ 17 ], VidMAP [ 18 ], VERSA [ 19 ], LTAR [20R], oboSherlock [ 21 ], Actor Prolog [ 16 ], etc. The idea is in that one applies logical formulae/rules for description and recognition of objects, situations, and events. One can explain the advantage of the logical approach to the intelligent video monitoring in the following way. The activity and behavior notions [ 22 ] differ in that the behavior of an object is the activity of the object related to the context information about the place, time, object attributes, etc. The information about the context allows deciding, for instance, whether the behavior of the object is abnormal and/or dangerous. Thus the analysis of the behavior is a more complicated problem than the analysis of the activity. It is necessary to describe and analyze the information about the context of the activity and the mathematical logic is perhaps the best instrument that can be used for this purpose.

The Actor Prolog language is an object-oriented logic language, that is, it combines expressiveness of the logical and object-oriented approaches to the programming [ 23–26 ]. This combination has increased the area of application of the logic programming. In particular, the object-oriented features enable to effort an opportunity to solve the problem of storing and processing big arrays of binary data (such as audio/video data) in the logic languages. The problem is in that plain logic languages do not implement data arrays directly, but use lists and structures for storing data because these data structures correspond to the Skolem functions in the first-order Predicate Calculus [ 27 ]. In the object-oriented logic languages, the arrays of data can be encapsulated in the instances of some specialized built-in classes. This enables fast and effective processing of the big data arrays in the logic languages.

The method of object-oriented logic programming of intelligent video surveillance was developed for the analysis of people behavior and recognition of abnormal activity [ 9–16 ]. The idea of the method is in the following: (i) The stages of low-level and high-level processing of the video stream are separated. (ii) The stage of the low-level video processing includes background subtraction, extraction of blobs, computing trajectories of the movements of the blobs, etc. The low-level processing is performed directly upon the video data arrays using special built-in classes of the logic language. The built-in classes are implemented in a procedural programming language to increase the speed of the data processing. (iii) The stage of the high-level video processing includes analysis of trajectories/graphs of the blob movements. The algorithms of the high-level analysis are implemented in the Actor Prolog logic language in a form of logical rules. The graphs of the blob movements are described using the terms of the logic language: structures, lists, and underdetermined sets [ 23 ]. (iv) The logic programs written in Actor Prolog are translated to Java [ 14, 28 ]. The Java language is used as an intermediate language in the translation scheme to provide high performance and stable work of the intelligent video monitoring software.

New built-in classes of the Actor Prolog language were developed to use the language for the intelligent video monitoring of the laboratory animals. These new built-in classes implement new means of the low-level video processing that is necessary to analyze simultaneously blobs of diferent k inds e xtracted u sing d iferent me thods. Th est andard me thods of bl ob extraction based on the background subtraction are not applicable in the case of sawdust background because the sawdust surface is changed permanently that is an effect of the movements of the rat and this is a cause of multiple false results during the background subtraction. It turned out, that the methods of blob extraction based on the analysis of the brightness and hue of objects are of little use in the recognition of the rats in the cage with the sawdust background too. The reason for this is in that the color of the rats is about the same as the color of the sawdust. Theoretically speaking, the computer vision methods can differentiate the hues of the rats and the sawdust, but, in practice, the illumination inside the cage often is non-uniform andthceolor of the rats and sawdust background is influenced by the shadows and reflections of the light from the colored objects and plastic walls of the cage. As a result of this, the standard methods of blob extraction based on the brightness and color also mistakes.

3. The Experimental Conditions

Let us consider a problem of intelligent video monitoring laboratory rats by the example of a neurophysiological experiment on the study of cognitive possibilities of the animal. By the terms of the experiment, one puts new objects into the cage with the animal. The animal explores the new objects and the experimenter estimates the total time spend by the animal to explore the objects. After a time, the experiment is to be repeated with the same objects. If the time spent by the animal to explore the objects is less than one in the first test, the experimenter can conclude that the animal remembers the objects. If the time is about the same, it means that the animal forgot the objects.

The exploratory activity of the rat is usually manifested in that it approaches the object and snifs around the object (see Figure 1). It is dificult to describe the exploratory activity of the rat in a formal way; thus a simplified approach to detection of the exploratory activity is often used in the neurophysiological experiments: the distance between the rat and the object is estimated and the cumulative time when the rat is situated close enough to the object is calculated. Sometimes one estimates just the number of approaches of the rat to the object. This simplified method leads to mistakes and is not applicable to the experiment under consideration because the cage is small and the rat is situated not far from the object about all the time. At that the rat can snif around the object or ignore it; it can just lie or dig the sawdust near the object.

An additional problem is in that we cannot detect the contour of the animal in the background of sawdust. Thus we cannot recognize in a reliable way the face and forelegs of the animal. Instead of this, we estimate the following attributes of the experimental setting with the help of the low-level image processing procedures: (i) The coordinates and the velocity of the centroid of the blob related to the body of the rat are estimated using the texture-oriented method of image analysis that is sensitive to the smoothness of the object surfaces. The body of the rat is well visible in the sawdust background because the rat hair has a smoother surface in comparison with the sawdust. (ii) The coordinates of EEG cap are detected using the color in the HSB space. (iii) Exact coordinates of the objects placed in the cage are estimated using the color information too. The only diference is in that the coordinates of the object are estimated permanently in all the frames of the video and then averaged to avoid mistakes when the rat covers the object.

Let B be the centroid of the blob corresponding to the body of the rat (see Figure 2). Let C be the centroid of the blob corresponding to EEG cap of the rat. Let O be the object that is nearest to the C point. Let A be a point on the contour of the object O that is nearest to the C point. Let D be a point in the contour of the same object O that is nearest to the B point. The following combinations of these attributes have been recognized as useful for the detection of the exploratory activity of the rats during the experiments: (i) If the distance between the A and C points is less than 1 cm, the rat probably snifs the object. If the distance between the A and C points is more than 4 cm, the rat probably does not explore the object. If the distance lies in the interval 1–4 cm, additional analysis is necessary to determine whether the rat explores the object or not. The analysis is complicated by the fact that the rat can lie down sideways during the snifing of the object; in this case, the A − C distance can be big enough. (ii) If the angle between the B − C and B − A lines is more than 50 degrees, the rat probably does not investigate the object. This heuristic rule reflects the fact that the rat turns his face to the object during the snifing. (iii) If the velocity of the B point is more than 3 cm per second, the rat probably just walks around the object, but does not explore it. (iv) If the ratio between the D − B and A − C distances is less than 1.3, the rat probably does not explore the object. This rule reflects the fact that the rat usually stands aside from the object and pulls his face to the object during the snifing.

These heuristic rules were implemented in a logic program for detection of the exploratory activity of the rats and tested in neurophysiological experiments.

4. The Logic Program for the Video Analysis

The Actor Prolog language has no built-in means of the fuzzy logical inference, however one can easily implement a kind of a fuzzy logical reasoning using the standard top-down resolution and standard built-in arithmetical operations. In particular, one can define the heuristic rules of rat behavior analysis described above in the following way: IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) PREDICATES: determ: sniffing_is_detected(REAL,REAL,REAL,REAL) imperative: fuzzy_metrics(REAL,REAL,REAL) = REAL - (i,i,i,i); - (i,i,i);

The snifing is detected predicate succeeds if an exploratory behavior of the rat is detected. The predicate has four input arguments: distance A − C; distance D − B; the angle between the B − C and B − A lines; and the velocity of the B point.

CLAUSES: sniffing_is_detected(AC,_,_,_):

AC < 0.01,!. sniffing_is_detected(AC,_,_,_):

AC > 0.04,!, fail. sniffing_is_detected(AC,DB,A,V):

M1== ?fuzzy_metrics(DB/AC,1.3,0.1), M2== 1-?fuzzy_metrics(A,50.0,10.0), M3== 1-?fuzzy_metrics(V,0.03,0.01), P== M1*M2*M3,

P > ?power(0.5,3). fuzzy_metrics(X,T,H) = 1.0 :

X >= T + H,!. fuzzy_metrics(X,T,H) = 0.0 :

X <= T - H,!. fuzzy_metrics(X,T,H) = V :

V== (X - T + H) * (1 / (2*H)).

The fuzzy metrics function is auxiliary one. It is used for definition of fuzzy thresholds. The function has three input arguments: the value to be checked; the threshold; and the width of the interval of uncertainty [ 14 ]:

In Figure 3, the user interface of the logic program intended for the analysis of the rat behavior is shown. Control elements of the dialog window of the program allow selection of the video fi les and examination of the video frames at various rates in the forward and backward directions. The information on the total distance of movement, average velocity, cumulative time expended by the rat for investigation of new objects in the cage is demonstrated in the top of the dialog window. In the graphics window, the program demonstrates the frames of the video, detected blobs, and auxiliary information related to the behavior analysis. The blob related to the body of the rat is indicated by the orange color. The blob related to the EEG cap is indicated by the cyan color. The green and blue blobs correspond to the new objects placed in the cage in the course of the experiment. The logical rules defined above are used for the recognition of the exploratory activity of the rat.

The logic program was tested on five r ats. T he r esults o f t he t ests a re g iven i n Table 1: the total distance passed by the rat during the test; the average velocity of the rat; and the time of the exploratory activity of the rat (in percents of the total time of the test).

All five videos were marked manually to estimate the quality of the algorithm of detection of the exploratory activity.

The results of the automatic recognition were compared with the results of the manual marking to compute the sensitivity and specificity of the detection (see Table 2). The results of the application of standard algorithms SVM [ 29 ] and ANFIS [ 30 ] are given in the table for reference.

The tests demonstrated that the logic program ensures about the same sensitivity and specificity ( ≈ 80%) as the SVM and ANFIS algorithms do on the base of the same data. This quality of detection is comparable with the quality of manual detection and is enough for conduction of the neurophysiological experiment. Note that an advantage of the logic programming approach to the detection is in that it does not require a preliminary training of the program and, that is more important, logical rules are understandable for the experimenter and can be manually fixed and/or improved in any time. The values of the attributes used in the rules (the thresholds of angles and distances, etc.) can be assigned manually or computed (automatically or semi-automatically) on the base of statistical analysis of marked videos.

5. Conclusions

A method of intelligent video monitoring the laboratory animals based on the object-oriented logic programming is developed. The method is intended for intelligent video monitoring the laboratory rats in non-standard experimental conditions when one cannot apply existed software for automation of neurophysiological experiments. In particular, this method is applicable for analysis of the behavior of rats in cages with a sawdust background when the contrast of the images is low and the illumination is non-uniform. In the framework of the method, heuristic rules and fuzzy definitions a re u sed f or d escribing a nd r ecognition o f t he a nimal b ehavior. The method is implemented on the base of the Actor Prolog [ 16 ] object-oriented logic language. Acknowledgments Authors are grateful to Natalia V. Gulyaeva, Ilya G. Komoltsev, Anna O. Manolova, Margarita R. Novikova, and Irina P. Levshina (IHNA and NPh RAS) for the experimental video data, to Yury V. Obukhov (IRE RAS) for the problem statement, and to Alexander F. Polupanov (IRE RAS) for the help in the research. We thank anonymous reviewers for the advertence and useful remarks on the paper. This research is funded by Russian Science Foundation (project No. 16-11-10258).

References

[1] Robinson L and Riedel G 2014 Journal of neuroscience methods 234 13 - 25

[2] Tscharke M and Banhazi T M 2016 Journal of Agricultural Informatics 7 23 - 42

[3] Spruijt B M , Peters S M , de Heer R C, Pothuizen H H and van der Harst J E 2014 Journal of neuroscience methods 234 2 - 12

[4] van Dam E A , van der Harst J E, ter Braak C J , Tegelenbosch R A , Spruijt B M and Noldus L P 2013 Journal of neuroscience methods 218 214 - 224

[5] Noldus L P , Spink A J and Tegelenbosch R A 2001 Behavior Research Methods 33 398 - 414

[6] Ohayon S , Avni O , Taylor A L , Perona P and Egnor S R 2013 Journal of neuroscience methods 219 10 - 19

[7] Weissbrod A , Shapiro A , Vasserman G , Edry L , Dayan M , Yitzhaky A , Hertzberg L , Feinerman O and Kimchi T 2013 Nature Communications 4 2018

[8] Giancardo L , Sona D , Huang H , Sannino S , Manag`o F , Scheggia D , Papaleo F and Murino V 2013 PloS one 8 e74557

[9] Morozov A A and Sushkova O S 2016 Real-time analysis of video by means of the Actor Prolog language Computer Optics 40 ( 6 ) 947 - 957 DOI: 10.18287/ 2412 -6179-2016-40-6- 947 -957

[10] Morozov A A , Sushkova O S and Polupanov A F 2017 Object-oriented logic programming of 3D intelligent video surveillance: The problem statement IEEE 26th International Symposium on Industrial Electronics (ISIE), Edinburgh , United Kingdom 1631-1636

[11] Morozov A A , Sushkova O S and Polupanov A F 2017 Towards the distributed logic programming of intelligent visual surveillance applications Proceedings Advances in Soft Computing: 15th Mexican International Conference on Artificial Intelligence 2 42 - 53

[12] Morozov A A 2015 Pattern Recognition and Image Analysis 25 481 - 492

[13] Morozov A A and Polupanov A F 2015 Development of the logic programming approach to the intelligent monitoring of anomalous human behaviour OGRW (Koblenz : University of Koblenz-Landau) 82 - 85

[14] Morozov A A and Polupanov A F 2014 Intelligent visual surveillance logic programming: Implementation issues CICLOPS-WLPE (Aachener Informatik Berichte ) 31 - 45

[15] Morozov A A , Vaish A , Polupanov A F , Antciperov V E , Lychkov I I , Alfimtsev A N and Deviatkov V V 2015 Development of concurrent object-oriented logic programming platform for the intelligent monitoring of anomalous human activities BIOSTEC 511 82 - 97

[16] Morozov A A and Sushkova O S 2018 The intelligent visual surveillance logic programming (Access mode : http://www.fullvision.ru)

[17] Haritaoglu I , Harwood D and Davis L S 1998 W 4: Who? When? Where? What? A real time system for detecting and tracking people FG ( Japan ) 222 - 227

[18] Shet V , Harwood D and Davis L 2005 VidMAP: Video monitoring of activity with Prolog IEEE AVSS 224-229

[19] O 'Hara S 2008 VERSA-video event recognition for surveillance applications (M.S. thesis . University of Nebraska at Omaha)

[20] Artikis A , Sergot M and Paliouras G 2010 A logic programming approach to activity recognition International Workshop on Events in Multimedia 3-8

[21] Worch J H , B´ alint-Bencz´edi F and Beetz M 2016 KI - K¨unstliche Intelligenz 30 21 - 27

[22] Borges P V K , Conci N and Cavallaro A 2013 IEEE Transactions on Circuits and Systems for Video Technology 23 1993 - 2008

[23] Morozov A A 1999 Actor Prolog : an object-oriented language with the classical declarative semantics IDL (Paris , France) 39 - 53

[24] Morozov A A 2002 On semantic link between logic, object-oriented, functional, and constraint programming MultiCPL 43-57

[25] Morozov A A 2007 Operational approach to the modified reasoning, based on the concept of repeated proving and logical actors CICLOPS 1-15

[26] Morozov A A 2003 Pattern Recognition and Image Analysis 13 640 - 649

[27] Chang C L and Lee R C T 1973 Symbolic logic and mechanical theorem proving (New York: Academic Press)

[28] Morozov A A , Sushkova O S and Polupanov A F 2015 A translator of Actor Prolog to Java RuleML DC and Challenge

[29] Cristianini N and Shawe-Taylor J 2000 An Introduction to Support Vector Machines and Other Kernel-based Learning Methods (Cambridge: Cambridge University Press)

[30] Jang J S R 1993 IEEE Transactions on Systems , Man, and Cybernetics 23 665 - 685