=Paper= {{Paper |id=Vol-2210/paper5 |storemode=property |title=On the problem of development of methods and algorithms based on the object-oriented logic programming for intelligent video monitoring of laboratory rats |pdfUrl=https://ceur-ws.org/Vol-2210/paper5.pdf |volume=Vol-2210 |authors=Alexei Morozov,Olga Sushkova }} ==On the problem of development of methods and algorithms based on the object-oriented logic programming for intelligent video monitoring of laboratory rats== https://ceur-ws.org/Vol-2210/paper5.pdf

On the Problem of Development of Methods and
Algorithms Based on the Object-Oriented Logic
Programming for Intelligent Video Monitoring of
Laboratory Rats

A A Morozov1 and O S Sushkova1

Kotel’nikov Institute of Radio Engineering and Electronics of RAS, Mokhovaya str. 11-7,
1

Moscow, Russia, 125009

Abstract. The problem of the video monitoring the laboratory rats by the means of the
object-oriented logic programming is considered. The main task of the video monitoring is
the analysis of the behavior of the animals in cognitive testing. An essential feature of the
video records is in that the experiments are conducted in the same cage where the animal
lives, that is, the background of the cage is sawdust. The color of the animals is about the
same as the color of the sawdust; thus the detection of the animals is not a simple task.
An additional difficulty is in that the videos were recorded simultaneously with
electroencephalograms (EEG) in the animals; thus the head of the rat is connected with EEG
cable that moves and causes false detections of recognition algorithms. In the paper,
development of low-level algorithms for video analysis as well as logical methods for the
analysis of the animal behavior is discussed. The methods and algorithms are implemented
in the Actor Prolog object-oriented logic language.

1. Introduction
In recent years, automation of neurophysiological experiments on laboratory animals was
recognized as an important direction in the computer vision and intelligent video monitoring [ 1,
2]. The methods of computer vision enable automation of routine of animal behavior analysis
and, that is more important, make the results of the analysis independent from the human factor.
Usually, a neurophysiological research is based on the relative change of the quantity of given
events (for instance, actions performed by the laboratory animal) under given experimental
conditions, but not on the absolute quantity of these events. Moreover, often it is not
possible to estimate the exact quantity of the events because the behavior of the animal is
not clearly expressed; in this case, the recognition of the events depends on the experience
and subjective opinion of the experimenter. The state of the experimenter including his
fatigue and his current ideas about the importance of various experimental events can influence
the results of the recognition too. Therefore it is important to provide the constancy and
uniformity of the recognition of required elements of animal behavior. This uniformity can be
provided by the methods of automatic video analysis (intelligent video monitoring).

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018)
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

Another important problem that can be solved using intelligent video monitoring is
standardization of neurophysiological experiments and providing reproducibility [3] of the
experimental results by independent researchers in various laboratories.
There are free and commercial software available for automation of video processing and
laboratory animal behavior analysis in biomedical experiments [4–8]. Nevertheless, new
problems arise constantly in the neurophysiological experiments that require video processing
that is out of capabilities of the existed software. It is expedient to use high-level programming
languages specialized for intelligent video monitoring for solving these problems. In this paper, a
video monitoring problem of such kind and video analysis methods used for solving this problem
are considered.
The videos considered below are produced in neurophysiological experiments on the study
of a convulsive electrical activity of the brain cortex. In the experiments, videos of a behavior
of laboratory rats were recorded simultaneously with EEG signals. A comparison of EEG data
with the behavior of the animals is necessary because sharp motions of the animals can result in
EEG artifacts that are very similar to the epileptic discharges. Thus, the first task of the video
monitoring is recognition of the sharp motions of the animals and using this information for
proper interpretation of the results of the experiments. The second task of the video monitoring
is the analysis of the behavior of animals in cognitive testing (in the tests on social recognition
and recognition of a new object). An essential feature of the video records is in that the
experiments are conducted in the same cage where the animal lives, that is, the background of
the cage is sawdust. The color of the animals is about the same as the color of the sawdust;
thus the detection of the animals is not a simple task.
Initial experiments with the video analysis have demonstrated that the methods of object
detection implemented in commercial software based on the analysis of brightness, analysis of
color, and background subtraction cannot provide stable recognition of the laboratory rats on
the sawdust background. Thus, we have applied more sophisticated texture-oriented methods
implemented in the Actor Prolog logic programming system for the recognition of the animals.
The texture-oriented methods provide stable detection of the animals at the expense of a
decrease of spatial resolution of the detection. In action, it implies a loss of information about
the coordinates of the contour of the animal and impossibility of using modern model-based
tracking methods for recognition of postures and actions of the animals. We have developed
logical methods for analysis of the behavior of laboratory rats based on the information about
the coordinates and velocity of objects that can be obtained using the low-level video processing
means implemented in the Actor Prolog system [9–16]:
(i) The coordinates and velocity of the centroid of the animal are computed using the texture-
based methods of image processing.
(ii) The coordinates of the EEG cap that connects the animal with the EEG cable are computed
using the color-based methods.
(iii) The exact coordinates of auxiliary objects placed in the cage are computed using the color-
based methods too.
We have developed a logical definition ( a s et of l ogical r ules) of t he exploratory b ehavior of the
laboratory rats that provides an acceptable quality of recognition of the required behavior in
the cognitive tests.
The logical approach to the definition and analysis of laboratory animal behavior is described
in the first section of the paper. The experimental conditions and peculiarities of the video data
to be processed are described in the second section. A description of an experimental program
for the video analysis implemented in the Actor Prolog logic language and the results of the
experiment are discussed in the third section.

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 30
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

2. The Logic Programming Approach to the Intelligent Video Monitoring
The idea of using mathematical logic and logic programming for intelligent video surveillance was
developed in research projects W4 [17], VidMAP [18], VERSA [19], LTAR [20], RoboSherlock [21],
Actor Prolog [16], etc. The idea is in that one applies logical formulae/rules for description and
recognition of objects, situations, and events. One can explain the advantage of the logical
approach to the intelligent video monitoring in the following way. The activity and behavior
notions [22] differ in that the behavior of an object is the activity of the object related to the
context information about the place, time, object attributes, etc. The information about the
context allows deciding, for instance, whether the behavior of the object is abnormal and/or
dangerous. Thus the analysis of the behavior is a more complicated problem than the analysis
of the activity. It is necessary to describe and analyze the information about the context of the
activity and the mathematical logic is perhaps the best instrument that can be used for this
purpose.
The Actor Prolog language is an object-oriented logic language, that is, it combines
expressiveness of the logical and object-oriented approaches to the programming [23–26]. This
combination has increased the area of application of the logic programming. In particular, the
object-oriented features enable to effort an opportunity to solve the problem of storing and
processing big arrays of binary data (such as audio/video data) in the logic languages. The
problem is in that plain logic languages do not implement data arrays directly, but use lists and
structures for storing data because these data structures correspond to the Skolem functions in
the first-order Predicate Calculus [27]. In the object-oriented logic languages, the arrays of data
can be encapsulated in the instances of some specialized built-in classes. This enables fast and
effective processing of the big data arrays in the logic languages.
The method of object-oriented logic programming of intelligent video surveillance was
developed for the analysis of people behavior and recognition of abnormal activity [ 9–16]. The
idea of the method is in the following:
(i) The stages of low-level and high-level processing of the video stream are separated.
(ii) The stage of the low-level video processing includes background subtraction, extraction of
blobs, computing trajectories of the movements of the blobs, etc. The low-level processing
is performed directly upon the video data arrays using special built-in classes of the logic
language. The built-in classes are implemented in a procedural programming language to
increase the speed of the data processing.
(iii) The stage of the high-level video processing includes analysis of trajectories/graphs of the
blob movements. The algorithms of the high-level analysis are implemented in the Actor
Prolog logic language in a form of logical rules. The graphs of the blob movements are
described using the terms of the logic language: structures, lists, and underdetermined
sets [23].
(iv) The logic programs written in Actor Prolog are translated to Java [14, 28]. The Java
language is used as an intermediate language in the translation scheme to provide high
performance and stable work of the intelligent video monitoring software.
New built-in classes of the Actor Prolog language were developed to use the language for the
intelligent video monitoring of the laboratory animals. These new built-in classes implement
new means of the low-level video processing that is necessary to analyze simultaneously blobs
of different k inds e xtracted u sing d ifferent me thods. Th e st andard me thods of bl ob extraction
based on the background subtraction are not applicable in the case of sawdust background
because the sawdust surface is changed permanently that is an effect of the movements of the
rat and this is a cause of multiple false results during the background subtraction.

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 31
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

It turned out, that the methods of blob extraction based on the analysis of the brightness and hue of
objects are of little use in the recognition of the rats in the cage with the sawdust background too.
The reason for this is in that the color of the rats is about the same as the color of the sawdust.
Theoretically speaking, the computer vision methods can differentiate the hues of the rats and the
sawdust, but, in practice, the illumination inside the cage often is non-uniform andthe color of the
rats and sawdust background is influenced by the shadows and reflections of the light from the
colored objects and plastic walls of the cage. As a result of this, the standard methods of
blob extraction based on the brightness and color also mistakes.

3. The Experimental Conditions
Let us consider a problem of intelligent video monitoring laboratory rats by the example of a
neurophysiological experiment on the study of cognitive possibilities of the animal. By the terms
of the experiment, one puts new objects into the cage with the animal. The animal explores
the new objects and the experimenter estimates the total time spend by the animal to explore
the objects. After a time, the experiment is to be repeated with the same objects. If the time
spent by the animal to explore the objects is less than one in the first test, the experimenter can
conclude that the animal remembers the objects. If the time is about the same, it means that
the animal forgot the objects.
The exploratory activity of the rat is usually manifested in that it approaches the object and
sniffs around the object (see Figure 1). It is difficult to describe the exploratory activity of the rat
in a formal way; thus a simplified approach to detection of the exploratory activity is often used
in the neurophysiological experiments: the distance between the rat and the object is estimated
and the cumulative time when the rat is situated close enough to the object is calculated.
Sometimes one estimates just the number of approaches of the rat to the object. This simplified
method leads to mistakes and is not applicable to the experiment under consideration because
the cage is small and the rat is situated not far from the object about all the time. At that the
rat can sniff around the object or ignore it; it can just lie or dig the sawdust near the object.

Figure 1. A laboratory rat investigates a new object in the cage. The logic program has
detected three blobs in the video: EEG cap that connects the head of the rat with EEG cable;
a new green object; and a new blue object placed in the cage.

An additional problem is in that we cannot detect the contour of the animal in the background
of sawdust. Thus we cannot recognize in a reliable way the face and forelegs of the animal.
Instead of this, we estimate the following attributes of the experimental setting with the help of
the low-level image processing procedures:

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 32
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

C A

O
B D

Figure 2. The attributes of the image to be used for detection of the exploratory activity of
the rat.
(i) The coordinates and the velocity of the centroid of the blob related to the body of the
rat are estimated using the texture-oriented method of image analysis that is sensitive to the
smoothness of the object surfaces. The body of the rat is well visible in the sawdust background
because the rat hair has a smoother surface in comparison with the sawdust.
(ii) The coordinates of EEG cap are detected using the color in the HSB space.
(iii) Exact coordinates of the objects placed in the cage are estimated using the color information
too. The only difference is in that the coordinates of the object are estimated permanently
in all the frames of the video and then averaged to avoid mistakes when the rat covers the
object.
Let B be the centroid of the blob corresponding to the body of the rat (see Figure 2). Let
C be the centroid of the blob corresponding to EEG cap of the rat. Let O be the object that is
nearest to the C point. Let A be a point on the contour of the object O that is nearest to the
C point. Let D be a point in the contour of the same object O that is nearest to the B point.
The following combinations of these attributes have been recognized as useful for the detection
of the exploratory activity of the rats during the experiments:
(i) If the distance between the A and C points is less than 1 cm, the rat probably sniffs the
object. If the distance between the A and C points is more than 4 cm, the rat probably
does not explore the object. If the distance lies in the interval 1–4 cm, additional analysis
is necessary to determine whether the rat explores the object or not. The analysis is
complicated by the fact that the rat can lie down sideways during the sniffing of the object;
in this case, the A − C distance can be big enough.
(ii) If the angle between the B − C and B − A lines is more than 50 degrees, the rat probably
does not investigate the object. This heuristic rule reflects the fact that the rat turns his
face to the object during the sniffing.
(iii) If the velocity of the B point is more than 3 cm per second, the rat probably just walks
around the object, but does not explore it.
(iv) If the ratio between the D − B and A − C distances is less than 1.3, the rat probably does
not explore the object. This rule reflects the fact that the rat usually stands aside from the
object and pulls his face to the object during the sniffing.
These heuristic rules were implemented in a logic program for detection of the exploratory
activity of the rats and tested in neurophysiological experiments.

4. The Logic Program for the Video Analysis
The Actor Prolog language has no built-in means of the fuzzy logical inference, however one can
easily implement a kind of a fuzzy logical reasoning using the standard top-down resolution and
standard built-in arithmetical operations. In particular, one can define the heuristic rules of rat
behavior analysis described above in the following way:
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 33
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

PREDICATES:
determ:
sniffing_is_detected(REAL,REAL,REAL,REAL) - (i,i,i,i);
imperative:
fuzzy_metrics(REAL,REAL,REAL) = REAL - (i,i,i);
The sniffing is detected predicate succeeds if an exploratory behavior of the rat is detected.
The predicate has four input arguments: distance A − C; distance D − B; the angle between
the B − C and B − A lines; and the velocity of the B point.
CLAUSES:
sniffing_is_detected(AC,_,_,_):-
AC < 0.01,!.
sniffing_is_detected(AC,_,_,_):-
AC > 0.04,!,
fail.
sniffing_is_detected(AC,DB,A,V):-
M1== ?fuzzy_metrics(DB/AC,1.3,0.1),
M2== 1-?fuzzy_metrics(A,50.0,10.0),
M3== 1-?fuzzy_metrics(V,0.03,0.01),
P== M1*M2*M3,
P > ?power(0.5,3).
The fuzzy metrics function is auxiliary one. It is used for definition of fuzzy thresholds. The
function has three input arguments: the value to be checked; the threshold; and the width of
the interval of uncertainty [14]:
fuzzy_metrics(X,T,H) = 1.0 :-
X >= T + H,!.
fuzzy_metrics(X,T,H) = 0.0 :-
X <= T - H,!.
fuzzy_metrics(X,T,H) = V :-
V== (X - T + H) * (1 / (2*H)).
In Figure 3, the user interface of the logic program intended for the analysis of the rat behavior
is shown. Control elements of the dialog window of the program allow selection of the video files
and examination of the video frames at various rates in the forward and backward directions.
The information on the total distance of movement, average velocity, cumulative time expended
by the rat for investigation of new objects in the cage is demonstrated in the top of the dialog
window. In the graphics window, the program demonstrates the frames of the video, detected
blobs, and auxiliary information related to the behavior analysis. The blob related to the body
of the rat is indicated by the orange color. The blob related to the EEG cap is indicated by
the cyan color. The green and blue blobs correspond to the new objects placed in the cage in
the course of the experiment. The logical rules defined above are used for the recognition of the
exploratory activity of the rat.
The logic program was tested on five r ats. T he r esults o f t he t ests a re g iven i n T able 1 : the
total distance passed by the rat during the test; the average velocity of the rat; and the time of
the exploratory activity of the rat (in percents of the total time of the test).
All five videos were marked manually to estimate the quality of the algorithm of detection
of the exploratory activity.

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 34
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

The results of the automatic recognition were compared with the results of the manual
marking to compute the sensitivity and specificity of the detection (see Table 2). The results
of the application of standard algorithms SVM [29] and ANFIS [30] are given in the table
for reference.
The tests demonstrated that the logic program ensures about the same sensitivity and
specificity (≈ 80%) as the SVM and ANFIS algorithms do on the base of the same data.

Figure 3. The user interface of the logic program intended for intelligent monitoring behavior
of laboratory rats. The program estimates cumulative time expended by the rat for investigation
of new objects in the cage.

Table 1. The results of analysis of laboratory rat behavior.
No. Total Distance [m] Velocity [cm/s] Exploratory
time [s] behavior [%]
1 181 1.572 0.9 63
2 189 2.97 1.6 52
3 184 3.17 1.7 36
4 181 3.355 1.9 28
5 171 1.12 0.7 16

Table 2. The sensitivity and specificity of detection of the exploratory activity of the rats.
No.
The logic program SVM ANFIS
sens. spec. sens. spec. sens. spec.
1 92 72 86 73 85 88
2 91 78 90 78 90 85
3 78 79 74 82 71 85
4 72 82 66 84 59 86
5 68 91 70 91 67 92
This quality of detection is comparable with the quality of manual detection and is enough
for conduction of the neurophysiological experiment. Note that an advantage of the logic
programming approach to the detection is in that it does not require a preliminary training of
the program and, that is more important, logical rules are understandable for the experimenter
and can be manually fixed and/or improved in any time. The values of the attributes used in
the rules (the thresholds of angles and distances, etc.) can be assigned manually or computed
(automatically or semi-automatically) on the base of statistical analysis of marked videos.
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 35
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

5. Conclusions
A method of intelligent video monitoring the laboratory animals based on the object-oriented
logic programming is developed. The method is intended for intelligent video monitoring the
laboratory rats in non-standard experimental conditions when one cannot apply existed software
for automation of neurophysiological experiments. In particular, this method is applicable for
analysis of the behavior of rats in cages with a sawdust background when the contrast of the
images is low and the illumination is non-uniform. In the framework of the method, heuristic
rules and fuzzy definitions a re u sed f or d escribing a nd r ecognition o f t he a nimal b ehavior. The
method is implemented on the base of the Actor Prolog [16] object-oriented logic language.

6. References
[1] Robinson L and Riedel G 2014 Journal of neuroscience methods 234 13-25
[2] Tscharke M and Banhazi T M 2016 Journal of Agricultural Informatics 7 23-42
[3] Spruijt B M, Peters S M, de Heer R C, Pothuizen H H and van der Harst J E 2014
Journal of neuroscience methods 234 2-12
[4] van Dam E A, van der Harst J E, ter Braak C J, Tegelenbosch R A, Spruijt B M and
Noldus L P 2013 Journal of neuroscience methods 218 214-224
[5] Noldus L P, Spink A J and Tegelenbosch R A 2001 Behavior Research Methods 33
398-414
[6] Ohayon S, Avni O, Taylor A L, Perona P and Egnor S R 2013 Journal of neuroscience
methods 219 10-19
[7] Weissbrod A, Shapiro A, Vasserman G, Edry L, Dayan M, Yitzhaky A, Hertzberg L,
Feinerman O and Kimchi T 2013 Nature Communications 4 2018
[8] Giancardo L, Sona D, Huang H, Sannino S, Manag`o F, Scheggia D, Papaleo F and Murino V 2013
PloS one 8 e74557
[9] Morozov A A and Sushkova O S 2016 Real-time analysis of video by means of the Actor
Prolog language Computer Optics 40(6) 947-957 DOI: 10.18287/2412-6179-2016-40-6-947-957
[10] Morozov A A, Sushkova O S and Polupanov A F 2017 Object-oriented logic
programming of 3D intelligent video surveillance: The problem statement IEEE 26th International
Symposium on Industrial Electronics (ISIE), Edinburgh, United Kingdom 1631-1636
[11] Morozov A A, Sushkova O S and Polupanov A F 2017 Towards the distributed logic
programming of intelligent visual surveillance applications Proceedings Advances in Soft Computing:
15th Mexican International Conference on Artificial Intelligence 2 42-53
[12] Morozov A A 2015 Pattern Recognition and Image Analysis 25 481-492
[13] Morozov A A and Polupanov A F 2015 Development of the logic programming
approach to the intelligent monitoring of anomalous human behaviour OGRW (Koblenz:
University of Koblenz-Landau) 82-85
[14] Morozov A A and Polupanov A F 2014 Intelligent visual surveillance logic
programming: Implementation issues CICLOPS-WLPE (Aachener Informatik Berichte) 31-45
[15] Morozov A A, Vaish A, Polupanov A F, Antciperov V E, Lychkov I I, Alfimtsev A N
and Deviatkov V V 2015 Development of concurrent object-oriented logic programming platform
for the intelligent monitoring of anomalous human activities BIOSTEC 511 82-97
[16] Morozov A A and Sushkova O S 2018 The intelligent visual surveillance logic
programming (Access mode: http://www.fullvision.ru)
[17] Haritaoglu I, Harwood D and Davis L S 1998 W 4: Who? When? Where? What? A
real time system for detecting and tracking people FG (Japan) 222-227

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 36
Image Processing and Earth Remote Sensing
A A Morozov and O S Sushkova

[18] Shet V, Harwood D and Davis L 2005 VidMAP: Video monitoring of activity with Prolog
IEEE AVSS 224-229
[19] O’Hara S 2008 VERSA-video event recognition for surveillance applications (M.S.
thesis. University of Nebraska at Omaha)
[20] Artikis A, Sergot M and Paliouras G 2010 A logic programming approach to activity
recognition International Workshop on Events in Multimedia 3-8
[21] Worch J H, Bálint-Benczédi F and Beetz M 2016 KI - Künstliche Intelligenz 30 21-27
[22] Borges P V K, Conci N and Cavallaro A 2013 IEEE Transactions on Circuits and
Systems for Video Technology 23 1993-2008
[23] Morozov A A 1999 Actor Prolog: an object-oriented language with the classical
declarative semantics IDL (Paris, France) 39-53
[24] Morozov A A 2002 On semantic link between logic, object-oriented, functional, and
constraint programming MultiCPL 43-57
[25] Morozov A A 2007 Operational approach to the modified reasoning, based on the
concept of repeated proving and logical actors CICLOPS 1-15
[26] Morozov A A 2003 Pattern Recognition and Image Analysis 13 640-649
[27] Chang C L and Lee R C T 1973 Symbolic logic and mechanical theorem proving (New
York: Academic Press)
[28] Morozov A A, Sushkova O S and Polupanov A F 2015 A translator of Actor Prolog to
Java RuleML DC and Challenge
[29] Cristianini N and Shawe-Taylor J 2000 An Introduction to Support Vector Machines
and Other Kernel-based Learning Methods (Cambridge: Cambridge University Press)
[30] Jang J S R 1993 IEEE Transactions on Systems, Man, and Cybernetics 23 665-685

Acknowledgments
Authors are grateful to Natalia V. Gulyaeva, Ilya G. Komoltsev, Anna O. Manolova, Margarita R.
Novikova, and Irina P. Levshina (IHNA and NPh RAS) for the experimental video data, to Yury V.
Obukhov (IRE RAS) for the problem statement, and to Alexander F. Polupanov (IRE RAS) for the help
in the research. We thank anonymous reviewers for the advertence and useful remarks on the paper.
This research is funded by Russian Science Foundation (project No. 16-11-10258).

IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 37