=Paper=
{{Paper
|id=Vol-2391/paper34
|storemode=property
|title=The peculiarities of interaction between the end-user and the remote sensing system for spatial objects detection and recognition
|pdfUrl=https://ceur-ws.org/Vol-2391/paper34.pdf
|volume=Vol-2391
|authors=Ruslan Brezhnev,Yuriy Maglinets,Ksenia Raevich,Gennady Tsybulsky
}}
==The peculiarities of interaction between the end-user and the remote sensing system for spatial objects detection and recognition ==
https://ceur-ws.org/Vol-2391/paper34.pdf
The peculiarities of interaction between the end-user and the
remote sensing system for spatial objects detection and
recognition
R V Brezhnev1, Yu A Maglinets1, K V Raevich1 and G M Tsibulsky1
1
Siberian Federal University, Institute of Space and Information Technology, Kirenskogo St.,
26, Krasnoyarsk, Russia, 660074
e-mail: brejnev.ruslan@gmail.com, ksenia_248@mail.ru
Abstract. The paper discusses the requirements for the system of remote sensing (hereinafter
referred to as “System”) which is focused on the end user (EU), based on the concepts of
object-oriented monitoring. The classification of simple queries generated with interactive
tools in the context of the end user’s subject area is presented. The research investigates the
features of the System needed for maintaining task-setting interaction, such as the subject
ontology, including knowledge about the object of interest, its static and dynamic properties,
the hierarchy of vector layers that describe spatial objects, the updated image database. The
authors indicate the resources necessary for solving specific tasks, including spatial data
repository and specification of computational procedures capable of interpreting the query in
terms of decision operators. Specifically, the paper considers the structure of task-setting
interaction tools, which ensures the object space-time localization, finding its meaning in
semantic space, performance specification parameters measured one time or in dynamics,
requirements for the visualization of the results and the activity of the System in relation to the
end user in the monitoring process. The solutions presented were tested on monitoring the
condition of agricultural lands in the multi-task space monitoring system of the Institute of
Space and Information Technology, Siberian Federal University, Krasnoyarsk.
1. Introduction
At present there is an intrinsic gap between the existing interfaces of geographic information systems
and the information need of the end user, who, as a rule, operates within a closed professional
environment. New tools are needed to reconcile the mental model of the end user, on the one hand,
and the totality of the processed data, the algorithms of data assimilation, presentation and processing
on the other. They should provide the end user with an interactive environment where he could set
monitoring tasks in the usual terms of objects, attributes, classifiers, indicators, charts, graphs, etc. The
most intuitive in this sense is the object-oriented paradigm to analyze complex subject areas, which is
often used in practice to set the task of implementing software solutions that perform specific
functions [1,2]. The paper discusses the possibility to use this concept for organizing interaction with
an end user, when it does not arise between the problem originator and the system analyst, but
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019)
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
between the problem originator and the decision system itself. At present, ontological approach, which
reflects many aspects of the object-oriented view on representing geographic knowledge, is widely
used in intellectual systems. However, due to the high degree of this approach formalization, direct
interaction between the end user and the systems such as, for example, Protégé [3] is cumbersome as it
involves intermediary – knowledge engineer. Next, we consider the issues of building an interactive
environment focused on the end user and supporting the capabilities of the System within the object-
oriented monitoring concept developed by the team of the authors.
2. Initial data for the task setting process
The main variables discussed in setting the task of spatial object remote research are: the localization
of the object of interest (context) and the informative characteristics to be determined.
The object is localized in the coordinate system of the earth’s surface, in time, and in semantic
space.
Spatial localization can be carried out based on the use of a priori information about the study
object (SO) position, represented as a pair of points defined in space by coordinates (x, y), which
identify rectangular search area, or a polygonal vector object represented in one of the GIS- formats.
In addition, the position of the object can be specified by the user directly, based on substrate maps, or
by specifying the elements of the existing vector maps, for example, administrative or cadastral
division, departmental affiliation, land use maps, etc. The position of an object can also be specified
indirectly, based on its non-spatial properties, defined by the user and applying the knowledge of the
System.
Localization in time can also be stated either explicitly, by specifying a time reference or interval,
or indirectly, based on the characteristics of the object life cycle. Depending on the nature of the
problem being solved, the following time localization options are possible: “now”, “exact date in the
past”, “last measurement results”, “forecast” (for a future date), “monitoring” (a series of
measurements with a specified interval).
The process of localization in semantic space, which, unlike the previous two methods of
localization, depends essentially on the nature of the object and the subject area of research, is
characterized by the greatest variability. In the first approximation, we assume that the specified
process comes to choosing one or more classes from a predetermined set of domain classifiers. The
examples presented in the article are given for the subject area of detecting agricultural land (AL) use
with remote sensing.
Substantive characteristics to be determined can be divided into static, dynamic and structural.
Depending on the nature of the problem being solved, it can be formulated as an object recognition
task, a measurement task, or a prediction of the numerical value of one or more object attributes, a
segmentation task (for heterogeneously structured objects). In the latter case, it is also possible to set
the task of recognition or measurement for each of the selected segments. The recognition task
requires the assignment of the observed object to a class defined in one of the domain classifiers. The
statement of the measurement (forecasting) task requires the correlation of the measured property with
the corresponding element of the attribute classifier.
3. The structure of interaction dialogue frames for the problem setting
Formally, the structure of interaction on setting the object monitoring task can be represented as a
directed graph:
Gr ( P,V ) , (1)
where P is the set of vertices defining the nature of the action (operation) in the interaction, V is the set
of directed edges defining the execution paths for operations. The set of the graph vertices consists of
two subsets:
P S EU , (2)
where S is a subset of the operational vertices of the system, EU is a subset of the operational
vertices of the user.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 251
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
The main stages of the task setting: 1) defining the context; 2) clarifying the goal; 3) developing the
requirements for representation of findings.
The object is localized along three “axes” of interaction: in the coordinate system of the earth’s
surface, in time and in semantic space, each of which is related to an interaction frame. The preferred
direction of interaction is determined by the user. For example, when choosing a spatial axis, the user
can specify the spatial coordinates of the desired point of the monitored object. If the object is
recognized by the system, it notifies the user about the main characteristics of this object, thereby
setting localization indirectly in the semantic space. In another possible scenario, the user can select
one of the object classifiers and specify the required classifier vertex. When switching to a frame with
a geospatial representation, the areas corresponding to the selected vertex at the current time are
activated. When switching to the time frame, the user can select the observation interval. If an interval
in the past is selected, corresponding changes should occur in the spatial frame, displaying the
required period of the object observation.
In the general case, various localization results are possible, affecting the further course of the task
solution process. Thus, the result of localization in space is the exact fixation of a single spatial object
(AL), a set of objects (AL sets) united by common features, for example, spring wheat fields of
“MAYAK" agricultural company, or indication of the region of interest where these objects can be
located. When setting the time interval, we determine the need for: a) obtaining immediate
information; b) obtaining retrospective information; d) monitoring; c) forecasting.
3.1. Clarifying the goal
For an object (class of objects) defined during the previous stage of interaction, we determine the set
of characteristics to be evaluated. A set of characteristics available for selection depends on the model
for describing semantic characteristics used in the System. In our example, these are properties
associated with the corresponding nodes of the objects classifiers, such as “area”, “degree of
heterogeneity” [4], “phytomass volume assessment”, “yield assessment”, “integral characteristic of the
crops condition”, “germination percentage", "surface layer temperature ", etc. Structural analysis is
possible for agricultural fields, with a high degree of heterogeneity during the field season, caused, for
example, by partial nonemergence of crops, or by extreme values of soil moisture in the local areas of
the field [5]. It includes segmentation of the zone of interest with the selection of homogeneous areas.
For each of the selected areas it is also possible to specify the properties required for measurement.
When setting the forecasting problem in the subject area under consideration, as a rule, we assume
the expected phytomass volumes, or the yield per unit area. At the considered stage of interaction, the
appropriate forecast model is selected from those available in the System.
When setting the monitoring task, as well as the task of current observation, one or more
characteristics to be monitored, as well as the method of organizing the monitoring process, are
determined.
3.2. Developing the requirements for representation of findings
At this stage of interaction, the forms of findings representation on request are refined - numerical
values, tables, elements of business graphics, thematic maps, and atlases. Since there is no complete
formalization of tasks in practice, the resulting images of the study objects are also of interest to the
user. They can be analyzed together with the information structured above.
When setting the monitoring task, particular interest consists in defining the probability of the
System prompt response to the anomalous behaviours of the study object, detected during the analysis
of timely remote sensing data. For example, when monitoring fallow lands, these are the excessive
values of vegetation indices, indicating the presence of phytomass in the field and, indirectly, the use
of inadequate tillage methods. The active method of monitoring provides for the possibility to notify
the user about negative scenarios of agricultural lands dynamics immediately when such information
is obtained by the System.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 252
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
4. Problem solution utilities of the System
A subject knowledge base pre-formed in the System, which restricts the class of study objects defining
the area of study and information about their spatial localization, is a prerequisite for organizing an
interaction. The research relies on the ontology-based approach to describe and extract the semantics
of agriculture objects, developed in [6,7]. To localize objects in the semantic space, a system of
thematic classifiers is presented there. To describe the measured attributes, the ontology properties are
used, which are characterized by the name, unit of measurement, allowable range of values and
computational measurement procedure.
The other necessary components of the system are: a cartographic base and a set of vector layers
characterizing the spatial localization of objects and their attribute characteristics describing factual
information. To organize monitoring, the System should have access to space survey data that arrives
at the intervals determined by the characteristics of the object's life cycle. Accordingly, it is crucial to
maintain a constantly updated database of remote sensing, which has assimilation, preprocessing,
indexing, storage utilities and access to the specified data, as described in [8]. The alternative is to use
web services that provide processing of information at place of its obtaining and delivering the results
of processing to the consumer (see for example, [9–16]).
For each of the attributes calculated in the system, we should present a computational procedure to
determine the sequence of spatial data processing operators and the conditions for their use [12]. This
functionality is supported by the availability of spatial information processing modules in the System
repository.
For active monitoring, as a supporting capability we need a formalized model of the object
development over time. Thus, the crops vegetation period includes an interconnected series of
phenophases changes, anthropogenic (farming operations) and environmental influences, which can be
interpreted as positive or negative in a specific context [13]. With software prototyping we designed
the interactive tools developed. The prototype is implemented as an additional module of the multi-
task space monitoring system of the Institute of Space and Information Technology, Siberian Federal
University, Krasnoyarsk [8,14].
5. Experimental Testing
The main provisions of the concept of man-machine interaction for setting and solving different tasks
of spatial objects remote monitoring were implemented for obtaining the crops condition information,
taking into account their uneven growth over time. Within the structure of interaction tools described
above, the task setting process can be decomposed into the following steps (Figure 1):
5.1. Design of a subject classifier by a knowledge engineer
The stage allows introducing priori descriptions in the form of thematic sub-classifiers containing the
monitoring feature classes, as well as their unary and binary properties. For the task in question, the
structure of the classifier is formally as follows [13]:
ci ID,{x, y}, Ф, K , E, Tm, F , NS , NP, T , N , (3)
where ci is the classifier of the AL objects, ci С , С is the global taxonomy classifier; ID – is
object identifier; {x, y} – is the set of geographic coordinates of the object in a given projection; K –
is the set of agricultural crops (property-object); Ф is the set of phenological phases throughout the
growing season; E – is the sequence of planned farming operations (FO); Tm is the set of reference
time intervals ei E , ei (k i , tmi ) and i Ф , i (k i , tmi , N i ) , where tmi is the time of
measurement, and k i K ; N - is the set of reference intervals of vegetation index values, associated
with phenological phases, allowing to interpret the current condition of vegetation; F is the set of the
object conditions. The relation f i ( N i ) is defined. In addition, the measured properties include
NS , NP, T – area, perimeter and thickness, respectively.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 253
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
Figure 1. The task decomposition pattern (Monitoring crop growth conditions throughout the growing
season, taking into account uneven growth over time).
5.2. Setting the position of an object in space
Setting the position of an object in space is the part of interaction. Based on (1) and (2), its model, can
be represented as a graph Gr1 (Figure 2). The object localization model includes the following
specific steps:
The system displays a map of the "AL" objects ( S11 ).
The end user localizes an object (or objects, by multiple selection) on the basis of the “AL” object
map ( EU 11 ).
The system identifies an object on the map visually ( S 21 ).
The system displays an object operation panel ( S 31 ).
The end user assigns the object to its agricultural company (AC) ( EU 21 ).
The Information System (IS) sets the ID j of the assigned object, fixes its coordinates {xi , yi }
( S 41 ).
The Information System displays a map of selected objects «AL» ( S51 ).
S11 S 21 S 31 S 41 S 51
EU 11 EU 21
Figure 2. Graph Gr1 : Spatial localization of an object.
5.3. The interaction of assigning a crop
The interaction of assigning a crop can be represented as a graph Gr2 (Figure 3), which includes the
following steps:
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 254
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
The system displays a map of the “AL” objects assigned to the agricultural company ( S12 ).
The end user localizes an object (or objects, by multiple selection) on the basis of the object map
of the company's agricultural lands ( EU 12 ).
The system identifies the object on the map visually ( S 22 ).
The system displays an object operation panel ( S 32 ).
The end user assigns the crop k i with the normative time points Tm for updating the state of the
object, i.e. measurement frequency of specified characteristics to the selected object ( EU 32 ) on
the basis of a given classifier. Each object can have individual time intervals between the state
changes, which are measured from the actual sowing date.
The system displays a map of "AL" objects with the given crops ( S 62 ).
S12 S 22 S 32 S 62
EU12 EU32
Figure 3. Graph 𝐺𝑟2 : Assigning a crop.
5.4. The process of setting the properties of an object
The interaction graph Gr3 depicts the process of setting the properties of an object to be monitored
(Figure 4). The main steps of interaction are:
The system displays a map of "AL" objects assigned to the agricultural company ( S 83 ).
The end user localizes an object (or objects, by multiple selection) on the basis of the “AL”
objects assigned to AC ( EU 13 ).
The system identifies the object on the map visually ( S 23 ).
The system displays an object operation panel ( S 73 ).
The end user sets restrictions on the values of size NS ( EU 43 ) and form ( EU 53 ) parameters to be
calculated.
The end user confirms the request ( EU 63 ).
S 83 S 23 S 73
EU 13 EU 43 EU 63
EU 53
Figure 4. Graph Gr3 : The interaction for setting the object properties to be monitored.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 255
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
5.5. The task solution stage
The task solution stage is considered as an information request which is impossible without prior
training a multi-task monitoring system. The learning processes lie outside the framework of the
considered interaction and consist in establishing the relation between subject classifiers and pre-
programmed algorithm being program modules. The modular architecture implements the principle of
reusing system components for solving various tasks, including those in parallel mode. At the physical
level, this approach enables to represent a given operation for processing or analyzing data as a
separate module, and a set of operations as a set of logically interrelated software modules running in a
given sequence, with a given set of input parameters, at a specified time.
Thus, when choosing an object of a certain class, the System knows in advance which algorithm is
required to be executed. If several algorithms are available for a class of objects, then we should
provide steps how to choose the relevant one. At the same time, taking into account the fact that this
interaction is user-focused, it is possible to use the natural language form of the search algorithm,
which will in essence have the form of interaction “NL request - function”.
So, in the considered task of monitoring crops condition, taking into account their uneven growth
over time, the object class “AL” is connected with an algorithm for localizing a spatial object with
heterogeneous structure. The practical result of this task solution is a series of “AL” object maps
displaying structural heterogeneity with the interpretation of conditions, some of which are presented
in Figure 5. The results are visualized to the user as vector layers in the Monitoring System. On the
one hand the number of the formed layers is limited by the number of time points of the object
properties being actualized, on the other hand, by remote sensing data, which are available under
favorable weather conditions.
21.05.2018 25.06.2018 03.07.2018
Figure 5. The time series of maps displaying the AL object inhomogeneous structure (ID = 164).
6. Conclusion
The paper proposes a concept of designing an interaction environment for the end user of the remote
sensing system for spatial objects detection and recognition within the paradigm of object-oriented
monitoring. The following issues were considered in detail: 1) Initial data and constraints for the
problem setting, which are grouped into the following categories: the representation of the object of
interest in time, in the coordinate system of the Earth’s surface, in semantic space, and in the target
characteristics. 2) Interaction dialogue frames needed for a System to interact with the end user,
including knowledge base for the subject area of research, a set of vector layers characterizing the
spatial localization of objects and their attribute characteristics, the remote sensing data stream,
implemented in the form of regularly updated image database or web-services, as well as functionality
for measuring the required properties of the object. 3) The process of task setting in terms of
interaction steps, which help to define the necessary context; goal clarification; development of
requirements for representation of findings.
The interactive tools were developed based on software prototyping within the project “Multi-task
space monitoring system of the Institute of Space and Information Technology, Siberian Federal
University, Krasnoyarsk”.
«This study was supported by the Russian Foundation for Basic Research (grant №: 18-47-242002
p_mk), the Government of the Krasnoyarsk Territory, the Krasnoyarsk Regional Science Foundation
as a part of the research project titled “Development of technology for creating intelligent information
systems of object-oriented monitoring of territories based on remote sensing data".
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 256
Image Processing and Earth Remote Sensing
R V Brezhnev, Yu A Maglinets, K V Raevich and G M Tsibulsky
7. References
[1] Nedoluzhko I V, Eremenko V S and Eremenko A S 2016 Modern problems of remote sensing
(Moscow) p102
[2] Korets M A 2014 Regional problems of Earth remote sensing (Krasnoyarsk) 294-298
[3] Brezhnev R V and Maltsev E A 2015 Pattern Recognition and Image Analysis 25 201-208
[4] Shukilovich A Yu, Fedotova E V and Maglinets Yu A 2016 J. Sib. Fed. Univ. 9(7) 1035-1044
[5] Brezhnev R V and Maglinets Yu A 2017 CEUR Workshop Proceedings 2033 247-251
[6] Raevich K V, Maglinets Yu A and Tsibulsky G M 2016 J. Sib. Fed. Univ. 9 1025-1034
[7] Maglinets Yu A, Raevich K V and Tsibulsky G M 2017 Proc. Eng. 201 331-340
[8] Brezhnev R V, Maglinets Yu A, Maltsev E A, Perfilyev S E, Sidorov A Yu, Tsibulsky G M and
Shocol A S 2012 J. Sib. Fed. Univ. 5 340-352
[9] Bartalev S A and Lupyan E A Current problems in remote sensing of the earth from space 10
197-214
[10] Tolpin V A, Lupyan E A, Bartalev S A, Plotnikov D and Matveev A M 2014 Optics of the
atmosphere and the ocean 27 581-586
[11] Lupyan E A, Bartalev S A, Tolpin V A, Zharko V O, Krasheninnikova Yu S and Oksyukevich
A Yu 2014 Current problems in remote sensing of the earth from space 11 215-232
[12] Brezhnev R V and Perevalova A A 2018 Regional problems of Earth remote sensing
(Krasnoyarsk) 40-43
[13] Brezhnev R V, Maglinets Yu A, Raevich K V and Tsibulsky G M CEUR Workshop
Proceedings 2210 316-322
[14] Maglinets Yu A, Tsibulsky G M and Noskov M V 2016 J. Sib. Fed. Univ. 9 1012-1018
[15] Plotnikov D E, Kolbudaev P A, Bartalev S A 2018 Identification of dynamically homogeneous
areas with time series segmentation of remote sensing data Computer Optics 42(3) 447-456
DOI: 10.18287/2412-6179-2018-42-3-447-456
[16] Varlamova A A, Denisova A Y, Sergeev V V 2018 Earth remote sensing data processing for
obtaining vegetation types maps Computer Optics 42(5) 864-876 DOI: 10.18287/2412-6179-
2018-42-5-864-876
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 257