<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Prediction and Visual Intelligence Platform for Detection of Irregularities and Abnormal Behaviour</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Konstantinos Demestichas</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Theodoros Alexakis</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nikolaos Peppes</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Konstantina Re- moundou</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ioannis Loumiotis</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Wilmuth Muller</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Konstantinos Avgerinakis</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute of Communication and Computer Systems</institution>
          ,
          <addr-line>9, Iroon. Polytechniou Str</addr-line>
        </aff>
      </contrib-group>
      <abstract>
        <p>Nowadays, (cyber)criminals demonstrate an ever more increasing resolve to exploit new technology so to achieve their unlawful purposes. Therefore, Law Enforcement Agencies (LEAs) should accommodate an approach that surpass the existing limits in policing practices. In this light, the authors introduce an innovative platform that provides near real-time advanced social behavior analytics using irregularities detection based on historical patterns.</p>
      </abstract>
      <kwd-group>
        <kwd>Big Data</kwd>
        <kwd>abnormal behaviour detection</kwd>
        <kwd>crime detection</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons</p>
      <p>License Attribution 4.0 International (CC BY 4.0).</p>
      <p>The proposed high-level system architecture provides the required tools for LEAs to
accelerate their investigations and remain careful in consideration of terrorist and
cyber-criminal threats by successfully integrating massive data streams from
heterogeneous sources. The system architecture is designed so to meet performance and
resiliency requirements at scale. Specifically, in the following paragraphs of this Section
they are presented with details all the main functional modules as well as the
components that the system generally is consisted of, in a coherent manner. The details supply
in a comprehensive way the structure of the system architecture based on the
combination of the different components into the common proposed platform which can be
deployed into LEAs’ and practitioners’ facilities. The main effort is being put on
standardization of the platform’s open architecture and its constituent components, interfaces
as well data exchange formats. In order to succeed, extensive monitoring studying and
contribution into activities related to standards of ISO/TC 292 (Security and Resilience)
in the area of security is foreseen. In short, the system is composed of: i) Data Mining
Module for Crime Prevention and Investigation, ii) Visual Intelligence Module, iii)
Semantic Information Representation and Fusion Module, iv) Trends Detection and
Probability Prediction Module for Organized Terrorism and Criminal Activities, v)
Detection Module of Cyber-Criminal Activities and Situation Awareness and vi) HMI
Module. The next figure illustrates the high-level architecture of the presented platform.
Data mining is used to extract valuable information from the existing data from various
sources such as web, dark web, social media, etc. as shown in Figure 2. Crawling and
mining take place by using crawl points (data sources) from which posts and references
are extracted and stored for analysis purposes. Eventually data from multiple sites are
processed and exported into a common format that is available to the other components
for further processing and analysis.</p>
      <p>To develop and study crime patterns we used existing open-source web and social
media mining tools in conjunction with API’s provided by social media as well as
extensive and scalable open-source web crawlers software projects. However, the
difficulty of accessing Dark Web sites as well as the existence of specific rate limits on
social media constitute a possible risk. The contingency plan pays attention to key Dark
Web Sites and narrow social media access by using specific keywords and phrases.
2.2</p>
      <sec id="sec-1-1">
        <title>Visual Intelligence Module</title>
        <p>
          The visual intelligence module is implemented by using available face recognition and
face detection algorithms such as YOLO [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ], SSD [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ], Faster R-CNN [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ], VITAL [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ]
and other widely used and tested algorithms. This specific module is engaged in order
to achieve identifications of persons and objects contained in images and videos
crawled from the Web, social media and footage from static or moving cameras as well
as surveillance systems. The face and object recognition through visual data is focused
on the detection of suspicious objects. Finally, suspicious or abnormal activities are
also being tracked by the adoption of crowd analysis and human action recognition with
spatio-temporal localization.
        </p>
        <p>
          The starting point for the prementioned implementations comprises Deep
Convolutional Network, Deep faces, SSD coupled with a YOLO architecture, KCF, Goal-based
descriptors and Swarm intelligence for crowd analysis and abnormal activity detection
[
          <xref ref-type="bibr" rid="ref8">8</xref>
          ]. Alongside with visual intelligence techniques, machine learning algorithms and
techniques are used in order to predict and estimate outliers and abnormal person and
object activities based on trends and motifs discovered by the gathered visual data.
2.3
        </p>
      </sec>
      <sec id="sec-1-2">
        <title>Semantic Information Representation and Fusion Module</title>
        <p>Another tool of the described platform is the Semantic Information Representation and
Fusion Module that gathers data from various and heterogeneous sources like:
geospatial data, web data, darknet data, video/image data, road traffic data, financial data,
telecom data, social networks data and information and security systems data. Thus, a
module dedicated to data and information fusion is mandatory. This module is able to
apply information fusion to the heterogeneous data that are collected from the different
sources listed above. The development and application of this module results to
information transformation into valuable knowledge. The baseline of this module includes
Knowledge graphs, JDL model, OWL language and Markov Logic Networks along
with the use of appropriate semantic information fusion models. Thus, the module can
extract useful patterns and hidden relationships among different datasets that can lead
to trends discovery and abnormal behaviour detection.
2.4</p>
      </sec>
      <sec id="sec-1-3">
        <title>Trends Detection and Probability Prediction Module for</title>
      </sec>
      <sec id="sec-1-4">
        <title>Organised Terrorism and Criminal Activities</title>
        <p>All the aforementioned modules gather data from various sources in order to create
various datasets. Generating all these datasets the next step is to predict organised
terrorism and criminal activities. In this direction big data analytics techniques are applied
over the collected data in order to identify hidden trends inside the datasets. Analytics
results lead to predictive models which can be considered as the link between data and
decision-making processes.</p>
        <p>This module uses machine learning algorithms which are developed by engaging
appropriate open source libraries alongside with predictive policing software. More
specific, the platform engages many widely used Artificial Intelligence Algorithms
such as Artificial Neural Networks (ANN), decision trees, pattern recognition and
Lifelong Learning Algorithms (LLA).</p>
        <p>The risks that the development engineers spotted are the inaccurate results that were
extracted from the prediction model and the false positive alert that a model generated
in some cases. These risks may cause an increasing false alarm rate (FAR) in the early
stages of deployment which can be tackled by the user feedback. A proposed
contingency plan foresees the use of data sources of higher degree of diversity as well the
creation of more sophisticated models for explaining deviant behaviours.
2.5</p>
      </sec>
      <sec id="sec-1-5">
        <title>Detection Module of Cyber-Criminal Activities</title>
        <p>In addition to trends detection and probability prediction module for organised
terrorism and criminal activities that presented in paragraph 2.4 there is the detection module
of cyber-criminal activities. This very module focuses not only to identify anomalies
but also on behavioural indicators as well as revealing previously unknown associations
and rules that are connected to cyber-criminal activities. For these purposes advanced
big data analytics techniques, based on artificial neural networks and classification
methods are applied to the collected data. In this light the three widely used machine
learning algorithms: K-means clustering, Support Vector Machines and Deep learning
algorithms, are being used in this module. The development and integration of these
algorithms was held with open source libraries for numerical computation in order to
achieve faster results.</p>
        <p>As presented above in paragraph 2.4 risks which appeared in module are not only
the inaccurate results of the model but also poor-quality model results over time. Thus,
again contingency plan gives attention to the selection of more complex model, fit the
training frequency as well test and modify the model.
2.6</p>
      </sec>
      <sec id="sec-1-6">
        <title>Situation Awareness and HMI Module</title>
        <p>Last but not least we have the situation awareness and HMI Module. This module
demonstrates to the end-users the gathered information and analysis results produced
by the aforementioned modules in order to increase the situation awareness of the
decision makers and practitioners. The baseline comprises open-source libraries for visual
analytics in addition to powerful, secure, and flexible end-to-end analytics platforms
for data visualization and representation purposes.</p>
        <p>Possible problems and risks that may occur and must be overcome could be the
inadequate offered visualization tools for some LEAs, the requirement for additional data
views in certain use cases and the difficulty in using and handling the visualization
environment. In addition, different LEAs use different tools so the adoption of a new
tool must be as close as possible to their tools. Thus, the developers of the platform take
into account the feedback from LEAs and end users in order to create a common
userfriendly HMI. The design of the HMI follows the main design principles of the LEAs’
HMIs and tries to simplify the environment in order to attract users to adopt it. It is of
outmost importance to receive the feedback from users in the early stage of deployment
in order to update and patch the visualization tools so to assure the credibility of the
platform’s results and increase the LEAs productivity.
3</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>Conclusion</title>
      <p>In conclusion, taking into account the increasing needs of LEAs for future-proof
solutions and expertise adoption so to fight crime, we presented a platform and its modules
which engage state-of-the-art tools and technologies. The modules presented are
interconnected and aim to enhance LEAs with supreme crime prediction and prevention
capabilities. Finally, it offers a future-proof framework that is open to the deployment
of new situation awareness applications, novel cognitive services and additional data
stream analytics tools, both from first- and third- parties, adopting standard and
welldocumented interfaces.</p>
      <p>It is worth noted that the development and the deployment of this very platform
acknowledges the legal, privacy, ethical and societal concerns of predictive policing
and data science method and integrates independent assessment, while participating
into an open dialogue with civil society organisations, security stakeholders,
practitioners and policy makers.</p>
      <sec id="sec-2-1">
        <title>Acknowledgement</title>
        <p>This work has been performed in the context of the PREVISION project, which has
received funding from the European Union's Horizon 2020 research and innovation
programme under grant agreement No 833115. The paper reflects only the authors'
view and the Commission is not responsible for any use that may be made of the
information it contains.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Hu</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ramanan</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Finding Tiny Faces</article-title>
          .
          <source>In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</source>
          , pp.
          <fpage>1522</fpage>
          -
          <lpage>1530</lpage>
          , (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Davis</surname>
            ,
            <given-names>P. K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walter</surname>
            ,
            <given-names>L. P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brown</surname>
          </string-name>
          , R. A.,
          <string-name>
            <surname>Douglas</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parisa</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Voorhies</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Using Behavioral Indicators to Help Detect Potential Violent Acts: A Review of the Science Base</article-title>
          , Santa Monica, Calif.: RAND Corporation, RR-215
          <string-name>
            <surname>-</surname>
            <given-names>NAVY</given-names>
          </string-name>
          , (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Jeelani</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Muqeem</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <article-title>Big data and semantic web, challenges and opportunities a survey</article-title>
          .
          <source>International Journal of Engineering &amp; Technology, 7</source>
          , pp.
          <fpage>631</fpage>
          -
          <lpage>633</lpage>
          , (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Hu</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ramanan</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <article-title>Finding Tiny Faces</article-title>
          ,
          <source>In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</source>
          , pp.
          <fpage>1522</fpage>
          -
          <lpage>1530</lpage>
          , (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Derpanis</surname>
            ,
            <given-names>K.G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lecce</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Daniilidis</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wildes</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <article-title>Dynamic scene understanding: The role of orientation features in space and time in scene classification</article-title>
          ,
          <source>In: Proceedings CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Conference on Computer Vision and Pattern Recognition</source>
          , pp.
          <fpage>1306</fpage>
          -
          <lpage>1313</lpage>
          , (
          <year>2012</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Ren</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>He</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Girshick</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sun</surname>
            ,
            <given-names>J. Faster R-CNN</given-names>
          </string-name>
          :
          <article-title>Towards Real-Time Object Detection with Region Proposal Networks</article-title>
          ,
          <source>In: IEEE Transactions on Pattern Analysis and Machine Intelligence</source>
          <volume>29</volume>
          ,
          <fpage>91</fpage>
          -
          <lpage>99</lpage>
          (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Song</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ma</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wu</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gong</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bao</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zuo</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shen</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lau</surname>
            ,
            <given-names>R.W.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yang</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>VITAL: VIsual Tracking via Adversarial Learning</article-title>
          ,
          <source>2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition</source>
          , pp.
          <fpage>8990</fpage>
          -
          <lpage>8999</lpage>
          , (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Kaltsa</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Briassouli</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kompatsiaris</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hadjileontiadis</surname>
            ,
            <given-names>L. I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Strintzis</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>Swarm Intelligence for Detecting Interesting Events in Crowded Environments</article-title>
          ,
          <source>IEEE Transactions on Image Processing</source>
          ,
          <volume>24</volume>
          , pp.
          <fpage>2153</fpage>
          -
          <lpage>2166</lpage>
          , (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>