<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Cyber range automation, a bedrock for AI applications</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Leonardo Gavaudan</string-name>
          <email>lgavaudan@nukk.a</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Swann Legras</string-name>
          <email>slegras@nukk.ai</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Véronique Ventos</string-name>
          <email>vventos@nukk.ai</email>
        </contrib>
      </contrib-group>
      <pub-date>
        <year>2021</year>
      </pub-date>
      <fpage>165</fpage>
      <lpage>176</lpage>
      <abstract>
        <p>This paper proposes an automated solution for conducting cybersecurity research. It shows how automation can improve the foundations of cybersecurity research, and consequently facilitate and bolster the development of artificial intelligence applications. The challenges that cybersecurity researchers face are discussed, as well as what automation ofers to tackle them at three diferent stages: provisioning, configuration, and attack simulation.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;cyber range</kwd>
        <kwd>security automation</kwd>
        <kwd>security testing</kwd>
        <kwd>adversary emulation</kwd>
        <kwd>threat hunting</kwd>
        <kwd>infrastructure-as-code</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>advanced AI applications remains an extremely dificult
task. An automated cyber range is a cyber range where
Over the last few years, the pace of cyber attacks has the deployment and configuration of the infrastructure,
particularly accelerated; their complexity and reach haveand initial installation of red teaming tools are automated.
relentlessly been growing. The 2020 Solarwinds attackB,y automating its setup, cybersecurity and AI researchers
an attack estimated to have infiltrated thousands of or- are empowered to focus on studying the core of an attack,
ganizations among which United States government sys- and building AI applications, rather having to worry
tems, is perhaps the best example of the trend. From the about the underlying groundwork.
start of 2020, the following major attacks can be already Given that the bottle neck to AI research and
develnamed: Solarwinds, Colonial Pipeline, JBS, Microsoft Ex- opment is a lack of good datasets, the aim of this paper
change Servers. The growing recognition of the need is to illustrate an automated cyber range platform that
for artificial intelligence applications in the realm of cy-researchers can use to easily generate and access high
bersecurity research in order to respond to the increased quality, and diverse attack simulation datasets. The
concomplexity and frequency of these attacks, is paralleledtribution of this paper is in the showcase of how and
with a lack of a good ecosystem for them to flourish. why existing open source technologies can be assembled
The current cybersecurity research process has more of to build an end to end platform solution for
cybersecua manual approach, and is not best suited for artificial rity automation. This paper explains how the choice
intelligence development. for each component of the solution is justified. More</p>
      <p>According to the European Defense Agency, a cyber importantly, it will compare the platform solution as a
range is ”a multipurpose environment in support of 3 whole to the current state of practice for end to end
auprimary processes: knowledge development, assurance tomation solution. As much as this paper is an abstract
and dissemination” composed of ”three complementary and theoretical explanation for cybersecurity automation,
functionality packages”: a Cyber Research Range (CRR),it is also a practical guide. That is why the paper will
a Cyber Simulation &amp; Test Range (CSTR), and a Cyber instantiate the proposed solution through an advanced
Training &amp; Exercise Range (CTER)1[]. It is a common persistent threat simulation example. The advanced
perissue for cybersecurity researchers who want to study sistent threat example will help depict the technologies,
a particular attack or technique to end up realizing justas well help researchers to start implementing, and using
how incredibly arduous the process of setting up such them.
a cyber range is. Cybersecurity professionals looking The plan of the paper is as follows: To begin with, in
to get started with AI development, and AI researchers section 2, we take a look at the APT29 attack simulation
looking to develop applications are confronted with aexample, and current end to end solutions. In section 3,
same problem. Without access to both a repository of we inspect the current way cybersecurity research takes
good datasets and a system to keep it up to date, the place and its shortcomings. We then provide, in section
mission for developing production ready, up to date and 4, a comprehensive automated solution that addresses
the challenges discussed in the previous section. Finally,
in section 5, we examine how cybersecurity automation
positively impacts artificial intelligence development.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Prerequisites &amp; related works</title>
      <p>2.2. Existing Cyber Ranges
In this section, we look at the context from which the 2.2.1. Splunk - Attack Range
scenario of APT29 is drawn, and why it is fit to describe The first end to end solution proposed is Splunk’s
Atthe cybersecurity automation landscape in2.1. We then tack Range[5]. Introduced in 2019 and developed by the
look at two existing end to end solutions, one of whom Splunk Threat Research Team, Attack Range is the state
represents current state of the art and practice in2.2. Both of art and state of practice when it comes to integrating
the pros and cons of the existing end to end solutions are multiple technologies to automate each stage in the
cyanalyzed, as well as how the solutions compare to thebersecurity research process. It takes care of deploying
one proposed in this paper. and configuring infrastructure in the cloud, to running
autonomous adversary emulation and extracting the
re2.1. APT 29 sulting logs. The Attack Range technology stack is very
similar to the one proposed in this paper, albeit the
soluMITRE Engenuity is a technology foundation and a portal tion presented here proposes some improvements. Attack
through which MITRE collaborates with the private sec-Range uses Terraform to deploy infrastructure in either
tor, and applies state of the art innovation that emerges AWS or Azure’s cloud, and Ansible to then configure
out of their research and development activity2[]. Op- it. Attack Range also deploys MITRE’s CALDERA as an
erating in MITRE Engenuity is the Center for Threat autonomous red teaming tool. Lastly, it uses Window’s
Informed Defense (CTID), a ”privately funded research WEF technology in order to recuperate the logs
generand development organization”3[] that has developed ated from an attack. Attack Range then indexes the logs
an adversary emulation library of advanced persistenton a Splunk Server and uses other Splunk technologies
threat attack plans. If an autonomous red teaming tool for security orchestration and rule based detection.
allows us to launch adversary simulations, then the plans However, although Splunk’s Attack Range allows us
produced by CTID are the important inputs needed to to test individual abilities and techniques, it does not
generate high quality attack simulation datasets. provide comprehensive attacks for researchers to study,</p>
      <p>
        One of the attack simulation plans that CTID has de-run, and develop on. Likewise, Attack Range’s default
veloped is a plan to simulate an APT29 attack. APT29environment setup is composed of 2 Windows machines,
is a ”threat group that has been attributed to Russia’s and 2 Linux servers. In order to execute complex attack
Foreign Intelligence Service (SVR). They have operated chains that require larger and convoluted environments,
since at least 2008, often targeting government networks one would need to firstly adapt to required the
environin Europe and NATO member countries, research in- ment by editing a configuration file provided in Attack
stitutes, and think tanks”[
        <xref ref-type="bibr" rid="ref10">4</xref>
        ]. By extensively studying Range. Then, one would need to gather, in the correct
orreal cyber attacks conducted by APT29, the CTID team der from the MITRE ATT&amp;CK framework, the list of
techbuilt an attack plan that simulates and draws on identicanlique IDs used in the attack, and feed it as a command
or similar techniques, tactics and procedures that the line argument argument when initiating the program.
Russian hacking group has previously used. We have Depending on an attack’s complexity, the configuration
chosen this particular attack simulation plan because thefile can quickly become bloated, hard to manage, and
attack’s complexity will further accentuate the benefits work against the initial intended goal that the research
of automation. The APT29 simulation plans provide both team had set: facilitating cybersecurity automation for
a manual and automatic implementation guide for how threat research.
to carry out the attack simulation. This in turn allows us Secondly, because Attack Range was developed by a
to start setting quantitative benchmarks for how much research team within Splunk, the data indexing and
visugain in time and productivity one can achieve through alization step, and SOAR step of the solution can only be
an automated solution. configured with Splunk technologies: Splunk and Splunk
      </p>
      <p>The CTID team continues to develop new emulation Phantom. Although Splunk is recognised as a leader in
plans, and refine already established ones. The plans arethe field of log collection, analysis, and detection,
avoidmapped to MITRE’s ATT&amp;CK framework, and used to ing vendor lock-in is a fundamental concept to consider
evaluate detection solutions both in the context of the when it comes to designing an end to end solution. It
al’ATT&amp;CK Evaluations’, a detection solution evaluationlows the solution to be easily modified in order to best fit
series led by MITRE, and for individually and separatelythe needs of users, and ensures that the solution doesn’t
evaluating one’s own detection solution. This ecosys-have a single point of failure.
tem allows researchers to develop applications on the Lastly, Splunk’s red teaming automation is based on
emulation library with confidence. CALDERA’s old version 2 release, and therefore any new
features developed and introduced in CALDERA will not
be compatible with Attack Range.
2.2.2. Microsoft - SimuLand attacker environment is required. The target
environment is a Windows domain sub-network with 2 Windows
The second tool we’re going to a look at is SimuLand6[], servers with the ’2019-Datacenter’ SKU, one serving as a
a tool developed by Microsoft and introduced in 2021. domain controller, and the other serving as a file server,
The tool isn’t a fully automated end to end solution. Itand three Windows workstations running Windows 10
automates the deployment and configuration of infras- 1903 with a “19h1-pro” or “1903-evd-o365pp“ SKU. The
tructure, but lacks red team automation tooling. Rather, domain controller and file server are usually controlled
it aims to guide researchers on how to manually simulate and under the supervision of IT professionals, whilst
diferent techniques on already deployed and configured workstations usually represent regular computers that
infrastructure. It deploys and configures an infrastruc- non-technical employees use. In this case, all VMs are
ture in the cloud using Azure Resource Manager Tem- “Standard B4MS” instances, with four vCPUs and 16GB of
plates, and has good integration with diferent security, RAM. The attacker environment is a second sub-network
DevOps, and cloud products within the Microsoft ecosys- with 2 Linux machines running 18.04.3 LTS Ubuntu, one
tem (Microsoft 365 Defender, Azure Defender, and Azure serving as a trafic redirector, and the other as a C2
(ComSentinel). On another hand, SimuLand’s design limits mand and Control) from where attack commands are
users to using Azure as a sole cloud provider, which is sent. The sub-network also has a workstation running
particularly problematic when studying security exploitsWindows 10 1903 with the same SKU as the target
workdirectly embedded in a cloud provider’s system. stations, it serves as a platform to replicate the target
environment, appropriately compile payloads. Lastly, a
3. Manual security research virtual peering network is needed to connect the two
sub-networks.</p>
      <p>In this section we will look at how current cybersecurity
research is being conducted at three diferent stages: pro- 3.2. Configuration
visioning (3.1), configuration ( 3.2), and attack simulation
(3.3). The section will analyze in detail the procedure thaTthe process of configuring an environment varies a lot
a researcher would go through for each step, and analyzedepending on the target operating system, and the attack
its weaknesses and disadvantages, all done through thesimulation. Without a clear list of settings and software
lens of a setup for an APT29 attack simulation. that need to be present, launching an attack is either
unachievable, or produces inaccurate results. The countless
3.1. Provisioning ways a configuration setup can go amiss, and the needed
technical knowledge and familiarity with the operating
The first step in conducting cybersecurity research is the system make configuration a daunting task.
deployment of an environment, a cyber range, in which To configure the environment for the APT29 scenarios,
we want to test out diferent abilities and techniques. A one needs to connect to each resource through Windows
common way to manually deploy the infrastructure is Remote Desktop, an application that allows one to
interby requesting the necessary resources through a cloud act with a GUI for your virtual machines. The domain
provider website’s graphical user interface. Another so- controller server needs to be setup by installing Active
Dilution is to build the required infrastructure with virtual-rectory (AD), creating a domain, adding the workstations
izing software like VirtualBox, or KVM and locally host to the domain along with creating a domain name service
the environment. (DNS), group policy objects (GPO), domain users, and</p>
      <p>Going through a cloud provider entails having to sepa-domain user groups. The workstations are then setup by
rately request each resource and specify instance detailsi.nstalling additional software like Google Chrome,
tamThe process of setting up an environment for a complex pering with the registries and firewall rules, disabling
attack can call for requesting to the cloud provider virtualWindows Defender, and ensuring that Windows Remote
machines, networks, sub-networks, network interface Management (WinRM) as well as other communication
controllers, network peerings, and additional resources.protocols and services are functioning correctly. Finally,
For each of these resource requests, one must specify a on a command and control server (C2), one needs to
ingroup of settings. For instance, a virtual machine willstall penetration testing tools such as Metasplo7it] [to
typically require the disk, image, CPU, RAM among other have a platform on which to launch the attack from.
details to be provided in order to be deployed. Individu- The measures taken on the domain controller,
workally deploying resources is prone to error, hard to debug,stations, and C2 are quite common for threat hunting
cytedious, and time consuming. bersecurity research. Additionally, APT29 also requires</p>
      <p>In order to set up the correct environment for the the Powershell execution policy set to ”Bypass”, the
regAPT29 scenario as depicted in Figure 11(), a target and istry modified to allow storage of wdigest credentials, the
ifrewall configured to allow SMB (Server Message Block),
a SMB share present and working, the UAC (User Access tration testing tools ready, the attack simulation can be
Control) set to never notify for all Windows hosts. initiated. This step requires general knowledge about</p>
      <p>Whereas the manual provisioning of the infrastructure what the attack is trying to achieve, how it accomplishes
was all initiated from a centralized cloud provider webits goals, what each step of the attack performs, also
platform with a user friendly interface, the manual con-known as ’Techniques, Tactics and Procedures’ (TTP), as
ifguration of an environment requires configuring the well as more in-depth knowledge about how to navigate
environment by connecting to diferent virtual machines and send commands from the penetration testing tool.
endpoints. That framework ofers a less controlled envi- The APT29 attack simulation is broken down into 2
ronment where the user has a harder time tracking the diferent scenarios in order to depict the two approaches
current state of configuration process and the remaining that the hacking group could deploy when they attack
steps. their targets. For both scenarios, the attacker uses a mix
of Metasploit and Pupy in order to communicate with
3.3. Attack Simulation infected workstations and send shell commands to carry
out the attacks. The first APT29 scenario represents a
Once the infrastructure is deployed and configured, a re- more aggressive, fast-paced, direct style that ’smashes
searcher can then proceed to launch attacks. Every attackand grabs’ in order to reach its goals. The goal is to
simulation requires an entry point from which malicious firstly collect and exfiltrate data, the focus then shifts to
commands are executed and payloads downloaded. The persistence, data collection, credential access, and lateral
entry point is an agent that lays in wait and listens formovement. The second scenario on the other hand is a
commands to execute from the C2 server. Therefore, stealthier and slower attack that looks at ”establishing
one needs to connect to at least one workstation, and persistence, harvesting credentials, then finally
enumerinitiate the agent process before starting the attack sim- ating and compromising the entire domain”.[9]
ulation. The use of already infected workstations as a More details about each step of the attack for both
starting point for conducting post-breach cybersecurity scenario 1 and 2 can be found in the appendix, where
research is common under the paradigm known as ”As- notes gathered from MITRE’s
adversary_emulation_lisume Breach Paradigm” [8]. Microso’fts Cyber Defense brary GitHub repository[9] can be found.
Operations Center describes it as such: ”despite all the
protections in place, we assume systems will fail or
people will make errors, and an adversary may penetrate our
infrastructure and services.”</p>
      <p>Once the workstation(s) are infected and the
pene</p>
    </sec>
    <sec id="sec-3">
      <title>4. Automated security research</title>
      <sec id="sec-3-1">
        <title>After studying how current manual security research is</title>
        <p>directed, we propose a way to conduct automated
security research, and how to implement each of its steps.</p>
        <p>The automated solution is split in 3 main steps:
provisioning (4.1), configuration ( 4.2), and attack simulation
(4.3) as shown on figure 2. Additionally, data collection
and reporting will be covered in4.4, and a comparison
between manual and automatic cyber security research
will be drawn in4.5.</p>
        <p>One important and pressing issue reoccurring in the
last section was the unavoidable complexity, and
subsequently the required technical know-how one needs
in order to carry out security research.This section is
now going to see how to abstract out the intricacies of
deploying, and configuring our cyber range through the
concept of Infrastructure as Code (IaC) 1[0, 11]. As for
attack simulations, the required technical knowledge that
comes with penetration testing software is abstracted
out through Caldera’s intuitive GUI for managing agents, Figure 2: The 3 step automation cycle for cybersecurity
readversarial profiles, and operations. search: Terraform, Ansible, and Caldera.</p>
        <p>Furthermore the automation technologies outlined for
infrastructure provisioning and configuring have
important attributes that make them all the more fit for cy- currently creates infrastructure in a way that respects
bersecurity research and development. The agentless the dependency graph, and in a manner that humans
nature of the technologies is an important step towards could not compete with. But the leading gain is in the
tackling the automation challenge as it avoids any unnecr-educed amount of workload and time someone has to
essary dependencies, limits the requirements to initiate spend in order to boot up the infrastructure. The whole
the automation process, and minimizes the probability of infrastructure can be built from 2 simple commands:
critical errors. Moreover, the declarative capabilities of
those technologies allow users to rapidly learn about andterraform plan -out {name_of_plan}
understand the diferent components that make up the terraform apply {name_of_plan}
automation process with little to no technical knowledgeThe user first creates a ’plan’ that represents the changes
of the tools used. By separating the user from software Terraform counts on implementing such as destroying or
implementation issues and edge case problems, a declar-creating a virtual machine, and then applies the planned
ative style approach requires very little efort to build changes. Finally, ”terraform state list” shows the current
functional and robust programs. state of the infrastructure. The user then doesn’t have to
spend 1 to 2 hours creating the resources by hand, but
4.1. Provisioning can spend that time on higher added value work whilst
waiting for Terraform to complete. The modular nature
In order to provision our environment in a fully auto- of Terraform means that we can share our code, or a
mated manner, we use Terraform [12], an open source portion of it for others to reuse. There are no manual
IaC software tool. The tool allows for the creation and equivalents when one looks for a way to share the
abilprovisioning of infrastructure using HashiCorp Configu- ity to launch an identical infrastructure, and show the
ration Language (HCL), a declarative configuration lan- desired infrastructure end-state. The code format of our
guage where the user ’declares’ or writes in HCL code infrastructure deployment, or IaC, enables us to use
feathe desired state for the infrastructure. tures that come with version controller systems (VCS)</p>
        <p>Terraform ofers the ability to write reusable and mod- hosting platforms like GitHub or GitLab. It allows us to
ular code. The re-usability feature gives Terraform aperform code reviews, work on particular branches of
sizeable advantage over deploying each resource manu-the code, look at history graphs, track issues and goals,
ally on a Cloud Provider. Users save on the amount of and more broadly work in a collaborative, structured
time the infrastructure takes to deploy: Terraform creates and fast-paced environment. Finally, in contrast to an
a resource dependency graph to set the order in which imperative approach, the user does not need to know
resources need to be deployed, it then instantly and con- how Terraform implements HCL code and deploys the
infrastructure. Subsequently, a great deal of complexityprovides an interface to inspect the collection of
abiliis taken out of the hands of the user given that HCL is ties that make up an adversary, or an adversarial attack.
easy to read, learn and execute. One can either analyze the attack by overviewing the
diferent abilities and getting a general understanding of
4.2. Configuration how the attack works, or dig deeper in each ability, and
inspect the commands launched. Lastly, the ’operations’
As for the automation of configuration for our environ- section is at the core of launching attacks, it allows for
ment, Ansible [13], another open source IaC software the configuration and management of operations, with
tool released in 2012, and developed by Red Hat Inc sincethe capability to manually add and execute commands
2015, provides a wide range of packages and functions to an on-going autonomous attack.
that allow for configuration management. Ansible uses The second advantage CALDERA has to ofer is a
plata hybrid between imperative and declarative style lanf-orm on which one can easily build variations out of a
guage where development should be as declarative asparticular adversarial attack, as well as producing,
experpossible but might still require imperative style code. imenting and sharing new adversaries. This feature is of</p>
        <p>
          Ansible’s modular programming capabilities are just as upmost importance as it comprehensively captures the
pronounced as Terraform’s, not only are functions used merit and spectrum of benefits that an automated system
within a source code file reusable, but so are larger and ab- provides. It provides an unparalleled flexible structure to
stract goals such as setting up and configuring Windows’ produce modular and automated attacks.
Active Directory can be packaged and encapsulated in The third advantage CALDERA brings is a catalogue of
a single package or ’role’. A role is a folder or packageattacks belonging to diferent threat actors, including the
that contains both what the user might conceive as the APT29 scenario. The utility for autonomous red teaming
main source code (tasks), as well as additional resourcesgrows with the complexity of the simulated attacks, and
such as template files, variable files, handler files (files given that the attacks available are of APT level
sophisthat manage exceptions and special conditions). They tication, CALDERA becomes an invaluable tool. An
adcan then easily be shared with the community or within vanced persistent threat (APT) is a threat group, usually
a workspace. Ansible’s flexibility becomes important associated with nation states, with advanced capabilities
when looking at requirements and interoperability. Its to penetrate systems and networks.
requirements are minimal, as it only requires for Python
to be installed, and for a mean of connection (WinRM or 4.4. Data collection and reporting
ssh [14]) to be available. Like Terraform, Ansible’s use
of IaC allows for code reviews and other perks, as wellThe data collection process is specific to an operating
as being easy and quick to launch. system, but can be setup automatically at the
configuration step through Ansible. Taking a deeper look at the
4.3. Attack simulation platform data collection process for a windows ecosystem, a way
to collect logs is through Windows Event Forwarding
There are various challenges with automatically mea(-WEF) [
          <xref ref-type="bibr" rid="ref12">17</xref>
          ], a service that comes as part of Windows 10,
suring aspects of a network’s security posture through and allows workstations to forward local logs to other
penetration testing, red teams, and adversary emulation Windows machines. The main steps in setting up WEF
and numerous way to go about implementing it [15]. Cy- can be split between setting up the workstations, also
ber Adversary Language and Detection Engine for Red known as WEF clients, on which the attacks are
unfoldteam Automation (CALDERA) version 3 [
          <xref ref-type="bibr" rid="ref11">16</xref>
          ] is a simu- ing, and the server listening for incoming connections
lated penetration testing platform for autonomous red from the WEF Clients. These steps include enabling the
teaming. It is an open source tool developed by MITRE. WinRM service, changing registry keys, changing
auCALDERA ofers three major advantages over a manual dit policies and system access controls, and uploading
style attack. XML files to configure the WEF service, all steps that can
        </p>
        <p>The first is that CALDERA has an interactive, friendly be completed automatically through Ansible packages
and graphical user interface, launching an attack is an ina-nd functions. There are plenty of services and products
tuitive and short procedure. The program has 3 main cat- that take care of the data reporting process, and are
usuegories: Agents, Adversaries, and Operations. One can ally chosen depending on the technology a user is most
easily toggle from one to the other without any interfer- familiar with, or already has setup.
ence between each other. The ’Agents’ section provides
a dashboard with a list of agents currently running, their4.5. Results
information and whether they have been terminated, as
well as code to implant agents on a target workstation be-The time results comparing a manual and automated
fore one can simulate an attack. The ’Adversaries’ section approach to simulating the APT29 attack plans can be</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>5. Relevance of automation for AI</title>
      <sec id="sec-4-1">
        <title>Generating and updating a collection of diverse datasets</title>
        <p>is especially important in the cybersecurity field where
threats, actors and their representations are constantly
changing, and where experts have to be persistently
Table 1 learning about new paradigms, heuristics and
technoloTime comparisons between manually and automatically com- gies. The automatic construction of a cyber range
prepleting APT29’s scenario 1 sented in this paper does not just provide solutions for
current threats, but a general framework in which one
can continually conduct research, and build new datasets.
found above in Table 11(). All in all, the deployment of In AI development, results can only be as good as the
the infrastructure, configuration of the environment, and quality of the data used. Therefore, having a limited
completion of the attack simulation for the first scenario amount of datasets to train good models and counting
of APT29 takes just under an hour from start to finish. on them to protect organisations is not a viable solution.
In contrast, for a cybersecurity researcher new to APT29 A fully automated cyber range ensures us to have access
attacks, the simulation would likely take between 6.5 andto diverse datasets. Its flexibility enables us to add
varia27 hours. tions on an attack, and create a wide array of diferent</p>
        <p>Using Terraform to deploy the infrastructure took 17 environments against which to generate datasets.
minutes, whilst deploying all of the infrastructure man- In this section we will discuss how such a framework
ually through Azure would take an amount of time in does not just enhance cybersecurity researchers but also
the scale of multiple hours. Configuration automation provide the necessary sandbox for AI researchers to
deallows researchers to have a environment ready in 30velop applications and train models of good quality. We
minutes. Configuring automatically with Ansible, here, will firstly see how such a cyber range can help build a
allows us to save time on a process that would usuallyhigh level ontology in the domain of cybersecurity i5n.1.
take between 5 - 10 hours. We then look at some commons pitfalls machine learning</p>
        <p>Manually running an attack simulation is diferent models encounter with poor data and how an automated
from deploying an infrastructure or configuring it, in the cyber range can help us avoid them in5.2.
sense that once an attack is mastered by a researcher,
he can complete the attack in the same time order as he 5.1. Ontologies
automatically would with an automated tool. CALDERA
took a total of 8 minutes to run scenario 1 of the APT29 ”MITRE ATT&amp;CK is a globally-accessible knowledge base
simulation, whilst a well versed researcher could finish of adversary tactics and techniques based on real-world
it in less than 30 minutes. observations”. The tactics and procedures provided by</p>
        <p>
          The automation of a cyber range does not come with- the framework allow us to paint a meaningful picture
out a price, cyber range automation allows us to acceler-for attacks [
          <xref ref-type="bibr" rid="ref13">18</xref>
          ]. If the MITRE ATT&amp;CK framework
proate the research development cycle but in turn takes awayvides the tools to build high level view attacks, and an
from potential expertise and knowledge that researchers automated system the low level log datasets of attack
would have developed in the process of creating a cyber simulations (see figure 3), then Ontology [
          <xref ref-type="bibr" rid="ref14">19</xref>
          ] is the key
range themselves. In the deployment and configuration to bridging the gaps between the two. It enables us to
step, the time savings justifies the expertise delegated map thousands of logs into a coherent and
comprehensito the automated platform. In contrast, spending time ble sequence of MITRE ATT&amp;CK techniques, tactics and
in the attack simulation stage to understand an attackprocedures.
and manually launch attack simulations is an important Ontology-based data access (OBDA) is a well
estabcomponent to preserve in an automated cyber range. If lished paradigm for querying incomplete and
heterogeexpertise was delegated in the deployment and config- neous data sources while incorporating knowledge from
uration stage, it is for researchers to spend more time a domain ontology [
          <xref ref-type="bibr" rid="ref15 ref16">20, 21</xref>
          ]. OBDA allows a user to
foron mastering the attack simulation stage. Nevertheless,mulate queries through a high-level ontology vocabulary,
automated red teaming presents an alternative and in-delegating to the algorithm the task of querying low level
teresting way of conducting attack simulations. Whilst data and mapping them back to high level concepts.
manually executing an attack simulation requires a grasp The ontological process represents the abstraction
exof every step before completing an attack, automated red ercise that cybersecurity experts perform each day when
teaming allows researchers to run full attack simulationslooking at a collection of security logs, whether it is in an
without understanding certain steps, which is useful for incident response, troubleshooting, or active monitoring
understanding the general operational flow of an attack. context. An ontology must consequently define semantic
concepts that cybersecurity analysts use and recognise attack risk, and should therefore be blocked. Secondly,
in order to abstract out the logs. the heavy use of abstraction allowed through the use of
        </p>
        <p>
          The diversity of our data allows us to test the levelontologies enables the models to limit the efects of
adof expressiveness an ontology has to ofer. Building an versarial artificial intelligence examples. Deep learning
ontology only with high level concepts in mind might models for instance, are known to be highly vulnerable
not scale to real data mapping. On the other hand, over-to adversarial examples2[
          <xref ref-type="bibr" rid="ref10">4</xref>
          ]. Introducing very superficial
iftting concepts on a limited amount of datasets puts us changes to an input can make predictions highly unstable
at risk of not being able to generalize when the paradigmand inaccurate, a situation where humans can reliably
shifts ever so slightly. Therefore, ontology building is understand that the input has not significantly changed.
an iterative process which is best served by an flexible Ultimately, the hope is for the machine learning models
and automated cyber range that can reliably produceto grasp abstract concepts and general pattern
recogniheterogeneous and realist as possible data. tion, and discover new heuristics for cybersecurity
analysts to integrate in their practice. As can be seen on
5.2. Machine learning ifgure 4 ( 4), among the most pressing issues that
cybersecurity analysts are trying to solve is the problem of
After developing an ontology, one can train machine overwhelming false positives (A2 and A3). The
problearning models that learn from the same semantic con- lem clouds analysts’ judgment for potential threats, and
cepts that cybersecurity experts use, enabling them to causes what is known as ’Alert Fatigue’, a fatigue
prointelligently interpret predictions, as opposed to tryingduced by a myriad of false positives that continually drain
to learn directly from datasets of innumerable logs. Al-analysts’ attention. Other important issues pointed out
lowing machine learning models to base themselves of in Panther Labs’ survey findings (Figure 4) are a lack of
of high level abstractions, it empower them to be much context for alerts and insights given by current SIEM
more robust both to overfitting problems, and adversarial systems to experts, and the sheer number of those alerts
examples. Firstly, it allows models to avoid overfitting on (A1 and A4). Using an approach to machine learning
meaningless features 2[
          <xref ref-type="bibr" rid="ref18 ref8">2, 23</xref>
          ] such as learning that com- based on ontology, the enrichment of our data from
inimunication with a particular IP address presents a high tial logs to high level data would allow models to put
(LDAP) queries to enumerate other hosts in the domain
(T1018) before creating a remote PowerShell session to
Details about each step of the APT29 attack simulationa secondary victim (T1021 / T1021.006). Through this
for both scenario 1 and 2 are compiled here, as refer- connection, the attacker enumerates running processes
enced in 3.3. The details were gathered from MITRE’s (T1057). Next, the attacker uploads (T1105) a new
UPXadversary_emulation_library GitHub repository9][. packed payload (T1027 / T1027.002) to the secondary
victim. This new payload is executed on the secondary
Scenario 1 victim via the PSExec utility (T1021 / T1021.002, T1035 /
        </p>
        <p>T1569.002) using the previously stolen credentials (T1078
The scenario begins with an initial breach, where a le/- T1078.002).
gitimate user clicks (T1204 / T1204.002) an executable The attacker uploads additional utilities to the
secpayload (screensaver executable) masquerading as a beo-ndary victim (T1105) before running a PowerShell
onenign word document (T1036 / T1036.002). Once exe- liner command (T1059 / T1059.001) to search for
filesyscuted, the payload creates a C2 connection over port tem for document and media files (T1083, T1119). Files
1234 (T1065) using the RC4 cryptographic cipher. The of interested are collected (T1005) then encrypted and
attacker then uses the active C2 connection to spawn in- compressed (T1002, T1022 / T1560.001 into a single file
teractive cmd.exe (T1059 / T1059.003) and powershell.exe (T1074 / T1074.001). The file this then exfiltrated over
(T1086 / T1059.001). the existing C2 connection (T1041). Finally, the attacker</p>
        <p>The attacker runs a one-liner command to search the deletes various files (T1107 / T1070.004) associated with
iflesystem for document and media files (T1083, T1119), that access.
collecting (T1005) and compressing (T1002 / T1560.001) The original victim is rebooted and the legitimate user
content into a single file. The file is then exfiltrated over logs in, emulating ordinary usage and a passage of time.
the existing C2 connection (T1041). The attacker now This activity triggers the previously established
persisuploads a new payload (T1105) to the victim. The pay-tence mechanisms, namely the execution of the new
serload is a legitimately formed image file with a concealed vice (T1035 / T1569.002) and payload in the Windows
PowerShell script (T1027 / T1027.003). The attacker then Startup folder (T1060 / T1547.001). The payload in the
elevates privileges via a user account control (UAC) by-Startup folder executes a follow-on payload using a stolen
pass (T1122 / T1546.015, T1088 / T1548.002), which ex- token (T1106, T1134 / T1134.002).
ecutes the newly added payload. A new C2 connection
is established over port 443 (T1043 using the HTTPS
protocol (T1071 / T1071.001, T1032 / T1573). Finally, the Scenario 2
attacker removes artifacts of the privilege escalation fromThe scenario begins with initial breach, where a
legitithe Registry (T1112). mate user clicks (T1204 / T1204.002) a link file payload,</p>
        <p>The attacker uploads additional tools (T1105) throughwhich executes an alternate data stream (ADS) hidden
the new, elevated access before spawning an interac-on another dummy file (T1096 / T1564.004) delivered
tive powershell.exe shell (T1086 / T1059.001). The addi-as part of the spearphishing campaign. The ADS
pertional tools are decompressed (T1140) and positioned on forms a series of enumeration commands to ensure it
the target for usage. The attacker then enumerates run- is not executing in a virtualized analysis environment
ning processes (T1057) to discover/terminate the initial (T1497 / T1497.001, T1082, T1120, T1033, T1016, T1057,
access from Step 1 before deleting various files (T1107 T1083) before establishing persistence via a Windows
/ T1070.004) associated with that access. Finally, theRegistry Run key entry (T1060 / T1547.001) pointing to
attacker launches a PowerShell script that performs aan embedded DLL payload that was decoded and dropped
wide variety of reconnaissance commands (T1016, T1033, to disk (T1140). The ADS then executes a PowerShell
T1063 / T1518.001, T1069, T1082, T1083), some of which stager (T1086 / T1059.001) which creates a C2 connection
are done by accessing the Windows API (T1106). over port 443 (T1043) using the HTTPS protocol (T1032 /</p>
        <p>The attacker establishes two distinct means of persis- T1573.002 , T1071 / T1071.001).
tent access to the victim by creating a new service (T1031 The attacker modifies the time attributes of the DLL
/ T1543.003) and creating a malicious payload in the Win-payload (T1099 / T1070.006) used in the previously
estabdows Startup folder (T1060 / T1547.001). The attacker col- lished persistence mechanism to match that of a random
lects screenshots (T1113), data from the user’s clipboard file found in the victim’s System32 directory (T1083).
(T1115), and keystrokes (T1056 / T1056.001). The attacker The attacker then enumerates registered AV products
then collects files (T1005), which are compressed and en- (T1063 / T1518.001) and software installed by the user
crypted (T1560 / T1560.001), before being exfiltrated to documented in the Windows Registry (T1012).
an attacker-controlled WebDAV share (T1048 / T1048). The attacker performs local enumeration using
variThe attacker uses Lightweight Directory Access Protocol ous Windows API calls, specifically gathering the local
computer name (T1082), domain name (T1016), current
user context (T1033), and running processes (T1057).</p>
        <p>The attacker elevates privileges via a user account
control (UAC) bypass (T1122 / T1546.015, T1088 / T1548.002).</p>
        <p>The attacker then uses the new elevated access to create
and execute code within a custom WMI class (T1047) that
downloads (T1105) and executes Mimikatz to dump
plaintext credentials (T1003 / T1003.001), which are parsed,
encoded, and stored in the WMI class (T1027). After
tracking that the WMI execution has completed (T1057),
the attacker reads the plaintext credentials stored within
the WMI class (T1140).</p>
        <p>The attacker establishes a secondary means of
persistent access to the victim by creating a WMI event
subscription (T1084 / T1546.003) to execute a PowerShell
payload whenever the current user (T1033) logs in.</p>
        <p>The attacker enumerates the environment’s domain
controller (T1018) and the domain’s security identifier
(SID) (T1033) via the Windows API (T1106). Next, the
attacker uses the previously dumped credentials (T1078
/ T1078.002) to create a remote PowerShell session to
the domain controller (T1028 / T1021.006). Through
this connection, the attacker copies the Mimikatz binary
used in Step 14 to the domain controller (T1105 / T1570)
then dumps the hash of the KRBTGT account (T1003 /
T1003.001).</p>
        <p>The attacker harvests emails stored in the local email
client (T1114 / T1114.001) before collecting (T1005) and
staging (T1074 / T1074.001) a file of interest. The staged
ifle is compressed (T1002 / T1560.001) as well as prepended
with the magic bytes of the GIF file type (T1027).</p>
        <p>The attacker maps a local drive to an online web
service account (T1102) then exfiltrates the previous staged
data to this repository (T1048 / T1567.002).</p>
        <p>The attacker deletes various files (T1107 / T1070.004)
associated with that access by reflectively loading and
executing the Sdelete binary (T1055 / T1055.002) within
powershell.exe.</p>
        <p>The original victim is rebooted and the legitimate
user logs in, emulating ordinary usage and a passage
of time. This activity triggers the previously established
persistence mechanisms, namely the execution of the
DLL payload (T1085 / T1218.011), referenced by the
Windows Registry Run key, and the WMI event subscription
(T1084 / T1546.003), which executes a new PowerShell
stager (T1086 / T1059.001). The attacker uses the renewed
access to generate a Kerberos Golden Ticket (T1097 /
T1558.001, T1558.003), using materials from the earlier
breach, which is used to establish a remote PowerShell
session to a new victim (T1028 / T1021.006). Through this
connection, the attacker creates a new account within
the domain (T1136 / T1136.001).</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <article-title>in the hands of cybersecurity experts meaningful and [5] Attack range github repository</article-title>
          ,
          <year>2019</year>
          . URL:https: contextual alerts. //github.com/splunk/attack_range. [6]
          <string-name>
            <surname>Simuland</surname>
            <given-names>github repository</given-names>
          </string-name>
          ,
          <year>2021</year>
          . URL: https:// github.com/Azure/SimuLand.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          6. Conclusion [7]
          <string-name>
            <given-names>D.</given-names>
            <surname>Kennedy</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. O'gorman</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Kearns</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Aharoni</surname>
          </string-name>
          ,
          <article-title>Metasploit: the penetration tester's guide, No The automation solution we brought forward in this pa-</article-title>
          Starch Press,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <article-title>per is designed to help cybersecurity researchers look-</article-title>
          [8]
          <string-name>
            <given-names>R.</given-names>
            <surname>Pompon</surname>
          </string-name>
          , Assume Breach,
          <article-title>Apress, ing to integrate AI in their operations, as well as AI re</article-title>
          - Berkeley, CA,
          <year>2016</year>
          , pp.
          <fpage>13</fpage>
          -
          <lpage>21</lpage>
          . URL:
          <article-title>searchers interested in contributing to the cybersecurity https://doi</article-title>
          .org/10.1007/978-1-
          <fpage>4842</fpage>
          -2140-
          <issue>2</issue>
          _
          <fpage>2</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          ifeld.
          <article-title>We used a scenario inspired from an APT29 attack campaign to better illustrate the benefits that the automa</article-title>
          <source>- [9] dAodiv:1e0r.s1a0r0y7 / 9e7 m8-u1l-a4t8i4o2n- 2</source>
          1l4i0b-
          <fpage>r2a</fpage>
          _
          <article-title>r2y. github repostion platform brings for researchers</article-title>
          .
          <source>The solution enables itory</source>
          ,
          <year>2019</year>
          . URL: https://github.com
          <article-title>/ researchers to operate and build software on top of an center-for-threat-informed-defense/adversary_ automated cyber range, allowing them to save time and emulation_library/tree/master/apt29/Emulation_ focus solely on the development of artificial intelligence Plan.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          tools.
          <article-title>Terraform automates the deployment of the infras-</article-title>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Rahman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Mahdavi-Hezaveh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Williams</surname>
          </string-name>
          ,
          <article-title>A tructure, Ansible automates its configuration, Caldera systematic mapping study of infrastructure as code provides autonomous red teaming capabilities for attack research, Information and Software Technology 108 simulations, and WEF helps centralize and collect the (</article-title>
          <year>2019</year>
          )
          <fpage>65</fpage>
          -
          <lpage>77</lpage>
          . URL: https://www.sciencedirect.
          <source>com/ attack simulation's data. science/article/pii/S095058491830250</source>
          .7 doi:h t t p s : / / d o i .
          <source>o r g / 1 0 . 1 0 1 6 / j . i n f s o f . 2 0 1 8 . 1 2 . 0 0 4 .</source>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <surname>References</surname>
            [11]
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Parnin</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          <string-name>
            <surname>Helms</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Atlee</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Boughton</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Ghattas</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Glover</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Holman</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Micco</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Murphy</surname>
          </string-name>
          ,
          <string-name>
            <surname>T.</surname>
          </string-name>
          Sa-
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <article-title>[1] Common staf target for military cooperation vor</article-title>
          , M. Stumm,
          <string-name>
            <given-names>S.</given-names>
            <surname>Whitaker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Williams</surname>
          </string-name>
          ,
          <article-title>The top 10 on cyber ranges in the european union, 2013. adages in continuous deployment</article-title>
          , IEEE Software URL: https://eda.europa.eu/docs/default-source/ 34 (
          <year>2017</year>
          )
          <fpage>86</fpage>
          -
          <lpage>95</lpage>
          .
          <source>doi:1 0 . 1 1 0 9 / M S . 2 0</source>
          <volume>1 7 . 8 6</volume>
          . procurement/annex-a--
          <string-name>
            <surname>-</surname>
            cyber-ranges-cst.pd.f [12]
            <given-names>Mitchell</given-names>
          </string-name>
          <string-name>
            <surname>Hashimoto</surname>
          </string-name>
          et al , Terraform website,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [2]
          <string-name>
            <surname>Mitre</surname>
            <given-names>engenuity website</given-names>
          </string-name>
          ,
          <year>2019</year>
          . URL: https:// URL: https://www.terraform.io/. mitre-engenuity.org/. [13]
          <string-name>
            <surname>Ansible</surname>
            <given-names>website</given-names>
          </string-name>
          ,
          <year>2012</year>
          . URL:https://www.ansible.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <article-title>[3] Center for threat informed defense website</article-title>
          ,
          <year>2019</year>
          . com/ . URL: https://ctid.mitre-engenuity.org/. [14]
          <string-name>
            <given-names>T.</given-names>
            <surname>Ylonen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Lonvick</surname>
          </string-name>
          , et al.,
          <article-title>The secure shell (ssh)</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Mitre</surname>
            <given-names>att</given-names>
          </string-name>
          &amp;ck website,
          <year>2015</year>
          . URL: https://attack. protocol architecture,
          <year>2006</year>
          . mitre.org/groups/G0016/. [15]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hofmann</surname>
          </string-name>
          ,
          <article-title>Simulated penetration testing: From ”dijkstra” to ”turing test++”</article-title>
          ,
          <source>in: Proceedings of the Twenty-Fifth International Conference on International Conference on Automated Planning and Scheduling</source>
          ,
          <source>ICAPS'15</source>
          , AAAI Press,
          <year>2015</year>
          , p.
          <fpage>364</fpage>
          -
          <lpage>372</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Caldera</surname>
          </string-name>
          , a scalable,
          <source>automated adversary emulation platform</source>
          ,
          <year>2021</year>
          . URL: https://caldera.mitre.org./
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [17]
          <article-title>Spotting the Adversary with Windows Event Log Monitoring</article-title>
          ,
          <source>Technical Report, NSA</source>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [18]
          <article-title>Best Practices for MITRE ATT</article-title>
          &amp;
          <article-title>CK Mapping</article-title>
          ,
          <source>Technical Report</source>
          , CISA, HSSEDI,
          <year>2021</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [19]
          <string-name>
            <surname>D. L. McGuinness</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Van Harmelen</surname>
          </string-name>
          , et al.,
          <article-title>Owl web ontology language overview</article-title>
          ,
          <source>W3C recommendation 10</source>
          (
          <year>2004</year>
          )
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [20]
          <string-name>
            <surname>M.-L. Mugnier</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.-C. Rousset</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Ulliana</surname>
          </string-name>
          ,
          <article-title>OntologyMediated Queries for NOSQL Databases</article-title>
          , in: DL:
          <article-title>Description Logics</article-title>
          , volume CEUR Workshop Proceedings, Cape Town, South Africa,
          <year>2016</year>
          , pp.
          <fpage>1051</fpage>
          -
          <lpage>1057</lpage>
          . URL: https://hal-lirmm.ccsd.cnrs.fr/ lirmm-01375093,
          <article-title>this paper is an extended abstract of the paper with the same title presented at AAAI 2016</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>A.</given-names>
            <surname>Poggi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Lembo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Calvanese</surname>
          </string-name>
          , G. De Giacomo,
          <string-name>
            <given-names>M.</given-names>
            <surname>Lenzerini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rosati</surname>
          </string-name>
          ,
          <article-title>Linking data to ontologies</article-title>
          , in: S. Spaccapietra (Ed.),
          <source>Journal on Data Semantics X</source>
          , Springer Berlin Heidelberg, Berlin, Heidelberg,
          <year>2008</year>
          , pp.
          <fpage>133</fpage>
          -
          <lpage>173</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>J.</given-names>
            <surname>Reunanen</surname>
          </string-name>
          ,
          <article-title>Overfitting in making comparisons between variable selection methods</article-title>
          ,
          <source>J. Mach. Learn. Res</source>
          .
          <volume>3</volume>
          (
          <year>2003</year>
          )
          <fpage>1371</fpage>
          -
          <lpage>1382</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>R. B.</given-names>
            <surname>Rao</surname>
          </string-name>
          , G. Fung,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rosales</surname>
          </string-name>
          ,
          <source>On the Dangers of Cross-Validation. An Experimental Evaluation, Society for Industrial and Applied Mathematics</source>
          ,
          <year>2008</year>
          , pp.
          <fpage>588</fpage>
          -
          <lpage>596</lpage>
          . URL: https://epubs.siam.org/ doi/abs/10.1137/1.9781611972788.54. doi:
          <volume>10</volume>
          .1137/1. 9781611972788.54.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>P.-A.</given-names>
            <surname>Moëllic</surname>
          </string-name>
          ,
          <article-title>The dark side of neural networks: an advocacy for security in machine learning</article-title>
          , Computer &amp; Electronics
          <string-name>
            <surname>Security Applications Rendezvous (C&amp;ESAR)</surname>
          </string-name>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>