<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Student Perceptions of Ethics in</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Timo Hynninen</string-name>
          <email>timo.hynninen@laurea.fi</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Conference on Technology Ethics - Tethics</institution>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Laurea University of Applied Sciences</institution>
          ,
          <addr-line>Vanha Maantie 9, Espoo, 02650</addr-line>
          ,
          <country country="FI">Finland</country>
        </aff>
      </contrib-group>
      <fpage>79</fpage>
      <lpage>93</lpage>
      <abstract>
        <p>Ethical considerations are of growing interest in the computing education field. In particular, cybersecurity is a field where a strong moral compass is required. Cybersecurity education is often recommended to use an ofensive approach as this has been demonstrated to lead to better learning results. At the same time, we must consider how students perceive the skills and knowledge gained from the ofensive approach. This paper presents a case study of organizing a cybersecurity basics course for computing students using mainly an ofensive approach. At the end of the course, we asked the students to reflect on their learning and what they gained from the course. As such, a thematic analysis of the 110 student reflections was conducted to find out how ethical issues are perceived by the students. We found that most students describe their learning as raised awareness of cybersecurity threats, and knowing defenses for them. While some students wrote to have learned practical hacking skills and tricks, the majority described the lessons learned in an ethically sound way. It was also noted that adding reflection questions to the learning tasks throughout the course may have increased the number of ethical considerations in the student reflections of learning.</p>
      </abstract>
      <kwd-group>
        <kwd>cybersecurity education</kwd>
        <kwd>hacking</kwd>
        <kwd>ethics</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        The author of this paper has taught cybersecurity in higher education for almost a decade. The
teaching method has combined ofensive skills (i.e. hacking) with defensive tactics to illustrate
how cyberattacks work. In my experience, this has been an efective and motivating approach
for students. The objectives of the cybersecurity courses have focused on how to defend against
attacks but nevertheless, some of these skills and knowledge picked up along the way could
also be used for illicit purposes. Some researchers have argued that the ethical considerations
of hacking and security education must be carefully considered (e.g. [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ]). As the personal
experiences of the author (and similar practices of many colleagues in the computing education
community) are somewhat tangential to the research on the ethical stance on teaching hacking
skills, perhaps there is a need for a deeper investigation of how these skills and knowledge are
perceived by students.
      </p>
      <p>
        Including ofensive activities to gain hands-on cybersecurity skills and knowledge is common
in computing education [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. In general, hacking skills are considered an essential component
CEUR
CEUR
Workshop
Proceedings
      </p>
      <p>
        ceur-ws.org
ISSN1613-0073
of computing education [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] as they help students gain a better understanding of computer
and information security [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. The argument for teaching hacking skills and knowledge in
technology education is straightforward: For example, Radziwill et al. [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] state, that ”regardless
of whether or not students are taught hacking skills, hackers will still exist,” therefore having
the knowledge is paramount for defending oneself in the digital age.
      </p>
      <p>
        At the same time, many researchers and educators argue for a better alignment of ethics in
the teaching of such ’unethical’ skills. Some claim that teaching hacking techniques could cause
institutions to be faced with ethical and legal dilemmas (for example, see Hartley et al. [
        <xref ref-type="bibr" rid="ref5 ref6">5, 6</xref>
        ]
and Curbelo &amp; Cruz [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]). Additionally, some argue that teaching students how to hack could
have negative consequences such as causing them to become cybercriminals (e.g. Smith et al.
[
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]). However, these claims seem to be mostly anecdotal, and not much evidence supporting
them exists in the computing education literature (to our knowledge). In addition, cybersecurity
(and other) education usually covers also legal and ethical implications of hacking [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
      </p>
      <p>
        Cybersecurity educators often employ hacking and ofensive skills because an ofensive
approach to the topic leads to better learning results [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. This prompts the question: Is the
ofensive approach suficient enough in terms of the ethical (and legal) repercussions? Are there
potential problems regarding ethics with this approach?
      </p>
      <p>The current paper presents an empirical case study investigating how students perceive
ethical considerations of hacking or ofensive technology skills in a cybersecurity course. To
achieve this goal, 110 student reflections from four consecutive years of teaching the course
were analyzed using the thematic analysis method. The objective is to gain a perspective into
how ethical considerations manifest in the students’ descriptions of what they learned. Thus the
following research questions were formulated:
• To what extent do written student reflections depict ethical considerations?
• What ethics-related issues emerge from the reflections?
• How to mitigate these issues?</p>
      <p>This paper is organized as follows. Section 2 presents extant literature on the ethical
considerations of hacking and cybersecurity education. Section 3 presents the research methods
and context of the study. Section 4 presents the findings which are then discussed in Section 5.
Finally, Section 6 concludes the paper.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Related work</title>
      <p>
        The extant work in cybersecurity education is well-established in the literature. The body of
knowledge is detailed in several literature reviews, for example: [
        <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref14 ref15 ref16 ref17 ref18 ref19 ref20 ref21 ref22 ref23 ref24 ref25 ref26 ref27 ref9">10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 9, 21, 22, 23, 24, 25, 26, 27, 28</xref>
        ]. In general, hacking skills and ofensive approaches to
teaching security are commonplace.
      </p>
      <p>
        Ofensive skills as part of ethical hacking are taught in cybersecurity programs to help
students protect themselves from the dangers of cybercrime. The practical skills associated
with hacking or cybersecurity are not easily learned by students on their own, so it is important
for schools to include ethical hacking in their curricula [
        <xref ref-type="bibr" rid="ref1 ref8">1, 29, 30, 8</xref>
        ].
      </p>
      <p>
        In terms of the negative stance on teaching hacking in schools, Radziwill et al. argue that
providing young people with the tools and knowledge of accessing secure networks while not
necessarily having developed skills in ethical reasoning is dangerous [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. This concern can also
be seen in faculty attitudes and opinions on teaching ofensive approaches to security [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Pike
[30] states point-blank that hacking draws students toward criminal acts. Sometimes hacking
curriculums are even described as undocumented sets of tips and tricks [31].
      </p>
      <p>
        Similarly, teaching ofensive skills or hacking skills can at times be perceived by
nonprofessionals as ethically unsound, or even banned by university administration [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
Cybersecurity professionals also share this concern to a degree; For example, the study of Smith et
al. on teaching ethical hacking to students calls for ”instructors to instill correct knowledge
of laws concerning hacking and their repercussions” to students [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. In a similar vein, the
study of Logan and Clarkson concludes that ’instructors should carefully consider the design
of all ”red team” (ofensive) exercises’ [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. However, the most vocal studies on the dangers of
teaching hacking are based on the opinions of security professionals (i.e. Pike [30]) or faculty
(i.e. Curbelo &amp; Cruz). Thus, the voices in extant literature have been mostly anecdotal, and do
not portray how students would use these skills and knowledge.
      </p>
      <p>
        The ofensive approach is generally considered more efective in students achieving learning
outcomes [
        <xref ref-type="bibr" rid="ref5 ref6">5, 6, 32</xref>
        ]. Many educators believe that institutions should teach ethical hacking to
undergraduate students [
        <xref ref-type="bibr" rid="ref3 ref7">3, 32, 7, 33</xref>
        ]. For example, Wilson [33] argues that security awareness
and defense can and should be taught through ofensive tactics. Likewise, Trabelsi &amp; McCoey
state that ofensive methods are becoming a must for security curricula [ 34]. In a similar vein,
Patrignani &amp; Kavathatzopoulos argue that teaching of technology is also teaching of ethics [35].
      </p>
      <p>
        Hacking and an ofensive approach can be used to motivate students, even in
noncybersecurity topics [
        <xref ref-type="bibr" rid="ref3">36, 3</xref>
        ]. Conti et al. suggest that hacking competitions are an untapped
resource in security education [37]. Dimkov et al. even challenged their students to break into
ofices and steal faculty laptops from the campus and found that their hands-on assignment
increased the students’ awareness of security mechanisms [36]. Students did not perceive the
exercises described in Dimkov et al. as harmful, although they were not seen as particularly
useful either [36]. This suggests that the best assignment for students is not only hands-on but
also incurring knowledge that is applicable in everyday life.
      </p>
      <p>
        Some research articles have explicitly discussed the ethical considerations in cybersecurity
education. Logan &amp; Clarkson [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] explore the potential ethical problems of introducing ”red
teaming” and attack-based exercises into information security courses. However, most students
are unaware of the university’s acceptable use policies, and thus, students should only train
ofensive activities under supervision [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. Pashel [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] discusses the ethical nature of teaching
computer students how to hack in an attempt to strengthen their skills in the field of information
systems security. Hartley [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] argues that an ethical hacking pedagogy may be efective in
preparation to combat unethical hacker intrusions associated with the Internet and computer
networks.
      </p>
      <p>Overall, extant research suggests that teaching hacking skills raises ethical concerns but
arguably the benefits outweigh the drawbacks, as these skills are necessary to better prepare
future information security professionals. However, it is unclear how the knowledge gained from
ofensive activities translates to learning results and students’ ethical perspectives. Additionally,
few studies investigating students’ responsible use of hacking skills exist [30]. Therefore, the
current paper takes steps to investigate this research gap.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Research method and context</title>
      <p>3.1. Cybersecurity course outline and content
The current study collected data from a cybersecurity fundamentals course arranged at a
polytechnic higher education institute. The data was collected during the past four years of running
the course for Information Technology (IT) and Software Engineering (SE) students, one course
implementation per year. Although the course is aimed at computing students, it is arranged
early in the studies with little or no prior computer science experience expected from the
participants. The high-level learning goals for the course were planned as follows. The students
know how to think of actions in terms of security, assess security risks, talk about security
using professional vocabulary, define a security policy, and protect personal communications.
After the course, the students know what cybersecurity is in computer systems, what kind of
threats are there in a digitalized world, and what are the current security needs and security
technologies.</p>
      <p>Overall the course was arranged four times during the past four academic years, from
20192020 to 2022-2023, with the course outline remaining the same with only minor changes. The
lecture content, exercises, and laboratory assignments remained identical, although deliverables
for students varied: In the first two years students turned in all exercises as small, weekly
assignment reports. Later the same weekly tasks remained but only four of them were submitted
for grading, and the reports were more comprehensive. The last assignment of the course was a
reflection of the course as-a-whole, and these reflections were used as the data source.</p>
      <p>Table 1 presents the course outline from the last implementation. The course was arranged
over 15 teaching weeks of which 13 lectures were covered. In 2019-2020 the lectures were held
live. In 2020-2021 the lectures were held online, and in 2022 the lecture videos were delivered
on YouTube giving students the freedom to watch them any time and any place.</p>
      <p>Weekly exercises and laboratory assignments (labs) were arranged to connect theory and
practice. The exercises contained various diferent activities relating to the course content. Most
of the exercises used an ofensive approach. The learning activities turned more technical as
the course progressed. In the beginning, students researched ways to forge physical documents
and types of online scams. Then, a virtual machine running a pre-configured Linux operating
system was introduced to complete more technical tasks, such as file encryption, message
authentication codes, and digital signatures in practice. Additionally, the students learned how
to (mis-)use an email system to forge email headers, retrieve password hashes and crack them
using Linux and Windows, and create a phishing site to harvest credentials using the Nginx [41]
web server. Finally, the learning tasks concluded with using reverse shells to gain remote access
to computer systems, performing SQL injections on a web service, and investigating other web
security methods such as clipboard poisoning and ransomware.</p>
      <p>During the last course implementation (2022-2023) a pedagogic intervention was designed
to help investigate the ethical perceptions of the students while studying the course: Several
small reflection questions about both the legal and illicit ways to use the tools presented were
added to the exercise and assignment instructions. The objective of this intervention was to see
Security principles and se- Forging signatures on
physcurity in practice ical documents
Week 2</p>
      <p>Cyberattacks
Week 3</p>
      <p>Attackers
Week 4</p>
      <p>Security needs
Week 5</p>
      <p>Encryption
Week 6</p>
      <p>Stream ciphers
Week 7</p>
      <p>Block ciphers
Week 8</p>
      <p>Integrity, message
authentication codes, and hashes
Week 9</p>
      <p>Security technologies
Week 10</p>
      <p>Computer security
Week 11</p>
      <p>Certificates and credentials
Week 12
Week 13
Week 14
Week 15</p>
      <p>Security tricks and human
factors
Web security
No lectures
Exam week, no lectures</p>
      <p>Physical penetration
testing with Malduino [38]
Creating a website for an
online scam
Using the Linux virtual
machine
Steganography and file
encryption on Linux</p>
      <sec id="sec-3-1">
        <title>No exercises</title>
        <p>Using hashes, message
authentication codes, and
digital signatures in practice</p>
      </sec>
      <sec id="sec-3-2">
        <title>No exercises</title>
        <p>Windows password
cracking with Ophcrack [39]</p>
      </sec>
      <sec id="sec-3-3">
        <title>No exercises</title>
        <p>Creating a phishing site for
credentials harvesting, and
modifying the Linux hosts
file</p>
      </sec>
      <sec id="sec-3-4">
        <title>No exercises</title>
        <p>SQL injections</p>
      </sec>
      <sec id="sec-3-5">
        <title>No exercises</title>
        <p>Clipboard poisoning and
trust on the web. Creating
ransomware</p>
        <p>Crafting phishing messages
and email spoofing
Linux password cracking
with John the ripper
Retrieving Windows
password hashes with
mimikatz [40] and cracking
them
Using a reverse shell to gain
remote access to a
computer
Final reflection
if explicit questions about the ethics of hacking prompt students to reflect on the ethical uses of
the skills learned at the end of the course. Thus, the tasks in the learning activities remained
the same but minor pedagogic scafolding was added to better support the ethical deliberation
along the way.
3.2. Reflections and the thematic analysis process
To assess how the course, its content, and teaching methods helped students achieve learning
goals, a summative final assignment was used throughout the course’s four-year life cycle. This
last learning task followed the form of a student reflection. In the assignment, we asked the
students to describe three examples of things they have learned on the security fundamentals
course in the following way:
• What (specific) thing did you learn?
• Why is this thing important?
• Why did you choose this thing or topic (to write about)?
• How will/can you make use of this knowledge now and in the future?</p>
        <p>Submitting the reflection task was voluntary but students were allowed to count this
assignment toward the course total (i.e. students were able to turn in this task instead of some other
laboratory assignment). The assignment was not graded in terms of content or correctness.
Students received a passing mark if the above criteria were fulfilled regardless of what they
wrote about (for example, a student could have said that they gained interpersonal and teamwork
skills, even though these are not cybersecurity-related topics).</p>
        <p>We used the thematic analysis research method to analyze the student reflections. The
thematic analysis method is a ”qualitative research method for identifying, analyzing and
reporting patterns (themes) within the data” [42]. The thematic analysis was deemed appropriate
as the data is open-ended. The thematic analysis had two objectives: First, the students’ chosen
topics (of learning) were collected and categorized. Second, we analyzed the essay texts to look
for explicit or implicit mentions of ethical considerations.</p>
        <p>The coding process was conducted by the author alone. Single-coder approaches to thematic
analysis are suficient if the coding is binary or checklist-based [ 43]. For this reason, a data
collection instrument was constructed prior to the coding phase. Therefore, the purpose of
the data collection instrument was to provide a semi-structured coding process to better avoid
researcher bias. Table 2 presents the data collection instrument.</p>
        <p>Overall the thematic analysis process consisted of five phases, presented as follows.
1. Familiarization with the data. An overview of the data was formed with an informal
inspection of all submissions. The students’ chosen topics were collected during this
initial review.
2. Generating initial codes. After the initial inspection, each essay was read, and codified
using the data collection instrument. Once an observation was noted backtracking was
employed to go through submissions that were already codified, in case something was
missed relating to the new observation.
3. Searching for themes. The resulting codes were inspected once the essays were codified.</p>
        <p>The prevalence of each code was calculated, and similar codes were merged when possible.
4. Reviewing themes. After themes were established explanations and features explaining
the codification were drafted.
5. Defining and naming themes. Evaluating and refining the themes, giving them succinct
names, and generating clear definitions.</p>
        <p>In the process of analyzing the data, we followed the ethical principles of research with
human participants by The Finnish National Board on Research Integrity TENK. The ethical
guidelines also afected the choice of research method; The work was (in part) limited to analysis
of the reports as surveys could not be used post-hoc.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Results</title>
      <p>Next, the results of the data collection based on the thematic analysis process are presented.
Overall, the data consists of observations from 110 student reflections. The codified topics (i.e.
”what did you learn”) are presented in Table 3. When reading the reflections, we listed all the
topics (”things”) the students reported to have learned from the course. The students chose
multiple diferent course topics in their learning reflections. In some cases, students would
choose more than three topics to describe and therefore, a total of 375 achieved learning goals
were codified from the submissions.</p>
      <p>The most prevalent topics were: Cryptography (theoretical knowledge of how cryptography
is used), (theoretical knowledge of understanding attacks), cyberattacks (theoretical knowledge
of understanding attacks), encryption (practical skill involving encrypting and decrypting
ifles), and email spoofing (practical skill of how to spoof email headers). Among the most
popular choices were also the CIA (confidentiality, integrity, and availability) security model,
risk management approaches, certificates and credentials, and computer security.</p>
      <p>It is notable that within the student choices, hands-on hacking skills, for example, Email
spoofing (N=28), Password cracking techniques (N=15), Linux password cracking (N=13), or
Windows password cracking (N=12), were no more popular than the security awareness and
defensive skills topics. In fact, most students would pick some generic umbrella topic within
security (for example, Cryptography or Cyberattacks) over practical hacking skills.</p>
      <p>The coding process continued with identifying statements related to the ethics of hacking.
We looked for paragraphs of text, where some explicit or implicit statement was made. For
example, an explicit statement could be ”I would never use hacking skills without permission.”
In contrast, anything that points towards ethical hacking or security awareness was considered
an implicit statement, for example, ”these skills are useful for a possible career in cybersecurity.”</p>
      <p>Table 4 describes the observations made from the student essays. First, we categorized the
essays based on if they described ofensive skills and knowledge (N=54), or defensive skills and
knowledge (N=82). We also recorded whether the essay described gaining practical skills (N=46)
or gaining security awareness and knowledge (N=90). Then, based on the given discussion
prompt ”how will/can you make use of this knowledge” we evaluated we recorded instances of
how the students described the usefulness of the learning. Particularly, we noted instances of:
• if there was an unclear or unknown future use of the knowledge (”unclear application”)
(N=12)
• if the text described usefulness in protecting the student themselves (”happens to self”).</p>
      <p>This was considered a positive ethical stance (N=27)
• if the student describes learning defensive skills (”learn to defend”). Likewise, this was
considered a positive ethical stance (N=64)
• if the student described increased awareness (”awareness raised”). This was considered a
neutral ethical stance (N=59)
• if the student described learning a practical hacking skill. This was considered an unclear
ethical stance (N=4)</p>
      <p>It should be noted that the categorization of all the above criteria was done for all essays.
Similarly, one student essay could be placed in multiple categories, for example, if the essay
described both defensive and ofensive skills or if it described both awareness and defensive
skills.</p>
      <p>Next, we focused on how the student essays described the ethics involved in hacking and
cybersecurity. The essays were categorized based on whether they contained an explicit
statement about ethical considerations, an implicit but clear statement, or no statement at all.
Cryptography
Attacks
Encryption
Email spoofing
CIA Model
Risk management
Importance of security
Certificates and credentials
Computer security
Password cracking
Linux password cracking
Windows password cracking
Digital signatures
Linux skills
Social engineering techniques
Clipboard hijacking
Steganography
Detecting vulnerabilities on Linux
SQL Injections and web security
Malduino
Password security
Reverse shell
Sandboxing
Network defences
Malware
Protocols
Creating a fake website
Principle of least privilege
Critical thinking
Forging documents
Total</p>
      <p>Detailed explanation
N
How encryption algorithms work, how hashing and 46
MACs work
How cyberattacks work (denial-of-service, brute-force 34
etc). Also to some degree mitigations of attacks
How to encrypt and decrypt files or data communications 30
(skill)
How to send email with fabricated headers 28
Meaning of confidentiality, integrity, and availability 24
Risk management, human elements, cybercrime 19
Importance of security as a topic. OR: Basics of cyberse- 18
curity
How cryptography is used to secure online communica- 17
tions in web servers
Generic computer security topics, such as using firewalls, 17
anti-virus, and up-to-date software
Password cracking in general as a skill 15
Specifically dictionary attack 13
Specifically rainbow tables 12
How to sign documents 11
Being proficient at using Linux 11
Topics such as phishing, spear-phishing, and whaling. 11
Copying text directly from a website can be harmful 9
(hidden text)
Hiding data inside innocent looking files 9
Vulnerability detection and compliance audit on Linux 7
using the Lynis software
Scripting and deploying bad USB attacks using Malduino 7
devices
General observations about password strength, reasons 7
to use two-factor authentication etc.</p>
      <p>Knowledge of how SQL injections work and how to test 7
a vulnerable web application
Scripting and deploying a reverse shell connection using 5
netcat (nc)
Use virtual machine as an isolated environment for safety 5
purposes
Network defences or network security concepts (for ex- 4
ample, firewall intrusion detection, dmz, honeypots)
Understanding what malware is and how malware pro- 2
grams work in general
How communications protocols are used to create inter- 2
connected systems, and how to secure those
communications
Creating a website for phishing or online scams 2
How compartmentalising and computer management is 1
used to improve security
Critical thinking skills, such as not falling for online 1
scams
How to make copies of digital or physical documents 1
375</p>
    </sec>
    <sec id="sec-5">
      <title>5. Discussion</title>
      <p>Next, the results are discussed in the context of the original research questions. To what extent do
written student reflections depict ethical considerations? We found that, overall, most students had
included some ethical aspects of using hacking skills and knowledge in their essays. Sometimes
this was an explicit statement of the ethical nature of the work (e.g. ”I will only try these
methods to audit my own systems”). Most often ethical considerations were expressed implicitly
(e.g. ”Now I have more awareness about cyberattacks, to defend from them better”). Implicit
and explicit mentions of the ethics of hacking were present in a total of 69% of the reflections.
When the course material was modified slightly to include more reflections throughout the
course, the number of ethical statements rose to 88%. Both these numbers can be considered
good, as the final essay did not specifically instruct students to write about the ethics of hacking.</p>
      <p>What ethics-related issues emerge from the reflections and how to mitigate them? 11% of the
essays were unsure about the practical applicability of the skills and knowledge gained from the
course. This could be mitigated with purposeful pedagogic design; For example, by investigating
real-world case studies which is often suggested in the cybersecurity education literature (see for
example [44]). Proven successful course designs also include situated learning [45, 46, 47], use
of real-world case examples (for example, news stories) [44, 48], or a special focus on industrial
environments [49].</p>
      <p>To answer the main research question how ethical considerations manifest in the students’
descriptions of what they learned: The most common themes from the student reflections were
related to security awareness and security technologies. Surprisingly, when asked about the
things learned in the course the students often chose topics related to security awareness,
security technologies, and defensive approaches more often than ofensive hacking skills. While
a portion of the students mentioned picking up ofensive skills, the overwhelming majority of
them (106 out of 110) described the learned skills in an ethically sound way. Conversely, only
a small minority of students (4) described just learning to use hacking tools. Therefore, the
horror scenarios about the dangers of teaching hacking depicted by some earlier research are,
for the most part for our students, a non-issue.</p>
      <p>Finally, the last ofering of the course included additional reflection questions in the weekly
exercises and labs. This may have helped with students’ ability to reflect on the ethical
considerations of the skills and knowledge from the course, as the number of observed ethics-related
statements grew to be the highest of all teaching years. However, without further validation,
this is only a cautious indication of the intervention’s results, and currently, the results may
not be statistically significant. Overall, this suggests that while we were able to prompt some
students to think more about the ethics of hacking using minor pedagogic scafolding, the
hacking content (ofensive skills) itself was not a significant factor in the course content.</p>
    </sec>
    <sec id="sec-6">
      <title>6. Conlusion</title>
      <p>Extant literature maintains that ethical considerations must be addressed in computing and
cybersecurity education. However, the current study suggests that, even for novices in the field,
there are seldom ethical issues regarding how to use hacking knowledge. It seems that in the
cybersecurity course, most students implicitly understand the context and the ’right’ way to
use the knowledge and skills in an ethically sound manner.</p>
      <p>Future work and limitations remain in the field. This paper presented only descriptive data
but no statistical analysis. As the data source was observations from the thematic analysis
process, a statistical analysis would have been dificult due to the many underlying confounding
factors and, probably spurious correlations. Additionally, the data analysis was completed by
the author alone. To mitigate researcher bias, a data collection instrument was designed before
data analysis was conducted.</p>
      <p>As the observations were recorded from data over several years, the analysis had to be
conducted post-hoc. Given that the data comes from formal student reports, students might
be simply writing what they think their teachers want to hear. However, the learning task
(reflection) did not specifically ask students to take an ethical stance. Finally, even if students
are trying to please the teachers by moderating how they write about the course, this on its
own suggests that students make conscious choices and ethical considerations while writing
the reflection.</p>
      <p>The increase in ethical considerations within the student reflections during the last
implementation of the course was an encouraging indicator. In future, the efect of the intervention
should be further explored by using a validated instrument and appropriate statistical analysis
method.
[28] M. Hendrix, A. Al-Sherbaz, B. Victoria, Game based cyber security training: are serious
games suitable for cyber security training?, International Journal of Serious Games 3 (2016)
53–61.
[29] Y. A. Younis, K. Kifayat, L. Topham, Q. Shi, B. Askwith, Teaching Ethical Hacking:
Evaluating Students’ Levels of Achievements and Motivations, in: International Conference
on Technical Sciences (ICST2019), volume 6, 2019, p. 04.
[30] R. E. Pike, The “ethics” of teaching ethical hacking, Journal of International Technology
and Information Management 22 (2013) 4.
[31] S. Bratus, A. Shubina, M. E. Locasto, Teaching the principles of the hacker curriculum
to undergraduates, in: Proceedings of the 41st ACM technical symposium on computer
science education, 2010, pp. 122–126.
[32] Z. Trabelsi, W. Ibrahim, Teaching ethical hacking in information security curriculum: A
case study, in: 2013 IEEE Global Engineering Education Conference (EDUCON), IEEE,
2013, pp. 130–137.
[33] B. Wilson, Teaching security defense through web-based hacking at the undergraduate
level (2017).
[34] Z. Trabelsi, M. McCoey, Ethical hacking in information security curricula, International
Journal of Information and Communication Technology Education (IJICTE) 12 (2016) 1–10.</p>
      <p>Publisher: IGI Global.
[35] N. Patrignani, I. Kavathatzopoulos, Teaching of technology is teaching of ethics. but how?,
in: Technology Ethics–Tethics 2021, University of Turku, 2021.
[36] T. Dimkov, W. Pieters, P. Hartel, Training students to steal: a practical assignment in
computer security education, in: Proceedings of the 42nd ACM technical symposium on
computer science education, 2011, pp. 21–26.
[37] G. Conti, T. Babbitt, J. Nelson, Hacking competitions and their untapped potential for
security education, IEEE Security &amp; Privacy 9 (2011) 56–59.
[38] Malduino, 2023. URL: https://malduino.com/.
[39] Ophcrack, 2023. URL: https://ophcrack.sourceforge.io/.
[40] mimikatz, 2023. URL: https://github.com/ParrotSec/mimikatz.
[41] Nginx: Advanced load balancer, web server, &amp; reverse proxy, 2023. URL: https://www.</p>
      <p>nginx.com/.
[42] V. Braun, V. Clarke, Using thematic analysis in psychology, Qualitative research in
psychology 3 (2006) 77–101.
[43] N. McDonald, S. Schoenebeck, A. Forte, Reliability and inter-rater reliability in qualitative
research: Norms and guidelines for cscw and hci practice, Proceedings of the ACM on
Human-Computer Interaction 3 (2019) 1–23.
[44] R. English, J. Maguire, Exploring student perceptions and expectations of cyber security,
in: Computing Education Practice, 2023, pp. 25–28.
[45] Z. A. Wen, Z. Lin, R. Chen, E. Andersen, What. hack: engaging anti-phishing training
through a role-playing phishing simulation game, in: Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems, 2019, pp. 1–12.
[46] T. Denning, A. Lerner, A. Shostack, T. Kohno, Control-alt-hack: the design and evaluation
of a card game for computer security awareness and education, in: Proceedings of the 2013
ACM SIGSAC conference on Computer &amp; communications security, 2013, pp. 915–928.
[47] M. Canepa, F. Ballini, D. Dalaklis, S. Vakili, Assessing the efectiveness of cybersecurity
training and raising awareness within the maritime domain, in: INTED2021 Proceedings,
IATED, 2021, pp. 3489–3499.
[48] T. Hynninen, On the learning activities and outcomes of an information security course,
in: Proceedings of the 19th Koli Calling International Conference on Computing Education
Research, 2019, pp. 1–2.
[49] T. E. Gasiba, K. Beckers, S. Suppan, F. Rezabek, On the requirements for serious games
geared towards software developers in the industry, in: 2019 IEEE 27th International
Requirements Engineering Conference (RE), IEEE, 2019, pp. 286–296.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>N.</given-names>
            <surname>Radziwill</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Romano</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Shorter</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Benton</surname>
          </string-name>
          ,
          <article-title>The ethics of hacking: Should it be taught?</article-title>
          ,
          <source>arXiv preprint arXiv:1512.02707</source>
          (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>L.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. M.</given-names>
            <surname>Chowdhury</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Latif</surname>
          </string-name>
          , Ethical Hacking: Skills to Fight Cybersecurity Threats,
          <source>EPiC Series in Computing</source>
          <volume>82</volume>
          (
          <year>2022</year>
          )
          <fpage>102</fpage>
          -
          <lpage>111</lpage>
          . Publisher: EasyChair.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>P. Y.</given-names>
            <surname>Logan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Clarkson</surname>
          </string-name>
          ,
          <article-title>Teaching students to hack: curriculum issues in information security</article-title>
          ,
          <source>in: Proceedings of the 36th SIGCSE technical symposium on Computer science education</source>
          ,
          <year>2005</year>
          , pp.
          <fpage>157</fpage>
          -
          <lpage>161</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>E.</given-names>
            <surname>Alashwali</surname>
          </string-name>
          ,
          <article-title>Incorporating hacking projects in computer and information security education: an empirical study</article-title>
          ,
          <source>International Journal of Electronic Security and Digital Forensics</source>
          <volume>6</volume>
          (
          <year>2014</year>
          )
          <fpage>185</fpage>
          -
          <lpage>203</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>R. D.</given-names>
            <surname>Hartley</surname>
          </string-name>
          ,
          <article-title>Ethical hacking pedagogy: An analysis and overview of teaching students to hack</article-title>
          ,
          <source>Journal of International Technology and Information Management</source>
          <volume>24</volume>
          (
          <year>2015</year>
          )
          <article-title>6</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>R.</given-names>
            <surname>Hartley</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Medlin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Houlik</surname>
          </string-name>
          ,
          <article-title>Ethical hacking: Educating future cybersecurity professionals</article-title>
          ,
          <source>in: Proceedings of the EDSIG Conference ISSN</source>
          , volume
          <volume>2473</volume>
          ,
          <year>2017</year>
          , p.
          <fpage>3857</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Curbelo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Cruz</surname>
          </string-name>
          ,
          <article-title>Faculty attitudes toward teaching ethical hacking to computer and information systems undergraduates students</article-title>
          ,
          <source>in: Proceedings of the Eleventh LACCEI Latin American and Caribbean Conference for Engineering and Technology</source>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>B. A.</given-names>
            <surname>Pashel</surname>
          </string-name>
          ,
          <article-title>Teaching students to hack: Ethical implications in teaching students to hack at the university level</article-title>
          ,
          <source>in: Proceedings of the 3rd annual conference on Information security curriculum development</source>
          ,
          <year>2006</year>
          , pp.
          <fpage>197</fpage>
          -
          <lpage>200</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>L.</given-names>
            <surname>Riihelä</surname>
          </string-name>
          ,
          <article-title>Teaching information security: A systematic mapping study</article-title>
          ,
          <source>Master's thesis</source>
          , LUT University, Finland,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>L.</given-names>
            <surname>Zhang-Kennedy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Chiasson</surname>
          </string-name>
          ,
          <article-title>A systematic review of multimedia tools for cybersecurity awareness and education, ACM Computing Surveys (CSUR) 54 (</article-title>
          <year>2021</year>
          )
          <fpage>1</fpage>
          -
          <lpage>39</lpage>
          . Publisher: ACM New York, NY, USA.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>V.</given-names>
            <surname>Švábenskỳ</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Vykopal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Čeleda</surname>
          </string-name>
          ,
          <article-title>What are cybersecurity education papers about? a systematic literature review of sigcse and iticse conferences</article-title>
          ,
          <source>in: Proceedings of the 51st ACM technical symposium on computer science education</source>
          ,
          <year>2020</year>
          , pp.
          <fpage>2</fpage>
          -
          <lpage>8</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>S.</given-names>
            <surname>Laato</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Farooq</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Tenhunen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Pitkamaki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Hakkala</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Airola</surname>
          </string-name>
          ,
          <article-title>Ai in cybersecurity education-a systematic literature review of studies on cybersecurity moocs</article-title>
          ,
          <source>in: 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT)</source>
          , IEEE,
          <year>2020</year>
          , pp.
          <fpage>6</fpage>
          -
          <lpage>10</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>X.</given-names>
            <surname>Mountrouidou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Vosen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Kari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. Q.</given-names>
            <surname>Azhar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bhatia</surname>
          </string-name>
          , G. Gagne,
          <string-name>
            <given-names>J.</given-names>
            <surname>Maguire</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Tudor</surname>
          </string-name>
          , T. T. Yuen,
          <article-title>Securing the human: a review of literature on broadening diversity in cybersecurity education</article-title>
          ,
          <source>Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education</source>
          (
          <year>2019</year>
          )
          <fpage>157</fpage>
          -
          <lpage>176</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>N. A. A.</given-names>
            <surname>Rahman</surname>
          </string-name>
          , I. Sairi,
          <string-name>
            <given-names>N. A. M.</given-names>
            <surname>Zizi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Khalid</surname>
          </string-name>
          ,
          <article-title>The importance of cybersecurity education in school</article-title>
          ,
          <source>International Journal of Information and Education Technology</source>
          <volume>10</volume>
          (
          <year>2020</year>
          )
          <fpage>378</fpage>
          -
          <lpage>382</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>H.</given-names>
            <surname>Aldawood</surname>
          </string-name>
          , G. Skinner,
          <article-title>Educating and raising awareness on cyber security social engineering: A literature review, in: 2018 IEEE international conference on teaching, assessment, and learning for engineering (TALE)</article-title>
          , IEEE,
          <year>2018</year>
          , pp.
          <fpage>62</fpage>
          -
          <lpage>68</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>R.</given-names>
            <surname>Roepke</surname>
          </string-name>
          , U. Schroeder,
          <article-title>The Problem with Teaching Defence against the Dark Arts: A Review of Game-based Learning Applications and Serious Games for Cyber Security Education</article-title>
          .,
          <source>in: Proceedings of the 11th International Conference on Computer Supported Education (CSEDU</source>
          <year>2019</year>
          ),
          <year>2019</year>
          , pp.
          <fpage>58</fpage>
          -
          <lpage>66</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>R. B. Sağlam</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Miller</surname>
            ,
            <given-names>V. N.</given-names>
          </string-name>
          <string-name>
            <surname>Franqueira</surname>
            ,
            <given-names>A Systematic</given-names>
          </string-name>
          <string-name>
            <surname>Literature</surname>
          </string-name>
          <article-title>Review on Cyber Security Education for Children</article-title>
          ,
          <source>IEEE Transactions on Education</source>
          (
          <year>2023</year>
          ). Publisher: IEEE.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>N.</given-names>
            <surname>Chowdhury</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Gkioulos</surname>
          </string-name>
          ,
          <article-title>Cyber security training for critical infrastructure protection: A literature review</article-title>
          ,
          <source>Computer Science Review</source>
          <volume>40</volume>
          (
          <year>2021</year>
          )
          <fpage>100361</fpage>
          . Publisher: Elsevier.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>W.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>He</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Tian</surname>
          </string-name>
          , W. He,
          <article-title>Exploring Cybersecurity Education at the K-12 Level</article-title>
          , in: SITE Interactive Conference,
          <article-title>Association for the Advancement of Computing in Education (AACE</article-title>
          ),
          <year>2021</year>
          , pp.
          <fpage>108</fpage>
          -
          <lpage>114</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>M.</given-names>
            <surname>Coenraad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Pellicone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. J.</given-names>
            <surname>Ketelhut</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Cukier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Plane</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Weintrop</surname>
          </string-name>
          ,
          <article-title>Experiencing cybersecurity one game at a time: A systematic review of cybersecurity digital games</article-title>
          ,
          <source>Simulation &amp; Gaming</source>
          <volume>51</volume>
          (
          <year>2020</year>
          )
          <fpage>586</fpage>
          -
          <lpage>611</lpage>
          . Publisher: SAGE Publications Sage CA: Los Angeles, CA.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>F.</given-names>
            <surname>Alotaibi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Furnell</surname>
          </string-name>
          , I. Stengel,
          <string-name>
            <given-names>M.</given-names>
            <surname>Papadaki</surname>
          </string-name>
          ,
          <article-title>A review of using gaming technology for cyber-security awareness</article-title>
          ,
          <source>Int. J. Inf. Secur. Res.(IJISR) 6</source>
          (
          <year>2016</year>
          )
          <fpage>660</fpage>
          -
          <lpage>666</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>J.</given-names>
            <surname>Jeong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Mihelcic</surname>
          </string-name>
          , G. Oliver,
          <string-name>
            <given-names>C.</given-names>
            <surname>Rudolph</surname>
          </string-name>
          ,
          <article-title>Towards an improved understanding of human factors in cybersecurity</article-title>
          ,
          <source>in: 2019 IEEE 5th International Conference on Collaboration and Internet Computing (CIC)</source>
          , IEEE,
          <year>2019</year>
          , pp.
          <fpage>338</fpage>
          -
          <lpage>345</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>R.</given-names>
            <surname>Ramirez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Choucri</surname>
          </string-name>
          ,
          <article-title>Improving interdisciplinary communication with standardized cyber security terminology: A literature review</article-title>
          ,
          <source>IEEE Access 4</source>
          (
          <year>2016</year>
          )
          <fpage>2216</fpage>
          -
          <lpage>2243</lpage>
          . Publisher: IEEE.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>S.</given-names>
            <surname>Das</surname>
          </string-name>
          ,
          <article-title>SoK: a proposal for incorporating accessible gamified cybersecurity awareness training informed by a systematic literature review</article-title>
          ,
          <source>in: Proceedings of the workshop on usable security and privacy (USEC)</source>
          ,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>J.</given-names>
            <surname>Pittman</surname>
          </string-name>
          ,
          <article-title>Understanding system utilization as a limitation associated with cybersecurity laboratories-A literature analysis</article-title>
          ,
          <source>Journal of Information Technology Education. Research</source>
          <volume>12</volume>
          (
          <year>2013</year>
          )
          <fpage>363</fpage>
          . Publisher: Informing Science Institute.
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>M.</given-names>
            <surname>Lamond</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Renaud</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Wood</surname>
          </string-name>
          ,
          <string-name>
            <surname>S. Prior,</surname>
          </string-name>
          <article-title>SOK: young children's cybersecurity knowledge, skills &amp; practice: a systematic literature review</article-title>
          ,
          <source>in: Proceedings of the 2022 European Symposium on Usable Security</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>14</fpage>
          -
          <lpage>27</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hautamäki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Karjalainen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Hämäläinen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Häkkinen</surname>
          </string-name>
          ,
          <article-title>Cyber security exercise: Literature review to pedagogical methodology</article-title>
          ,
          <source>in: INTED Proceedings, IATED Academy</source>
          ,
          <year>2019</year>
          . Issue:
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>