Cryptography as a dual-faceted instrument of security and vulnerability Maksim Iavich1,†, Sergei Simonovi1,† and Tetiana Okhrimenko2,*,† 1 Caucasus University, 1 Paata Saakadze str., 0102 Tbilisi, Georgia 2 National Aviation University, 1 Lubomyra Guzara ave., 03058 Kyiv, Ukraine Abstract Cryptography is an important field dedicated to securing data transmission through advanced algorithms and techniques. Everyday applications of cryptographic algorithms such as TLS for encrypted web traffic and Diffie-Hellman or RSA for secure remote server management showcase their critical role in protecting information. However, the same cryptographic techniques that protect data can also be misused by malicious actors for malicious purposes. This research focuses on analyzing the innovative applications of cryptographic methods in both safeguarding data and facilitating cyberattacks, emphasizing the dual-edged nature of these technologies in the evolving landscape of cybersecurity. The research underscores the critical need to recognize encrypted traffic as a significant threat and provides targeted recommendations for improving defensive and offensive strategies. Keywords cryptography, IDS, DLP, penetration testing. 1 1. Introduction The objective of this research is also to conduct experimental evaluations to assess the effectiveness of In our everyday life, we utilize cryptographic methods and defensive software in detecting and mitigating encrypted algorithms to secure the data and prevent eavesdropping. malicious traffic. Additionally, the study aims to explore the This can be achieved with various algorithms, for example: dual role of cryptography as both a defensive mechanism TLS, RSA, AES, and many more. With the help of and a tool exploited by penetration testers and cybersecurity cryptography, we can be almost sure that even if our data is criminals. By examining how cryptographic techniques are stolen, it cannot be read by the threat actor. Though, this employed in both safeguarding and attacking digital also works in the opposite direction: if the malicious user systems, this research seeks to provide insights into the sends malicious data over the encrypted channel, the so- strengths and limitations of current cybersecurity defenses called blue team will struggle with identifying such traffic, in the face of sophisticated encryption-based threats [4]. as without decrypting the traffic, it is rather problematic to The research also aims to show the critical need to recognize conclude, whether it is malicious indeed or not. The encrypted traffic as a significant threat and to provide development of machine learning can help with the targeted recommendations for improving defensive and problem: the artificial intelligence is trained on the datasets offensive strategies. and learns to identify the malicious encrypted traffic, though, if the attacker utilizes self-written encryption or 2. Review of the literature obfuscation algorithm, the artificial intelligence will fail to spot it, as the data signature will be unmatched. This Cryptographic methods are pivotal for ensuring data situation causes the dilemma: cryptography, a savoir of security, employing algorithms such as Transport Layer confidentiality, can be used as a double-edged sword to hide Security (TLS), RSA, and Advanced Encryption Standard malicious traffic and data transfers. Cryptography can be (AES) to protect information from unauthorized access used to hide the following attacks: web-based attacks, [5, 6]. These techniques are essential in maintaining reverse shells and remote code execution, and data confidentiality and integrity in data transmission. Despite exfiltration. This paper discusses the effectiveness of their effectiveness, encrypted communication poses network Intrusion Detection Systems (IDS) and Data Loss significant challenges to network security, particularly in Prevention (DLP) tools against encrypted malicious traffic. detecting malicious activities. Traditional IDS and DLP tools The paper also discusses the nested encryption, and often struggle with encrypted traffic, as these systems obfuscation techniques and their usage in penetration tests require decryption to analyze the content, making detection [1–3]. of malicious activities complex [7, 8]. CPITS-II 2024: Workshop on Cybersecurity Providing in Information 0000-0002-3109-7971 (M. Iavich); and Telecommunication Systems II, October 26, 2024, Kyiv, Ukraine 0009-0000-0124-2931 (S. Simonovi); ∗ Corresponding author. 0000-0001-9036-6556 (T.Okhrimenko) † These authors contributed equally. © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). miavich@cu.edu.ge (M. Iavich); s_simonovi@cu.edu.ge (S. Simonovi); t.okhrimenko@nau.edu.ua (T. Okhrimenko) CEUR Workshop ceur-ws.org ISSN 1613-0073 378 Proceedings Machine learning and artificial intelligence have emerged as As a result, current cybersecurity measures face significant potential solutions to enhance detection capabilities. These challenges: technologies can identify patterns and anomalies in network traffic, even when encrypted [9]. However, their 1. Inadequate Detection: Traditional IDS and DLP effectiveness is compromised when attackers employ systems frequently fail to identify malicious custom encryption or obfuscation techniques, which can activities within encrypted traffic, leading to render traffic patterns unrecognizable and evade detection potential blind spots in network security. [10]. This illustrates a critical challenge: while cryptography 2. Complex Analysis: Even when encrypted threats ensures data confidentiality, it can also be leveraged to are detected, the difficulty in decrypting and obscure malicious activities. analyzing the traffic impedes the ability to The use of cryptography in concealing cyberattacks is understand and mitigate attacks effectively. increasingly prevalent. Attackers can exploit encryption to 3. Advanced Obfuscation: Attackers' use of nested hide various types of malicious activities, including web- encryption and proprietary obfuscation based attacks, reverse shells, remote code execution, and techniques introduces additional layers of data exfiltration [11]. Techniques such as nested encryption complexity, making it difficult for security and obfuscation further complicate the detection and professionals to reconstruct attack timelines and analysis of malicious traffic, presenting substantial assess the full scope of threats. challenges for defensive strategies [12]. To address these This problem needs a critical evaluation of how current issues, research suggests integrating SSL certificates into defensive technologies must be improved to address these IDS solutions, employing anomaly-based and Indicator of challenges. There is a serious need to create and implement Compromise (IOC) detection, and deploying Endpoint advanced detection techniques that can efficiently handle Detection and Response (EDR) systems to enhance overall encrypted malicious traffic and ensure that cybersecurity security [13, 14]. defenses can keep pace with evolving threats. Recent studies have also explored obfuscation techniques that challenge malware detection and analysis. Techniques such as code obfuscation and metamorphism 4. Laboratory are used to hide malware from detection systems, To validate the hypothesis regarding the effectiveness complicating the analysis and remediation process [15–17]. of encrypted malicious traffic, we constructed a virtual These methods demonstrate how attackers can leverage laboratory comprising several key components. The setup encryption and obfuscation to enhance the stealth of their included: activities, further emphasizing the need for advanced defensive strategies.  VirtualBox is the hypervisor, providing the The authors of the papers also emphasize the necessity virtualization environment necessary for the lab. for adaptive defensive and offensive strategies to keep pace  Kali Linux serves as the attacker machine, with evolving cryptographic threats [18–21]. Recognizing equipped with tools for executing and managing encrypted traffic as a significant threat and developing attacks. targeted recommendations for improving detection and  pfSense 2.7.0 is configured as the router and response capabilities are essential for strengthening firewall, facilitating network traffic management cybersecurity posture in the face of sophisticated and security. encryption-based threats [21–25].  Ubuntu 22.04.3 runs Suricata 6.0.4 as the network IDS and Damn Vulnerable Web Application 3. Problem statement (DVWA) as the target vulnerable software. The increasing number and complexity of cyber threats and the prevalence of cryptographic techniques in securing The network topology of this virtual laboratory is communications have created a significant challenge for depicted in Fig. 1. cybersecurity protection measures. While encryption The goal is to emulate the global network environment, technologies such as TLS, RSA, and AES are essential for where the victim machine is located behind the NAT router, securing data from unauthorized access, they also can pose and the attacker machine is outside of the victim’s local serious difficulties in the detection and analysis of malicious network. Kali Linux is a part of 192.168.1.0/24 network and activities conducted over encrypted channels. has a network interface in a “bridged” mode. The main issue appears from the dual role of The victim host is a part of 172.16.0.0/24 network and cryptography: while it protects legitimate data, it also has a network interface in an “internal network” mode. allows attackers to obfuscate malicious payloads, rendering PfSense plays the role of a NAT router and is a part of traditional IDS and DLP tools less effective. These systems both networks, having two network interfaces: WAN often struggle to analyze encrypted traffic comprehensively, (192.168.1.150, works in a “bridged network mode”) and as they are unable to inspect the content without decrypting LAN (172.16.0.1, works in an “internal network” mode). it. This limitation is exacerbated when attackers employ To make a victim server reachable from 192.168.1.0/24 advanced techniques such as nested encryption or custom network, the port forwarding rules for ports 80 and 443 are obfuscation algorithms, which further obscure the nature of added on PfSense. Also, “block private networks” checkbox the malicious traffic and complicate the reconstruction of is unchecked. All egress traffic is permitted, all ingress attack sequences. traffic, except port forwarding, is prohibited. 379 The victim machine uses apache2 2.4.52 and Suricata 6.0.4. Suricata uses custom rules and the rules are taken from the following GitHub repository [26]. 5. Experiments In the virtual laboratory, the following experiments were executed:  Running SQL injection over HTTP.  Running SQL injection over HTTPS.  Running reverse shell over the unencrypted socket.  Running reverse shell over the encrypted socket. The first experiment is running the SQL injection over HTTP. Being straightforward, the attack signature Figure 1: Virtual laboratory network topology. is well known, and as it is unencrypted, is easily detected by the IDS. The results of the attack can be observed in Figs. 2 and 3. Figure 2: Running the SQL injection over HTTP. Figure 3: Suricata detects the attack. The second experiment is running the SQL injection over HTTPS. To perform this, a self-signed SSL certificate will be generated. The Apache will be configured to use a domain name “dvwa.local”. The according line (192.168.1.150 dvwa.local) will be added to the “/etc/hosts” on Kali Linux. This will solve the problem with IP hostnames over the NAT. The virtual host configuration file can be seen in Fig. 4. When running the attack, Suricata can detect the attack no more, as the payload is encrypted with SSL. This can be seen in Figs 5 and 6. Timestamps are Figure 4: DVWA virtual host. included. 380 Figure 5: Running the SQL injection over HTTPS. Figure 6: Suricata fails to detect the attack performed over HTTPS. The third experiment involves running the reverse shell unencrypted connection opens between a victim and the over the unencrypted socket. To perform this, attacker. “command injection” tab in DVWA will be used. The The Suricata rule to detect the malicious traffic is: payload running the reverse shell will be alert tcp any any -> any any (msg:”WHOAMI bash -c "bash -i >& /dev/tcp/192.168.1.100/443 0>&1" issued”; flow:not_established,to_server; where 192.168.1.100 is an address of the Kali Linux content:”whoami”; nocase; sid:4000006; rev:1;) and 443 is a port on which the attacker will “catch” the The results of the attack and the detection are shell. Even if the payload is run over HTTPS, the new depicted in Figs. 7 and 8. Timestamps are included. Figure 7: Received reverse shell and issued “whoami” command. Figure 8: Suricata detects the attack. 381 The fourth experiment involves running the reverse signed certificate generated by “openssl”. Results of the shell over the encrypted socket. To perform this, “socat” attack can be observed in Figs. 9, 10, and 11. tool will be used. The encryption is achieved using a self- Figure 9: Running the encrypted reverse shell. Figure 10: Receiving the reverse shell and running the “whoami” command. Figure 11: Suricata fails to detect the encrypted malicious traffic. 6. Results discussions The paper highlights that even if an IDS identifies the payload, the complexity of analyzing the attack The research presented in this paper explores the increases considerably when dealing with encrypted effectiveness of cryptography as a tool for executing traffic. The challenge is further exacerbated by attacks and evaluates the efficacy of IDS in defending techniques like nested encryption. For instance, the against such attacks when they occur over encrypted research describes a scenario where a blue team channels. The study demonstrates that encrypted traffic successfully obtained the certificate used for encrypting can effectively evade detection by widely used IDS the data. However, upon decrypting the malicious software, such as “Suricata.” While these IDS systems traffic, they discovered that the data had been further may be capable of detecting the presence of payloads, scrambled using an unknown algorithm, complicating they face significant challenges in analyzing the attack the process of understanding and mitigating the attack. timeline and reconstructing the sequence of actions The findings underscore the potential threats posed performed by the attacker. by any encrypted traffic, which can be exceedingly 382 difficult to detect and analyze using traditional security Acknowledgments tools. This reveals a critical gap in current cybersecurity measures and highlights the need for more advanced This work was supported by the Shota Rustaveli techniques to address the evolving challenges posed by National Foundation of Georgia (SRNSFG) (NFR-22- encrypted attack vectors. 14060). 7. Conclusions References In conclusion, the research presented in this [1] R. Marusenko, V. Sokolov, P. Skladannyi, Social whitepaper underscores the dual role of cryptography as Engineering Penetration Testing in Higher both a protective measure and a potential attack tool, Education Institutions, Advances in Computer highlighting significant limitations in the efficacy of Science for Engineering and Education VI, vol. 181 traditional IDS when faced with encrypted traffic. The (2023) 1132–1147. study provides substantial evidence that well- [2] R. Marusenko, V. Sokolov, V. Buriachok, established IDS solutions, such as Suricata, can be Experimental Evaluation of Phishing Attack on bypassed by encrypted communications. Furthermore, High School Students, Advances in Computer even if such encrypted payloads are detected, the Science for Engineering and Education III, vol. process of reconstructing the attack timeline and 1247 (2020) 668–680. doi:10.1007/978-3030-55506- understanding the sequence of actions undertaken by 1_59. the attacker remains profoundly challenging, [3] R. Marusenko, V. Sokolov, I. Bogachuk, Method of particularly when nested encryption techniques are Obtaining Data from Open Scientific Sources and employed. Social Engineering Attack Simulation, Advances The findings of this research emphasize the critical in Artificial Systems for Logistics Engineering, vol. need to acknowledge encrypted traffic as a sophisticated 135 (2022) 583–594. doi: 10.1007/978-3-031-04809- threat that traditional security tools may inadequately 8_53. address. To mitigate these challenges, the whitepaper [4] R. Chernenko, et al., Encryption Method for offers specific recommendations for both blue and red Systems with Limited Computing Resources, in: team practitioners: Workshop on Cybersecurity Providing in For Blue Team Members: Information and Telecommunication Systems, CPITS, vol. 3288 (2022) 142–148.  Import SSL Certificates: Integrate SSL [5] I. Ristic, Bulletproof SSL and TLS: Understanding certificates used by web servers into IDS and deploying SSL/TLS and PKI to secure servers solutions to enhance visibility and detection and web applications. Feisty Duck (2014). capabilities. [6] M. Iavich, et al., Comparison and Hybrid  Enable Advanced Detection Techniques: Implementation of Blowfish, Twofish and RSA Implement anomaly-based and Indicator of Cryptosystems, in: IEEE 2nd Ukraine Conf. on Compromise-based detection methods to Electrical and Computer Engineering (UKRCON) improve the identification of malicious (2019) 970–974, doi: 10.1109/UKRCON.2019. activities. 8880005.  Deploy Endpoint Detection and Response [7] R. Oppliger, SSL and TLS: Theory and Practice. (EDR): Utilize EDR solutions on both servers Artech House (2023). and client systems to strengthen endpoint [8] A. S. Ashoor, S. Gore, Importance of Intrusion protection and response mechanisms. Detection System (IDS), International Journal of Scientific and Engineering Research, vol. 2, no. 1 For Red Team Members: (2011) 1–4. [9] O. Depren, et al., An Intelligent Intrusion  Leverage Encrypted Channels: Use encrypted channels to execute payloads and attacks, Detection System (IDS) for Anomaly and Misuse making it more challenging for IDS systems to Detection in Computer Networks, Expert systems detect and analyze the traffic. with Applications, vol. 29, no. 4 (2005) 713–722.  Apply Nested Encryption: Employ nested [10] H. J. Liao, et al., Intrusion Detection System: A encryption strategies to further complicate Comprehensive Review, Journal of Network and detection efforts and hinder the blue team's Computer Applications, vol. 36, no. 1 (2013) 16–24. ability to reconstruct attack sequences. [11] D. Day, B. Burns, A Performance Analysis of Snort and Suricata Network Intrusion Detection and By applying these recommendations, organizations can Prevention Engines, in: 5th International greatly enhance their preparedness and responsiveness Conference on Digital Society, Gosier, Guadeloupe to the evolving threats posed by encrypted traffic, (2011) 187–192. therefore improving their overall cybersecurity level in [12] J. Donadio, G. Guerard, S. B. Amor, Collection of an increasingly complex threat landscape. the Main Anti-Virus Detection and Bypass Techniques, in: Network and System Security: 15th International Conference (NSS) (2021) 222–237. 383 [13] E. Albin, N. C. Rowe, A Realistic Experimental Comparison of the Suricata and Snort Intrusion- Detection Systems, in: 26th International Conference on Advanced Information Networking and Applications Workshops (2012) 122–127. [14] K. Wong, et al., Enhancing Suricata Intrusion Detection System for Cyber Security in SCADA Networks, in: IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE) (2017) 1–5. [15] S. Liu, R. Kuhn, Data Loss Prevention. IT Professional, vol. 12, no. 2 (2010) 10-13. [16] S. Alneyadi, E. Sithirasenan, V. Muthukkumara- samy, A Survey on Data Leakage Prevention Systems, Journal of Network and Computer Applications, vol. 62 (2016) 137–152. [17] J. Singh, Challenge of Malware Analysis: Malware Obfuscation Techniques, International Journal of Information Security Science, vol. 7, no. 3 (2018) 100–110. [18] I. You, K. Yim, Malware Obfuscation Techniques: A Brief Survey, in: International Conference on Broadband, Wireless Computing, Communication and Applications (2010) 297–300. [19] B. B. Rad, M. Masrom, S. Ibrahim, Camouflage in Malware: From Encryption to Metamorphism, International Journal of Computer Science and Network Security, vol. 12, no. 8 (2012) 74–83. [20] D. Maiorca, et al., Stealth Attacks: An Extended Insight into the Obfuscation Effects on Android Malware, Computers & Security, vol. 51 (2015) 16– 31. [21] D. Park, H. Khan, B. Yener, Generation & Evaluation of Adversarial Examples for Malware Obfuscation, in: 18th IEEE International Conference on Machine Learning and Applications (ICMLA) (2019) 1283–1290. [22] M. Christodorescu, S. Jha, Testing Malware Detectors. ACM SIGSOFT Software Engineering Notes, vol. 29, no. 4 (2004) 34–44. [23] M. I. Sharif, et al., Impeding Malware Analysis Using Conditional Code Obfuscation, in: NDSS (2008). [24] G. Canfora, et al., Obfuscation Techniques against Signature-based Detection: A Case Study, in: Mobile Systems Technologies Workshop (MST) (2015) 21–26. [25] E. Jintcharadze, M. Iavich, Hybrid Implementation of Twofish, AES, ElGamal and RSA Cryptosystems, in: IEEE East-West Design & Test Symposium (EWDTS) (2020). doi: 10.1109/ ewdts50664. [26] M. Daffa, Suricata Rules (2023). URL: https://github.com/daffainfo/suricata-rules 384