Cryptography is an important field dedicated to securing data transmission through advanced algorithms and techniques. Everyday applications of cryptographic algorithms such as TLS for encrypted web traffic and Diffie-Hellman or RSA for secure remote server management showcase their critical role in protecting information. However, the same cryptographic techniques that protect data can also be misused by malicious actors for malicious purposes. This research focuses on analyzing the innovative applications of cryptographic methods in both safeguarding data and facilitating cyberattacks, emphasizing the dual-edged nature of these technologies in the evolving landscape of cybersecurity. The research underscores the critical need to recognize encrypted traffic as a significant threat and provides targeted recommendations for improving defensive and offensive strategies.
In our everyday life, we utilize cryptographic methods and
algorithms to secure the data and prevent eavesdropping.
This can be achieved with various algorithms, for example:
TLS, RSA, AES, and many more. With the help of
cryptography, we can be almost sure that even if our data is
stolen, it cannot be read by the threat actor. Though, this
also works in the opposite direction: if the malicious user
sends malicious data over the encrypted channel, the
socalled blue team will struggle with identifying such traffic,
as without decrypting the traffic, it is rather problematic to
conclude, whether it is malicious indeed or not. The
development of machine learning can help with the
problem: the artificial intelligence is trained on the datasets
and learns to identify the malicious encrypted traffic,
though, if the attacker utilizes self-written encryption or
obfuscation algorithm, the artificial intelligence will fail to
spot it, as the data signature will be unmatched. This
situation causes the dilemma: cryptography, a savoir of
confidentiality, can be used as a double-edged sword to hide
malicious traffic and data transfers. Cryptography can be
used to hide the following attacks: web-based attacks,
reverse shells and remote code execution, and data
exfiltration. This paper discusses the effectiveness of
network Intrusion Detection Systems (IDS) and Data Loss
Prevention (DLP) tools against encrypted malicious traffic.
The paper also discusses the nested encryption, and
obfuscation techniques and their usage in penetration tests
[
Cryptographic methods are pivotal for ensuring data
security, employing algorithms such as Transport Layer
Security (TLS), RSA, and Advanced Encryption Standard
(AES) to protect information from unauthorized access
[
0000-0002-3109-7971 (M. Iavich); 0009-0000-0124-2931 (S. Simonovi); 0000-0001-9036-6556 (T.Okhrimenko) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
As a result, current cybersecurity measures face significant challenges:
Inadequate Detection: Traditional IDS and DLP systems frequently fail to identify malicious activities within encrypted traffic, leading to potential blind spots in network security.
Complex Analysis: Even when encrypted threats are detected, the difficulty in decrypting and analyzing the traffic impedes the ability to understand and mitigate attacks effectively.
Advanced Obfuscation: Attackers' use of nested encryption and proprietary obfuscation techniques introduces additional layers of complexity, making it difficult for security professionals to reconstruct attack timelines and assess the full scope of threats.
This problem needs a critical evaluation of how current defensive technologies must be improved to address these challenges. There is a serious need to create and implement advanced detection techniques that can efficiently handle encrypted malicious traffic and ensure that cybersecurity defenses can keep pace with evolving threats.
To validate the hypothesis regarding the effectiveness of encrypted malicious traffic, we constructed a virtual laboratory comprising several key components. The setup included:
VirtualBox is the hypervisor, providing the virtualization environment necessary for the lab. Kali Linux serves as the attacker machine, equipped with tools for executing and managing attacks. pfSense 2.7.0 is configured as the router and firewall, facilitating network traffic management and security.
Ubuntu 22.04.3 runs Suricata 6.0.4 as the network IDS and Damn Vulnerable Web Application (DVWA) as the target vulnerable software.
Machine learning and artificial intelligence have emerged as
potential solutions to enhance detection capabilities. These
technologies can identify patterns and anomalies in
network traffic, even when encrypted [
The use of cryptography in concealing cyberattacks is
increasingly prevalent. Attackers can exploit encryption to
hide various types of malicious activities, including
webbased attacks, reverse shells, remote code execution, and
data exfiltration [
Recent studies have also explored obfuscation
techniques that challenge malware detection and analysis.
Techniques such as code obfuscation and metamorphism
are used to hide malware from detection systems,
complicating the analysis and remediation process [
The authors of the papers also emphasize the necessity
for adaptive defensive and offensive strategies to keep pace
with evolving cryptographic threats [
The increasing number and complexity of cyber threats and the prevalence of cryptographic techniques in securing communications have created a significant challenge for cybersecurity protection measures. While encryption technologies such as TLS, RSA, and AES are essential for securing data from unauthorized access, they also can pose serious difficulties in the detection and analysis of malicious activities conducted over encrypted channels.
The main issue appears from the dual role of cryptography: while it protects legitimate data, it also allows attackers to obfuscate malicious payloads, rendering traditional IDS and DLP tools less effective. These systems often struggle to analyze encrypted traffic comprehensively, as they are unable to inspect the content without decrypting it. This limitation is exacerbated when attackers employ advanced techniques such as nested encryption or custom obfuscation algorithms, which further obscure the nature of the malicious traffic and complicate the reconstruction of attack sequences. 1. 2. 3.
The network topology of this virtual laboratory is depicted in Fig. 1.
The goal is to emulate the global network environment, where the victim machine is located behind the NAT router, and the attacker machine is outside of the victim’s local network. Kali Linux is a part of 192.168.1.0/24 network and has a network interface in a “bridged” mode.
The victim host is a part of 172.16.0.0/24 network and has a network interface in an “internal network” mode.
PfSense plays the role of a NAT router and is a part of both networks, having two network interfaces: WAN (192.168.1.150, works in a “bridged network mode”) and LAN (172.16.0.1, works in an “internal network” mode).
To make a victim server reachable from 192.168.1.0/24 network, the port forwarding rules for ports 80 and 443 are added on PfSense. Also, “block private networks” checkbox is unchecked. All egress traffic is permitted, all ingress traffic, except port forwarding, is prohibited.
The second experiment is running the SQL injection over HTTPS. To perform this, a self-signed SSL certificate will be generated. The Apache will be configured to use a domain name “dvwa.local”. The according line (192.168.1.150 dvwa.local) will be added to the “/etc/hosts” on Kali Linux. This will solve the problem with IP hostnames over the NAT. The virtual host configuration file can be seen in Fig. 4.
When running the attack, Suricata can detect the attack no more, as the payload is encrypted with SSL. This can be seen in Figs 5 and 6. Timestamps are included.
The victim machine uses apache2 2.4.52 and Suricata
6.0.4. Suricata uses custom rules and the rules are taken
from the following GitHub repository [
In the virtual laboratory, the following experiments were executed:
Running reverse shell over the encrypted socket.
The first experiment is running the SQL injection over HTTP. Being straightforward, the attack signature is well known, and as it is unencrypted, is easily detected by the IDS. The results of the attack can be observed in Figs. 2 and 3. The third experiment involves running the reverse shell over the unencrypted socket. To perform this, “command injection” tab in DVWA will be used. The payload running the reverse shell will be bash -c "bash -i >& /dev/tcp/192.168.1.100/443 0>&1" where 192.168.1.100 is an address of the Kali Linux and 443 is a port on which the attacker will “catch” the shell. Even if the payload is run over HTTPS, the new unencrypted connection opens between a victim and the attacker.
The Suricata rule to detect the malicious traffic is: alert tcp any any -> any any (msg:”WHOAMI issued”; flow:not_established,to_server; content:”whoami”; nocase; sid:4000006; rev:1;)
The results of the attack and the detection are depicted in Figs. 7 and 8. Timestamps are included. The fourth experiment involves running the reverse shell over the encrypted socket. To perform this, “socat” tool will be used. The encryption is achieved using a selfsigned certificate generated by “openssl”. Results of the attack can be observed in Figs. 9, 10, and 11.
The research presented in this paper explores the effectiveness of cryptography as a tool for executing attacks and evaluates the efficacy of IDS in defending against such attacks when they occur over encrypted channels. The study demonstrates that encrypted traffic can effectively evade detection by widely used IDS software, such as “Suricata.” While these IDS systems may be capable of detecting the presence of payloads, they face significant challenges in analyzing the attack timeline and reconstructing the sequence of actions performed by the attacker.
The paper highlights that even if an IDS identifies the payload, the complexity of analyzing the attack increases considerably when dealing with encrypted traffic. The challenge is further exacerbated by techniques like nested encryption. For instance, the research describes a scenario where a blue team successfully obtained the certificate used for encrypting the data. However, upon decrypting the malicious traffic, they discovered that the data had been further scrambled using an unknown algorithm, complicating the process of understanding and mitigating the attack.
The findings underscore the potential threats posed by any encrypted traffic, which can be exceedingly difficult to detect and analyze using traditional security tools. This reveals a critical gap in current cybersecurity measures and highlights the need for more advanced techniques to address the evolving challenges posed by encrypted attack vectors.
In conclusion, the research presented in this whitepaper underscores the dual role of cryptography as both a protective measure and a potential attack tool, highlighting significant limitations in the efficacy of traditional IDS when faced with encrypted traffic. The study provides substantial evidence that wellestablished IDS solutions, such as Suricata, can be bypassed by encrypted communications. Furthermore, even if such encrypted payloads are detected, the process of reconstructing the attack timeline and understanding the sequence of actions undertaken by the attacker remains profoundly challenging, particularly when nested encryption techniques are employed.
The findings of this research emphasize the critical need to acknowledge encrypted traffic as a sophisticated threat that traditional security tools may inadequately address. To mitigate these challenges, the whitepaper offers specific recommendations for both blue and red team practitioners:
For Blue Team Members:
This work was supported by the Shota Rustaveli National Foundation of Georgia (SRNSFG) (NFR-2214060).
Enable Advanced Detection Techniques: Implement anomaly-based and Indicator of Compromise-based detection methods to improve the identification of malicious activities.
Deploy Endpoint Detection and Response (EDR): Utilize EDR solutions on both servers and client systems to strengthen endpoint protection and response mechanisms.
Apply Nested Encryption: Employ nested encryption strategies to further complicate detection efforts and hinder the blue team's ability to reconstruct attack sequences.
By applying these recommendations, organizations can greatly enhance their preparedness and responsiveness to the evolving threats posed by encrypted traffic, therefore improving their overall cybersecurity level in an increasingly complex threat landscape.