As the complexity and scale of systems continue to increase, enterprises are placing ever higher demands on system security, making comprehensive security analysis particularly important. Among the various methods, penetration testing is regarded as one of the most direct means for assessing security. In order to explore the resistance (to attacks) standard within the SQuaRE series standards concerning product quality, this paper proposes a penetration path discovery method based on an improved deep reinforcement learning algorithm, Adaptive Soft Actor-Critic (ASAC). This method involves modeling the penetration process and quantifying the penetration benefits, and it leverages the Soft Actor-Critic (SAC) algorithm, which is suitable for complex state spaces, to construct an intelligent penetration agent. The goal is to solve for the optimal penetration path, thereby enabling the evaluation of a system's resistance to attacks during system validation.Experimental results demonstrate that the attack efectiveness of the proposed ASAC algorithm surpasses that of commonly used reinforcement learning algorithms such as Q-learning and DQN. It can quickly identify the most critical penetration paths in a network and maintain high performance across diferent network environments. This approach provides efective theoretical and technical support for the comprehensive assessment of the robustness of the cybersecurity of the system.
As business scales grow, many enterprises choose to store data across multiple physical devices (such as servers) to enhance access speeds, with unified management and access via networks. Attackers may infiltrate the data storage local area network through boundary nodes, thereby stealing sensitive information and causing significant losses to the enterprise. It is crucial for enterprises to periodically conduct penetration testing based on the state of their network environment, to assess the system’s risk and robustness. Penetration testing is a black-box security testing method, in which the tester adopts the mindset and technical means of an attacker to detect vulnerabilities in business system targets. It helps enterprises uncover hidden security flaws and vulnerabilities in normal business processes, breach the system’s security defenses, and deeply assess the potential impacts of vulnerabilities.
In real-world enterprise internal networks and sensitive core networks, initial stages of penetration testing may not provide valuable information due to the lack of direct insight into the system. The Markov process (Markov Process) is a type of stochastic process characterized by the property that future states depend only on the current state, not on past states. This paper formalizes the penetration testing process as a Markov Decision Process (MDP), utilizing network information to build a reward function that guides the agent to discover hidden attack paths from the attacker’s perspective. This method does not require prior validation of network structures or software configurations and can autonomously discover attack paths, extracting essential 6th International Workshop on Experience with SQuaRE family and its Future Direction,3rd December 2024,Chongqing, China(Hybrid) ⋆ You can use this document as the template for preparing your publication. We recommend using the latest version of the ceurart style. * Corresponding author. † These authors contributed equally. $ lymon1004@163.com (Y. Liu); liuxj@bjut.edu.cn (X. Liu); 759301040@qq.com (X. Yu) 0009-0009-6580-9840 (Y. Liu); 0000-0002-0666-4102 (X. Liu) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). penetration testing information. Furthermore, reinforcement learning is employed as the agent’s learning strategy, enabling it to interact with the environment to maximize its reward. Penetration testing can also be viewed as a dynamic decision-making process based on the current environmental state. Given the similarity between penetration testing and reinforcement learning mechanisms, reinforcement learning is well-suited to describe penetration testing in unknown environments.
This paper proposes an improved reinforcement learning algorithm, ASAC, for penetration path planning in unknown environments. By improving search strategies and action selection policies, the algorithm enhances the planning eficiency.
Additionally, the proposed method aligns with the ISO/IEC 25000 series standards, specifically the SQuaRE (Software Quality Requirements and Evaluation) framework, particularly the security resistance sub-characteristics. According to the SQuaRE standard, security, as an important aspect of software quality, requires that the system be able to resist attacks and ensure the confidentiality, integrity and availability of data. Our proposed penetration testing method, by optimizing the attack path discovery, can evaluate the system’s resistance and defense capability according to the calculation time of the model and the success rate of multiple simulated attacks. In the ISO/IEC 25010 standard corresponding security features such as confidentiality, integrity, availability and resistance have been efectively improved.
The main contributions of this paper are as follows: • We combines complex multi-step attack characteristics to conduct penetration test modeling, quantifies action gains and network environment vulnerability losses, and more accurately simulates the process of attackers gradually breaking through various firewalls and gateways, providing accurate reward signals for the generation of multi-step and multi-host attack paths. • SAC reinforcement learning algorithm suitable for complex discrete action or even continuous action space is introduced into the penetration testing process, and the improved intelligent penetration testing agent (ASAC) can cope with the complex and changeable network environment and the exponential growth of the action space.
Traditional penetration testing (PT) relies on manual methods, which become impractical as the systems grow in size and complexity. By simulating 20 attack strategies of real attackers, automated penetration testing technology uses various algorithm models to automate the penetration of 21 target networks, significantly reducing test cost and improving penetration eficiency [1] [2]. Automated penetration testing is an important application of artificial intelligence technology in the field of network security [3]. Zang et al. [4] summarized the current research progress of attack path discovery in automated penetration testing, and proposed future research directions. Modeling the attack environment through Markov process can well help us describe and calculate the penetration test path. Literature [5] modeled the environment as Markov decision process diagram based on the attack graph, and used the value iterative algorithm to find the optimal penetration path. This approach can help penetration testers find the most efective attack path, thereby improving the eficiency and success rate of penetration testing. Literature [6] proposes an automatic attack planning algorithm (NIG-AP) based on network information gain, which formalizes penetration testing into a Markov decision process, finds potential attack paths from the attacker’s perspective, and uses network information to optimize attack strategies. Zhou et al. [7] proposed an attack path planning algorithm based on network information gain, which formalized penetration testing into a Markov decision process, used network information to obtain rewards, and guided agents to choose the best response actions. The method based on reinforcement learning, which can simulate the uncertainty of ofense and defense in the real world by designing the probability of success of action execution, is an important research direction in this field. Schwartz et al. [8] designed a lightweight network attack simulator NASim, which provides a benchmark platform for network attack defense simulation test. They verify the efectiveness of basic reinforcement learning algorithms, such as Deep Q-learning Network(DQN), in the application of penetration path discovery. In order to improve the convergence speed of DQN algorithm in path discovery problems, Zhou Shicheng et al. [9] proposed an improved reinforcement learning algorithm, noise-double-dueling DQN. Hoang Viet Ngueyen et al. [10] introduced an A2C algorithm with dual agent architecture, which is responsible for path discovery and host utilization respectively. Zeng Qingwei et al. [11] suggested using hierarchical reinforcement learning algorithm to solve the problem of path discovery and host utilization being handled separately. In summary, current research on the discovery of intelligent penetration test paths is still in the preliminary stage. The advantage of MDP model is that it can model the uncertainty in the process of penetration testing well, but it brings the increase of computational complexity, which is dificult to apply to large-scale network scenarios. The method based on reinforcement learning is only experimentally verified in simple network scenarios, and there is still much room for improvement in the convergence speed and scalability of the algorithm.Based on the above research, we further study penetration testing in complex scale scenarios. In this paper, the infiltration test attack path is modeled from the perspective of multi-step attack, and it is combined with MDP. On this basis, an intelligent decision method for complex computing network attacks is proposed.
3.1.
• represents the set of penetration states observed
by the agent, such as network topology, host
information, and vulnerability details. These states
correspond to various stages in the penetration testing
process, which typically proceeds in the following
sequence:
– a) Information Gathering: The initial
phase, where information regarding the
network, systems, and potential vulnerabilities
is collected.
– b) Vulnerability Assessment: The process
of identifying and evaluating weaknesses or
vulnerabilities within the network and
systems.
– c) Exploitation: The stage where discovered
vulnerabilities are exploited to gain
unauthorized access to systems.
– : d) Privilege EscalationThe process of
eleDamage to the network as (
(′|, ), which defines the probability of transitioning from one state to another after performing a given action. This is typically associated with the success rate of the attack actions.
The objective of the MDP is to select the optimal policy
= () that maximizes the long-term cumulative
reward (0) for the current state, as expressed in equation
(
Designing a quantitative attack cost model can
comprehensively consider the investment and return of attack behavior
from multiple dimensions, so as to provide a more objective
basis for network security evaluation. For the MDP process
discussed above, the penetration test adopts multiple attack
modes to penetrate the network. Assuming that the threat
of a certain type of attack is , the launching cost of such
an attack is , the attack cost of node i is , and the attack
value is , then the attack cost of a certain node is
attacked. Suppose that on the optimal attack path, the state of
node relative to the attacker is S,After the attack fails, the
network will detect the attack, thus increasing the attack
cost.LC(limit cost) a given cost, it is assumed that the
resources held by the attacker can consume LC at most, which
can help the intelligent agent accelerate the convergence
speed during the network search attack.
it satisfies inequality as (
<
<=
(
<
So the attack reward function (
Reinforcement learning algorithms can be categorized into three main types: value-based, policy-based, and actorcritic-based. SAC is an of-policy reinforcement learning algorithm that combines maximum entropy learning with the actor-critic framework. It learns a policy network to select optimal penetration actions while estimating state and action values, thereby maximizing the policy’s entropy to encourage exploration and diversification in attack path selection. Currently, SAC is an eficient model-free reinforcement learning algorithm capable of learning stochastic policies, achieving state-of-the-art results in many standard environments. Figure 2 shows the architecture of the improved Attack Soft Actor-Critic(labelled as ASAC) algorithm. The ASAC algorithm consists of a policy network and four value function networks. The policy network acts as the actor, outputting attack actions toward the environment, while the four value networks evaluate the policy network. Since it is a model-free algorithm, SAC uses an experience replay bufer to store all actions and environmental feedback data, and updates the networks through random sampling. Given that actions in reinforcement learning are often highly correlated, this approach allows the neural network to achieve more efective training.Diferent from other RL algorithms, in order to encourage exploration, the concept of entropy is added in SAC algorithm, and entropy regularization increases the exploration degree of reinforcement learning algorithm. The greater is, the stronger the exploration is, which helps to accelerate the subsequent strategy learning and reduce the possibility of the strategy falling into poor local optimal.
It can be seen that the input of the policy network Actor is the network status , the output is (|) action policy, and the update loss expression of its neural network is : MSELoss = − 1
∑︁ |ℬ| (,,+1,+1)∈ℬ ︀[ 0 (︀ , ′)︀ − ln (︀ ′|; )︀] ′∼ (·| ; ) E′∼ (·| ; ) 0
︁( , ′; (0))︁ − ln (︀ ′|; )︀ ]︁ [︁ =
∑︁ ,ln (′|; ) in these terms perfectly match the loss cal
The output of the V critic network is used as a predicted value, and finally, MSE loss is applied as the loss function, training the V neural network. ︁( , ′; ())︁
As we mentioned earlier,entropy represents a measure of the degree of randomness of a random variable. Specifically, if is a random variable and its probability density function is , then its entropy () is defined as () = E∼ [− log ()].
In a large-scale network environment, direct use of a single experience playback pool may cause the experiences of diferent subnets or nodes to interfere with each other, thus reducing training eficiency. To solve this problem, the experience playback pool can be divided into multiple regional playback pools (each region corresponds to a subnet or node) to store and utilize the experience more targeted.
Divide the experience pool: The experience pool is divided into multiple subpools based on the network topology. Each subpool is dedicated to storing experiences from a particular subnet or node. This division ensures that the experience within each region is more homogeneous and avoids the mixing of diferent regional experiences.
Intra-zone sampling: At each policy update, SAC agents can sample experiences from the playback pool associated with the current zone of operation. For example, if the agent is currently in subnet A, it will preferentially sample from subnet A’s experience pool. This targeted sampling method can improve the learning eficiency of the model, because the experience of sampling is more relevant and can optimize the strategy within the region faster.
Cross-region sampling: In a partial update step, a pool of experience from diferent regions can be sampled to enhance the model’s adaptability to the global network. This approach balances the relationship between regional focus and global exploration, so that the model can not only focus on the strategy optimization of a specific region, but also understand the dynamics of other regions and enhance the generalization ability.
Mitigate non-stationarity issues: In complex network environments, the dynamics of subnets may not be synchronized. For example, some subnets may change frequently, while others are relatively stable. By storing the experience pool in diferent regions, the negative impact of nonstationarity on the model can be prevented and the stability of training can be efectively maintained.
and ln (′|; ) — are fully aligned with the Loss
function depicted in the graph. It’s important to note that
0 , ′; (0))︁ can be replaced by 1
︁(
︁(
, ′; (
The symbol ℬ represents the experience bufer, meaning that when calculating the Loss, you need to take the average of the samples drawn from the bufer. This ensures that the expected average meaningfully represents the sample’s overall outcome.
Here, is the entropy coeficient, which controls the significance increases as increases. importance of the entropy term ln (+1|; ), and its
Based on the optimal Bellman equation, we use () = + (+1) as the true value estimate for the state , while the (, ) value (where = 0, 1) is used as the predicted value estimate for state with the actual action . Finally, the MSE loss is used as the loss function to train the neural networks 0 and 1.
Note that using MSE loss implies taking the average of the data sampled from the experience bufer (denoted as across a batch, as follows: ℬ ) Loss = 1 |ℬ| (,,,+1)∈ℬ [︁ (︁ , ; ())︁ − ()]︁2
Using the following equation with entropy for state value estimation, the V critic network outputs the true value: ∑︁
︂[ () = E′∼ (·| ; ) =0,1 min ︁( , ′; ())︁ − ln (︀ ′|; )︀ ︂] impact on policy improvement. Prioritized experience replay selects experiences that contribute more significantly In reinforcement learning, not all experiences have the same to policy improvement, helping the SAC algorithm make eficient use of critical experiences and accelerate learning progress.
1. Priority Calculation: Each experience is assigned a priority, typically calculated based on the Temporal Diference (TD) error: ⃒ ⃒
′ = ⃒⃒ + max (′, ′) − (, )⃒⃒ The greater the TD error, the higher the potential for policy improvement, and therefore, the higher the priority. 2. Experience Sampling: During sampling, experiences with higher priority are more likely to be chosen. This can be achieved by probability sampling, where the sampling probability () is set proportional to the priority level . For example, the sampling probability () can be defined as: ⃒ ⃒ () = ∑︀
where is the priority of experience , and controls the level of prioritization. When = 1, sampling is fully prioritized by priority; when = 0, sampling is uniform. 3. Importance Sampling Weight Correction: To correct for sampling bias, higher-priority experiences are given lower weights in the gradient calculation. The importance sampling weight () can be used to balance this bias: () = ︂(
1 · ()
︂) where is the size of the experience bufer, and is a parameter that adjusts the degree of importancesampling correction. When
= 1, the correction is fully applied. Typically, starts from 0 and gradually increases to avoid instability due to high weights early in training.
The flow of the improved ASAC(Attacker Soft Actor Critic) algorithm is as Algorithm 1.
Line 1-4: Initialize the Critic, Actor, and target network parameters, along with regional experience replay bufers and temperature parameter. Line 5: Start the loop for each episode, sampling the initial state and determining the region. Line 7-10: At each time step, sample an action, execute it, observe the outcome, and store the experience in the respective regional replay bufer. Line 11-19: Perform training steps by sampling a mini-batch with prioritized experience replay, calculating TD errors and priorities, updating Critic and Actor networks, and adjusting temperature. Line 20: Update the target networks with a soft update mechanism.
To simulate a realistic observation-based penetration
process, we assume that both topology and host information
must be obtained through scanning or feedback from attack
actions. Therefore, the Information Gathering step is
carried out through 4 steps: (
Initialize
regional ℛ1, ℛ2, . . . , ℛ Initialize temperature parameter for each episode do Initialize Critic network parameters 1, 2 and Actor network parameters Initialize target network parameters 1− , 2− experience replay bufers Sample initial state 1, determine the current region for each time step = 1 → do
Sample action ∼ (·| ) from the policy network +1 Execute action , observe reward and next state Store (, , , +1) in the corresponding regional replay bufer for training steps = 1 → do
ℛ Sample a sampling from all regional replay bufers using prioritized
mini-batch {(, , , +1)}=1 Calculate the TD error = + min=1,2 − (+1, +1) − (, ) Update priority for each sample based on and adjust sampling probabilities Calculate target value
min=1,2 − (+1, +1) Update Critic networks: for = 1, 2, minimize the loss function Update Actor network using reparameterization trick to sample action ˜ Update temperature parameter to minimize entropy objective = + end for Update target network parameters:
− ← + (1 − )− , = 1, 2
end for
end for
discover all hosts in the subnet; (
The network topology diagram of the simulation
experiment in this paper is shown in Figure 3: A total of 6 subnets,
each subnet is set between the firewall and access protocol,
the yellow host is the sensitive host in the network, the
attacker has mastered the two available nodes (
To validate the performance of our model and decision method, we focus our experiments on answering the following three Research Questions (RQs):
In this part, we first perform detailed parameter tuning for the ASAC algorithm, and a total of four groups of parameters are set, with detailed data shown in Table 3. The goal of parameter tuning is to find an optimal set of hyperparameters to ensure the stability and efectiveness of the algorithm in the network environment. We used a combination of grid search and random search to explore diferent combinations of hyperparameters, including learning rate, discount factor, batch size, and so on. Through many experiments, we identify a set of optimal hyperparameters and verify them in the network environment.
Figure 4 shows the performance comparison of ASAC algorithm under diferent parameter Settings. The experimental results of each group are smoothed. The experimental results show that the proxy training obtained by group4 group parameters cannot achieve balance, and basically each training round cannot reach the optimal condition. But it doesn’t converge yet. After multiple tuning, the optimal parameter group group1 is found. After parameter tuning, the ASAC algorithm shows good performance in the network environment and can converge in about 300 rounds. It can be seen that the optimized algorithm has a significant improvement in convergence speed and final performance.
In order to verify the performance advantages of the improved ASAC algorithm in the network environment, we compare the ASAC algorithm with several classical network optimization algorithms, including Q-learning and DQN (Deep Q-Network), which have been studied in intelligent penetration testing.
Figure 4 shows the change of the cumulative reward value of each round of diferent algorithms with the number of training rounds in the network environment. The learning goal of the agent is to learn to use fewer steps to obtain the permissions of all sensitive hosts in the target network, so as to obtain the reward value, so the reward value can be used to measure the level of the agent’s strategy. It can be found from the Figure 5 that in the initial stage of training, the reward value that the agent can obtain in each turn is small, but with the progress of training, the reward value keeps increasing, indicating that the agent gradually learns the strategy of obtaining the maximum reward value. In the early exploration process, the cumulative reward value of ASAC algorithm per round is significantly lower than that of DQN and Q-learning algorithm. This is because in the early exploration process of ASAC algorithm, the agent has no experience to rely on and can only use random strategies for exploration. It also wastes a lot of attack steps, but with the progress of training Q-learning can not get better results due to the explosion of Q table, and the convergence speed and efect of DQN are not as good as that of ASAC algorithm. Finally, both ASAC and DQN algorithms converge to a stable cumulative reward value within 600 rounds, and ASAC algorithm has the best performance and can converge to the optimal value within 300 rounds.
The experimental results show that the improved ASAC algorithm has obvious advantages in network environment. Including convergence speed, decision quality and resource utilization. ASAC algorithm outperforms other algorithms in all indexes, especially in the environment dealing with large-scale networks and high dynamic changes.
In order to verify the scalability of ASAC algorithm, we conducted experiments in diferent scale network environments. The experiment was carried out in a network simulation environment to simulate network topologies of diferent sizes 1e-3 1e-3
As can be seen from Figure 6, in the three heterogeneous networks A, B and C with the same sensitive host and slightly diferent summary points and subnets, the ASAC algorithm agent has good robustness, and its average cumulative reward value can converge to the optimal value within 400 rounds in the three similar scenarios.
As the number of subnets and hosts increases, the convergence rate of the algorithm slows down as the number of hosts and actions that the agent needs to attempt in each turn increases rapidly. Figure 7 shows the performance of ASAC algorithm under diferent network scales. It can be seen that the performance of ASAC algorithm decreases slightly with the increase of network scale, but it can still efectively deal with complex problems in large-scale network environment. In the simulated network environment of 40 hosts,ASAC algorithm can also converge to the optimal solution. The experimental results show that the ASAC algorithm maintains good performance when the network size increases gradually. In addition, ASAC algorithm also performs well in resource utilization, and can maintain high resource utilization under diferent network sizes. When dealing with complex network topology and high load, ASAC algorithm can make decisions efectively to ensure the stability and eficiency of the network.
Through the experimental analysis, we can draw the following conclusions:
These experimental results validate the efectiveness and superiority of ASAC algorithm, provide an efective tool for network intelligent penetration attack agents, and also provide evaluation support for the security subfeature "resistance" defined in ISO/IEC 25000 SQuaRE product quality model. It provides strong support for the future application in the field of network optimization.
Quantitative analysis reveals that the ASAC algorithm not only outperforms traditional methods like Q-learning and DQN but also demonstrates robust performance under varying network conditions, contributing to a more reliable and secure network optimization solution. The ability to adapt and maintain high security performance across different network scales and configurations provides strong support for future applications in network security and optimization.
However, the algorithm has certain limitations. One notable drawback is the lack of comparison between the ASAC algorithm and other currently optimized attack agent algorithms, which would provide a clearer benchmark for its relative performance. Despite its promising performance and stability, the algorithm may still face convergence speed issues in highly dynamic environments. Moreover, the parameter tuning process remains relatively complex, potentially increasing the dificulty of practical implementation. Future work should focus on developing more eficient adaptive parameter adjustment mechanisms to improve the ASAC algorithm’s adaptability and robustness in even more complex and unpredictable environments.