The increasing need for security assurance in 5G mobile networks is a priority at both national and European levels. However, implementing SCAS (Security Assurance Specifications) tests in the 5G Core (5GC) presents significant challenges, particularly when it comes to verifying the temporal sequence of events. Based on our own experience, we outline the key requirements for accurately implementing SCAS tests in such complex environments. We propose a general approach for executing these tests, emphasizing the critical issue of correctly verifying the sequence of events amidst concurrent and interdependent operations. Our approach is validated by testing three open-source 5GC implementations, highlighting the limitations of subjective interpretation of results.
The rapid growth of mobile applications and services across various sectors has driven the development
of advanced communication systems capable of handling vast amounts of data generated by millions of
devices. This evolution necessitates robust regulations, best practices, and methodologies to ensure
both high network performance and strong security. At the national level, Italy’s National
Cybersecurity Agency (ACN) is defining procedures for 5G security evaluations, focusing on two key aspects:
verifying the implementation of network security features and conducting vulnerability assessments
and penetration tests. The National Assessment and Certification Centre (CVCN), part of ACN, is tasked
with creating a methodology for assessing 5G components critical to national security, as identified
by the DPCM of 15 June 2021 [
However, in practical terms, conducting SCAS-defined security tests by third-party specialists presents
significant challenges. These tests often demand interaction with network functions that go well beyond
simple “black-box” testing. Addressing these challenges, our research focuses on bridging the gap
between SCAS specifications and their efective implementation in testing. We provide insights and
define key requirements to enable testers to deploy tests that accurately reflect SCAS specifications.
Our analysis identifies critical aspects of the implementation process, including handling the temporal
sequencing of events – a non-trivial aspect of such tests. To validate our approach, we implemented a
series of representative tests using the ScasDK toolkit [
The paper is structured as follows. Sec. 2 provides the background on the European regulatory framework, 5G network architecture, and related work. Sec. 3 delves into the 3GPP SCAS methodology, emphasizing the challenges of implementing the specified tests and identifying key requirements. Sec. 4 details the test implementation process, while Sec. 5 presents the proof-of-concept evaluation and the results of the implemented tests. Finally, Sec. 6 concludes the paper by summarizing the key findings and contributions of this work.
The security of 5G networks is a priority for the European Commission and central to its Security Union
Strategy, given their role as critical infrastructure for essential services and societal functions. To address
5G cybersecurity risks, the Member States’ authorities participating in the NIS Cooperation Group,
supported by the European Commission and ENISA, published the “EU Toolbox of Risk Mitigating
Measures” [
The 3GPP technical specification (TS) 23.501 [
In this paper, we will focus on the 5GC NF which terminates the N1 and N2 reference points, that is,
the AMF. As the entry point of the entire service-based architecture (SBA) of the 5GC control plane, the
AMF plays a crucial role in the implementation of various functionalities such as, e.g., the management
of the UE registration to the network (cf. Clause 5.3.2, [
The security assurance of 5G networks necessitates a comprehensive approach that encompasses
various aspects of security testing, as emphasized by ENISA [
Let us consider the perspective of a tester tasked with implementing a set of SCAS tests to certify the implementation of an AMF. As discussed in Sec. 2.2, the AMF is among the most complex NFs in the 5GC1. The process begins with the tester reviewing the relevant Technical Specifications. These documents define a suite of tests designed to verify that the 5G-specific security requirements are correctly implemented in the NF under test. For each security feature, the specification includes one or more tests that evaluate both the correctness of its basic functionality and its resilience in handling corner cases. Each test is structured into three main sections.
• Pre-conditions, i.e., the necessary conditions that must be met before executing the test. • Execution Steps, i.e., a sequence of actions the tester must perform to carry out the test. • Expected Results, i.e., the outcomes that the tester needs to observe and document to validate the test.
The tester begins by examining the Pre-conditions section, which outlines the configuration requirements for the testing network and the components supporting the NF under test. These often include other NFs, base stations (gNBs), and UEs, whether real or emulated. At first glance, the tester might assume that deploying standard network components would sufice to support test execution. However, the complexity of the 5GC architecture becomes quickly apparent. The distributed SBA in the control plane introduces the need for precise configuration and coordination of multiple network elements, even in a test environment.
The tester reviews the Execution Steps, which describe actions and events in natural language. These often require supporting components to enforce anomalous behaviors, such as altering messages or managing precise timings. The complexity grows due to multiple communication protocols (e.g., NAS, NGAP, HTTP) and requirements like message delays or timeouts. This calls for programmable, adaptable components capable of handling heterogeneous protocols and synchronizing actions. The tester delves into the test steps, recognizing the need for specialized expertise to implement them efectively. In some test scenarios, a single operation may translate into a series of intricate sub-steps. For example, in the TC_Sync_Fail_Seaf test, the tester must handle sequence number resynchronization, ensuring fresh messages are used to prevent replay attacks. Constructing a synchronization failure message involves extracting data from the authentication procedure, which is not explicitly stated in the test description. In other cases, the necessary actions are implicit, requiring the tester to deduce them. For example, in case B of TC_Res_Star_Verification_Failure test, the tester must ensure the UE completes a registration and deregistration procedure to obtain a valid 5G-GUTI before initiating the test. These preparatory actions, essential for the test, are not explicitly detailed in the test specification, increasing the implementation efort for the tester.
The last operational section of a test case consists of the Expected Results. This section specifies the results the tester must provide to prove the success of the test. It typically defines three possible types of results:
• a screenshot of the execution; 1While this example focuses on the AMF, the challenges and motivations are similarly relevant to SCAS tests for other NFs. • the log files of the component under test; • capture of the test network trafic in a standard format (i.e., a pcap file).
The first type of format, while visually intuitive, lacks standardization, making it challenging to use as definitive proof of a test’s success. Additionally, it is unsuitable for automated result evaluation systems. Log files, on the other hand, ofer a format that is potentially machine-readable. However, their structure varies significantly depending on the implementation. Some logs may be structured (e.g., JSON), others unstructured, and in certain cases, logs might not be stored as files but instead managed through centralized logging systems. Even when accessible, logs may not contain all the necessary information to validate the success of a test. Furthermore, testers may not always have permission to retrieve the required data from the logs. Network trafic generated during the test could serve as a more reliable proof of success, as it captures the interactions directly. However, many communications in the 5GC are encrypted (e.g., HTTP trafic in the SBA is often secured with TLS). In such cases, testers must decrypt the trafic before saving it in a pcap file for analysis.
Based on the lessons learned from implementing SCAS tests, we identified key features commonly required by most test cases, as far as our analysis indicates. These requirements are grouped into categories, each one addressing a specific aspect of test implementation. The main requirements are as follows.
• Actors Involved, i.e., the number of active participants required during the test. • Action Types, i.e., the types of actions the tester must perform during the test. • Verifications, i.e., the verification processes needed to validate test outcomes and how they should be conducted.
A test may require a single actor that actively interacts with the NF under test. This case also include scenarios in which the component under test interacts with other NFs only for supporting its operations (e.g., while testing UDM, it requires to access to UDR to retrieve subscribers information but these component does not actively participate to the test). If a test requires multiple actors (e.g., an UE and an AUSF for testing the authentication procedure in the AMF), the tester has to keep in mind that has to implement a dedicated mechanism to coordinate such components. The actions performed during a test can be categorized as functional or non-functional. Functional actions involve operations on the content of communication, such as modifying, replaying, or dropping messages. Non-functional actions, on the other hand, address the quality of communication, including introducing delays or triggering timeouts in the NF under test. Many tests require a combination of both functional and non-functional actions. Verification requirements involve two main aspects: determining whether specific events occur (positive verification) or do not occur (negative verification), and assessing the relationships between events, especially their sequence. While some tests only require event occurrence validation, others necessitate confirming the correct temporal order of events, particularly for those originating from diferent parts of the network, where proper sequencing cannot be assumed. Tab. 1 summarizes these requirements.
In this section, we present our approach to implementing SCAS tests and address how we overcame the challenges associated with meeting the aforementioned requirements. Specifically, we focus on ensuring the correctness of temporal event sequence verification. While this task is straightforward when the tester controls a single component, we will demonstrate in the following sections that in distributed scenarios, such as ours, this assurance becomes significantly more complex. To provide a clear framework for creating support tools to implement these tests, we have abstracted our approaches from the specifics of the actual implementation.
A SCAS test can be broadly represented as a sequence of the following phases.
• Configuration – The tester sets up the network or configures the support components required for the test. • Action – The tester initiates an action within the test network, such as starting a procedure or triggering an anomalous behavior. • Verification – The tester verifies that specific events (or a subset of them) have occurred correctly, ensuring compliance with the test requirements.
These phases are not intended to follow a rigid, sequential waterfall model. Instead, they are designed to be executed iteratively, ofering the flexibility to adapt based on the specific demands of each test. For instance, a tester may begin by configuring a test component, then trigger an action, verify that an expected event occurs, perform another action, and finally validate that all required events have been correctly executed. This iterative approach allows the phases to be mixed, repeated, or adjusted as necessary to ensure comprehensive test coverage and accuracy.
All of the SCAS tests discussed in this paper were implemented and applied to virtualized 5GC networks
using an enhanced version of the ScasDK framework originally introduced in [
ScasDK architecture is inspired by the conceptual architecture widely adopted in software defined networking, and entailing the separation between a distributed data plane and a centralized control plane. In details: • the Data Plane Layer consists of the test actuators, encompassing all the entities actively participating in the test. In our context, these include UEs, gNBs, and NFs; • the Control Plane Layer implements the test logic, managing interactions with the data plane for configuring components, triggering test behaviors, capturing events, and analyzing test outcomes.
The control plane layer in our approach incorporates features designed to meet the outlined SCAS test requirements. For the implementation of the test, i.e., their software programming, ScasDK exposes an interface based on the Robot Framework, a widely-used open-source test automation tool suitable for industrial applications. Robot Framework ofers testers a user-centric environment, enabling them to write tests in a comprehensible manner, generate detailed logs, and identify root causes of failed tests. We enhanced the framework with specialized libraries for 5G SCAS testing, categorized as follows: • Configuration Libraries , i.e., tools for setting up test components (e.g., UEs, gNBs, NFs); • Action Libraries, i.e., tools for initiating actions in the network under test; • Event Libraries, i.e., tools for defining, capturing, and analyzing test events.
Further descriptions of ScasDK and the test controller can be found in Annex A and B.
One of the critical requirements in SCAS testing is verifying that specific events occur in a defined temporal sequence. Often, this requirement is inherently satisfied by standard-defined procedures. For instance, the 5G-AKA protocol employs a request-response model that inherently linearizes the sequence of events. In such cases, testers primarily validate event occurrence rather than explicitly confirming temporal order, as the protocol itself guarantees correct sequencing. However, deeper verification becomes crucial in scenarios where distributed NFs generate events concurrently. For example, in case B of TC_Sync_Fail_Seaf test, the test verifies whether the AMF correctly handles resynchronization under abnormal conditions. In this scenario, the AUSF intentionally delays its response to the AMF to induce a timeout. The AMF must wait for the timeout to expire before restarting a new authentication procedure. Here, testers must validate the event sequence by analyzing timestamps to ensure the timeout precedes the subsequent authentication request. A tester can apply the following two approaches to timestamp attribution.
• Component-Generated Timestamps – Each component in the network assigns its own timestamps. However, this approach becomes infeasible when testers lack control over the components, such as the AMF under test. Even if logs are accessible, clock drift between hosts can create inaccuracies, potentially misrepresenting the actual order of events. Additionally, there may be rare cases where the timestamps of two events are identical, complicating sequence verification. • Tester-Generated Timestamps – The tester assigns timestamps upon event detection. This approach eliminates discrepancies caused by clock drift since a single, unified clock source is used. However, network delays can distort the tester’s perspective, leading to potential misinterpretation of the event sequence. For instance, an event transmitted before a timeout might appear to occur after it due to delays in communication.
To address these challenges, we enhanced ScasDK to include a mechanism capable of verifying the temporal sequence of events using Tester-Generated Timestamps within the proxy-based architecture. This architecture intercepts and relays trafic to the test controller while temporarily holding the flow until event processing is complete, ensuring that event sequences are linearized under controlled conditions. In cases involving parallel events where network delays could impact sequence accuracy, we implement synchronization techniques combined with protocol-level logic to deduce relationships between events.
In scenarios involving concurrent events where network delays might distort the perceived event order, we utilize synchronization techniques augmented with protocol-level insights to interpret event relationships. For instance, in the case B of TC_Sync_Fail_Seaf test, delays could cause the tester to observe events in reverse order, making it seem as though the AMF waited for the timeout before sending a new authentication request. However, the AMF should include a fresh challenge in the new request, which can only be obtained from the AUSF after the timeout or upon receiving the new challenge. If the AMF sends the new authentication request prematurely without waiting for the AUSF, it will reuse the same challenge. This behavior violates protocol requirements and indicates that the new request was sent before the timeout expired. While our method improves the reliability of event sequence verification, scenarios with highly concurrent events still present challenges. Figure 2 illustrates how our approach addresses the limitations of basic methods, enhancing accuracy in event order validation.
We validated our approach by implementing a subset of SCAS tests for the AMF component. Table 2 demonstrates how each requirement is addressed by at least one test, highlighting the efectiveness of our analysis in identifying meaningful requirements. We applied these tests to three open-source 5GC implementations: free5GC v3.3.0, Open5GS v2.7.0, and OpenAirInterface (OAI) v2.1.0. To ensure the reliability of our results, we compared test outcomes with the actual communication flows captured during execution. For our case study, we focused on the case B of TC_Sync_Fail_Seaf test, as it incorporates the most requirements. For each implementation, we executed the test and captured network trafic at the AMF to serve as a ground truth. The following analysis underscores the challenges of interpreting results, especially in light of ambiguities introduced by test specification limitations. Additional details, including test trafic and implementation notes, are provided in the annex C.
We present the analysis of the outcomes, while in the annex C.1 we present the detailed info of the trafic extracted. Our findings validate our approach and implementation, as all test outcomes align with the observed trafic flows. However, they also expose limitations in the SCAS test specifications, which lack clear guidance for addressing ambiguities and interpreting results in cases of communication flow deviations or unforeseen network behaviors. This underscores the need to enhance test specifications to minimize subjective interpretations and promote consistent evaluation across diferent implementations.
In free5GC implementation the test passed, with communication behavior precisely matching the test specifications. The UE sent a sync_failure message, which the AMF forwarded to the AUSF. The AUSF delayed its response, as specified, triggering the timeout. Upon receiving the delayed response, the AMF immediately initiated a new authentication request to the UE. This behavior satisfied all test requirements and validated the approach.
Also in Open5GS implementation the test passed, but with deviations in the communication flow. The AMF sent a Registration Reject message before the timeout, citing an internal error. After a delay, the UE restarted the authentication procedure and completed it successfully. Despite these deviations, the AMF initiated a new authentication request after the timeout, as required by the test specification. However, the unexpected Registration Reject message introduces ambiguity, which is not clarified by the test specification. This raises the question: Does the test remain valid if the communication flow deviates from the expected sequence?
TC_SYNC_FAIL_SEAF_AMF – Case A TC_SYNC_FAIL_SEAF_AMF – Case B TC_NAS_NULL_INT_AMF - Case A
TC_NAS_NULL_INT_AMF - Case B
TC_NAS_INT_SELECTION_USE_AMF TC_UE_SEC_CAP_HANDLING_AMF - Case 1 TC_UE_SEC_CAP_HANDLING_AMF - Case 2 TC_UE_SEC_CAP_HANDLING_AMF - Case 3
TC_UE_SEC_CAP_HANDLING_AMF - Case 4
Actors Actions Verifications
Multiple Functional Nonfunctional Occurrence Temporal – – – – • • • • •
AMF tests • • • • – – – – – • • • • • • • • • – • – – – – – – – – – • • • • • • • • • – – – – – – –
In contrast, in OAI the test failed due to behavior that contradicted the specifications. The AMF sent an Authentication Failure message to the UE, attributing the failure to the UE itself. Consequently, the UE did not initiate a new authentication procedure, leading to disconnection from the network. This prevented the UE from accessing services despite being authorized. These deviations violate test requirements and highlight significant issues in the AMF’s implementation. Furthermore, the ambiguity in interpreting whether the observed behavior satisfies the test objectives stems from insuficient guidance in the test specification.
This paper examined the application of 3GPP SCAS tests to virtualized 5G network architectures, emphasizing test analysis, implementation complexity, and temporal accuracy. Using a centralized clock and protocol knowledge, our framework enhances event sequencing and validation in distributed systems like the 5GC. Validation against free5GC, Open5GS, and OAI revealed significant behavioral variations not addressed by SCAS specifications, highlighting gaps that reduce consistency and requires subjective interpretation. Future work will focus on automating test case generation through machine learning, improving scalability and efectiveness in 5G security assurance.
This work was partially supported by the European Union - Next Generation EU under the Italian National Recovery and Resilience Plan (NRRP), Mission 4, Component 2, Investment 1.3, CUP F83C22001690001, E83C22004640001 and B53C22004050001, partnership on “Telecommunications of the Future” (PE00000001 - program “RESTART”), and CUP F83C22001670001 and B53C22003990006, partnership on “SEcurity and RIghts In the CyberSpace” (PE00000014 - program “SERICS”).
During the preparation of this work, the authors used GPT-3.5 in order to: grammar and spelling check. After using these tools, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.
The ScasDK framework provides a practical solution for testing the security of 5G networks by using proxies to monitor and modify network trafic between diferent components. These proxies act as intermediaries, intercepting communications between network functions without altering their original software. They can be dynamically configured to adapt to specific test scenarios, enabling detailed examination and manipulation of protocols unique to 5G. A centralized controller orchestrates the entire process, setting up proxies to perform tasks such as altering data, analyzing messages, or introducing delays. This flexible approach allows testers to simulate various conditions and behaviors in real-time, ensuring comprehensive security evaluations. When tests reveal issues, the system ofers detailed feedback to help identify and address potential vulnerabilities in the network.
Proxies Proxies in the ScasDK framework play a key role in managing and analyzing network trafic during tests. They intercept messages between 5G network components, forwarding them unchanged or modifying them by using user-defined hooks. This adaptability allows testers to record data, manipulate communications, and introduce delays. For instance, the NGAP/NAS proxy is tailored for interactions between gNB and AMF, while the HTTP proxy employs the open-source Envoy platform to facilitate control-plane communication among network functions. Both proxies can be dynamically configured via an interface, allowing external tools to adjust and analyze messages in real-time. This approach ensures seamless communication between network components, even when they come from diferent vendors or lack support for standard protocols.
Controller The test controller is the central hub of the ScasDK testing framework, responsible for managing and coordinating the entire testing process. It sets up the environment by configuring key components like proxies and emulated devices with necessary parameters (e.g., network addresses or operator IDs). During a test, the controller ensures all elements work together seamlessly, reconfiguring them dynamically as needed. It can also generate network trafic directly or delegate this task to other components, depending on the complexity of the interactions required. By overseeing operations, collecting data, and evaluating results, the controller ensures that each test runs smoothly and provides meaningful insights into network behavior and security.
The Robot Framework is an open-source automation tool designed to simplify test creation and execution. Its keyword-driven approach, paired with extensibility through libraries written in Python or Java, makes it highly adaptable to specific testing needs, including those defined in SCAS tests. As part of our implementation, we developed the following specialized libraries: • Network Components Libraries: These libraries represent supported test components, such as UEs, RANs, and NFs, within the Robot Framework. They provide functionality to configure these components and enable them to perform actions, including sending messages or initiating specific procedures. • Proxy Libraries: Proxy libraries model instances of proxies deployed in the network. They allow testers to configure them and define hooks to be executed during testing. • Hook Libraries: These libraries empower testers to design custom hooks to log, monitor, and modify network trafic. Users can define events triggered by these hooks, with the library automatically logging event occurrences and associating timestamps. Furthermore, the library provides functions to validate the occurrence of events and retrieve their timestamps in order to verify the temporal sequence.
In this section we present the description of two representative test SCAS for AMF we implemented, namely “Tc_Sync_Fail_Seaf - Case B” and “Tc_Ue_Sec_Cap_Handling” . We focus on implementation detail we applied in order to accomplish the test requirements.
The first test implemented with ScasDK, called “Tc_Sync_Fail_Seaf- Case B”, focused on simulating a synchronization failure and testing how the AMF handles specific timing and event sequences. This required addressing several challenges, such as creating a failure scenario, managing a timeout, and ensuring events happened in the correct order. The test involved three main components: the UE (user equipment), AMF, and AUSF. Communication between the UE and AMF relied on the NAS protocol, while the AMF and AUSF interacted using the HTTP protocol. To simulate the failure, a modification was made to the NAS messages, changing the authentication token and triggering a mismatch that caused the desired synchronization failure. To test the timeout, the communication between the AMF and AUSF was delayed for 15 seconds, according to T3520 timer definition, by intercepting and holding HTTP messages. A custom tool was used to manage this delay. Additionally, the ScasDK framework was updated to record timestamps for key events, ensuring that their timing could be accurately measured. This made it possible to confirm whether the AMF waited the correct amount of time before proceeding. Finally, the test checked that the AMF didn’t start a new authentication process before the timeout expired and the delayed response was processed. If the AMF behaved correctly—only continuing the process after the timeout and response—the test was considered successful.
In the following we further analyze the reaction of the three Core Network implementations we tested. free5GC. In the free5GC implementation, the observed network flow aligns with the expected test specifications, as shown in Figure 3. Specifically, the UE sends the synchronization failure message at 25.835698 seconds after the capture begins (packet number 293). Approximately 15 seconds later, we observe the delayed response from the SEAF (packet number 437). Immediately afterward, the AMF sends a new authentication request to the UE (packet number 438), completing the flow as specified in the test case.
Open5GS. In Open5GS, the test also passes, albeit with a diferent network flow, as depicted in Figure 4. Here, the delayed synchronization failure request is sent at 58.509908 seconds after the capture starts (packet number 866). The AMF subsequently responds with an authentication request and a registration failure message approximately 10 seconds later (packet number 996). This behavior is attributed to an internal timeout in the AMF’s HTTP client. However, the registration failure cause, specifically “Payload Not Forwarded”, allows the UE to initiate a new registration request. Following this, the new authentication request is received, thereby satisfying the test requirements.
OAI. In the case of OAI, the test fails. The flow, illustrated in Figure 5, appears similar to that of Open5GS initially, with the AMF responding 1 second after the synchronization failure request due to its internal HTTP client timeout. However, the failure’s root cause difers: the AMF responds with an “Illegal UE” cause, leading to the disconnection of a valid UE from the network. Unlike in Open5GS, the UE does not attempt to reconnect, resulting in a complete failure to meet the test requirements.
The “Tc_Ue_Sec_Cap_Handling” focused on testing how the AMF handles user registration when the security parameters are incomplete or invalid. Specifically, it examined how the AMF reacts to messages missing encryption or integrity algorithms or using unsupported mandatory algorithms. According to security standards, the AMF should reject these registrations with a “Registration Reject” message. This test involved two main components: the UE and the AMF, communicating through the NAS protocol. The challenges included identifying the required algorithms, modifying registration requests, and verifying the AMF’s response.
To cover all scenarios, four separate tests were performed: i) No 5GS encryption algorithms; ii) No 5GS integrity algorithms; iii) Mandatory 5GS encryption algorithms not supported, and iv) Mandatory 5GS integrity algorithms not supported. Proxies were used to intercept and modify the registration request messages sent by the UE. These proxies adjusted the encryption and integrity algorithm fields, either setting them to zero or excluding mandatory algorithms, based on oficial 3GPP guidelines. The AMF’s response was then monitored. If it sent a “Registration Reject” message to the UE in each scenario, the test passed, confirming the AMF correctly handled the invalid security parameters.
The results showed that the free5GC Core Network passed all four scenarios, demonstrating full compliance. In contrast, OAI failed in every scenario, indicating a lack of proper handling for invalid security parameters. Open5GS presented mixed results: it passed the second and fourth scenarios (those involving integrity algorithms) but failed the first and third scenarios related to encryption algorithms.