It is important to clearly distinguish the combinations of security testing and security risk analysis depending on whether it is viewed from a security testing perspective or a security risk analysis perspective. The main focus in the former view is security testing in which test objectives are to be achieved, while the main focus in the latter view is security risk analysis with the aim to fulfill risk acceptance criteria. The literature's lack of addressing this distinction is accompanied with the lack of addressing two immediate problems within this context, namely the gap between high-level security risk analysis models and low-level security test cases, and the consideration of investable effort. We present initial ideas for methods that address these problems followed by an industrial case study evaluation in which we have gathered interesting results.
Security testing is a process to determine that an information system protects
data and maintains functionality as intended [
The literature collectively refers to the combinations of security testing and security risk analysis as risk-based security testing. It is, however, important to make a clear distinction between such combinations depending on whether it is viewed from (A) a security testing perspective, or (B) a security risk analysis perspective. In (A) the main focus is security testing in which test objectives are to be achieved while risk analysis is used as a means to make security testing ⋆ This work has been conducted as a part of the DIAMONDS (201579/S10) project funded by the Research Council of Norway, as well as a part of the NESSoS network of excellence funded by the European Commission within the 7th Framework Programme. more effective. We name this approach Risk-driven Security Testing (RST). In (B) the main focus is security risk analysis with the aim to fulfill risk acceptance criteria while security testing is used as a means to develop and/or validate risk models. We name this approach Test-driven Security Risk Analysis (TSR). Furthermore, we address two key challenges within both RST and TSR. The first challenge is to bridge the gap between high-level security risk models and low-level security test cases, while the second challenge is to make sure that the investable effort is correctly reflected in RST and TSR. As an initial evaluation of our proposed methods for RST and TSR, we have carried out an industrial case study evaluation of a TSR based approach in which we have gathered interesting results.
The paper is structured as follows: Sect. 2 outlines the research objectives followed by a brief description of the research method, Sect. 3 gives an overview of the work done to date, Sect. 4 presents preliminary results from an industrial case study in which a TSR approach was tried out, and Sect. 5 concludes the paper. 2
Security risk analysis models are often at a high-level of abstraction (e.g. business level), while security test cases are at a low-level of abstraction (e.g. implementation level), and the challenge in RST is to identify the most important security test cases, while the challenge in TSR is to identify an accurate security risk model of the target of evaluation.
These challenges are important to address in order to define exactly what to
test, produce only the necessary security test cases and to obtain an accurate
security risk model. The two first points provide the security testers with an
indication for when to terminate the security testing process, i.e. to limit the
scope of the security testing process. The necessity to limit the scope of the
security testing process is due to the fact that security is often constrained
by cost and time – one example from the industry is the author’s personal
experience of conducting security testing in a European organization [
The author seeks to address the particular task of developing an industrial
guideline for effort dependent risk-driven security testing and test-driven security
risk analysis. The initial research question is the following:
{ What is a good industrial guideline for effort dependent risk-driven security
testing and test-driven security risk analysis?
The research work will adopt an iterative incremental approach where the
industrial partners provide industrial case studies on RST and TSR. There will,
in total, be carried out six case studies. In this light, the research process will
be conducted using the Technology Research Method [
Risk-driven Security Testing (RST) Test-driven Security Risk Analysis (TSR) Security Testing Security Risk Analysis
Security Testing Security Risk Assessment Security Risk Assessment Security Testing Fig. 1. The steps in Risk-driven Security Testing and Test-driven Security Risk Analysis. 4
We have carried out an industrial case study using a TSR based approach. In particular, we carried out the alternative for validating the risk model, i.e. step 1, 4, 5, 6 and 7 of the TSR approach, as explained in Sect. 3. The target system analyzed was a multilingual, web-based e-business application, which serves as the backbone for the system owner’s business goals and is used by a large amount of users every day. The case study was carried out in a period of four months in form of meetings and security testing sessions. The presented results are limited to the experiences obtained in the case study due to confidentiality reasons.
Table 1 outlines the process undergone during the case study. The first column specifies the meeting sequence (SRA denotes a security risk analysis meeting, while ST denotes a security testing meeting). The second column lists the participants (C denotes participants from the customer organization, while A denotes participants from the analysis team). The third column describes the contents and achievements in each meeting. Finally, the fourth column shows the approximate time spent (in man-hours) for each meeting. The time spent on work before and after the meetings is not included in the table.
Security risk analysis was conducted using the CORAS approach [
Table 2 presents the overall TSR results obtained in the case study. The leftmost column specifies the information security assets taken into consideration during the security risk analysis process. The row named Common denotes the number of security risk analysis elements that are common for all assets. The topmost row specifies the number of security risk analysis elements: R denotes the number of risks, TS denotes the number of threat scenarios, RT denotes the number of tested risks, TST denotes the number of tested threat scenarios (the tested threat scenarios that addressed confidentiality also addressed integrity and are therefore not counted two times in the total sum), R Upd. denotes the number of updated risks based on the security testing results and TS Upd. denotes the number of updated threat scenarios based on the security testing results.
A total number of 31 risks and 43 threat scenarios were identified during the risk assessment, and from these, 11 risks and 7 threat scenarios were tested. Approximately 80% of the identified security tests that explored these risks and threat scenarios uncovered some form of security vulnerability. This is an indication that the risk model contributed to identify relevant security tests. Furthermore, the security testing results helped us to validate and update the risk model; approximately 20% of all risks and 14% of all threat scenarios had
R TS RT TST R Upd. TS Upd. `````SR`A`e`le`me`n`ts
Assets Con dentiality of information 8 2 5
Integrity of information 8 7 5
Availability of information 11 13 1 Accountability of information 4 6 0
Common 0 15 0
Total 31 43 11 to be adjusted, with respect to likelihood values, based on the security testing results. 5
The combinations of security risk analysis and security testing must be clearly distinguished depending on whether it is viewed from a security testing perspective (RST), or a security risk analysis perspective (TSR). Additionally, the immediate challenges that need to be addressed in these approaches are: (1) the gap between high-level security risk analysis results and low-level security test cases, and (2) the correct reflection of investable effort. An industrial case study evaluation of a TSR based approach showed how security testing can be used as a significant means to validate and update the risk model – i.e. approximately 20% of all risks and 14% of all threat scenarios had to be adjusted, with respect to likelihood values, based on the security testing results.