=Paper=
{{Paper
|id=Vol-2019/modevva_6
|storemode=property
|title=Hierarchical Model Exploration for Exposing Off-nominal Behaviors
|pdfUrl=https://ceur-ws.org/Vol-2019/modevva_6.pdf
|volume=Vol-2019
|authors=Kaushik Madala,Hyunsook Do,Daniel Aceituna
|dblpUrl=https://dblp.org/rec/conf/models/MadalaDA17
}}
==Hierarchical Model Exploration for Exposing Off-nominal Behaviors==
https://ceur-ws.org/Vol-2019/modevva_6.pdf
Hierarchical Model Exploration for Exposing
Off-Nominal Behaviors
Kaushik Madala Hyunsook Do Daniel Aceituna
Computer Science and Engineering Computer Science and Engineering DISTek Integration, Inc.
University of North Texas University of North Texas Daniel.Aceituna@distek.com
kaushik.madala@my.unt.edu hyunsook.do@unt.edu
Abstract—Finding and addressing off-nominal behaviors in implemented the enhanced causal component model (ECCM)
the later phases of the software development life cycle is costly by considering hierarchical model exploration to reduce the
and time consuming. In our previous work, we proposed the state and rule explosion problem [11]. The main intention
causal component model (CCM), a semi-automated approach
that can expose off-nominal behaviors in requirements. The of proposed approach is to reduce scalability issues because
empirical study with CCM showed promising results, but CCM of sub-states. The main idea of our approach is that, as the
has limitations with scalability. To reduce the scalability issues, component state transitions among super-states do not require
in particular because of hierarchical relationships, we propose the details of sub-states, when they are analyzed separately, the
an enhanced causal component model that uses hierarchical number of system-states and system transition rules generated
model exploration. The results of our case study indicate that
the proposed approach reduces the state/rule explosion problem are small, and thus it simplifies the model analysis. To evaluate
and detects off-nominal behaviors. our proposed approach, we performed a case study using the
pacemaker requirements document [12]. Our results indicate
I. I NTRODUCTION that the ECCM can reduce the state and rule explosion problem
Embedded systems provide enormous benefits[1]. However, and detect ONBs.
most embedded systems are susceptible to off-nominal behav- The rest of paper is organized as follows. Section II provides
iors (ONBs) [2], [3] that can lead to system failure. ONBs background information on the CCM, and Section III presents
are unexpected or unintended behaviors [2], [3]. ONBs can the proposed approach. Sections IV and V present a case study
be caused by various reasons such as a user performing an and discuss the results of the study. Sections VI and VII present
operation or triggering an event without following the correct related work, and conclusions respectively.
procedure, two components of system being in contradictory
states of operation, or environmental factors affecting the II. BACKGROUND
system’s operations. For example, in a microwave oven, an
unexpected behavior would be the oven cooking while the door The causal component model (CCM) [7] is generated from
is open. requirements to expose ONBs. It models the system behavior
To date, various approaches for detecting and correcting using three model elements: component, state, and transition
ONBs have been proposed, but many of these approaches condition. A component is an entity that is a part of the
require product implementation [4], [5], [6] and only a few of system’s composition that can change states or trigger a state
them are at requirements level [2], [7]. However, catching and change in other components. In the case of embedded systems,
correcting defects in the later phase of software development components can be both hardware and associated software
is usually more time consuming and costly than catching them functional units. For example, in a robot system, components
in an early phase (e.g., requirements engineering phase) [8], might include the motor, gyroscope and ultrasonic sensor. Each
[9]. component has its own set of possible states, such as a motor
To address this lack, our previous work proposed the causal in a robot having off and on states. These states are component
component model (CCM) [7], which generates a model from the states. Components’ states can be grouped together to form a
rules created from natural language requirements and performs system state. For example, in a robot system, while motor in
a model analysis against undesired behaviors. While empirical its ‘on’ state is referred as component state, the state of robot
studies with the CCM approach showed promising results, when motor is ‘on’, gyroscope is ‘on’ and ultrasonic sensor is
the approach has limitations with scalability. CCM suffers ‘on’ represents the system state of robot . Transition condition
from state explosion and rule explosion problems, and cannot is the event or condition that trigger for a state transition.
handle sub-states without performing state flattening [10]. State These transition conditions can be environmental (i.e, a person
flattening removes hierarchical relationships and results in or environmental factor triggers events), or system related
atomic states, which can affect dependencies in a hierarchy (i.e, component states trigger state transitions). These model
and lead to a large number of state transitions in CCM. To elements are identified in natural language requirements and
handle these limitations, we improved the CCM approach and transition rules are written. The transition rules are in the form
of mapping function Transition Condition: Component(current graph-based technique for analyzing off-nominal behaviors
state) → Component(next state). (ONBs), the dependencies between the components are not
The component states are combined into system states, lost.
converting component state transition rules to system state
transition rules. The ONBs that could be caused by non- Algorithm 1: Generation of Rule Sets
recoverable and undesired states are found by checking the input : A set of components in system C = {C1 , C2 , C3 , ..., Cn }
A set of states at level 0 for each component
model against undesired rules. Once the ONBs are determined, maxlevel, maximum level of sub-states
the stakeholders can address the problems. The process is Sets of sub-states for each component at level 1, 2, 3, ..., maxlevel
Rules R = {R1 , R2 , R3 , R4 , ..., Rw }
repeated until the outcome satisfies the stakeholders. More output : Rulesets
detailed discussion of CCM can be found in [7].
1 rulesetnum = 0;
III. T HE P ROPOSED A PPROACH : ECCM 2 for currentlevel ← 0 to maxlevel do
3 if currentlevel == 0 then
To address the limitations of the CCM, we propose the 4 foreach Rule Ri in R do
enhanced causal component model (ECCM). Figure 1 illustrates 5 if states in Ri ∈ any of component-states at level 0 then
6 Rulesets[rulesetnum] ← Rulesets[rulesetnum] ∪ Ri ;
an overview of the proposed approach. The ovals represent the
processes and the rectangles represent the software artifacts 7 rulesetnum ← rulesetnum+1;
(inputs and outputs of the processes). The numbers just outside 8 else
9 foreach Component Ci in C do
the ovals indicate step numbers. Each step is detailed as follows: 10 if Component Ci has sub-states at level = currentlevel
1: Convert natural language requirements to ECCM specifica- then
11 foreach Rule Ri in R do
tions: The first step is to convert natural language requirements 12 if component in Ri == Ci then
into ECCM specifications. To convert the requirements to 13 if state in Ri ∈ sub-states of Ci at level =
currentlevel then
the input specification to generate ECCM, we identify model 14 truncate states with lower level
elements: components, components’ states, and transition sub-states;
15 Rulesets[rulesetnum] ←
conditions. The explanation on these model elements can Rulesets[rulesetnum] ∪ Ri ;
be found in Section II. The input specification of ECCM is
16 else
component state transition rules. A transition rule of ECCM 17 if states in Ri ∈ any of states at level 0
is in the form of Transition Condition: Component(current then
18 Rulesets[rulesetnum] ←
state) → Component(next state). After identifying the model Rulesets[rulesetnum] ∪ Ri ;
elements, the transition rules are created. Because ECCM 19 rulesetnum ← rulesetnum+1;
considers sub-states differently than CCM approach (where
states are flattened), we represent sub-state as parent state
followed by dot (‘.’) followed by sub-state. For example, a
component motor has ‘off’ and ‘on’ states and ‘on’ state
has ‘accelerate’ and ‘decelerate’ states. ‘On’ state of motor Algorithm 1 partitions the rules into rule sets. The algorithm
is represented as Motor(on), where as ‘accelerate’ sub-state generates rule sets until all levels of sub-states are covered. It
of ‘on’ state is represented as Motor(on.accelerate). If there initially generates rule set containing the higher level (abstract)
are multiple levels of sub-states, then the representation will representation of system. Once, this rule set is generated, each
be the outer most state followed by dot (‘.’) followed by its component that has sub-states is taken, and for each level of sub-
sub-state followed by dot followed by its sub-state and the states in component, treating states of other components to be at
process is repeated until the inner-most sub-state which needs abstract level, rule sets are generated. Once all the rule sets are
to be represented is found. generated, each rule set is allocated to a thread in multithreading
2.) Partition rules into rule sets: The rules generated may environment to perform rule expansion concurrently.
result in too many system states, and consequently the number 3.) Rule expansion: To analyze ONBs, we generate the
of system transition rules created may also be increased. In system states from the component-states. We generate system
order to reduce state explosion and rule explosion, we partition states from component-states because, at any given instance,
the rules into rule sets based on hierarchy levels of sub-states the state of system is dependent on all its components’ states. In
and components (see step 2 of Figure 1). addition, we aim to find system states where components’ states
The advantages of partitioning are that we analyze only a are conflicting or that might result in catastrophic behavior of
part of system at a time and if the users are not domain experts system.
or non-technical experts and need to understand the level of To simplify the system state generation process, we converted
operation at higher or abstract level, then they need to go the rules into numerical form. The components are represented
through only one rule set (first rule set) which represents the as rows and the states as columns and the ordinal positions of
abstract representation of system. In addition, the generation component and state are used to represent numerical values as
on rule sets based on hierarchy eases the analysis process of shown in the upper table of Figure 2. For example, if there
technical experts. They can consider only part of system which are 3 components in a system with 3 states each, then state
has issues and analyze it easily. Because our approach uses 2 of component 3 is represented as (0, 0, 2) The sub-states
2
Fig. 1. Enhanced Causal Component Model Overview
are represented as sub-column within a column and the their (0, 2, 0) : (2.1, 0, 0) → (2.4, 0, 0), it will be converted to (0, 2,
position in the sub-column is appended to the parent state 0) : (2.1, 2, 0) → (2.4, 0, 0). Propagation involves transferring
position with a separator ‘.’ as shown in the sub-state table the information from the current state to the next state. That
in Figure 2). For example, if state 1 of component 1 has 5 is, after propagation, the above rule becomes (0, 2, 0) : (2.1,
sub-states then we represent sub-state 2 of state 1 of component 2, 0) → (2.4, 2, 0). After the absorption and propagation,
1 as (1.2, 0, 0). we consider the 0 values in the state, which means that the
component-state can be any arbitrary state of that component.
This process is referred to as an expansion and is performed
State 1 State 2 ………. State n
to see all possible component state configurations possible for
Component 1 1, 0, 0 2, 0, 0 ………. n, 0, 0 that given transition. As a result, we replace all the zeroes with
possible component-states. We replace the system cause with
Component 2 0, 1, 0 0, 2, 0 ………. 0, n, 0 a notation, for example, cause (0, 2, 0) is now denoted as T1.
The rule (0, 2, 0) : (2.1, 2, 0) → (2.4, 2, 0) after expansion
Component 3 0, 0, 1 0, 0, 2 ………. 0, 0, n
becomes T1: (2.1, 2, 2) → (2.4, 2, 2) and T1: (2.1, 2, 1) →
(2.4, 2, 1).
Sub-State 1 Sub-state 2 …… Sub-state k 4.) Undesired state detection: After the rules are expanded,
0, 2.1, 0 0, 2.2, 0 …… 0, 2.k, 0 the model is generated and a state profiling algorithm [7] is
used to generate the number of incoming transitions due to
Fig. 2. Table for Numerical Representation of States and Sub-States of
environmental causes (EID), and due to system causes (TID),
Components and the number of outgoing transitions due to environmental
causes(EOD), and due to system causes(TOD) of a state in
We also converted any transition conditions with component- the model. The state profiling algorithm generates the above
states into numerical form. These numerical states are used mentioned values by going through each state in the rule
to generate numerical rules. Once the numerical rules are and creates or modifies corresponding profile. The algorithm
generated, the rules are expanded by performing absorption and identifies difference between system causes and environmental
propagation operations similar to CCM. We perform absorption causes by looking over if the transition condition has any
and propagation to consider the component state information component state information. We perform this by checking
in transition condition and transfer it to the current and next the transition condition information with the database of
system states. Absorption involves moving information from components and states that we collected while creating the
transition condition to the state if the transition condition is a rules.
system cause, system cause is the transition condition in which Based on the number of incoming transitions and the number
a component state or set of components’ states result in state of output transitions of each system state, we classified states
transition of other component); for instance, if there is rule like as unrecoverable or unreachable.
3
TABLE I
C OMPONENTS AND S TATES
Component States
Pulse Generator (PG) off, on
Fig. 3. State Profile with In-Degree and Out-Degree Device Controller-Monitor (DCM) off, on
Magnet outofplace, inplace
Printer off, on
Unrecoverable states are system states which cannot recover
without external input, i.e, TOD = 0. These states can be
undesired because the system is stuck in these states till external 1.) Convert natural language requirements to ECCM spec-
input is provided. Unreachable states are system states whose ifications: Two graduate students analyzed the pacemaker
TID and EID are zero, which means that no system state can requirements document and identified the elements of ECCM:
transition to this state. It is necessary to analyze such states components, states, and transition conditions. Once the states
for off-nominal behaviors, because while those states might were identified, hierarchical relationship among them is de-
be desired they can never be reached, leaving the system in a termined. Table I shows components and their higher level
problematic state. However, all unrecoverable states might nor states that we identified. The pulse generator component has
result in ONBs. Any unrecoverable state that does not lead to three levels of sub-states. The components, parent states, and
ONB is considered as false positive. These unrecoverable and sub-states are shown in Table II. The meaning of the acronyms
unreachable states are used for finding ONBs. of the sub-states of permanent, temporary, pace-now, magnet
5.) Analyze the detected undesired states: Not all unrecov- and power-on-reset states can be found in Table III. These sub-
erable/unreachable states are necessarily undesired states that states represent the operating modes of the states of operation
result in ONBs. The requirements engineers and stakeholders of the pacemaker.
should examine the detected off-nominal system states and Once the hierarchy of states is determined, the rules were
label them as undesired or not. After the undesired states are created. In total, 71 rules are created, which serves as ECCM
finalized, their associated rules are examined to find ONBs. specifications.
6.) Correct requirements: The rules that result in ONBs 2.) Partition into rule sets: In this step, we generated rule
and respective requirements are examined by requirements sets based on the sub-states generated in step 1. Using the rule
engineers and stakeholders, and corrective actions are taken set generation algorithm detailed in Section III, 4 rule sets are
based on stakeholders decisions. The entire analysis process generated from 71 rules. Due to space limitations, we present
is repeated with the corrected requirements until no undesired a subset of rules from the first two rule sets as follows:
states are found. Rule Set 1 - First Two Rules:
1) U serP ress(on_P G) : P G(of f ) → P G(on)
IV. C ASE S TUDY 2) U serP ress(of f _P G) : P G(on) → P G(of f )
Rule Set 2 - First Two Rules:
To evaluate our proposed approach, we implemented the
1) P atient(chronic_incompetence) :
ECCM tool. The tool allows users to enter information about P G(on.preimplant) → P G(on.implant)
components, component states, component sub-states, and 2) P atient(sick_sinuss yndrome) :
ECCM specifications (rules of component-state transitions). P G(on.preimplant) → P G(on.implant)
The tool utilizes a multi-threading environment to enhance While the two rules in rule set 1 refer to user switching on
performance. We performed case study using pacemaker using and off the pulse generator (PG). The two rules in rule set
ECCM and compared it with CCM. The following subsections 2 detail two conditions in which PG must be implanted into
present the details of the pacemaker requirements, the procedure person. We can observe that all the rules illustrated above
of the study, and the results. have environmental causes (Patient and UserPress are not
components of system).
A. Pacemaker Requirements
The first rule set has 7 rules out of 71 rules and the second
The pacemaker requirements document [12] is a 35-page
rule set has 18 rules. The third and fourth rule sets contain 14
document that describes the device’s system requirements and
and 59 rules, respectively.
provides details about diagnostics and therapy. The pacemaker
3.) Rule expansion: After the rule sets were generated, each
system consists of a pulse generator (PG), a device controller
rule set was allocated to a thread and the rule expansion is
monitor (DCM), leads, and a magnet. In this paper, we analyze
performed as explained in Section III to generate system states
the pulse generator, which plays a vital role in the system’s
and their transition rules. We converted our rules into numerical
operation; any malfunction of this component would result in
rules, and the following are numerical rules created for the
complications for the patient.
first two rules in rule set 2:
B. Procedure of the Study Rule Set 2 - First Two Numerical Rules:
1) P atient(chronic_incompetence) : 2.1, 0, 0, 0 → 2.2, 0, 0, 0
This section describes the procedure for our study. 2) P atient(sick_sinus_syndrome) : 2.1, 0, 0, 0 → 2.2, 0, 0, 0
4
TABLE II
C OMPONENTS AND S UB -S TATES
Component Parent state Parent state Sub-states
level
PG on 0 pre-implant, implant, pre-discharge-follow-up, routine-follow-up, ambulatory, explant
PG ambulatory 1 permanent, temporary, pace-now, magnet, power-on-reset
PG permanent 2 off, DDDR, VDDR, DDIR, DOOR, VOOR, AOOR, VVIR, AAIR, DDD, VDD, DDI, DOO, VOO, AOO, VVI,
AAI, VVT, AAT
PG temporary 2 off, DDDR, VDDR, DDIR, DOOR, VOOR, AOOR, VVIR, AAIR, DDD, VDD, DDI, DOO, VOO, AOO, VVI,
AAI, VVT, AAT, OOO, OAO, OVO, ODO
PG pace-now 2 VVI
PG magnet 2 off, DDD, DOO, AOO, VOO, OOO
PG power-on-reset 2 VVI
TABLE III when the magnet’s state is in place, the system requires the
E XPLANATION OF S UB -S TATES WITH ACRONYMS IN P ULSE G ENERATOR operating mode node not to have any sensing operation. Having
the sensing operation when the magnet’s state is in place might
Category Chambers Chambers Response Rate result in energy interference, which can cause the electrical
paced sensed to sensing modulation problem with the pacemaker.
Meaning O-None O-None O-None R-Rate 6.) Correct requirements: Once the undesired states are
A-Atrium A-Atrium T-Triggered modulation
V-Ventricle V-Ventricle I-Inhibited identified, their corresponding rules and requirements are
D-Dual D-Dual D-Tracked identified. In our study, we manually traced back from rules
to requirements. Consider a case that shows an undesired
state, 2.5.4.4, 2, 1, 2, which indicates magnet is out of
Once the numerical rules are generated, absorption, propaga- place but PG is in magnet state. Its corresponding rule is
tion, and expansion are performed as explained in Section III. U ser(remove_magnet) : 2.5.4.4, 2, 2, 2 → 2.5.4.4, 2, 1, 2
The result of these operations are the expanded rules that and its requirement is “When the magnet is removed the device
contain system states. The following are some of the expanded shall assume pretest operation”. Because the pretest operation
rules from the rule set 2: is never defined in the document, it is hard to know what
Rule Set 2 - First Four Expanded Rules: transition occurs from magnet state and what the destination
1) P atient(chronic_incompetence) : 2.1, 1, 1, 1 → 2.2, 1, 1, 1 state is. To clarify this requirement, we corrected it as follows:
2) P atient(chronic_incompetence) : 2.1, 1, 1, 2 → 2.2, 1, 1, 2 When the magnet is removed, the device shall assume pretest
3) P atient(chronic_incompetence) : 2.1, 1, 2, 1 → 2.2, 1, 2, 1
4) P atient(chronic_incompetence) : 2.1, 1, 2, 2 → 2.2, 1, 2, 2 operation and comes out of the magnet state.
4.) Undesired state detection: The expanded rules from each We also found ONBs from the states that are associated with
rule set were analyzed to detect possible undesired states by PG and DCM. It is not safe for DCM to be switched off when
generating state profile for each state as mentioned in step 4 PG is on if there is a necessity for continuous monitoring. When
of Section III. Using these state profiles, we found undesired continuous monitoring is required, this situation will result in
states that aid in finding off-nominal behaviors (ONBs) using complications of patient’s condition because the change of
the process detailed in step 4 of Section III. The detailed results pacing and sensing functions can no longer be controlled and
will be discussed in Section IV-C. it will affect the heart beat rate of the patient. If such incomplete
5.) Analyze detected undesired states: The detected undesired requirements are not addressed, it is likely to produce a product
states were analyzed to eliminate false positives. False positives that can threat the patients’ safety.
are the states which are desired but are identified as undesired.
C. Results
In our study, we found 146 false positives and they were easy
to find due to the hierarchical relationship between super-state Table IV shows the results of our study. RS in the table refer
and sub-states. For example, states where PG is ‘on’ state or to the rule set. Because CCM does not use the hierarchical
sub-states of ‘on’ state and DCM is ‘on’ state but the magnet is approach, some items are not applicable such as the number of
‘out of place’ and the printer is ‘off’ are detected as undesired sub-states and rule sets. We explain the results obtained from
states. However, printer in ‘off’ state and magnet in ’out of the ECCM first and then compare them with those from the
place’ state will not result in any off-nominal behaviors. Thus, CCM.
these cases are be considered as false positives. As Table IV shows, using ECCM approach, we identified 4
After eliminating the false positives, we obtained the components, 8 states (level 0 states or parent states or super
finalized undesired states. We found 250 undesired states in states), 61 sub-states for 3 levels of hierarchy. 71 rules were
the pacemaker requirements. For example, when the magnet created and 4 rule sets were obtained. The total number of
state is ‘in place’ and pulse generator is not in ‘magnet’ state system states generated is 504, and the number of expanded
of operation, it is considered as an undesired state because rules for all rule sets is 1880. The ECCM approach detected
5
TABLE IV TABLE V
R ESULTS OF ECCM AND CCM L IST OF O FF -N OMINAL B EHAVIORS (ONB S ) F OUND
ECCM CCM S.No Off-nominal behavior
Number of components 4 4 1 Magnet is inplace when PG is off
Total number of states 8 62 2 DCM is switched off when Pulse Generator is on
Total number of sub-states 61 N/A 3 Pulse generator is in magnet state when magnet is
Number of rules from NL 71 1192 out of place or Pulse generator is in other state
requirements when magnet is in place
Number of rule sets generated 4 N/A
Number of system states analyzed 16 (RS1), 448
48 (RS2),
40 (RS3), undesired state detection followed by false positive analysis,
400 (RS4)
Number of expanded rules 56 (RS1), 9496
the analysis of ONBs in CCM involves manual identification
generated and analyzed 232 (RS2), of undesired rules. These undesired rules are used to detect
160 (RS3), possible ONBs in expanded rules automatically. Thus, for the
1432 (RS4)
Number of state analyzed for 396 448 ONB analysis time, ECCM took 30 minutes, but CCM took 4
finding undesired states hours (due to large number of rules). Because we used only
Number of ONBs 3 3
Time taken to convert NL to rules 1 hour 4 hours
a small set of requirements (pulse generator) in this study, if
(after reading requirements) we apply our approach to a larger set of requirements, ECCM
Time taken for analysis of 30 minutes 0 will produce far greater benefits than CCM.
false positives
Time taken for generation of 0 4 hours
undesired rules V. D ISCUSSION
By examining the results, we drew the following observations.
First, overall, the proposed technique helps to determine ONBs
396 undesired (off-nominal) states and 146 of them were false in systems with complex behaviors during the requirements
positives. We found 3 ONBs from 250 undesired states. The phase. Second, the proposed approach reduces the number of
identified ONBs are detailed in Table V. Every undesired state rules and states being analyzed, as we divide the input rules
is caused by one of those 3 off-nominal behaviors. We consider into rule sets. The results indicate that state and rule explosion
the detected ONBs as low to moderate risk as they can be can be reduced by combining state abstraction and state space
easily addressed, and are not likely to lead to fatal accidents restriction.
unless some additional factors come into play such as energy We believe that our hierarchical approach can help the
inference. requirements engineers and stakeholders to understand the
When we applied the CCM approach to the same set of dependencies among the states more easily and address of
requirements, because the process of identifying components ONBs compared to non-hierarchical approaches such as CCM.
is identical to ECCM, we obtained the same number of In addition, our proposed approach can give system behavior
components, which is four. However, the number of rules at abstract level which we believe make it comprehensible for
created for CCM (1192) is far larger than for ECCM (71). non-technical stakeholders as abstract level representation is
Also, the number of expanded rules bt CCM (9496) is much simple.
larger than ECCM (1880). Unlike ECCM, CCM does not detect Although our results are promising, our approach has some
possible undesired states but requires human intervention to limitations that we want to address. One limitation is that our
analyze all states to find undesired states, which in turn are approach cannot guarantee exposure of all ONBs as it does
used for generating undesired rules to check the reachability of not consider states that recover to another undesired state. We
the undesired states. Thus, it does not produce false positives. plan to address this limitation by analyzing all possible states
By analyzing 52 undesired rules, we were able to detect 3 and then profiling rules similar to CCM.
ONBs that were found from ECCM. We are able to find two The second limitation of our approach is that it is still prone
ONBs at level 0 of hierarchy and one ONB at level 2 while to state explosion problem. If there are too many states at a
the total number of levels are 3. While the two approaches single level of hierarchy, the approach can have scope for state
found the same number of ONBs, the systematic way of the explosion. We plan to mitigate it by considering combinatorial
ECCM approach located them more easily and faster. Further, hierarchical exploration. The third limitation is the high number
CCM does not guarantee to locate ONBs. of false positives during automated undesired state detection.
Also, we found that our approach is more efficient than the We plan to address it by creating a knowledge base which aids
CCM approach. As shown in Table IV, the ECCM approach in identifying domain specific off-nominal properties.
took 1 hour to convert natural language requirements into
rules while the CCM approach took 4 hours. This indicates VI. R ELATED W ORK
that creating rules using ECCM is much easier than CCM. To date, some amount of research has been done in finding
Analyzing ONB is done differently for each approach. While ONBs [3], [2]. For example, Sukhwani et al. [6] proposed
the analysis of ONBs in ECCM involves automated possible software reliability growth models to find ONBs during the
6
maintenance phase. Other researchers have tried to analyze ACKNOWLEDGMENT
ONBs at the testing phase. Foyle and Hooey [5] proposed
off-nominal testing by incorporating off-nominal scenarios into This work was supported, in part, by NSF CAREER Award
experimental design. CCF-1564238 to University of North Texas.
Some researchers have analyzed ONBs by developing
simulation models [4]. A few researchers have tried to detect the
ONBs at the requirements level. For example, Tsai et al. [13] R EFERENCES
analyzed scenario specifications for off-nominal scenarios using
automated event tree analysis, in which the scenarios are
[1] A. K. Mostafa, “How embedded systems benefit people and in what
executed by considering failure models to generate event trees. fields ?” Tech. Rep. July, 2015.
Day et al. [2] used system modeling language(SysML) to [2] J. Day, K. Donahue, A. Kadesch, A. Kennedy, and E. Post, “Modeling
model generic patterns of ONBs and used them to check the off-nominal behavior in SysML,” in AIAA Infotech, Jun. 2012, pp. 19–21.
[3] D. Firesmith, “The Need to Specify Requirements
completeness of the system. for Off-Nominal Behaviors,” 2012. [Online]. Avail-
These methods consider off-nominal situations from human able: https://insights.sei.cmu.edu/sei_blog/2012/01/the-need-to-specify-
factor point of view. Unlike these methods, in our work, we requirements-for-off-nominal-behavior.html
consider exposing the causes that result in ONBs from system’s [4] G. Fraccone, V. Volovoi, A. Colon, and M. Blake, “Novel air traffic
procedures: Investigation of off-nominal scenarios and potential hazards,”
perspective. Journal of Aircraft, vol. 48, no. 1, pp. 127–140, 2011.
[5] D. Foyle and B. Hooey, “Improving evaluation and system design through
VII. C ONCLUSIONS AND F UTURE W ORK the use of off-nominal testing: A methodology for scenario development,”
In this paper, we presented the model-driven requirements in Internaltional Symposium on Aviation Psychology, 2003, pp. 397–402.
[6] H. Sukhwani, J. Alonso, K. S. Trivedi, and I. Mcginnis, “Software
analysis approach that reduces state and rule explosion prob- reliability analysis of nasa space flight software: A practical experience,”
lems using a hierarchical model exploration strategy, and its in 2016 IEEE International Conference on Software Quality, Reliability
empirical results with the pacemaker requirements. and Security (QRS), Aug 2016, pp. 386–397.
[7] D. Aceituna and H. Do, “Exposing the susceptibility of off-nominal
Although our empirical results showed the promising results, behaviors in reactive system requirements,” in Requirements Engineering,
our approach also has several limitations as we discussed Aug. 2015, pp. 136–145.
in Section V. For future work, we intend to investigate and [8] B. Boehm and V. Basili, “Software defect reduction top 10 list,” IEEE
improve our approach further considering those limitations. Computer, pp. 135–137, 2001.
[9] J. M. Stecklein, J. Dabney, B. Dick, B. Haskins, R. Lovell, and
First, we plan to use natural language processing to ease process G. Moroney, “Error Cost Escalation Through the Project Life Cycle,”
of identifying model elements, add analysis of orthogonal states Incose -Annual Conference Symposium Proceedings, 2004.
to proposed ECCM model, and reduce false positives using a [10] X. Devroey, G. Perrouin, M. Cordy, A. Legay, P. Schobbens, and
P. Heymans, “State machine flattening: Mapping study and assessment,”
knowledge base for identifying undesired states. Second, we CoRR, 2014.
plan to perform various empirical studies to explore further [11] A. Valmari, The state explosion problem. Berlin, Heidelberg: Springer
advantages and limitations of our approach. Third, we also Berlin Heidelberg, 1998, pp. 429–528.
plan to adapt and use the rule set generation algorithm for [12] Boston Scientific, “Pacemaker system specification,” Boston Scientific,
Tech. Rep., 2007.
system of systems. Finally, we plan to take possible contexts [13] W. Tsai, W. Song, R. Paul, Z. Cao, and H. Huang, “Services-oriented dy-
into consideration when analyzing and exposing off-nominal namic reconfiguration framework for dependable distributed computing,”
behaviors. in COMPSAC, vol. 1. Citeseer, 2004, pp. 554–559.
7