Introduction
techniques are used alongside the passive testing, while other studies are focused
on passive testing only.
1.1
Methodology of the overview
Here we present an overview of recent research on passive testing. In order to
gather the collection of relevant papers for the analysis the following three sources
were used:
{ Google Scholar4,
{ ACM digital library5,
{ IEEE Xplore6.
Search results for \passive testing" query were analyzed and 104 the most
relevant papers were selected. Then, the papers were classi ed according to their
approaches, methods, and application areas. Each class is summarized in a
dedicated section below.
1.2
Publications timing and authorship
As it was already mentioned above the data set consists of 104 research papers.
Fig. 1 shows distribution of the papers by publication year.
The co-authorship graph is presented on Fig. 2. The graph nodes represent
authors. Size of node is proportional to the number of papers in the collection,
which are co-authored by the researcher. Also, this number is explicitly displayed
4 https://scholar.google.ru/
5 https://dl.acm.org/
6 https://www.ieee.org/publications/xplore/
after the author name. Edge between nodes stands for the collaboration relation.
Thickness of the edges indicates the number of the joint papers. If it is 2 or more,
then the number is displayed next to the edge.
Word cloud diagram on Fig. 3 is used to visualize the most frequent topics
in the collection. One can see that majority of papers are devoted to network
protocols, algorithms for trace analysis, formal requirements and speci cations,
nite state machines, invariants etc.
1.3
Structure of the paper
The remaining part of this paper presents compilation of manually processed
abstracts of the selected papers. From the collection of 104 relevant papers on
passive testing we included here those, which have clear focus on application
of the technique in practical settings. The text is organized in the following
way. Section 2 summarizes papers focused on network protocol testing. Section
3 surveys research on testing of web services. Section 4 is devoted to internet of
things applications. Security related testing is covered in section 5. Other speci c
cases of application of passive testing are collected in section 6.
This section summarizes papers focused on application of passive testing to
network protocols.
In [
36
] Protocol Integrated Test System is introduced to test routing
protocols. The system under test is modeled as a black box with windows. The testing
covers multiple channels and ports. A passive testing is applied in production
eld for conformance, interoperability and performance testing. Protocol state
machine is used to cover routing information manipulation. OnLine Test System
(OLTS) presented in [
39
] additionally introduces topology analysis and internal
process simulation to perform testing of BGP, OSPF and RIP protocols.
In [
25
] passive testing algorithm based on extended FSM is applied to TCP
protocol. The paper runs experiment to analyse the resulting test coverage for
state transitions.
Passive testing on protocols can be applied to detect faults in network devices
[
26
]. Linear programming is used to determine the ranges of the variables to
reveal transitions covered by the testing and monitoring.
Practical aspects of testing and monitoring are considered in [
9
]. The paper
presents its own independent approach to practical testing, own protocol monitor
and own syntax description tool.
In [
6
] a passive testing approach based on invariants is introduced. A set of
invariants represent the most relevant expected properties of the implementation
under test. The presented pattern matching algorithm allows to decide
correctness of the proposed variants with respect to a given speci cation. In additional
to theoretical framework a TestInv tool is developed and applied to Wireless
Application Protocol (WAP).
Papers [
30
] and [
18
] address non-deterministic network protocol speci cations
and both present case studies applied to Internet protocol and Managed Session
Protocol accordingly. The rst one is based on implementation machines and the
second one enhances the use of a set of invariants that implementation under
test should ful ll.
A methodology based on contextual signatures and passive testing with
invariants is presented in [
38
]. The concept of contextual signature o ers a
framework to add information on the states, the values of parameters and logical
connectors to increase the expressive power of invariants. This allows to cover
properties related to di erent layers of the protocol stack and end-to-end
communication between distant entities. A correlation algorithm is de ned to enable
interoperability testing between events collected from di erent network views.
Process mining and passive testing are applied in [
35
] to detect anomalies in
DNS protocol traces. Multi-actor modeling is based on the sequence of structured
activities, queries and responses by clients and servers. A practical example
illustrated approach application to identify mail bonnet attack in Internet Life
DNS traces.
3
Testing of Web Services
A detailed survey on formal approaches to active and passive testing with
applications to the cloud is presented in [
19
]. Here we mention the most relevant
articles.
An approach for EFSM-based passive testing of web services was suggested
in see [
7
]. The authors present an approach to speed up state recognition of
EFSM-based observers designed for observation of Web Services. The approach
is based on combining observed events and backward walks in the EFSM model
to recognize states and appropriately initialize variables.
E cient traces collection mechanisms for passive testing of web services was
suggested in [
8
]. The authors consider management of Web Services by passive
testing where the tester itself is a Web Service. They propose di erent
architectures for observation of simple and composite Web Services. They also study
a set of online traces collection mechanisms and discuss their performances in
terms of required CPU/RAM resources and introduced network overhead. These
performances are then maximized by selecting best locations of observers.
Observation considers both functional and non-functional (QoS) properties of Web
Services.
Paper [
12
] proposes an approach to test (actively and passively) Web services
composition described in BPEL using TGSE (Test Generation, Simulation and
Emulation), that is a tool for generating test cases for Communicating Systems
(CS). TGSE implements a generic generation algorithm allowing either test cases
derivation or traces checking. It supports the description of one or several
components with data and temporal constraints. First, in order to model the BPEL
behaviors, the timing constraints, and data variables, the BPEL speci cation is
transformed into the Timed Extended Finite State Machines (TEFSM) model.
TGSE can check whether a trace is valid according the speci cation or not. The
Loan Web Service is used as a case study.
In paper [
31
] the authors choose a non-intrusive approach based on
monitoring to propose a conformance passive testing methodology to check that a
composed Web service respects its functional requirements. This methodology is
based on a set of formal invariants representing properties to be tested including
data and time constraints. Passive testing of an industrial system (that uses a
composition of Web services) is brie y presented to demonstrate the e ectiveness
of the proposed approach.
Paper [
3
] presents a methodology to perform passive testing based on
invariants of distributed systems with time information. This approach is supported
by the following idea: A set of invariants represents the most relevant expected
properties of the implementation under test. Intuitively, an invariant expresses
the fact that each time the system under test performs a given sequence of
actions, then it must exhibit a behavior re ected in the invariant. These invariants
are called local because they only check the correctness of the logs that have
been recorded in each isolated system. The type of errors that are undetectable
by using only local invariants is discussed. In order to cope with these
limitations, this paper introduces a new family of invariants, called globals to deal
with more subtle characteristics. They express properties of a set of systems, by
making relations between the set of recorded local logs. It is shown that global
invariants are able to detect the class of undetected errors for local invariants.
Paper [
15
] presents a methodology and a set of tools for the modelling,
validation and testing of Web service composition, conceived and developed within the
French national project WebMov. This methodology includes several modelling
techniques, based mainly on some variations of Timed Extended Finite State
Machines (TEFSM) formalism, which provide a formal model of the BPEL
description of Web services composition. These models are used as a reference for
the application of di erent test generation and passive testing techniques for
conformance and robustness checking. The whole WebMov methodology is
integrated within a dedicated framework, composed by a set of tools that implement
the model representation, the test generation and passive testing algorithms.
This framework also permits the interaction of these tools to achieve speci c
modelling and testing activities in a complementary way. A case study based
on a real service, a Travel Reservation Web Service, is presented as well as the
results of the application of the proposed WebMov methodology and tools.
Paper [
14
] presents a methodology to perform passive testing of behavioural
conformance for the web services based on the security rule. The proposed
methodology can be used either to check a trace (o ine checking) or to
runtime veri cation (online checking) with timing constraints, including future and
past time. In order to perform this: rstly, the authors use the Nomad language
to de ne the security rules. Secondly, they propose an algorithm that can check
simultaneously multi instances. Afterwards, with each security rule, graphical
statistics is proposed, with some xed properties, that helps the tester to easy
assess about the service. In addition to the theoretical framework a software
tool is developed, called RV4WS (Runtime Veri cation engine for Web Service),
that helps in the automation of the passive testing approach. In particular the
algorithm presented in this paper is fully implemented in the tool. The authors
also present a mechanism to collect the observable trace in this paper.
Article [
5
] describes the application of the TestInv-P passive testing tool as
part of the testing phase of TXT e-tourism Web application. TestInv-P is a
passive testing tool that monitors communication traces of an application
during run-time and veri es whether it satis es certain security-related invariants
derived from SHIELDS models.
Paper [
2
] presents a formal framework to perform passive testing of
serviceoriented systems. The approach uses the historical interaction les between web
services to check the absence of faults. The authors assume that a global log is
not available. They show how to use local logs (recorded in each web service)
in order to check local properties and how to combine them in order to check
global properties.
Paper [
13
] presents two tools for conformance testing of web services. One
tool for unit testing that is implemented by an on-line approach. This tool can
be used to test a web service-based and/or an orchestration by simulating its
partners. The other focuses on veri cation of a timed trace with respect to a set
of constraints. Specially, one can use this tool to verify on-line or o -line a timed
trace. A real-life case study, Product Retriever, is presented by combining the
two tools.
Two related papers, [
34
] and [
33
] on conformance testing of web service
choreographies were published in 2012. The passive testing approach is prefered due
to its non-intrusiveness, support for black-box peers without source code being
available, and both local and global conformance. In [
34
] Chor speci cation
language is chosen, which can be seen as an abstraction of the standard Web service
choreography language, WS-CDL. The formal framework of the approach and
the tool support for one possible implementation model, Web service
choreographies, are presented. Collaborations within a chaoreography are usually achieved
through information exchange, thus taking data into account during the testing
process is necessary. In [
33
] the authors address this issue by using a non-intrusive
passive testing approach based on functional properties. A property can express
a critical (positive or negative) behavior to be tested on an isolated peer (locally)
or on a set of peers (globally). They support online veri cation of these kind of
properties against local running traces of each peer in a distributed system where
no global clock is needed.
4
Testing for Internet of Things
In a series of papers published in 2009 the authors apply a passive conformance
testing technique to a Mobile ad hoc network (MANET) routing protocol, OLSR,
which supports a dynamically changing topology and absence of centralized
management. In [
16
] a formal passive testing method to test the conformance and
reliability of the protocol is proposed. Paper [
17
] is taking into account the OLSR
formal speci cation, formal description of properties and collected traces of the
implementation. In order to precisely express new properties in multi-node
environments a new kind of invariants is introduced in [
4
].
In [
21
] and [
22
] the authors present a logic-based approach to test the
conformance and performance of XMPP protocol through real execution traces and
formally speci ed properties. Two related papers, [
24
], and [
23
], propose a
logicbased approach to formal speci cation of functional requirements for WSN
routing protocol. An algorithm to evaluate these properties on collected protocol
execution traces is suggested.
Prof. Brzezinski in [
11
] presents a tester that is built around an Arduino-class
microcontroller and is programmed in a test language that re-creates the basic
semantics of the standardized test language TTCN-3. The approach is intended
for the validation of IoT-class distributed systems involving humans-in-the-loop,
especially for long-term unobtrusive supervision of in-the-wild behaviour change
experiments in an instrumented home/work facility.
5
Security testing
A process model of a security-aware computer and network system is presented
in [
37
]. For each entity in the model a history of activities that have occurred to
the entity is recorded. Two major categories of attack-detection techniques are
discussed: anomaly detection and attack signature recognition. Anomaly
detection is used as a complement, to detect novel attacks with unknown signatures.
In [
10
] the author suggests using Testing and Test Control Notation as a
platform for Intrusion Detection systems on the example of the Smurf attack.
In [
29
] a passive testing approach to check whether a system respects its
security policy is proposed. To specify this policy Nomad formal language is
used, which is based on deontic and temporal logics. The methodology is applied
to an industrial case study provided by SAP group.
In [
20
] a passive testing approach is used for security monitoring of web
services. The authors propose a passive testing approach for SOA, encompassing
a non-intrusive module that gathers selected traces for web services for
centralized and decentralized work ows, and also a passive tester that analyzes the
distributed collected traces against security requirements. The proposed
methodology is applied to a Loan Origination Process using BPEL work ow.
In [
32
] the authors propose a novel approach to de ne protocol properties in
terms of Input-Output Symbolic Transition Systems (IOSTS) and show how they
can be tested on real execution traces taking into account the both, data portions
and control portions. These properties can be designed to test the conformance
of a protocol as well as security aspects. A parametric trace slicing approach
is de ned to match trace and property. The approach is illustrated on a set
of execution traces extracted from an automotive Bluetooth framework with
functional and security properties.
6