=Paper=
{{Paper
|id=None
|storemode=property
|title=Quality Assurance Aspects in Automation Systems Engineering Projects - Challenges and Lessons Learned from Industry Prototype Applications
|pdfUrl=https://ceur-ws.org/Vol-821/paper6.pdf
|volume=Vol-821
}}
==Quality Assurance Aspects in Automation Systems Engineering Projects - Challenges and Lessons Learned from Industry Prototype Applications==
https://ceur-ws.org/Vol-821/paper6.pdf
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
Quality Assurance Aspects in Automation Systems
Engineering Projects
Challenges and Lessons Learned from Industry Prototype Applications
Dietmar Winkler
Christian Doppler Laboratory “Software Engineering Integration for Flexible Automation Systems”
Institute of Software Technology and Interactive Systems,
Vienna University of Technology, Vienna, Austria
{Dietmar.Winkler}@tuwien.ac.at
Abstract— Automation systems, e.g., hydro power plants and support provided by individual tools (i.e., technical heterogene-
industrial automation systems include heterogeneous engineering ity) and various applied data models (semantic heterogeneity)
disciplines, e.g., mechanical, electrical, process, and software collaboration becomes more difficult and results in a high
engineering, and raise additional challenges for quality assurance manual effort for synchronizing data models during the life-
activities, e.g., identifying defects in change management process- cycle phases [25]. Efficient automation-supported data ex-
es where different disciplines are involved. Our observations in change across disciplines and domain borders is an important
industry shows various tools and data models with limitations in key challenge to increase quality and decrease defects, effort
collaboration and data exchange that hinder effective and effi- and cost.
cient quality assurance across disciplines where experts have to
collaborate and exchange data. The Automation Service Bus Quality assurance is a comprehensive activity across all
(ASB) offers a middleware platform with focus on (a) integrating project phases to enable the construction of high quality prod-
tools from difference disciplines and (b) bridging the gap between ucts. In addition results of quality assurance activities help pro-
data models coming from different sources. Based on technical ject and quality managers in identifying defects early [25], as-
and semantic integration quality assurance across disciplines and sessing individual project states frequently, and – based on
domain borders becomes possible. In context of quality assurance individual project states – enables product quality evaluation
and project management this paper presents selected research with respect to monitoring and controlling the engineering pro-
challenges and shows results of implemented application proto-
ject [13]. Nevertheless, loosely coupled tools and data models
types of an ongoing project in industry context.
hinder efficient quality assurance and require high manual ef-
Keywords-Quality Assurance, Automation Systems fort for data collection, aggregation, and analysis. As a conse-
Development Processes, Defect Detection, Integration Testing, quence, quality assurance tasks are executed less frequently
Project Management and can raise quality issues. Figure 1 highlights the need for
automation supported data exchange for change management
and quality assurance in heterogeneous engineering environ-
I. INTRODUCTION ments.
Large-scale automation systems engineering projects, like Mechanical & process
hydro power plants, steel mills and large manufacturing sys- properties
tems include heterogeneous disciplines during design and de-
Software (SW) properties,
n
velopment, construction and test, commissioning, and opera-
tio
en &
Electrical Change
e.g., Logical Behavior
ta
ru ipe
Software Change
P
tion [4][18]. During the product life-cycle [4] various stake-
m
P&ID Change
Process ?
st
Change
In
holders from heterogeneous disciplines have to collaborate and Engineer
Management
P&ID Change
Electrical Change
oc on
exchange data across domain-specific tools applying domain P&ID Change Software Change
B l c ti
Software Change
k
n
Fu
specific data models [12]. For instance, electrical engineers Electrical Change Quality?
gr it
Assurance
ia rcu
Software
am
Electrical
focus on circuit diagrams and wiring, process engineers pro- Engineer
i
C
Engineer
D
vide process workflows for the plant, and software engineers
develop control system to observe and control the automation Electrical properties, e.g.,
Transmission lines.
system under construction. Our observations showed that pro- Figure 1: Changes and Defects - Challenges in
cess engineers and domain experts focus on bridging the gap heterogeneous Engineering Environments.
between heterogeneous environments manually to support de-
velopment, commissioning (i.e., system launch at the custom- Change Management (CM) is a success critical issue in
ers site) and operation (i.e., operation and maintenance at the software and systems development projects during develop-
plant site). Note that these manual activities are error prone ment and runtime [22]. CM addresses the need to respond to
(e.g., caused by media breaks) and require a high effort provid- changes coming from individual disciplines by other related
ed by experts when linking various data models and sources disciplines. For instance a modified sensor type (from the elec-
manually. Because of limited interaction and collaboration trical discipline) can require modified control flows (change
33
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
within the process discipline) and modified data visualization In automation systems development projects various disci-
approaches (software engineering discipline), e.g., modified pline apply heterogeneous tools and data models. Changes can
value ranges of the changed sensor. Changes have to be passed occur frequent (in different disciplines) and have to be syn-
to all involved disciplines within a short time interval to enable chronized to enable a consistent engineering project. Our ob-
addressing these changes within assigned individual models by servations in industry projects showed a traditional engineering
related engineers. Thus, change management is a crucial use process approach following a (sequential) product life-cycle
case for engineering projects and quality assurance. Quality process model. Figure 2 presents the basic process approach at
assurance focuses on identifying defects in general and in our industry partner – a large scale systems development and
change management processes specifically. Thus, we observed integration organization – including six basic steps. The indi-
three main challenges: vidual process phases are organized strictly sequentially. A
major challenge is to synchronize various disciplines within
Quality Assurance support for early defect detection in individual phases as often as possible to (a) identify defects
engineering processes across domain borders [22]. early, (b) enable efficient handling of changes, (c) enable a
Testing signal chains from hardware sensor to software consistent engineering repository involving artifacts from all
variable to address defects across disciplines in later phas- related disciplines, and (d) enable a comprehensive view on the
es of systems development [25]. overall project progress. Today this synchronization is execut-
ed mainly manually or in a less automated way [21].
Comprehensive view on the project progress including
signal change observations and project course assessments B. Change Management Processes in ASE Projects
[13].
Figure 3 presents the challenges of Change Management in
The remainder of this paper is structured as follows: Sec- heterogeneous environments [6][22]. Electrical engineers per-
tion 2 introduces to common automation systems development form changes in electric circuit diagrams including electric
projects and project dashboards with focus on quality assurance planning tools and data models (1). Typically individual tools
and project management. Section 3 presents the research is- are able to handle changes and quality assurance tasks individ-
sues. Section 4 presents candidate solution approaches imple- ually. Nevertheless, change management and quality assurance
mented at our industry partners. Finally, Section 5 summarizes, across systems borders and tools are not sufficiently solved.
presents lessons learned and highlights future work. Changes have to be passed to affected engineers/disciplines to
enable efficient synchronization and defect detection (2). In
II. RELATED WORK addition notification of changes (3) can support engineers in
better understanding the individual change and its consequenc-
This section summarizes related work on (a) Automation es, and – in case of conflicts – enable an efficient problem-
Systems Engineering (ASE) processes, (b) change management solving between involved engineers.
in ASE projects, and (c) project management with project
cockpits/dashboards. Electrical Plan
Changes
Tool Data
A. Automation Systems Engineering Processes 1 Electrial Engineer
Lessons learned from business IT software development
Software Dev .
show a variety of different software processes as framework Environment
Synchronization /
for project management support [20]. Traditional software de- Tool Data 2 Defect Detection
velopment projects typically follow a sequential or V-model Software Engineer
approach (e.g., Waterfall Model and V-Modell XT1). A lack of 3 Notification
Pipe &
flexibility with respect to changes caused by customer requests Instrumentation
(e.g., frequent changing requirements) and/or defects lead to Tool Data
the application of more flexible and agile process approaches, Process Engineer
e.g., Scrum [19] or eXtreme Programming [2].
Figure 3: Challenges in Change Management in
Defects
Heterogeneous Engineering Environments.
Changes
Frontend
Engineering Basic Detail Procurement
Start-Up Operation
and Design Engineering Engineering Construction In industry practice we observed less frequent synchroniza-
Sync. Sync. Sync. Sync. tion (because of a high effort by experts who a familiar with at
least two disciplines), informal discussions during prob-
lem/conflict solving processes [11][22] and quality assurance
Figure 2: Sequential Systems Development Process in tasks. Based on interviews with engineers with our industry
partners we learned that a set of defects are identified during
Automation Systems Engineering Projects.
the start-up phase (commissioning) by introducing system
components stepwise (including on-site quality assurance
tasks). In addition we learned that common defects, e.g., miss-
ing links between (hardware) sensors and (software) variables
1 See http://www.v-modell-xt.de for a detailed description of or defective component descriptions could have been found
the V-Modell for software and systems development projects. earlier by applying a systematic and frequent synchronization
34
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
of different disciplines, systematic quality assurance tasks (e.g., efficient quality assurance (i.e., focused reviews and integra-
by supporting focused reviews), and testing approaches [25] on tion testing) and (b) project observation and control with the
integration test level. Engineering Cockpit [13].
C. Comprehensive Project Management and Control A. Automation Service Bus and Common Concepts
Project and quality management, i.e., project observation The Automation Service Bus (ASB) has been developed as
and control, are key activities of project and quality managers a middleware platform [4] to bridge the technical gap between
across individual phases of systems development to capture the various tools and the semantic gap between individual data
current project state and introduce counter measures in case of models [5]. Technical and semantic integration of tools and
derivations in terms of resource problems and quality issues data models are the foundation for comprehensive quality as-
[20]. Software cockpits [16] and software project control cen- surance and project monitoring and control.
ters [14] have been developed to enable project and quality
management in business IT software projects. Main objectives Figure 4 presents the common concepts [6] represented as
of engineering cockpits include (a) the collection of engineer- the virtual common data model (VCDM), aiming at bridging
ing data, (b) the aggregation and analysis of the collected data, the gap between various and heterogeneous sources [12]. The
and (c) the presentation of the analysis results according to the main idea is focusing on common data sets that enable the col-
need of project and quality managers [15][16]. Nevertheless, laboration of related disciplines. For instance, electrical engi-
heterogeneity of engineering disciplines, tools, and data models neers apply dedicated tools for constructing electrical plans (1);
include an additional challenge in establishing a comprehen- on the other hand side software engineers apply tools (5) for
sive view on an automation engineering project. modeling software functionality, e.g., function plans (common
graphical language for implementing control applications).
Because of less transparent changes in automation system Common concepts (3) located in a centralized data base – the
engineering projects (due to less frequent synchronization be- engineering data base (EDB) – holds data which are relevant
tween disciplines and limited data exchange capabilities), for both disciplines for synchronizing and collaboration pur-
changes and the impact of changes remain unclear and depend poses. Note that the common concepts are limited to a small
on the individual estimation of experts, i.e., project and quality number of data elements which are mandatory for synchroniza-
managers. A key question of project managers focuses on the tion. Other tool-specific data (not required for synchronization
number of changes in an engineering project per time interval, steps) are left within related individual tools. Data changes are
project phase and the impact of/on disciplines raising changes passed and synchronized via transformers (T) to and from re-
and/or defects. lated disciplines in a bidirectional way (see (2) and (4)). Thus,
this concept enables data exchange across different tools using
III. RESEARCH ISSUES different data models.
Virtual common data model
Based on observations and discussions with our industry
Tool A Data Model Tool A Data Extract Tool B Data Extract Tool B Data Model
partner we identified a set of research questions:
Electrical Plan T T Function Plan
Tool Data
RQ1: How can we enable data exchange between heterogene- Electrical
Tool Data
Cust_Signal
2 4
FB_Signal
Software
Engineer Engineer
ous data sources? Efficient data exchange between tools and -------------
+ Address
+ Description
X
-------------
3
5
-------------
+ Location
+ FB_Info
+ ...
various data sources is the foundation for efficient change + Value Range
+ Voltage
+ Digitl/Analog
Y
-------------
Engineering Data Base
FA
-------------
+ ...
+ Value Defs
+ Input
+ Datatype
FZ
-------------
+ ...
1 + ...
management, quality assurance, and project management.
RQ2: How can we enable efficient quality assurance across Figure 4: Common Concepts [6].
disciplines? Individual quality assurance activities are located
within the individual discipline and tool. Today, these quality Analyzing typical data models used in hydro power plants
assurance tasks are executed by experts, who are familiar with at our industry partner, we identified signals as underlying and
at least two disciplines and requires a high effort. Automation common concepts linking various disciplines [6]. For instance,
supported QA across disciplines can increase QA efficiency electrical engineers see signals as wires and measurable voltage
and product quality. levels, process engineers identify signals as input/output values
of pipes and valves, and software engineers see signals as
RQ3: How can we enable a comprehensive view on the engi- software variable used to control and visualize data within the
neering project? Traditional project management approaches control application. Thus, signals (as common concepts) repre-
focuses on an isolated discipline within homogenous environ- sent a light-weight conceptual approach that can enable effi-
ments. We observed limitations in heterogeneous environments cient collaboration of various disciplines and enable efficient
involving loosely coupled tools and data models in various data exchange and synchronization between heterogeneous
disciplines. One key question is how we can develop a project engineering environments.
cockpit to address cross-disciplinary engineering projects.
B. Synchroniation between Various Disciplines
IV. USE CASE AND INDUSTRY PILOT APPLICATION
Figure 2 presented a basic engineering process for con-
This section presents the key use case “Change Manage- structing large-scale automation systems, i.e., hydro power
ment” [22] and the pilot application at our industry partner to plants, observed at our industry partner. Frequent synchroniza-
enable data exchange across disciplines as foundation for (a) tion of engineering artifacts (across tools and data models) can
35
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
increase construction efficiency, increase quality, and reduce (see Section II-A for an overview of observed project phases at
cost [22][25]. our industry partner). Note that the electrical engineer applies
an electrical planning tool and modifies the underlying electri-
Figure 5 presents a basic synchronization step of individual cal data model, e.g., circuit diagram (1), by changing and/or
disciplines (see Figure 5a), i.e., mechanical, electrical, and removing signals (or engineering objects). The engineering
software engineering. Note that every discipline applies do- data base (EDB) holds the current project data from previous
main specific quality assurance activities [17] with limitations phases and/or check-ins conducted by engineers coming from
on overlapping quality assurance tasks, i.e., defect detection different disciplines. The modified signal information is passed
across disciplines, tools, and data models. Thus, a key chal- via a defined and tool-specific transformer (T) to the ASB (2)
lenge is bridging the gap between heterogeneous environments. and is compared with the current signal information located at
The ASB approach enables an automation supported synchro- the EDB (3). Differences are presented to an engineer, typically
nization step, i.e., based technical integration of tools and se- a systems integrator and/or project manager who are responsi-
mantic integration of data models (see Figure 5b) [25]. Note ble for a consistent data base. Changes can be accepted or re-
that frequent synchronization will lead to a more consistent jected by the systems integrator via web frontend.
engineering data base (EDB) applicable for all related disci-
plines. Figure 7 presents a sample snapshot of the pilot application:
three changes have been identified (e.g., highlighted in Figure
(a) Synchronization across various Disciplines including comprehensive QA activities (b) Automation Service Bus
Tool Mec. SCADA
7) and can be accepted or rejected by the systems integrator.
Tech. Interop.
Tool-Specific Change
Change Mechanical Engineering QA
Tool Elec. Analysis
After a successful check-in of the modified data in the EDB the
Management
Tool SW Workflow
Change Electrical Engineering QA
Synchronization
Change & Conflict Resolution
Technical Integration
of Tools
database holds the updated (consistent) signal information for
Change Software Engineering QA QA Model Mec.
further usage including the changes conducted (and checked-
Model
Elec.
Model
SW
in) by the electrical engineer. Related engineers from other
Synchronized Data Models
Semantic Integration of
disciplines (e.g., software and mechanical/process engineers)
heterogeneous data models
can check-out the latest signal versions and update their local
Figure 5: Synchronization between Disciplines [25].
data models, assigned to their individual tools (6). Note that
this check-out requires a corresponding and tool-specific trans-
This synchronization step addresses all systems engineering former. Following this basic workflow (step 1, 2, 3, and 6), it
phases, typically separated by milestones and signal states. is easy to handle new and modified signal information. Never-
Note that signals are the common concepts used for synchroni- theless, handling removed signals is still challenging and needs
zation, change management, and signal change handling. to be solved.
C. Object Change Management
Change management of common concepts, i.e., signals in
hydro power plants or – more general, engineering objects in
other domains, is the most critical use case in engineering pro-
jects. Figure 6 presents the implemented change management
workflow at our industry partner based on the common concept
of change handling (presented in Figure 3) involving electrical,
mechanical and process, and software engineers [22].
2 ASB
Changes
Electrical Plan T
1 Tool Data
Elec. Engineer
Checkin & Difference
3
Figure 7: Prototype Implementation of Object Change
Difference
Analysis
Analysis
Engineering
Management with three changes.
Database
Removed
Signals 4 Removed signals (4) are specific type of changes and re-
Notification Engineering
Ticket quire a more specific handling within an engineering project.
Assuming that an electrical engineer removes defined signals,
5 Checkout e.g., obsolete sensor signals, by applying the presented change
management workflow (step 1, 2, 3, and 6), the workflow will
Software Dev.
Environment
T force related tools to remove the signal in the corresponding
Tool Data
data models as well. For instance, the signal will be removed in
Software Engineer 6
the function block diagram of the software engineer without
Pipe &
Instrumentation
Tool Data
T any warning – the signal will disappear and cannot be used for
Process Engineer further process steps. A more critical result of signal removal is
that a connection point might become disconnected (because of
Figure 6: Object Change Management [22]. a missing sensor) caused by the workflow rather than by the
software expert. To overcome these “unintentional surprises”
Basically, an electrical engineering executes a change in a (signals will be disappear in the related data models) removed
defined project phase, e.g., during the basic engineering phase
36
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
signals will initiate an engineering ticket (5), which (a) notifies tional process step for identifying changes) the presentation of
related engineers about the signals which are going to be re- signal changes enables focusing discussions on real deviations.
moved and (b) enable the response to these actions before Note that expert estimations of our industry partners assume
checking out the modified data models. Major benefit of apply- that typical projects include approximately 20% of signal
ing engineering tickets is that engineers can discuss and re- changes along the project course. Focusing on real changes and
sponse to changes and removed signals early, i.e. during every defects will significantly reduce synchronization and defect
check-in process. Note that removed signals can be rejected by detection effort and will increase project and product quality.
the system integrator as well; an appropriate notification of the Highlighted changes and the presentation of changes will re-
decision must also be passed to all related engineers. duce the number of signal comparisons and signal checks sig-
nificantly and – as a consequence – will decrease effort and
The pilot application at our industry partners showed that cost. With respect to quality assurance, reviewers and inspec-
selective notification of engineers based on removed signal tors [9] can focus directly on the changes to identify defects
information (and as a next step notification of all critical and deviations.
changes) turned out to be most valuable for project and quality
managers to enable early discussion on selected changes and Lessons learned from business IT software development
increased product quality (early identification, prediction and and software inspection [8] as a well-established static quality
prevention of candidate defects caused by changes). Thus the assurance technique in software engineering [3][9] might lead
implemented change management process is a promising start- to software inspection and reading technique [1] and reading
ing point for applying quality assurance tasks within an engi- technique variants [23] with focus on automation systems de-
neering project in the automation systems domain. velopment projects in heterogeneous engineering environ-
ments. In addition, next steps can also include the analysis of
D. Quality Assurance Support with Focussed Reviews engineering objects/signals and the impact of changes with
Assuming that a hydro power-plant includes a set of up to respect to other related components which are not directly
40.000 signals, change management and quality assurance be- changed by an individual engineer, e.g., the impact on compo-
come challenging [25]. Manual synchronization of lists of sig- nents which might have been affected by the change (but were
nals (and engineering objects) and defect detection in case of not covered by the change request so far).
changes are time consuming and error prone. Thus, the auto-
mation supported synchronization process (presented in Sec- E. Integration Testing in Heterogeneous Environments
tion IV-C) based on common concepts can reduce effort for Software testing is well-known in common software engi-
synchronization and increase project and product quality. neering practice to identify defects based on executable soft-
ware code [7] systematically. Based on software testing best-
In context of ASB application quality assurance in hetero- practices, several test levels apply, e.g., unit test, integration
geneous engineering environment focuses on two important tests, and system and acceptance tests [10]. For instance inte-
aspects [25]: (a) individual and isolated quality assurance tasks gration tests from business IT Software development focus on
conducted by experts in their assigned disciplines (e.g., me- testing the interfaces, integrated components and the data ex-
chanical and process, electrical, and software engineering) and change between components. Concerning automation systems
supported by isolated tool solutions; (b) quality assurance on engineering projects, similar test levels are applicable [24].
overlapping areas, i.e., common concepts, where different dis- Thus, we see the concept of integration testing (learned from
ciplines have to collaborate and synchronize data. Figure 8 business IT software development) as “comparable” to hetero-
presents the focus of quality assurance (also introduced in Fig- geneous automation systems development, where different
ure 5) by example. disciplines should be integrated and tested.
Mechanical & process
properties
„process“
QA
For instance, a hardware sensor is connected to a systems
Mechanical Software (SW) properties,
interface (e.g., a switchboard) and connected to a software in-
n
tio
terface (e.g., control application or visualization software) rep-
en &
equipment properties „software“ e.g., Logical Behavior
ta
ru ipe
Process Engineer QA
P
m
Requirements
resenting the software behavior [6][25]. Considering individual
st
Location IDs Machine vendor
In
Components
Interfaces
catalogue
disciplines as components (and individually) tested via “unit
oc on
tests” within the related isolated discipline, linking and testing
B l c ti
Electrical Software
k
n
Fu
Engineer Engineer
these individual component and disciplines on architecture
gr it
Transmission lines Data Types
ia rcu
am
Terminal points Logical Behavior
i
C
Signals (I/O) level might refer to integration tests including assigned and
D
„electrical“ QA across tested components and interfaces. Similar to integration tests in
Electrical properties, e.g., QA tools and data models
Transmission lines. software engineering, integration tests in automation systems
engineering projects will find different defects on architecture
Figure 8: Focused Reviews on Common Concepts. level which could hardly be identified by inspection and/or
isolated QA activities within one discipline. Examples of can-
Highlighting the differences during signal check-in (see didate defects in the automation systems domain are presented
Figure 7 for an example) and selective notification of affected in Figure 9:
engineers by applying engineering tickets (see Figure 6 for the D1: Missing link between sensor (Sx) and system interface
basic workflow) will enable focused discussions and problem- (I2). No value will be available at the software control ap-
solving activities of engineers affected by the change and/or plication.
defect. Instead of discussing 40.000 signals (including an addi-
37
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
D2: Correctly wired sensor (S2) to the system interface I3 F. Project Observation and Control
but no link to a software variable, i.e., sensor value will Observing and controlling engineering projects are key ac-
not be analyzed and used for further applications. tivities of project and quality managers. Project observation in
D3: Multiple connected sensors (S2 and S4) to I5 and V3. homogenous environments, e.g., business IT software projects,
is supported by individual methods and tools. Software cock-
D4: Correctly wired sensor S5 to system interface I6 and pits have been established as project control centers aiming at
wired connection at I6. Nevertheless the connection got summarizing and presenting most relevant project information
lost on the way to the software variable, e.g., caused by a for project and quality managers, e.g., schedule, budget, project
cable break. progress, and quality [14][16]. Note that the collection and
D5: Software Variable V5 not linked to system interface analysis of project related data is typically limited to homoge-
(Ix) and sensor (Sx). neous engineering environments like in software engineering
projects. Additional challenges arise in heterogeneous envi-
Figure 9 shows related stakeholders/disciplines and inter- ronments, e.g., in automation systems development projects,
faces [6] and illustrate highlighted defects across disciplines when linking data models and data from various sources.
which cannot be identified easy during check-in processes
and/or individual and local quality assurance activities. Based on discussions with managers at our industry partner,
the key challenge is to measure the impact of changes within
System Software Software an engineering project, i.e., identifying the number of changes
Sensor Interface Interface „Behavior“ per time interval and/or per project phase. Today expert estima-
S1 I1 V1 tions regarding the number of changes along the project course
S2 D1 I2 V2 are the foundation for capturing change management metrics
I3 (e.g., 20%) because no detailed data regarding the real number
S3 D2 I4 V3
of changes per project exists.
S4 D3 I5
V4
S5 D4 I6 ? The ASB approach enables a more detailed view on chang-
V5
es by providing real data according to change management
D5
processes along the automation systems development process
Wiring Configuration Use of Data (see Section II-A for a detailed description of a common engi-
neering process and Section IV-C for a change management
process). Common concepts (see Section IV-A) enable effi-
Electrical Engineer Configurator Software Engineer cient data exchange between various tools and data sources and
Figure 9: Testing Signal-Chains from Hardware Sensors to are the foundation for process observation and control. Note
Software Variables. that common concepts (i.e., signals) are used to observe and
analyze the process across disciplines and tool borders and to
Integration testing in automation systems development is control the next steps of the engineering process. Based on
typically embedded within the commissioning phase, where common concepts frequent synchronization between different
individual disciplines are linked to each other manually. In- disciplines becomes possible (see Section IV-B) and enable
spection and testing are applied stepwise during this ramp-up process analysis based on real and captured data.
phase. Defects have to be located manually by consulting indi-
vidual plans from heterogeneous sources. This manual activity, The analysis and presentation of change management data
conducted by integration engineers and experts is time- enable a more detailed view on the engineering process within
consuming and require high effort and cost (e.g., tracing indi- an engineering cockpit. Figure 10 presents a snapshot of the
vidual signals across the plant manually). The ASB approach implemented engineering cockpit, a “window to engineering
enables engineer in testing plans from different sources based data”, based on captured signal change management data from
on the common concepts. This so-called “End-to-End” test a real engineering project. The engineering cockpit from the
enables engineers in testing the overall chain of signals across perspective of managers (Figure 10a) enables an overview on
heterogeneous environment from hardware sensors to software the project state (visualized by the project progress) summariz-
variables [6]. In addition efficient navigation features have ing various groups of engineering objects (e.g., components)
been implemented (a) to navigate to signals in various plans to over time. Note that the visualization includes the current state
see the implementation of the functional requirements (without of engineering objects (i.e., signals) per project months and the
consulting the paper work) and (b) to link runtime data during number of expected engineering objects of the completed au-
the commission phase within the affected engineering plan tomation system. Thus, the project progress becomes traceable
(e.g., presentation of runtime and/or simulation within a PDF and observable. A “drill-down” feature enables a more detailed
document)2. Automation-supported integration testing of engi- view on an individual component to see subcomponents on
neering plans from various sources and efficient navigation various levels of detail.
between affected plans can improve systems development pro- Figure 10b illustrates the volatility of engineering objects
cesses and the commissioning phase significantly. (i.e., signals) e.g., the number of new, removed and modified
signals per project months. This metric enables project assess-
ments and evaluations of the engineering objects within a pro-
2See http://cdl.ifs.tuwien.ac.at/en/node/29 for a detailed de- ject. A high number of changes (modified and removed sig-
scription of these use cases and screen casts. nals) might indicate improvement options of the engineering
38
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
process because some components might have been reused captured data and metrics. The initiator of the change, i.e., a
during system development (including some required changes). tool or engineer, might be an indicator for improving individual
Nevertheless, a more details investigation of project manage- disciplines and the application of components. Thus, the engi-
ments will be necessary to fully understand the signal change neering cockpit provides a view on the tool impact of changes
results; the engineering cockpit provides a visualization of the (Figure 10c).
(b) Volatility of Engineering Objects
(a) Engineering Cockpit Overview with „Drill-Down“ (c) Tool Impact of Changes
Figure 10: Project Observation with an Engineering Cockpit Prototype.
the virtual common data model (VCDM) that bridges the gap
V. SUMMARY AND LESSONS LEARNED between different sources [5][6]. Based on various tool data
Development projects of large scale automation systems, models (from various sources) the VCDM holds the common
e.g., manufacturing plants, steel mills, and power plants, typi- concepts, e.g., signals, used for collaboration and data ex-
cally include a large and heterogeneous group of stakeholders change based on an agreed subset of data elements.
who participate in systems engineering projects. These stake- RQ2: How can we enable efficient quality assurance across
holders come from different disciplines, e.g., mechanical and disciplines? Common concepts represent overlapping infor-
process engineering, electrical engineering, and software engi- mation areas (see Figure 8) where engineers have to collabo-
neering, using different tools (technical heterogeneity) and data rate and exchange data. Furthermore, these common concepts
models from various sources (semantic heterogeneity). Never- are used for synchronizing disciplines along the engineering
theless, heterogeneous tools and data models hinder (a) effi- process. Note that frequent synchronization supports change
cient collaboration between disciplines, (b) effective and effi- management and defect detection. In common industry practice
cient quality assurance activities across disciplines and domain every discipline applies individual quality assurance activities.
borders, and (c) make a comprehensive project and process In addition, the ASB enables two novel quality assurance ap-
observation and control more difficult. The Automation Ser- proaches for defect detection: focused inspection and end-to-
vice Bus (ASB) provides a middleware platform that aims at end integration testing. Focused inspection enables the identifi-
bridging the technical and semantic gap between tools and data cation of defined defects in the overlapping information areas
models from various disciplines. where two or more disciplines collaborate by applying com-
Based on previous publications (e.g., [4]) this paper sum- mon concepts. The end-to-end test enables semi-automated
marized the basic concepts of the ASB approach and illustrates testing of the signal chain from hardware sensors to software
a prototype application at an industry partner to improve col- variables.
laboration, quality assurance, and project observation and con- RQ3: How can we enable a comprehensive view on the engi-
trol in a real world application context. neering project? Based on the VCDM process observations we
RQ1: How can we enable data exchange between heterogene- introduced an Engineering Cockpit [13], aiming at (a) provid-
ous data sources? Efficient data exchange approaches between ing stakeholder related data derived from the engineering pro-
data models from heterogeneous sources are the foundation for ject databases and (b) enabling control of project steps based
(a) efficient quality assurance activities and (b) a comprehen- on the analysis results [25]. In addition to process related in-
sive process observation capabilities across tools and domain formation and metrics, project context information and quality
borders. To overcome semantic heterogeneity we introduced
39
First Workshop on Industrial Automation Tool Integration for Engineering Project Automation
data might be useful to support project and quality managers in [11] Moser T., Waltersdorfer F., Zoitl A., Biffl S.: „Version
better assessing and controlling the automation systems project. Management and Conflict Detection Across Heterogene-
ous Engineering Data Models”, Proc of the 8th Int. Conf
Lessons Learned & Future Work. Based on discussion with on Industrial Informatics (INDIN), Osaka, Japan, 2010.
our industry partners and our experiences in developing [12] Moser T., Biffl S., Sunindyo W.D., Winkler D.: “Inte-
business IT software products, we see the technical integration grating Production Automation Expert Knowledge across
of tools and semantic integration of data models as very Engineering Stakeholder Domains”, Proc of the 4th
promising approaches to support different types of projects, CISIS Conference, Krakow, Poland, 2010.
where different stakeholders and disciplines are involved, [13] Moser T., Mordinyi R., Winkler D., Biffl S.: “Engineer-
and/or heterogeneous engineering environments can be ing Project Management using the Engineering Cockpit:
observed. A collaboration platform for project managers and engi-
Future work of this ongoing research project will focus on neers”, Proc of the 9th Int. Conf on Industrial Informatics
(a) different aspects of common concepts (in other engineering (INDIN), Lisbon, Portugal, 2011.
domains), (b) automation supported process modeling, obser- [14] Münch J., Heidrich J.: “Software project control centers:
vation, and validation, and (c) enhanced and automation- concepts and approaches”, Journal of Systems and Soft-
supported quality assurance methods. ware, 70, (1-2), pp. 3-19, 2004.
[15] Neumann R., Zbrog F., Dumke R.: “Cockpit Based Man-
ACKNOWLEDGMENT agement Architectures”, in Abran A., Braungarten R.,
Dumke R.R., Cuadrado-Gallego J.J., and Brunekreef J.
This work has been supported by the Christian Doppler
(Eds.): “Software Process and Product Measurement”,
Forschungsgesellschaft and the BMWFJ, Austria. We also
want to thank the domain experts at our industry partners for Springer, pp. 87-100, 2009.
their valuable insight into engineering projects and their feed- [16] Ramler R., Beer W., Klammer C., Wolfmaier K.,
back on the prototypes presented in this paper. Larndorfer S.: “Concept, Implementation and Evaluation
of a Web-Based Software Cockpit”, Proc. 36th
EUROMICRO Conference on Software Engineering and
REFERENCES
Advanced Applications (SEAA), pp. 385-392, 2010.
[1] Basili V.: “Evolving and Packaging Reading Tech- [17] Sage A.P., Rouse W.B.: “Handbook of Systems Engi-
niques”, JSS, 38(1), pp3-12, 1997. neering and Management”, 2nd Edition; Wiley, 2009.
[2] Beck K., Andres C. “Extreme Programming Explained - [18] Schäfer W., Wehrheim H.: “The Challenges of Building
Embrace Change”, Addison-Wesley, 2004. Advanced Mechatronic Systems”, Pros of ICSE, Future
[3] Biffl S.: “Software Inspection Techniques to support of Software Eng., Washington DC, 2007.
Project and Quality Management”, Shaker, 2001. [19] Schwaber K.: “Agile Project Management with Scrum”,
[4] Biffl S., Schatten A., Zoitl A.: “Integration of Heteroge- Microsoft Professional, ISBN 978-0735619937, 2004.
neous Engineering Environments for the Automation [20] Sommerville I.: “Software Engineering”, 8th Edition,
Systems Lifecycle”, Proc of INDIN, Cardiff, UK, 2009. Addison Wesley, 2007.
[5] Biffl S., Sunindyo W.D., Moser T.: “Bridging Semantic [21] Sunindyo W.D., Moser T., Winkler D., Biffl S.: “Foun-
Gaps Between Stakeholders in the Production Automa- dations for Event-Based Process Analysis in Heteroge-
tion Domain with Ontology Areas”, Proc of the 21 st neous Software Engineering Environments”, Proc of
SEKE Conference, Boston, USA, 2009. 36th Euromicro Conference, SEAA, France, 2010.
[6] Biffl S., Moser T., Winkler D.: “Risk Assessment in [22] Winkler D., Moser T., Mordinyi R., Sunindyo W.D.,
Multi-Disciplinary (Software+) Engineering Projects”, Biffl S.: “Engineering Object Change Management Pro-
Int. Journal of Software Engineering and Knowledge En- cess Observation in Distributed Automation Systems
gineering (IJSEKE), Special Session on Risk Assess- Projects”, Proc of the 18th EuroSPI Conference, Den-
ment, Volume 21(2), March 2011. mark, 2011.
[7] Kaner C., Falk J., Hguyen H.Q.: “Testing Computer [23] Winkler D.: “Improvement of Defect Detection with
Software”, Wiley, 1999. Software Inspection Variants : A Large-Scale Empirical
[8] Kollanus S., Koskinen J.: “Survey of Software Inspection Study on Reading Techniques and Experience”, VDM,
Research: 1991-2005”, Working Paper WP 40, Universi- 2008.
ty of Jyväskylä, 2007. [24] Winkler D., Biffl S., Östreicher T.: “Test-Driven Auto-
[9] Laitenberger O., DeBaud J.M.: “An encompassing life mation – Adopting Test-First Development to Improve
cycle centric survey of software inspection”, Journal of Automation Systems Engineering Processes”, Proc. of
Software and Systems (JSS), 50(1), pp5-31, 2000. the 16th EuroSPI Conference, Madrid, 2009.
[10] Meyers G.J., Sandler C., Badgett T., Thomas T.M.: “The [25] Winkler D., Biffl S.: “Improving Quality Assurance in
Art of Software Testing”, 2 nd edition, John Wiley & Automation Systems Development Projects”, in “Quality
Sons, 2004. Assurance”, INTEC Open Publishing, 2012 (accepted for
publication).
40