The paper declares a novel approach for Embedded Software (ES) development based on Embedded DevSecOps enhancing with Software Product Line paradigm. While retaining DevSecOps benefits to cope its everyday challenges an approach firstly focuses on the new ones that (non)anticipated diversity, interdependence and volatility of DevSecOps environment pose in today's rapidly evolving technological landscape, such as comprehensive Variability managing and Reuse enabling. To this end an approach prescribes: consider end-to-end ES Continuous Testing over DevSecOps CI/CD pipeline as its Foundation mostly assuring ES quality; fix its key blocking Issues; represent it as a feedback-based series of the stages where regular ES testing procedures known as X-in-Loops should be executed with simulator/emulator; finally, each stage explicit modeling in a Test driven development style i.e. as a process of dedicated Test Product Line (TPL) engineering up to ISO/IEC 26554 while assuming Embedded DevSecOps process to be in turn modeled as a target ES Product line (SPL) engineering up to ISO/IEC 26550. Each of successive TPL modeling should be tightly aligned with all TPLs and SPL assumptions just defined and with current refinement of SPL assumptions to enable it drive adjoining TPL To justify the approach proposed two results are presented of applying it to ES Continuous Testing subprocess being performed on host at the very beginning of Software in open Loop (CT) just producing first ES code for it. The first is sound (i. e. compliant, transparent, ES Quality-focused and automated) CT's Technological Model composing its Context, Parameters, Sub-processes. Pilot Kit for CT automation combining proven tools for the minimal set of necessary TPL processes tasks up to the Model proposed is thus the second one. Drafting the Model proposed benefits and future work to refine it concludes the paper.
In today’s rapidly evolving technological and social landscape conventional challenges of
Embedded Software (ES) Development being adequately coped with Embedded DevSecOps
dedicated practices [
safety and secure managing complex and highly distributed ES development environment.
Recognizing Embedded DevSecOps insufficiency to cope them both academia [
The purpose of the paper is two-fold. First, in line with ideas of [
Secondly, the paper sums up the results of the author’s attempt to justify their approach. It
describes draft Technological Model for ES Continuous Testing critical sub-process (CT from now
on) that should be performed on host at the very beginning of Software on open Loop stage [
The process of end-to-end Software Continuous Testing over СI/CD pipeline is introduced with ISO/IEC/IEEE 32675 as implementation of core DevOps Principle “Left shift and continuous everything”. Within Embedded domain it purposes at constantly meeting unique requirements for ES safety, security, privacy, reliability, performance and extremely long life cycles thus becoming the Foundation of DevSecOps. As such, being well-designed and implemented, its key promising features are as follows: • • • •
Just it should ensure both ES appropriate quality and its generation efficiency (speed,
security, and coordination among Dev, Sec, Ops and Test teams) enabling its usage within
target system as expected [
It should implement feedback among ES versions within a single build (in single Embedded CI/CD pipeline round) as well as traceability and controllability among ES versions of different builds (across multiple rounds) thus implementing Variability Management for ES and their (non)executable artifacts.
It have to highlight defects at Continuous Integration, Continuous Deployment and Operation stages; concerning information obtained after ES careful static analysis [8], particularly Advanced Static Application Security Testing (SAST) [8, 9, 10] for (non)directed fuzzing, followed further by automated (and appropriate manual) testing of safety and security, reliability, compatibility, performance, load, maintainability, and portability. It should set up and maintain appropriate conceptual and information environment for Embedded DevSecOps.
Realizing the above promises could however be blocked with key Issues. Some of them are largely
originated in embedded systems overarching architecture and inherent complexities of their design
and deployment [
These are listed beneath.
Common conventional processes and techniques for dynamic testing as prescribed with ISO/IEC/IEEE 29119-3 and ISO/IEC/IEEE 29119-4 effective usage within Embedded domain. 2. ES Continuous Testing assets effective and efficient configuration, management and controllable reuse both over single Embedded DevSecOps rounds and among them. 3. Full-aspect traceability enablement among ES Continuous Testing assets themselves on the one hand and over DevSecOps process as a whole on the other one. 4. ES testsing on embedded targets from Software in the Loop to Hardware (or in automotive domain even vehicle) in the Loop Initiation, Monitoring and Management. 5. Due ES quality, especially Safety and Security, timely and carefully monitoring, reporting and assuring. 6. Proper responsibilities division and control among various Test and Dev, Sec, Ops teams. 7. Current maturity assessment of ES Continuous Testing process and gain insights concerning the needs, aspects and ways to increase it.
As far as proper ES Continuous Testing process should be the Foundation for Embedded
DevSecOps its intrinsic above-listed Issues 1-7 become also crucial for DevSecOps too thus
demanding be effectively and efficiently addressed. Analysis of Embedded domain [
Table 1
Comparing Product line and Microservice Architecture for ES Engineering Paradigm Key Idea Benefit Limitation Use case
Focus on Reusability
Quality and Consistency
With above-mentioned ultimate purpose in mind of Embedded DevSecOps enhancing with Product
Line and Microservice Architecture, proper ES CT process is proposed to introduce in Test driven
development style [
The first such concept is Embedded DevSecOps. It could be considered as two-stepped
refinement of conventional DevOps defined with ISO/IEC/IEEE 32675-2022 as a set of principles
and practices which enable better communication and collaboration between relevant stakeholders
for the purpose of specifying, developing, and operating software and systems products and
services, and continuous improvements in all aspects of the life cycle. — for further delivering and
maintaining higher-quality ES overcoming the challenges above. It brings together three core
fundamentals: development, operations, quality assurance [
ES safety and security become a next-gen challenge that transforms Embedded DevOps [
Embedded CI/CD pipeline is thus the second
The third one is PL being defined with ISO/IEC 26550 as a set of products and/or services sharing explicitly defined and managed common and variable features and relying on the same domain architecture to meet the common and variable needs of specific markets. ISO/IEC 26550 also fixes PL engineering peculiarities useful hereafter such as: • • • • •
Problem and Solution spaces being formed with anticipated common and variable domain products’ features (that are referred to as commonality and variability) together with the rules of composing them for a product and, respectively, with reusable assets (both nonexecutable and software) together with corresponding rules of configuring assets for terminal or compound features implementing within a sample product.
Problem space de-facto standardized representation with Feature Model (FM) being defined as a tree of commonalities and variabilities with their additional dedicated interrelations. In turn, Solution space de-facto standardized representation with Platform being defined as a PL architecture, a configuration management plan and domain assets enabling Application engineering to effectively reuse and produce a set of derivative products. Domain and Application engineering lifecycles where the above reusable resources are engineered (for reuse) and, respectively, iteratively configured within sample products with commonalities and anticipated variabilities (with their respective Verification and Validation and also Realization processes where ES development specificity will be primarily implemented).
Both Organizational and Technical management process groups with the process of PL Variability management within the second one where Variability is considered as the ability of PL’s product or artefact to be extended, changed, customized or configured for use in a specific context.
To cope ES Continuous Testing Issues 1—7 from Section 2 let’s define CT (being considered as earliest and simplest sub-stage) with its dedicated Technological Model trough three successive steps. First, assume ES is being developed for various hardware systems with explicitly define common and variable characteristics as an extensible set of software products possessing hardware data—driven common and variable features with appropriate PL (SPL) where: 1. SPL engineering process is constituted with the series of PL environment initial Set-up and further Reinvention rounds with its FM and Platform initial defining and then evolving (where PL products creating is prohibited) that interchange with unified production rounds (where FM and Platform are conversely fixed and options are especially provided to create ES up to them). Just this round is considered further in the article as a context of TPL defining within process perspective. 2. Unlike ISO/IEC 26550, SPL Domain Engineering and Application Engineering life cycles are implemented by means of appropriate Domain and Application Embedded DevSecOps cyclic processes being synchronized with Organizational and Technical Management processes that are however left conventional. In particular, Domain and Application Realization processes are implemented by means of Embedded CI/CD pipelines from corresponding DevSecOps processes.
Secondly, list requirements for the Model to make it constructive. These are inspired with the requirements [8, 9] or reliable analysis of ES code data and control flows as follows. 1. Compliance — applicability for any ES target PL and CI/CD pipelines implementation technologies, test automation strategy and Dev, Sec, Ops, Test teams responsibilities and skills. 2. Transparency — clear representation of CT’s tasks, tests and artifacts being produced as their solutions, teams’ roles producing them and all the traceability links between and target ES PL artifacts. 3. Focus on ES Quality — enablement tests for compliance checking with critical ES Quality characteristics, foremost Safety and Security, Performance efficiency, Reliability, Compatibility, Maintainability and, not compromising, Functional suitability prescribed with ES quality model adopted (ISO/IEC 25010 by default or its refinements e. g.
Z.Tamrabet’s ESQuMO). 4. Lean automatability — acceptability any test automation strategy (from classical R. Martin’s Test Pyramid to DevOps Hourglass to Spotify’s Honeycomb) and tools (from online and opensource tools to proprietary software). 5. CT estimability — enablement its sound, informed and consistent maturity profile assessment in accordance with TPI Next [11] industrial testing maturity model.
Lastly, to meet the requirements listed, seamlessly combine industrially proven best practices and techniques from ES Engineering and business-driven industrial testing domains, namely: • • • • • •
Embedded DevSecOps process and its CI/CD pipeline effective patterns [
Resulted high-level CT’s Model is defined from the process perspective to be three-leveled structured triple: , ; (1) (2) (3) where is CT’s context — organizational , developmental and methodical ; denotes CT’s parameters proposed to tailor it up to , namely CT’s Strategy and extensible sets of quality characteristics to be necessarily tested, adopted testing techniques and automated tools , TPL reinvention rules and CT’s automation levels ; is a structured sextet (2) of TPL generic processes and TPL information environment where they should execute that includes Domain Testing and Application Testing , Test Management , Asset Management in Testing and Variability Management in Testing as defined with ISO/IEC 26554 while being enhanced with appropriate reference CT workflows using advanced static analysis and embedded fuzzing for CI/CD pipelines within Domain and Engineering DevSecOps being assumed [14];
composes domain assets and application assets that and (2) processes produce with own internal structure (presented further in Figure 1); , (4) where fixes TPL features with their interrelations thus capturing its variability; and contains patterns for static analysis of SPL Platform elements and respectively autonomous tests and scenarios for dynamically testing them thus constituting TPL Platform; , are the same as , but for admissible configurations of SPL Platfom elements; fixes rules for , , elements within Application Testing; and store reports being produced with and ; contains CT maturity profiles up to TPI Next [11].
At the second TM’s level its constituents from (1) are further detailed as follows: is an extensible set of paired personal and team roles of CT’s actors from Dev (tester, programmer, analyst, Tec writer, manager by default), Quality Assurance (leader, analyst, technician by default), Security (CISO, analyst, technician by default) and Operations (CIO, analyst, technician by default);
fixes an extensible set of Embedded DevSecOps with CI/CD pipeline proven patterns where
J. Beningo’s “ideal” CI/CD pipeline [
where extensible set includes reference modes of testing being performed by actors with roles from at unit (u), integration (i) and system (s) levels within Domain engineering(d) or Application one (a) that by default are: product by product testing with tests immediately generating ( ); product by product testing with tests proactively generating as a Domain engineering reusable assets ( ); incremental testing where the first product is tested with or mode but for the next only tests for new features are generated while for the features yet previously tested existing tests are used as reusable assets of Domain engineering ( ) or Application one ( );
contains initial level (where only open source, free trial and as-a-service solutions should be used along with initial requirements for automation kit eliciting), interim level (additional subscription solutions and requirements refining) and the last level of full CT’s automation with proprietary and custom tools meeting elicited requirements that altogether are the core.
In turn, for CT’s sub-processes from (2), (3) unified view is proposed focusing at ’s tasks: where just lists these ’s tasks; includes workflows of ’s operations being executing over ’s sub-environment to perform its tasks .
Table 2 presents all the tasks from obtained through CT-based refinement of conventional PL testing sub-processes’ tasks as prescribed with ISO/IEC 26554. (5) (6)
Domain tests for SPL domain artefacts initiation including requirements and conditions for them and design for unit, integration and system testing (above u, i, s levels from (5)) while accounting TPL variability Environment Implementing or setting up TPL domain test environment including test data
Set-up and and tools, enabling interoperability with SPL information environment in order Maintenance DS to access domain artefacts to test and proper feedback, changing TPL domain test environments and sharing its status with the relevant stakeholders
DE
Preparing review, inspection or static analysis to conduct static testing of SPL domain artefacts, executing domain static tests and recording their results while differentiating those for commonality from for variability ones
Running an ordered set of test cases, determining the pass/fail of them, document test execution and results related to variability Trace the status of defects in both SPL domain artefacts commonality and variability reported during domain and application testing, produce reports on domain test status and results
Setting up the readiness to Application Testing, deriving application-specific test cases and defining test procedures based on Domain Testing assets Setting up and maintaining Application Testing environments based on Domain Testing environment established with DS Performing application static and dynamic testing on Application testing environments established with AS Reporting application-specific and domain test incidents and how they will be managed, analyzing the failures and locations where they have occurred
Providing technical guidelines for performing domain and application testing. Elaborating Test Strategy (primarily as (5)) that provides the scope of domain testing (of SPL domain artefacts) and application testing, where these tested artefacts will be deferred to Selecting and tailoring the rest TPL processes from (2) to prevent and solve TPL-specific test problems Providing test plans for performing TPL tests while accounting variability model of both domain and application testing
Control MC
Accounting domain and application test progress and sharing it with the relevant stakeholders, monitoring and controlling its compliance with test plans
Mechanism Category VC
VA
Variability in
Test TV
Management in Testing AM
Maintaining a set of TPL variability implementation mechanisms that can be used for expressing or realizing variability in Domain Testing artefacts Defining TPL variability types and ways to express variability included in TPL artefacts and products such as test plans, test cases and scenarios within both Domain and Application Testing Establishing and maintaining trace links between variability in test artefacts and variability models in test defined separately to support the reuse of test artefacts in Application Testing (foremost with the authors’ dedicated Integrated three-dimensional variability assessment sub-model [14]) Managing Domain Testing artefacts that will be reused there and in Application Testing such as test cases for commonalities, variabilities and their interactions Managing TPL Application Testing artefacts that will be referred in regression testing due to the application evolution or as inputs in other applications
It’s worthwhile to note that besides process-oriented view (1)-(6) it should also be useful to consider CT within staged perspective as series beginning with Set-up round and then interchanging Production and Reinvention rounds inspired and tightly driven with SPL evolution.
Internal structure of CT’s Model outlined above is depicted with Figure 1 where notation of (2)(4) is kept
As Figure 1 shows, SPL, when defined, in turn defines and then drives TPL by means of its FM in Problem space and through integration CT workflows as and composition into Domain CI/CD pipelines as well as and ones — into Application CI/CD pipelines.
It’s worthwhile to note that besides process-oriented view (1)-(6) it should also be useful to consider CT within staged perspective as series beginning with Set-up round and then interchanging Production and Reinvention rounds inspired and tightly driven with SPL evolution.
While be formally enabled with (1) — (6) expressions above expected benefits of CT being modeled accordingly to them couldn’t be achieved without its proper automation.
As the first step to this end pilot CT automation Toolkit is proposed based on both the maximal shortening the list of tasks to be automated from Table 2 and generic requirements for necessary tools in foremost ISO/IEC 23643 and 30130 as well as ISO/IEC 20741, 23396, 23531, 24766, 33060.
Resulting Toolkit aligned with the Model (1)-(6) above is presented with Figure 2.
The Technological Model and Prototype Toolkit for ES Continuous Testing being now considered only on host at the very beginning of Software in the open Loop stage provides Dev, Sec and Ops teams with significant benefits such as: • • •
Ensures readiness of tested software for Software in-the-Loop testing Provide a background for comprehensive embedded testing process model elaborating Leverages and coordinates various Test and Dev, Sec, Ops teams efforts at all the testing levels among embedded CI/CD stages over SPL Domain and Application engineering.
Their additional bonus is the possibility to further enhance CT with GenAI potential related to TMap Organizing and Performing topics [13] included in the Model.
Extending the outlined Model and its automation Kit for Embedded Testing process as a whole for further implementing an approach proposed is the authors’ future work.
The author(s) have not employed any Generative AI tools. [8] M. Becker, J. Palczynski, Automatic Verification of (un)intended Data and Control Flows in Embedded Software (2024). URL: https://www.mathworks.com/content/dam/mathworks/conference-or-academicpaper/automatic-verification-of-unintended-data-and-control-flows-in-embedded-softwareew24.pdf [9] M. Becker, J. Palczynski, Increasing Resilience to Cyberattacks through Advanced Use of Static Code Analysis (2021). URL: https://www.mathworks.com/company/technicalarticles/increasing-resilience-to-cyberattacks-through-advanced-use-of-static-codeanalysis.html [10] M. Becker, J. Palczynski, Reconciling Software Development Speed and Robustness with Optimally Balanced Static Application Security Testing (2023). URL: https://www.mathworks.com/company/technical-articles/reconciling-software-developmentspeed-and-robustness-with-optimally-balanced-static-application-security-testing.html [11] A. van Ewijk et al., TPI® NEXT Business Driven Test Process Improvement, Sogeti, 2013 [12] M. Huang, C. Lemieux, Directed or Undirected: Investigating Fuzzing Strategies in a CI/CD Setup (Registered Report), in; Proceedings of the 3rd ACM International Fuzzing Workshop (FUZZING ’24), September 16, Vienna, Austria. ACM, New York, NY, USA, 2024, pp. 33-41. doi: org/10.1145/3678722.3685532 [13] Topics plotted on the High-performance IT delivery models, 2025. URL: https://www.tmap.net/page/topics-plotted-high-performance-it-delivery-models [14] O. Slabospitskaya, A. Kolesnik, The Model for Enhanced Variability Management Process in Software Product Line, in: Mayr H.C., Kop C., Liddle S., Ginige A. Information Systems: Methods, Models and Applications. Revised selected papers of 4-th International United Information Systems Conference (UNISCON 2012). Yalta, Ukraine, 2012, pp. 162–171. doi: https://doi.org/10.1007/978-3-642-38370-0_15