Towards a Quality Framework for Enterprise Architecture Models

Abstract

-While Enterprise Architecture Management is an established and widely discussed field of interest in the context of information systems research, we identify a lack of work regarding quality assessment of enterprise architecture models in general and frameworks or methods on that account in particular. By analyzing related work by dint of a literature review in a design science research setting, we provide twofold contributions. We (i) suggest an Enterprise Architecture Model Quality Framework (EAQF) and (ii) apply it to a real world scenario.

Keywords

• Enterprise Architecture
• model quality
• quality framework
• EA modeling

I. INTRODUCTION

Enterprise Architecture Management (EAM) aims to maintain flexibility, cost efficiency and transparency within an enterprise and addresses effective and efficient Business-ITalignment [ 1 ]. Research and practice in this discipline provide a profound pool of frameworks, tools and guidelines to master this complex task of EAM in order to systematically develop IT landscapes tailored to the business context [ 2 ].

One central artefact of EAM is the enterprise architecture (EA) model. It provides a holistic view on the enterprise with respect to its elements and dependencies that are required for value creation. Several EA modeling languages like ArchiMate exist, that are used in practice [ 3, 4 ].

In general, the EAM discipline is an extensively discussed research field regarding EA methodologies, management or lifecycle processes [ 5 ]. Nevertheless, to our knowledge no widely accepted approach exists, that enables stakeholders of EA to completely assess the EA model’s quality [ 6 ]. Still, the benefits of EA management highly depend on the model quality [ 7 ]. As we found out during our research, only a few articles address this research gap with the specification of EA quality attributes, but without providing a holistic framework how to actually use them in an EAM context. Thus, we address with this work the following research goal: to develop a holistic framework that reveals what EA practitioners have to consider when assessing their EA model’s quality. In contrast to other works (cf. [ 7 ]), we therefore solely focus on the EA model and define the following research question (RQ): What aspects does a framework for assessing the quality of EA models have to contain? From our point of view this includes the analysis of related work, which also may root in other domains than EA modeling, the structure of the framework and guidelines for the framework’s application to real-world contexts.

We structure our work as follows: After setting up our research design in section II, we discuss related work in section III. In the main part, we present the resulting EA model quality framework (EAQF) (section IV) and how we applied it in a business context (section V). In section VI we conclude our work, discuss limitations and derive future work in this research field based on our findings.

II. RESEARCH DESIGN

Design science research (DSR) is a widely applied and accepted mean for developing artefacts in information systems (IS) research. It offers a systematic structure for developing artefacts, such as constructs, models, methods, or instantiations [ 8 ]. As our research question indicates the development of means, the application of a DSRM is appropriate. We stick to the approach of Peffers et al. [ 9 ], since it transpired as effective in former research. It is split up into six single steps and two possible feedback loops (cf. Figure 1).

For the development of means to solve our research question, we conducted a systematic literature review (SLR) by combining the approaches by Kitchenham et al. [ 10 ] and Webster and Watson [ 11 ]. After defining the SLR scope, which is in line with our research question from section I, we searched for the combination of the terms “enterprise architecture”, “model” and “quality” in abstracts of articles on the Scopus1 and AISeL2 databases from 2007 to the present. After analyzing the titles and abstracts of the 209 results, we gathered a first pool of four directly relevant articles, that discussed the quality of EA models [ 6, 12–14 ]. In a next step we searched back- and forward [ 7 ] with this basis and completed the literature base with further related work known to us [ 3, 15 ].

The demonstration and evaluation is put into practice by applying the proposed means to a single case study. Single case studies gain a first, in-depth reflection on means in real life scenarios [ 16 ]. Moreover, single case studies are a feasible instrument to show applicability. Our case study does not ensure that our quality attributes are sound and complete. Consequently, future feedback loops have to take this into account.

The communication is done with this paper itself, since this got published. We performed two feedback loops within our research to improve our framework. Moreover, advanced feedback on this paper can be facilitated for further feedback loops and will influence future research elaborating on this topic.

III. RELATED WORK

Before giving an overview on related work, we want to clarify the term of EA model quality. Regarding to ISO/IEC 25010 quality “is the degree to which a product or system can be used by specific users to meet their needs to achieve specific goals with effectiveness, efficiency, freedom from risk and satisfaction in specific contexts of use” [ 17 ]. In the context of EA research Ylimäki states that “a high-quality EA conforms to the agreed and fully understood business requirements, fits for its purpose […] and satisfies the key stakeholder groups’ […] expectations in a cost-effective way understanding both their current needs and future requirements” [18, p. 30]. In general, research regarding EA quality agrees that it is defined by the ability to meet the EA users’ requirements [ 7, 14, 19, 20 ]. Most of the related work divides quality aspects of EA into the quality of EA products (e.g. EA models of current state or future vision), its related services, and EA processes (e.g. management tasks like EA planning) [ 7, 14 ].

Since model quality is not a research topic solely related to the EA discipline, we also relate to relevant work from other information systems disciplines. In the context of software engineering, the ISO/IEC organization defines five characteristics to assess a system’s quality and further divides them into sub-characteristics, namely, effectiveness, efficiency, satisfaction, freedom from risk, and context coverage [ 17 ]. A well-known framework for determining the success of information systems is the IS success model by DeLone and McLean, last updated in 2003 [ 21 ]. Lange et al. adapted this model to the EA domain and depict EA product quality, EA function setup quality, EA service delivery and EA cultural aspects as the drivers that influence EA’s user satisfaction and the intention to use it [22, p. 4234].

1 https://www.scopus.com/ 2 http://aisel.aisnet.org

At this point, we want to emphasize that the quality framework presented in this work focuses on assessing the quality of EA models. The EA model is related to the prior explained concept of EA product quality. We thus understand EA model quality as the degree of fulfilment towards a set of attributes a model has to fulfil regarding its purpose and requirements defined by its stakeholders.

In the discipline of enterprise modeling there are approaches that discuss model quality in general, without focusing on a certain modeling structure. Becker et al. define six principles that have to be considered when assessing an enterprise model’s quality (e.g., business process model, entityrelationship diagram). These principles are namely the principle of validity, the principle of relevance, the principle of economic efficiency, the principle of clarity, the principle of systematic model structure and the principle of comparability [ 15 ]. Although, these principles do not provide explicit measures, they offer a thorough quality frame from different perspectives regarding a certain model type, e.g. an EA model. Sandkuhl et al. also apply them to evaluate the quality of their modeling language 4EM and further depict concrete quality attributes: unambiguity, flexibility and stability, homogeneity, completeness, scope, integration and simplicity [ 23 ]. Moreover, Pitschke provides a list of quality attributes for IS models and discusses them [ 20 ]. This list is mainly related to a prior work by Rauh and Stickel from the data modeling domain [ 24 ]. Pitschke expands the quality attributes and explains them in relation to business process models.

Although literature identifies a lack of research in the topic of assessing EA quality (cf. [ 6, 7, 25 ]), some articles investigate EA quality related issues. Ylimäki defines twelve critical success factors for EA and relates them to maturity levels [ 18 ]. In addition, Ravazi et al. propose a quantitative approach to assess the maintainability and interoperability of a certain EA [ 26 ]. Another generic approach for EA quality assessment is proposed by Lakhrouit et al. who define a generic evaluation concept model that can be used for several metrics to assess an EA’s quality [ 6 ].

As explained above, general EA quality does not necessarily directly relate to the EA model as an artefact, but also EA management processes or other services. In the majority of the related work only general statements on EA quality are made. Still, some articles focus on the investigation of certain attributes that can be used to assess the EA model’s quality. Lim et al. provide a list of EA quality attributes, which were derived from six established EA frameworks [ 14 ]. Likewise, Niemi et al. provide a further list of EA quality attributes based on 14 interviews in [ 7 ]. The authors relate identified attributes to EA product and EA service quality (cf. [ 22 ]). Further, Davoudi and Aliee define measures to assess EA maintainability [ 12 ]. Next to their generic model, Lakhrouit et al. also discuss EA quality indicators they deem reasonable [ 6 ].

After analyzing the relevant literature, it becomes obvious that a thorough quality of EA models includes both quantitative and qualitative metrics. Khayami suggests a list of qualitative characteristics of EA models [ 13 ]. In contrast, Spence and Mitchell develop quantitative metrics for defining an EA models syntactic and semantic correctness as well as its completeness using insights from set theory [ 25 ].

As discussed earlier, the common sense of all articles is that the EA model’s quality has to be evaluated regarding its purpose and the stakeholders’ concerns [ 15 ]. Hence, Lankhorst et al. emphasize that the establishment of the EA’s purpose and its stakeholders is a vital aspects, each EA model should follow [ 3 ]. As can be seen in this section, numerous research relates to the topic of EA quality. Still, most of the identified articles do not provide a holistic approach how to assess the quality of an EA model [ 7 ]. Therefore, we aim to provide a framework that structures all relevant information from related work and helps enterprise architects to reflect on their EA models. This EA quality framework groups both qualitative and quantitative attributes, explains them and gives guidelines how to measure them. We present it in the next section before applying it to a certain use case afterwards.

IV. A FRAMEWORK FOR ASSESSING THE QUALITY OF EA

MODELS

In contrast to the other research on EA quality, we propose a framework that aims to guide architects how to assess their EA’s quality. Thereby, the framework solely focuses on the EA model as one of the EA products. Thus, we do not address EA management processes or services related to an EA product. The framework was built by (i) identifying and using an appropriate conceptual framework categorizing different quality aspects of EA models and (ii) identifying relevant EA model attributes within these categories. In contrast to related work (cf. [ 6, 7, 13, 14, 20 ]) this goes beyond merely measure quality attributes and guides architects how they should apply them to their concrete EA context.

For (i) identifying a conceptual quality framework approaches from disciplines beyond EA research can be facilitated. Semiotics theory provides a general framework for assessing model quality [ 27 ]. It addresses model quality from the perspectives of syntax, semantic and pragmatism and is applied in research related to enterprise reference model quality [ 28 ] or conceptual modeling [ 29 ]. While the syntactic quality discusses the alignment between an IS model and the modeling language it uses, semantic quality refers to the similarity between the IS model and the domain it reflects on. Further, the pragmatic quality aspect considers choices made during modeling in terms of comprehensibility [ 29 ].

We understand this threefold conceptualization of model quality sufficient when it comes to detailed IS models like entity relation graphs or business process models. EA models, on the other hand, are means for decisions regarding businessIT alignment or other strategic issues related to IT. They focus on a broader and more aggregated extract of reality. In order to support EA related decisions, architects may also include economic aspects in their EA models. Further, due to their complexity and the heterogeneity of their addressees, the structure and the documentation of EA models seem to have a distinctive influence on their quality. Therefore, we assess the framework from semiotics theory as too narrow and aim to conceptualize EA model quality from a more holistic point of view. Therefore, we apply the framework of modeling principles proposed by Becker et al. Besides using the dimensions of semiotics theory they explicitly define quality aspects addressing economic efficiency and the structure of enterprise models. They name six principles for proper modeling [ 15 ] as explained in the following:      

Principle of validity: Does the model match the segment of reality? Syntactic and semantic correctness are the most important criteria for this principle.

Principle of relevance: It says that it is not necessary to model all elements from the real world, just the ones that are needed for the modeling purpose. The decision for relevance has to be made with aim and purpose of the model.

Principle of clarity: All stakeholders of the EA model have to comprehend the model, even without being involved in the modeling process itself.

Principle of economic efficiency: Modeling should follow a clear purpose or aim. Even the costeffectiveness should be taken into account.

Principle of systematic model construction: All model parts should follow a general documented structure and should be held consistent.

Principle of comparison: The model should be comparable in semantic and syntactic against others, even with different model notations. This includes possible transformations into different modeling languages.

These six principles of model quality form the basis we used to fill with quality attributes from the literature analysis. Therefore, we analyzed relevant research we found for EA quality attributes that focus on the EA model in concrete and related them to the appropriate quality principle. Differently named attributes were aggregated, when they addressed the same aspect of EA model quality. The following articles were identified: [ 3, 6, 7, 13–15, 20, 23, 25 ]. For each attribute we defined a concrete description and identified assessment methods that help architects to measure them. Thereby we used both qualitative and quantitative as metric types. Although they relate to quantitative metrics, we also defined yes/no questions as a separate type, as well as assessment methods that could be performed by dint of modeling tools.

Before presenting the EA quality framework (EAQF) in detail, we explain how to use it. According to Lankhorst et al. EA modeling is goal-driven and, thus, highly depends on its purpose, its different stakeholders, and their concerns towards the EA model. Therefore, an EA model repository stores all model elements and their relationships among each other. In order to address the manifold stakeholder concerns, different views on this complete EA model exist, that address different aspects [ 3 ]. We deem it vital to align our EAQF with this approach. Thus, we structure it by three dimensions: (i) EA purpose, objective, stakeholders, (ii) EA model as a whole, and (iii) certain EA model views. Statements should be made regarding these dimensions. Thus, for each of these dimensions we identify relating quality attributes from the different principles from [ 15 ]. This is illustrated by Fig. 2. Fig. 2. The Enterprise Architecture Model Quality Framework (EAQF) As shown in the illustration, the (A) EA purpose, its objectives and the stakeholders’ concerns form the basis to assess the actual EA model’s quality. After clarifying this, the EAQF defines quality attributes addressing the two EA products of interest: (B) the EA model as a whole and (C) each EA model view in particular. While the attributes related to the basis help to assess whether the EA’s scope is sufficiently determined, the architect can use the results from (B) and (C) to balance them against the determined EA scope, if e.g. the level of detail is appropriate. This supports the idea that different EA models focus on different aspects. For example, if an enterprise uses EA models to analyze the complexity of their IS landscape, a highly detailed business architecture layer does not seem appropriate. Although the quality attribute “level of detail” (see TABLE 4) for the business architecture model view in this case would be rated as “bad” in isolated consideration, in balance against the EA’s purpose it is appropriate.

The complete EAQF is shown by dint of several tables that are described in the following. TABLE 1 gives an overview of all EA quality attributes that were identified in literature. They are described and classified towards the respective quality principle from [ 15 ]. Further, the table reveals from what source the attributes were taken. After relating the attributes to the quality principles we decided whether an attributes is to be assessed regarding (A) the EA purpose, the (B) complete EA model or (C) for each single EA view. We therefore initially clarified for each attribute, if it addresses (A) contextual characteristics (e.g., a clear EA purpose, stakeholders) or the (B/C) EA model in particular (e.g., semantic and syntactical correctness). To distinguish between (B) and (C) it was clarified whether the attribute can only be assessed on global model level or depends on a single EA view. For instance, while statements for the attribute “completeness vs. conciseness” can only be made on global level, the attributes “Level of Detail” should be discusses for each EA view. Here, some attributes may also be assessed on both, (B) and (C) level (e.g., “comprehensibility”). The resulting allocation of attributes can be seen in TABLE 2 (attributes addressing the EA purpose), TABLE 3 (attributes addressing the whole EA model) and TABLE 4 (attributes addressing a specific EA model view). Each attribute can consist of several assessment methods, that can be either of qualitative, quantitative, Yes/No or Tool Support-related nature. For each model view a separate assessment should be done. Furthermore, we want to emphasize that these attributes offer a guideline to the architect, who is conducting the EA model quality assessment. Thus, depending on the model’s purpose, he may exclude irrelevant attributes.

V. APPLYING THE FRAMEWORK IN PRACTICE

A. Case Environment

A single case study fits our purpose of gaining a first, indepth reflection of applicability of our framework in a real life scenario [ 16 ]. A case study is further suitable to shed light on the phenomenon of interest from different perspectives [ 16 ], which fitted our research objective of empowering EA practitioners assessing EA model’s quality.

The case organization is one of the leading insurance providers in the German-speaking market. About 30,000 employees and 16,000 associated agents count toward the workforce of the company. The organization gains revenues of over 16 billion Euro and manages investments of 135 billion Euro. Furthermore, the case organization has several subsidiaries: One of them is the internal IT service provider, in which we conducted our case.

The IT service provider employs around 1,400 employees. These are responsible for operations and development of technological solutions for the whole organization, including all of its subsidiaries. The IT provider began establishing EAM initiatives in 2008 and currently hosts two EAM units: The first unit, “architecture management”, employs twelve members, being responsible for application development. The second unit, “infrastructure architecture management”, hosts fifteen internal and thirty external (e.g., consultants) members, who are responsible for infrastructure management (e.g., operations of servers). As regulatory instances, both units are responsible for all EA related questions, ranging from EA development to EA implementation and EA maintenance.

There exist mainly two processes to maintain the elements of the EA model. First, the EA model is used to execute an application lifecycle management (ALM). Second, the EA model is used to execute an infrastructure lifecycle management (ILM), whose results are incorporated into the ALM.

The EAM department utilizes the ALM process to calculate the technical fit and the conformance to business demands once a year. Depending on these results the EAM department either determines areas of activity to improve technical fit and business conformance or decides in accordance with the business to shut down the application. Moreover, the ALM process is employed to trigger the responsible persons to Abgleichsliste

BSW Hypokick

Vertrieb

AD-DB EVITA

VESPA

Markt und Kunde CRM Longial

EBIS-KDB MAX-S

SCOUT

AV-DB BAV AZUR LB AIDA LM pro

Vertrag Produkt

BS GAntrVer LM AT Fig. 3. Exemplary Extract of the AWP View update all information to applications which are not relevant for the ALM process.

The ILM process is quite similar utilized compared to the ALM process. It is also executed once a year and employed to trigger the responsible persons to update infrastructure information. Especially, the status of the infrastructure is emphasized, i.e. planned, phase in, active, phase out, or end of life. Since several infrastructure components are exploited to realize applications, this status is also essential for the ALM process. Therefore, the latest status of all exploited infrastructure components is included into the ALM rating.

B. Exemplary Framework Application

We applied EAQF at the beforehand presented case. To answer the questions regarding EA’s purpose (

TABLE 2), we reused the results of the stakeholder interviews in [ 30 ]. Following Patton [ 31 ] we conducted a series of open-ended interviews, using a fixed set of questions for all interviewees. These questions dealt with stakeholder concerns. However, questions regarding EA products in general and the EA model in particular were also taken into account. Moreover, the results of [ 30 ] could also be used to answer several qualitative questions of the other EAQF parts (TABLE 3 and TABLE 4). Where the results of [ 30 ] were not sufficient to answer EAQF questions, we conducted deepening expert interviews with members of the EAM unit.

Answering questions regarding the whole EA model (TABLE 3), we took the EA model of the organization into account as well as the results of the interviews. The EAM unit notates their model using ArchiMate 2.1 [ 32 ] with slight changes. For example, they introduced so called work areas which are used to determine operation costs on the host system and to assign them to certain applications. Last, we answered EAQF’s questions regarding a specific EA model view (TABLE 4). Therefore, we chose the so called Anwendungssystemportfolio (AWP) which illustrates all business domains and which applications are used to realize this business domain. An exemplary extract is depicted in Figure 3. The business domains are modeled as business functions according to ArchiMate 2.1 within the repository. The applications are modeled as application collaborations. However, both elements are represented as stacked boxes in the view which is not in conformance to ArchiMate 2.1, since the AWP view is older than the decision for ArchiMate 2.1.

1) Quality Attributes addressing the EA Model's Purpose The EA model’s purpose is defined at our case. On the one hand, it is used to do a guided lifecycle management, i.e., ALM and ILM. On the other hand, it is used for information and decision demands, e.g., to spread who is in response for a certain application or to offer needed data for an informed decision of the management.

According to EAQF the EA model’s purpose quality lacks only in one point: the purpose is not regularly revised. The EAM unit performed stakeholder interviews in which, inter alia, stakeholders’ demands towards the EA model were enquired. However, it is not planned to perform such interviews regularly and to revise the purpose meanwhile. 2) Quality Attributes addressing the whole EA Model As already mentioned, the EA model is notated in ArchiMate 2.1 with slight changes. Consequently, the validity is not perfect, which is also reflected in Spence and Michel’s ratios [ 25 ]: Qs = 53,6%, Qa = 71,4%, and Qc = 100%. The value of Qs indicates that nearly the half of the elements of ArchiMate 2.1 is not used within the repository. Moreover, almost one third of the contained element types in the repository is not actively used (Qa). This reveals potential improvements of the model. It is arguable if it is necessary to use all elements of ArchiMate 2.1, but at least the number of not used element types can be reduced. This would lead to a clearer structure of the repository and would, consequently, heighten its quality.

Another shortcoming of the model identified by EAQF can be situated in the principle of relevance. EAQF asks for modeling guidelines which lay down what (not) to model. Those guidelines are not explicitly formulated in our case. Rather, there exists some kind of oral tradition to pass on what (not) to model to new architects.

Further quality flaws can be found in the context of economic efficiency. For example, the repository contains, apart from unused element types, elements which have not been updated for a long time, even though, their representation has been changed. However, those elements were not considered for reports or the like. Consequently, EA model’s quality could be raised by removing those.

Regarding clarity, the stakeholder concerned especially more up-to-date information. They stated that the used communication channels could be useful, but as long as the information is not up-to-date the communication channels are useless.

In the cases of systematic model structure and comparability we could not uncover any issues related to EA model’s quality. This is grounded in the fact that ArchiMate 2.1 is used as modeling language and, consequently, ArchiMate supports all requested quality attributes for these two principles.

3) Quality Attributes addressing a specific EA Model View Stakeholders perceive the validity of the considered view as sufficient. They remarked only a lack of up-to-dateness as a negative characteristic. Updating the view up to three times a year is not satisfactory to stakeholders’ needs. The stakeholder concern as short update cycles as possible.

Relevance and Clarity are two principles which evoke no issues related to quality. According to EAQF, references to external material are needed. However, the stakeholder interviews have shown that contained information is sufficient and external references for the purpose of the view are not necessary.

In the field of systematic model structure, the relations between various views are not made explicit. This is an existing flaw of this view and should be corrected soon consonant with stakeholders’ opinion. Moreover, the intention of the view is not explicitly formulated. This should be aligned as well.

Syntactical Properness Semantical Properness

Up-To-Dateness Quality of Information Sources

Uniformity and Cohesion Yes/No Yes/No Yes/No Yes/No

Yes/No quantitative Tool Support

Yes/No Yes/No Yes/No

Yes/No Tool Support qualitative Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No

SOURCE

EA Purpose and Objectives

EA Stakeholders Concerns

Is there a clear purpose for the EA defined? Does the EA team define objectives to fulfill the EA purpose Are purpose and related objectives regularly revisited? Is there a thorough assessment of stakeholders involved? Are the concerns of the stakeholders determined? Syntactical Properness

Uniformity and Cohesion

Reduction of Redundancy

EA Purpose and Objectives

EA Stakeholders Concerns

Completeness vs. Conciseness

Reusability

Flexibility

Calculate the ration according to [ 25 ] Validate towards Language Syntax Is the EA model based on a EA framework? Is a EA development method used? Does the EA model conform to a predefined model structure? Does the EA model hold any duplicates Is the EA model repository free from duplicates? Conduct Expert Interviews for EA model to identify implicit duplicates Is there a clear purpose for the EA defined? Does the EA team define objectives to fulfill the EA purpose Are purpose and related objectives regularly revisited? Is there a thorough assessment of stakeholders involved? Are the concerns of the stakeholders determined? Are there modeling guidelines defined addressing what (not) to model? Does the EA repository only store used elements? Are reoccurring phenomena reused in the model? Does the EA team agree on and use reference models? Does the EA model show alternative paths for organizational development?

Is a process implemented how EA models support decisions?

VALIDITY

Within our research we identified a research gap regarding the quality of EA models. Consequently, we formulated our research question what a quality framework for EA models should contain. To answer this question, we applied DSR according to Peffers et al. [ 9 ]. Based on a SLR combined by the approaches of Kitchenham [ 10 ] and Webster and Watson [ 11 ], we facilitated the framework of Becker et al. [ 15 ] and adapted it to our purpose.

We came up with a structure consisting of three parts.

One part forms the basis on which the other both parts are established. In this basis the purpose, objectives, and stakeholder are determined. The other parts are utilized to either rate the quality of the whole model or the quality of a certain view.

In a next step, we performed two feedbacks loops within the DSR process. Therefore, we applied the framework twice at our partners EA model. The results of each application are exploited to enhance the framework. Yes/No Yes/No Yes/No Yes/No Yes/No qualitative Yes/No Yes/No Yes/No

Yes/No Tool Support qualitative qualitative quantitative quantitative

CASE APPL.

No No Yes Yes Yes No Conduct Expert Interviews Conduct Validation Workshops with relevant Stakeholders Date of Last Change Frequency of Change Conduct Expert Interviews qualitative Conduct Validation Workshops with relevant Stakeholders qualitative Conduct Expert Interviews qualitative Conduct Validation Workshops with relevant Stakeholders qualitative Is the EA model repository free from duplicates? Tool Support Conduct Expert Interviews for EA model to identify implicit duplicates qualitative Are the goals of the EA model view clearly defined? Yes/No Is there a "EA supply chain" present? Yes/No How many levels of detail are used by the model view? quantitative Are the different levels of detail made transparent? Yes/No Conduct Feedback Interviews with Stakeholders qualitative How many elements are documented/explained in ratio to all elements. quantitative Does the EA model vocabulary follow a clear taxonomy (e.g. of a certain domain)? Yes/No Does each model view follow a clear layout? Yes/No Are modeling convention developed for certain situations to ensure consistent layout? Yes/No Do view templates exist that can be used for certain views? Yes/No Show number of model view elements. quantitative Depending on the view's purpose, is the amount of elements reasonable? Yes/No Is the structure of the EA model made transparent? Yes/No Is further material attached that explains ambiguous elements? Yes/No Yes Is external material referenced? Yes/No No Is the intention of each model view explicitly documented? Yes/No No Does every model view relate to the model structure? Yes/No Yes Are interrelations among model views made transparent? Yes/No No

Our main output is a framework for EA model quality assessment (EAQF), which closes a gap in existing research.

While literature on EA quality discusses certain quality attributes, the EAQF puts existing attributes into context and provides a mean to assess EA model quality depending on its purpose and stakeholders’ concerns. A use case reveals its significance to decision makers and identifies needs for improved EA development.

Apart from uncovering quality flaws, EAQF supports a better EA development. This is grounded by the fact that it can be facilitated as a setting to guide through the development. For instance, architects can choose several quality principles and pursue to raise the affected parts of the EA model to the needed quality level.

Our research still offers different improvement potentials.

First, our conducted SLR covers limited number of search terms. For example, a further review should contain ancillary phrases like synonyms for the used terms. This could identify further quality attributes EAQF may include. Second, the external validity of EAQF needs further investigations.

Therefore, supplementary DSR loops in other contexts should be executed. E.g., EAQF should be applied in different organizations from different industries or with different maturity grades. Third, a case study does not ensure that the quality attributes are sound and complete.

Consequently, other evaluation methods should be applied in future work as well.

The maturity grade of the EAM unit may be an important point, since for organizations with a low grade other quality attributes can be interesting compared to those with a higher grade. Consequently, EAQF should be aligned according to the maturity grade of the unit under inspection.

This stresses also another aspect for future research: the configurationally of EAQF. Every organization has special demands towards EAM. Therefore, the demands of each organization should be reflected in EAQF properly.

Nevertheless, organizations from equal industries may have similar demands which can represent as standard configurations within EAQF.

Last, executing EAQF has shown that questions are interrelated with each other. Though, these relations are not made explicit. This should be explored in future work, since this can reduce the needed effort to execute EAQF significantly.

References

[1] F. Ahlemann , E. Stettiner, and M. Messerschmidt , Strategic Enterprise Architecture Management: Challenges, Best Practices , and Future Developments, 2012nd ed. Berlin Heidelberg: Springer Berlin Heidelberg, 2012 .

[2] D. Matthes , Enterprise Architecture Frameworks Kompendium: Über 50 Rahmenwerke für das IT-Management . Berlin Heidelberg: Springer-Verlag Berlin Heidelberg, 2011 .

[3] M. Lankhorst , Enterprise Architecture at Work: Modelling, Communication and Analysis . Berlin, Heidelberg, s.l.: Springer Berlin Heidelberg, 2017 .

[4] The Open Group, ArchiMate 3 .0 specification: Open Group standard, 2016 .

[5] D. Simon , K. Fischbach , and D. Schoder , “ An exploration of enterprise architecture research,” Communications of the Association for Information Systems , vol. 32 , no. 1 , pp. 1 - 71 , 2013 .

[6] J. Lakhrouit , K. Baïna , and K. Benali , “ Model and Application Architecture Indicators of Evaluation the Enterprise Architecture,” in Advances in Intelligent Systems and Computing, New Perspectives in Information Systems and Technologies , Volume 2 , Á. Rocha , A. M. Correia , F. B. Tan , and K. A. Stroetmann, Eds., Cham: Springer International Publishing, 2014 , pp. 63 - 71 .

[7] E. Niemi and S. Pekkola , “ Enterprise Architecture Quality Attributes: A Case Study,” in 2013 46th Hawaii International Conference on System Sciences, Wailea , HI , USA, 2013 , pp. 3878 - 3887 .

[8] A. R. Hevner , S. T. March , J. Park, and S. Ram, “ Design science in information systems research,” MIS quarterly , vol. 28 , no. 1 , pp. 75 - 105 , 2004 .

[9] K. Peffers , T. Tuunanen , M. A. Rothenberger , and S. Chatterjee , “ A Design Science Research Methodology for Information Systems Research,” Journal of Management Information Systems , vol. 24 , no. 3 , pp. 45 - 77 , 2007 .

[10] B. Kitchenham , “ Procedures for Performing Systematic Reviews, Version 2.3: Technical Report EBSE-2007-01 , Department of Computer Science, Keele University and National ICT , Australa Ltd ,” 2004 .

[11] J. Webster and R. T. Watson, “ Analyzing the Past to Prepare for the Future: Writing a Literature Review,” MIS Q, vol . 26 , no. 2 , pp. xiii -xxiii, http://dl.acm.org/citation.cfm?id= 2017160 .2017162, 2002 .

[12] M. R. Davoudi and F. S. Aliee , “ Characterization of Enterprise Architecture quality attributes , ” in 2009 13th Enterprise Distributed Object Computing Conference Workshops , 2009 , pp. 131 - 137 .

[13] R. Khayami , “ Qualitative characteristics of enterprise architecture , ” Procedia Computer Science , vol. 3 , pp. 1277 - 1282 , http://www.sciencedirect.com/science/article/pii/S18770509110000 56, 2011 .

[14] N. Lim , T.-g. Lee, and S.-g. Park, “ A Comparative Analysis of Enterprise Architecture Frameworks Based on EA Quality Attributes,” in 2009 10th ACIS International Conference on Software Engineering , Artificial Intelligences, Networking and Parallel/Distributed Computing, Daegu, Korea, 2009 , pp. 283 - 288 .

[15] J. Becker , W. Probandt , and O. Vering , Grundsätze ordnungsmäßiger Modellierung: Konzeption und Praxisbeispiel für ein effizientes Prozessmanagement . Berlin Heidelberg: Springer Berlin Heidelberg, 2012 .

[16] R. K. Yin , Case Study Research: Design and Methods , 5th ed. Thousand Oaks , London, New Delhi: Sage Publications, 2013 .

[17] ISO/IEC 25010, Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models . Geneva: ISO , 2011 .

[18] T. Ylimäki , “ POTENTIAL CRITICAL SUCCESS FACTORS FOR ENTERPRISE ARCHITECTURE,” in Journal of Enterprise Architecture , 2006 , pp. 29 - 40 .

[19] T. Tamm , P. B. Seddon , G. Shanks, and P. Reynolds , “ How does enterprise architecture add value to organisations ?,” vol. 28 , pp. 141 - 168 , 2011 .

[20] Dr . Juergen Pitschke, “ Gute Modelle - Wie die Qualität von Unternehmensmodellen definiert und gemessen werden kann ,”

[21] W. H. DeLone and E. R. McLean , “ The DeLone and McLean model of information systems success: A ten-year update , ” J. Manage. Inf. Syst. , vol. 19 , no. 4 , pp. 9 - 30 , 2003 .

[22] M. Lange , J. Mendling , and J. Recker , “ A comprehensive EA benefit realization model - An exploratory study ,” in Proceedings of the Annual Hawaii International Conference on System Sciences , 2012 , pp. 4230 - 4239 .

[23] K. Sandkuhl , J. Stirna , A. Persson , and M. Wißotzki , Enterprise Modeling . Berlin, Heidelberg: Springer Berlin Heidelberg, 2014 .

[24] O. Rauh and E. Stickel , Konzeptuelle Datenmodellierung . Wiesbaden: Vieweg+Teubner Verlag; Imprint, 1997 .

[25] C. Spence and V. Michell , “ Measuring the Quality of Enterprise Architecture Models,” JEA , vol. 12 , no. 3 , pp. 64 - 74 , 2016 .

[26] M. Razavi , F. Shams Aliee , and K. Badie, “ An AHP-based approach toward enterprise architecture analysis based on enterprise architecture quality attributes,” Knowl Inf Syst , vol. 28 , no. 2 , pp. 449 - 472 , 2011 .

[27] C. W. Morris , Foundations of the theory of signs , 12th ed. Chicago: Univ. of Chicago Press, 1970 .

[28] J. P. Van Belle , “ Evaluation of selected enterprise reference models,” in Reference Modeling for Business Systems Analysis , 2006 , pp. 266 - 286 .

[29] O. I. Lindland , G. Sindre, and A. Sølvberg , “Understanding Quality in Conceptual Modeling,” IEEE Softw, vol. 11 , no. 2 , pp. 42 - 49 , http://dx.doi.org/10.1109/52.268955, 1994 .

[30] S. Hacks , M. Brosius , and S. Aier , “ A Case Study of Stakeholder Concerns on EAM,” in IEEE EDOC Proceedings - Trends in Enterprise Architecture Management Research , Quebec, Canada, 2017 .

[31] M. Q. Patton , Qualitative Research & Evaluation Methods , 3rd ed. Thousand Oaks: Sage Publications , 2002 .

[32] The Open Group, ArchiMate 2.1 Specification , 2013 .