=Paper=
{{Paper
|id=Vol-479/paper-1
|storemode=property
|title=Early Requirements Engineering for Public Private Partnerships: Aligning Agents' Mental Models
|pdfUrl=https://ceur-ws.org/Vol-479/paper1.pdf
|volume=Vol-479
}}
==Early Requirements Engineering for Public Private Partnerships: Aligning Agents' Mental Models==
https://ceur-ws.org/Vol-479/paper1.pdf
Early requirements engineering for public private
partnerships: Aligning agents’ mental models
Brigitte Burgemeestre
Faculty of Economics and Business Administration,
Vrije Universiteit Amsterdam, The Netherlands
cburgemeestre@feweb.vu.nl
Abstract. Developing decision support systems is a complex process. It
involves stakeholders with diverging interpretations of the task and domain. In
this paper, we propose to use ontology mapping to make a detailed analysis of
the overlaps and differences between mental models of stakeholders. The
technique is applied to an extensive case study about EU customs regulations.
Companies which can demonstrate to be in ‘control’ of the safety and security
in the supply chain, may become ‘Authorized Economic Operator’ (AEO), and
avoid inspections by customs. We focus on a decision support tool, AEO
Digiscan, developed to assist companies with an AEO self-assessment. We
compared the mental models of customs officials, with mental models of the
developers of the tool. The results highlight important differences in the
interpretation of the new regulations, which will lead to adaptations of the tool.
Keywords: e-government, shared mental models, decision support systems
1 Introduction
The Dutch Tax and Customs Administration (Dutch TCA) aims to modernize and
reorganize its processes through the introduction of a new form of governance, called
“horizontal supervision”, which is based on mutual cooperation and trust. Dutch TCA
engages in horizontal supervision with companies that are trustworthy and in control
of their business processes. The companies receive more responsibilities: they have to
exercise a form of self control supported by their (IT) infrastructure and openly report
the results to Dutch TCA. If companies are able to exercise self control in a
responsible way, Dutch TCA can assign fewer resources to these companies and
exercise strict control on unreliable companies.
To build software that supports the processes of horizontal supervision, one has to
take the new roles of the participants and their changed tasks into account. Norm
enforcement has changed into a more collaborative activity. Dutch TCA can therefore
be seen as one of the actors within the system, instead of an external entity that
imposes norms upon the system [1] [2]. Requirements engineering methodologies
such as [19] [4] are intended for modeling processes which involve multiple
(autonomous) actors. A contribution of these methodologies is that they include an
Proceedings of CAISE-DC 2009
analysis of the actors’ goals and dependencies before addressing detailed system
requirements. This gives developers a deeper understanding of the environment in
which the software must operate, and of the kind of interactions that should occur
between the actors [4].What is often implicitly assumed in methodologies, like [19]
[4], is that mental models of the task and domain are shared among actors. In practice
however, this assumption is not always met. Especially in public-private partnerships
like horizontal supervision, where the parties involved have different interests and
backgrounds, differences in the interpretation among various actors can exist. Overlap
in task-specific knowledge structures or having a „shared mental model is argued to
have a positive influence on performance and effectiveness in collaborative situations
[5]. We suggest that for successful collaboration actors must have either a similar
interpretation of the task, or have accessible mental models, so that other actors can
adapt their models. We argue therefore that early requirements engineering should
involve the identification of the differences and similarities that exists among the
mental models of the stakeholders. With the differences clarified, the stakeholders
become aware about each other’s mental model constructs, which they in turn can use
to align their approaches already in an early stage in IT development. Unlike some of
the empirical work on shared mental models, however, we are not satisfied with mere
lists of differences. Instead we propose to build conceptual models, to detect
divergent or synonymous mental model constructs in a systematic and precise way.
2 Related work
Semantic heterogeneity is a phenomenon that emerges in distributed heterogeneous
environments as separate systems are often engineered based on different and
sometimes even incompatible conceptualizations [14]. General domain ontologies
were seen as a solution to overcome the problem: separate systems would need to
match their own conceptualizations against a common ontology of the application
domain, such that all systems would be semantically interoperable. Nowadays the
increasing number of distributed domain ontologies itself is a source for semantic
interoperability problems [11]. Ontology matching [17] is seen as a promising
solution to overcome the semantic heterogeneity problem [11]. It aims at finding
correspondences between semantically related entities of different ontologies There
are different matching techniques available [11] [13]. Most techniques require the
existence of some commonly shared body of knowledge, structure or syntax.
However in an innovative public private partnership such as horizontal supervision
where both the businesses and government have to adapt to their new roles,
responsibilities and ways of interaction such a thing does not exist. The shared body
of knowledge is evolving as best practices are developed, procedures are maturing
and lessons are learned based on experiences in the field.
Multi agent systems research addresses this issue through meaning negotiation or
semantic negotiation [14] [3]. They try to offer a dynamic and flexible form of
semantic coordination for situations in which no priory coordination exists. Bouquet
et al. introduce in [3] a method that makes the meaning of nodes in structured
semantic models explicit by combining three types of knowledge: lexical, domain and
Proceedings of CAISE-DC 2009
structural knowledge. They combine the knowledge sources to build a new
representation of the problem, where the meaning is encoded as a set of logical
formulae. The problem of semantic coordination is no longer tackled as a problem of
computing structural similarities but as a problem of deducing relations between the
formulae that represent the meaning of each concept in a given semantic structure.
Van Diggelen et al focus in [6] on the use of ontology negotiation protocols to
overcome communication problems between agents with heterogeneous ontologies.
The idea is that a negotiation protocol enables agents to gradually build towards a
semantically integrated system by sharing parts of their ontologies. In their solution
they combine a normal agent communication protocol with an ontology alignment
protocol. In [6] they propose that an ontology negotiation protocol should enable
sound and lossless communication between the agents. The agents should furthermore
deal with communication problems at the moment they arise and should built up a
relatively small communication vocabulary such that it remains easy to learn and to
process.
Another approach to match ontologies are the instance based methods [7] [16]. These
methods determine the similarity between concepts of different ontologies by
examining the extensional information of concepts [7]. Various approaches to
instances based methods exist: in [7] machine learning techniques are used to identify
mappings and in [16] a lexical search engine is used to map instances from different
ontologies. Concept classification information is exchanged between these mapped
instances, to generate an artificial set of common instances shared by concepts from
two ontologies, so that simple similarity measures can be applied. The advantages of
this method are that it does not depend on the availability of concept labels or a rich
ontology structure.
3 Research approach
To identify the differences and similarities that exist between the mental models of
actors, we propose a combination of ontology matching and semantic negotiation
techniques. A combined approach is needed to compensate for the absence of shared
domain ontologies and the likelihood of different conceptualizations by the actors. As
a starting point we propose the use of generic knowledge model templates, from
knowledge engineering methods such as CommonKADS[15]. These generic
templates can function as a basis for the actor specific mental model constructs. In
line with the CommonKADS method, the mental model of the actors we construct
will therefore consist of three knowledge categories: domain knowledge, task
knowledge and inference knowledge [15]. After we have constructed the actors’
mental models we like to compare them. Since the actors in a public private
partnership have a different background it is important to thoroughly assess the
meaning of the concepts implemented in the mental models before comparing them.
Therefore we will use an approach similar to [3] which makes use of distinct sources
of semantic knowledge to focus on the meaning of the concepts used by the actors
when matching their ontologies. Furthermore we will explore matching techniques
based on instance-matching to derive general concepts based on instances observed in
Proceedings of CAISE-DC 2009
the field. To conclude our research we will compare the actors mental model
constructs with each other to construct a model that presents the encountered
differences and similarities and provides means for the agent models to converge.
4 Case study: AEO self assessment support
We conducted a case study on AEO self assessment, which is part of the application
procedure for companies to qualify for Authorized Economic Operator (AEO). AEO’s
benefit from reduced customs inspections while for non-certified enterprises customs
will continue the traditional supervision.
To qualify for the AEO status a company must assess itself on a number of criteria,
which are described in the community customs code and [9] [10]. The company
reports its findings to customs. Customs then visits the company to check if the self
assessment is performed correctly and to gather additional information. The customs
then determine whether the AEO certificate is granted or not. The self assessment is a
nice example of public-private collaboration, because a traditionally public task
(compliance assessment) is partly delegated to a private party (a company). The
private party therefore needs insight in the mental model of the public party (customs
authority) to perform the task according to their standards. The customs, on the other
hand, are interested in the mental model of the company, because the legislation is
new and customs need to learn from best practices of early AEO applicants.
Consultancy firms offer services and tools to assist companies in performing the self
assessment. The AEO Digiscan developed by Deloitte’s Tax Advise unit, is an online
tool that works as a classic expert system and is based on the AEO guidelines. Experts
of Deloitte contributed to the development of the AEO Digiscan, by specifying the
guidelines, and turning them into clear questions. The questions that a company has to
answer depend on the company’s role in the supply chain and on answers to earlier
questions. Scores are expressed on a 5 point scale ranging from “Potential risk can be
considered high” till “Potential risk could be considered low and acceptable”.
4.1 Case analysis
This section presents the initial results of the research towards the mental models of
AEO self assessment. We compared a mental model embedded in a decision support
system with the mental models of customs experts. For the data collection we used the
following methods: document analysis and semi-structured interviews [8] [20]. We
studied internal and public documents from both Dutch TCA and Deloitte on AEO
certification and self assessment. To elicit detailed expert knowledge in the
interviews, we showed the experts the AEO application of a petrochemical company,
which had used the Deloitte AEO Digiscan, and asked them how they would have
assessed this company (if there would have been no AEO self assessment) and if they
could point out points of interest.
Domain, task and inference model. To analyze the interview results, we use an
adapted version of the knowledge model templates for the assessment task of the
Proceedings of CAISE-DC 2009
CommonKADS methodology [15]. As the self assessment task is concerned with
identifying risks, implementing and evaluating control measures to mitigate risks we
consider the IT risk management model of NIST [18] an appropriate starting point for
a domain model. Risk management is the activity of continuously assessing risks,
defining and implementing control measures to mitigate risks and evaluating and
improving the results. A risk assessment identifies the threats facing a company given
its line of business and its environment. The vulnerability of a company to threats
depends on its current control measures. A risk assessment therefore contains an
estimation of the likelihood of threats having an impact, and the expected size of the
impact. Control measures either reduce the likelihood, by dealing with vulnerabilities
(preventative controls), or reduce the impact (detective and corrective controls).
Consider for example the risk of smuggling: someone places an additional item in a
container, along with the rest of the cargo, without the trader, the shipping company
or customs knowing about it. The vulnerability can be reduced by limiting physical
access to all premises where containers are loaded and unloaded, on the basis of the
principle of ‘least privilege’. Only those employees are allowed to have access to the
containers, who need it because of their job.
We also include the AEO criteria and the AEO guidelines in our domain model.
They are merely attention points, which – given a business environment – indicate the
main risks for the company. It is however the responsibility of the company to set
their own internal norms, depending on the actual risks encountered.
Fig. 1. Inference structure of the assessment task [15]
The purpose of a domain model is to specify key concepts and indicate how they are
related. The implementation of these relationships is then further worked out in the
inference structure, which we show in figure 1. Figure 1 depicts the inference
structure adapted for AEO self assessment. Basis is the generic assessment model,
Proceedings of CAISE-DC 2009
taken from [15]. The input is a company that applies for the AEO status, the “Case”.
The “Abstracted Case” is the data relevant for the self assessment: the potential risks,
the measures that mitigate these risks, and the quality level of the measures, related to
the company’s business activities and its role in the supply chain. The AEO
guidelines provide an overview of risk indicators that companies can use in the
identification of potential risks. After the abstraction the company must specify which
(sub) sections addressed in the AEO guidelines are applicable to the company’s
specific situation and need to be evaluated and reported in the AEO summary to the
customs. From this set of (sub) sections a company selects a single subsection that
needs to be evaluated.
For each subsection a company determines if the risk mitigation is sufficient and
evaluates the implementation of the measures. The output value is an integer (0-5)
indicating the implementation level of the measure according to the COSO definition
of levels of internal control. A company has to report the values in the AEO summary.
The “Match” function checks whether the scores on the self assessment summary lead
to a decision if a company is AEO compliant or not. The “Match” function only stops
prematurely in case of (clear) incompliance. A company is only AEO compliant if it
scores well on all the (sub) sections that are applicable. We will not present a
CommonKADS task model as the domain and inference diagram contain all the
necessary information.
Constructing and comparing mental models. We found that the interpretations of
Deloitte and Dutch TCA of the task and domain model for AEO self assessment
overlap. Both are based on the AEO guidelines, and therefore use similar attention
points. Both make use of risk analysis methods. However, important aspects of the
self assessment are interpreted differently. In general we found that the approach
offered by the AEO Digiscan is more structured and requires less expertise on AEO
legislation, than the Dutch TCA approach posted on their website. However the scope
of the AEO Digiscan is limited; it focuses on risk assessment (identifying risks and
measures) while Dutch TCA’s approach focuses on all parts of risk management,
including implementation of measures. We also observe a difference in scoring: a
measure of the implementation of control measures by Dutch TCA and a risk-based
scoring by Deloitte. The risk management versus risk assessment view also is in line
with the views that Dutch TCA and Deloitte have on the AEO certification. Dutch
TCA sees the AEO self assessment as a means to judge the quality of companies’
internal control system, and to create awareness of potential risks. In contrast, Deloitte
efficiently provides companies with an indication of their position with respect to
achieving the AEO status. The Deloitte approach is therefore more aimed at
compliance with AEO legislation, whereas the Dutch TCA approach aims at
companies being ‘in control’ of their internal procedures regarding safety and
security.
These are all important aspects should have been addressed during the early
requirements phase of the development of the Digiscan tool. These aspects greatly
influence the kind of tool that is developed and the role the tool will fulfill within the
task of “self assessment”. They lead to different system requirements. Figure 2
summarizes our findings. The grey concepts are only covered by the Dutch TCA
Proceedings of CAISE-DC 2009
approach; the concepts depicted in white are part of both the Deloitte AEO Digiscan
and the Dutch TCA approach.
Fig. 2. Model of differences (dark) and overlap between DTCA and Deloitte
5 Discussion and conclusion
Charting the differences between mental models of stakeholders is an important
element of developing a complex decision support system, because it helps to identify
differences in expected functionality, and in the way the system is expected to be
used. Differences in task and domain models will lead to different system
requirements, consider for example the difference in scoring. Where most approaches
only identify the difference in scoring, mental models help to unravel the underlying
issues that contributed to these differences, such as the differences in scope and the
perception of the task. Therefore mental model mapping should be part of the early
requirements engineering phase [19] [4]. Note that expectations may be too complex
to implement. It is easier to design and implement an expert system about compliance
(rule-based), than about risk assessment in context (principle-based). Once such
expectation gaps have been identified, it is important that the stakeholder, who is
having the system developed, makes clear choices about the intended functionality of
the system, and communicates these to the other stakeholders. A less ambitious
system, with a task that naturally aligns with one or more sub-tasks of the task model,
may be easier to get accepted, than an overly ambitious system which will disappoint
some stakeholders. An interesting side-effect of our research is that the stakeholders
themselves have now realized what their respective positions are. The differences are
not insurmountable. In fact, some Deloitte experts have expressed a willingness to
adapt their tool, and especially the risk-based scoring model, to address concerns of
Dutch TCA about the implementation of control measures.
Acknowledgments. This research is part of the EU project ITIADE. We are grateful
for the open and insightful discussions with representatives of Dutch TCA and
Deloitte
Proceedings of CAISE-DC 2009
References
1. Boella, G., Hulstijn, J., Tan, Y-H., Van der Torre, L. Modeling Control Mechanisms with
Normative Multiagent Systems: The Case of the Renewables Obligation. In: Lindemann et
al. (eds.) Proceedings of Workshop on Agents, Norms and Institutions for Regulated
Multiagent Systems, LNCS vol. 3913, pp. 109–121, Springer-Verlag, Berlin (2006)
2. Boella, G. and van der Torre, L.: Norm negotiation in multiagent systems. International
Journal of Cooperative Information 16(2), pp. 97-122 (2007)
3. Bouquet, P. Serafini,L. and Zanobini, S.: Semantic Coordination: A New Approach and an
Application, Proceedings ISWC, LNCS vol. 2870, pp. 130-145, Springer Heidelberg (2003)
4. Bresciani, P., Perini, A., Giorgini,P., Giunchiglia,F., and Mylopoulos,J. Tropos: An agent
oriented software development methodology. Autonomous Agents and Multi-Agent
Systems Journal, 8:203–236 (2004)
5. Cannon-Bowers, J.A. & Salas, E.: Reflections on Shared Cognition. Journal of
Organizational Behavior, Vol. 22, No. 2, pp. 195-202 (2001)
6. Van Diggelen, J., Beun, R., Dignum,F., van Eijk, R., and Meyer, J.: Ontology Negotiation:
Goals, Requirements and Implementation. International Journal of Agent-Oriented
Software Engineering, Volume 1 , Issue 1, pp. 63-90 (2007)
7. Doan, A.H., Madhavan, J., Domingos, P., and Halevy, A.: Learning to map between
ontologies on the semantic web. In: Proceedings of the 11th international conference on
World Wide Web, pp. 662–673, ACM (2002)
8. Eisenhardt, K. M.: Building theories from case study research. The Academy of
Management Review 14 (4), pp. 532-550, (1989)
9. European Commission: Authorised Economic Operators–Guidelines, TAXUD/2006/1450,
Brussels (2007)
10. European Commission: Authorised Economic Operators–The AEO Compact model,
TAXUD/2006/1452, Brussels (2006)
11. Euzenat, J., Shvaiko, P. Ontology Matching. Springer, Heidelberg (2007)
12. Kalfoglou, Y. & Schorlemmer, M.: Formal support for representing and automating
semantic interoperability. In: ESWS, LNCS vol. 3053 pp. 45–60, Springer, Heidelberg
(2004)
13. Kalfoglou, Y. & Schorlemmer, M.L.: Ontology mapping: The state of the art. The
Knowledge Engineering Review, Vol. 18:1, pp. 1–31, (2003)
14. Schorlemmer, M. & Kalfoglou, Y.: Progressive ontology alignment for meaning
coordination: An information theoretic foundation. In: AAMAS ’05, Utrecht, the
Netherlands (2005)
15. Schreiber, G.., Akkermans, H., Anjewierden, A. , de Hoog, R., Shadbolt, N., Van de Velde,
W. and Wielinga, B., Knowledge Engineering and Management, MIT Press, Cambridge
(2000)
16. Schopman, B.A. C. Wang, S. and Schlobach, S.: Deriving Concept Mappings through
Instance Mappings, In: John Domingue and Chutiporn Anutariya, (eds.) ASWC. LNCS vol.
5367, pages 122-136, Springer, Heidelberg (2002)
17. Sowa, J.: Knowledge Representation: Logical, Philosophical, and Computational
Foundations. MIT Press, Pacific Grove, CA: Brooks/Cole (2000)
18. Stoneburger, G., Goguen, A. and Feringa, A.: Risk Management Guide for Information
Technology Systems. NIST Special Publication 800-30, National Institute of Standards and
Technology, (2005)
19. Yu, E.K.S.: Towards Modelling and Reasoning Support for Early-Phase Requirements
Engineering, In: Proceedings of the Third IEEE International Symposium on Requirements
engineering, pp. 226-235, IEEE Press, New York (1997)
20. Yin, R. K.: Case study research: Design and methods. Sage Publications Inc, London(2003)