An Approach to Capture Design-induced Error Using an Ontology InJae Shin1, Sanghee Kim2, and Chris McMahon1 1 Department of Mechanical Engineering, University of Bath, U.K. {enpijs, enscam}@bath.ac.uk 2 Engineering Design Centre, Department of Engineering, University of Cambridge, U.K. shk32@cam.ac.uk Abstract. Since engineered systems, e.g. aviation control, have increasingly equipped with automated and computer-supported artifacts, human-system interaction has been an important issue. Understanding the infleunce of design on human performance requires phycological theories that explain human behaviour and cognition. One of the challenges of modelling such theories is to identify complex relations between systems and humans including different task perspectives. To address this problem, we have developed an ontology that specifies which relations are better comprehended when assisted with the theories and have used it to model human errors induced by designs. This paper presents the results of developing such ontology using a supporting tool, i.e. PCPACK. 1. Introduction An accident or incident event report contains useful information about evidence and it summarises the investigation results of probable causes [1]. Mostly the causes are due to either human errors or engineering failures. Whereas the number of accidents caused by the engineering failures has been reduced, the number of accidents caused by human errors is steadily increased. It is known that over 70% of accidents have human factors. An event report is a good information source to identify such human errors. The descriptions of how these errors were made are often mentioned implicitly making it difficult to understand the real reasons of the errors. Understanding the infleunce of design on human performance requires phycological theories that explain human behaviour and cognition [2]. In particular, such theories help understand an operator’s mistakes within his or her task perspectives. The accident report is important for designers and safety analysts because knowing the reasons of accidents can help them to design more reliable systems. There are methods like Fault-Tree Analysis (FTA) that helps visualise information contained in accident reports. However, these methods are not designed to capture relevant concepts directly from reports. Manual identifications by human experts are thus required. The Semantic Web technique can reduce the reliance on the experts through automatic semantic annotations. A challenge is to identify central concepts and their relations that are essential for designers in understanding the users’ behaviours affected by their designs [3]. An ontology is an explicit specification of conceptualization [4]. One of the main reasons of developing ontologies is to make domain knowledge explicit leading to more sharable. The ontology consists of concepts and relationships. In this research, the ontology is designed to support information sharing and extracting information from information sources. The main objective of this research is to develop an ontology that specifies information about human error and its relationships with designs. This paper presents the results of developing an ontology for the concept of Design-Induced Error (DIE). This research is based on the previous research that proposed a categorization of DIE [5]. 2. Purposes of developing an ontology of DIE Aviation accident reports are maintained by governments and published on the Web sites. Most of the reports are downloadable to end-users. For example, the Australian Transport Safety Bureau (ATSB) posts investigation reports into an official website [6]. Each report contains background information about an accident, e.g. aircraft model or accident date, and content in a free-text. The ATSB Web site provides a limited search that enables to search based on the background information. That is, it is difficult to retrieve the accidents only caused by Human Error that have relationship with Design Error. A reason is that extracting specific information from the unstructured texts requires analysing the texts from semantic perspectives. To address, this research has developed an ontology for the following objectives: (1) To formalise relations between design of a system and human error by analysing accident reports (2) To visualise a model of the relations (3) To examine the possibility to capture a psychological theory from the reports 3. Ontology development Through discussions with knowledge engineering experts and examination of current methodologies, the following development process was adapted as shown in Figure 1. applied development methods stages source determining the domain Noy & McGuinness ontology and scope of the development method ontology knowledge elicitation: description analysis and meta- defining a domain theory of design-induced error documents accident reports (web) mark-up technique knowledge extraction from (PC PACK protocol tool) the domain documents knowledge analysis technique knowledge analysis (generating concepts and (PC PACK ladder tool) relations) refining concepts and relations knowledge representation technique knowledge modelling (PC PACK ladder, diagram tool) web browser publishing technique (PC PACK annotation and publish validation and publishing tool) complete Figure 1. Ontology development and applied methods This is based on the development process proposed by Noy and McGuinness: [7] (1) determine ontology domain and scope, (2) consider reusing existing ontologies, (3) enumerate important terms in the ontology, (4) define the classes and class hierarchies, (5) define the properties of classes-slots, (6) define the facets of the slots, and (7) create instances. For a concept annotation, the PCPACK tools were used [8]. Figure 2 shows an overview of various tools within the PCPACK software that accepts documents in txt or html formats and generates the annotation outcomes into xml form. It is used:  To annotate information extracted from documents  To structure the annotation using various knowledge models (such as trees, diagrams, grids and hypertext)  To acquire and validate knowledge from experts Figure2. a diagram of PC PACK Toolkits 3.1 Data set This study is based on the reports downloaded from the ATSB. The ATSB was chosen since: (1) the reports are easily accessible; and (2) investigators tend to pay more attention to identify and describe human error. In particular, the investigators have expressed the importance of recognising human error and the underlying reasons of it. The analysis is based on Reasons’ human error theory [9]. 3.2 Development stages The development consists of six stages each of which is described below. Stage 1: Determining the domain and scope of the ontology This stage is to determine the domain and scope of the ontology. The main role of this ontology is to make information explicit in order for easier extractions. In order to determine the scope of the ontology, a list of questions that the ontology should answer was sketched as competency questions [10]: (1) Which design-induced error characteristics should I consider when designing a system? Mise en forme : Puces et (2) How does a design lead operators to make an error? - which design concepts numéros easily make operators fall into design-induced error phenomena e.g. gulf of evaluation? (3) Can an inadvertent action of an operator be influenced by design error? (4) Why did an operator fail to follow designed rules? (5) Which characteristics of a phenomenon affect its appropriateness for design? (6) What are different perspectives between designer and operator in human system interaction failures? These questions will help determine whether the ontology developed could provide answers to these questions and the answers require a particular level of details or representations. Reuse of existing ontologies was considered but unfortunately a relevant reusable ontology was not found. The ontology had to be developed from scratch. Stage 2: Knowledge elicitation: Defining testing documents Given the results of the Stage 1, the decision about what concepts and relations have to be modelled in the ontology has to be made. There are a number of knowledge elicitation methodologies e.g. an interview with domain experts or collecting documents that contain domain knowledge [11]. This research tried to collect documents (i.e. accident reports) that contain domain knowledge because it is cost- effective and can reduce human intervention. It was conducted by the first author and based on the discussion between the author and experts in design and human error at the Innovative Manufacturing Research Centre (IMRC) in the University of Bath. It has the following steps: (1) Screening for documents that contain “human-system interaction failure” by picking up cases that were caused by “operator error” (2) Analysing the screened cases in terms of design-induced error by applying theories (3) Clustering necessary concepts (e.g. error inducing design, human error) for the ontology development (4) Enumerating important terms (or phrase) by categorising terminology (keywords) that frequently appear or are used to express a concept (5) Classifying documents according to evidence (6) Selecting domain documents (testing document) that will be used for the knowledge acquisition and representation process The authors have examined a total number of 556 accident reports found until February 2005. After conducting the manual description analysis, 48 reports were selected as domain documents (i.e. testing documents) for an ontology construction. These reports were stored into a database system (i.e. Microsoft Access) for further analysis. Stage 3: Knowledge extraction This stage is to identify ontology concepts from domain documents. The Protocol Tool of PCPACK was used to analyse transcripts of accident reports chosen in the previous stage. The tool helps identify concepts as well as instances. The tool simulates the way someone would mark-up a page of text using highlighter pens. Each concept is associated with a different colour, for example, blue for design concept, red for human error as shown in Figure 3. This process was conducted simultaneously with a knowledge analysis (described in the next stage). Basic concepts, attributes, and relationships were predefined by the knowledge analysis. Terms and phrase in the domain document were extracted as instances of concepts. Figure 3. A screen shot of the PC PACK protocol tool for knowledge acquisition (mark-up) Stage 4: Knowledge Analysis: Generating concepts and relations This stage is to define the concepts and relations and organise them into a hierarchy. It is to answer the questions of: What are relating concepts located in this process? How many of them we can capture in order to formalise? (1) Defining the classes (concepts) and class hierarchy This ontology is based on the meta-theory and on accident reports. The classification of concepts and the class hierarchy are therefore categorised and defined according to how the classification and terminology effectively capture the concepts. Concepts: - Design - Operator - Designer - HumanErrorInducingDesign - HumanSystemInteractionMethod - Artefact - ProblemArea - HumanError - Theory - Accident Figure 4. A screen shot of the PC PCACK ladder tool for constructing an ontology template With these preliminary defined concept categories, the classes and the class hierarchy were constructed using PCPACK ladder tool as shown in Figure 4. Knowledge captured in the previous step, a knowledge extraction step, can be automatically put into the related concepts. (2) Defining the relationships between classes In order to define the relationship between classes, an Entity and Relation (ER) diagram was drawn with classes. This ER diagram shows how the classes are related to each other. A total number of 16 relations were defined. Relation: - HasError - IsUnsovledBy - HasPerspective - HasEffectOn - HasErrorInducingDesign - HasExplanation - IsFailedBy - IsCausedBy - HaComponent - Provide - Interact The PCPACK Ladder Tool and Diagram Template Editor were used to build concept and relation hierarchies. The ladder tool helps creating a tree-like hierarchical diagram by putting the classes into ladders. There are a concept ladder, a relation ladder, and an attribute ladder in the tool. Stage 5: Knowledge Modelling The Diagram Tool is used to create and edit diagrams. Concepts and relationships can be represented in the diagrams in the form of nodes and links. Nodes in a diagram represent knowledge objects in the knowledgebase, and links represent relationships between the knowledge objects. A diagram template determines the types of nodes and links used in a diagram. 48 accident documents cases were reconstructed with the tool as shown in Figure 5. The Annotation Tool allows a page of information to be created and edited for each knowledge object (e.g. concept, attribute, task). The user can enter text or pictures to annotate what is known about that particular knowledge object. This tool uses a hypertext (html) format, hence words can be highlighted and linked to other pages. This allows web-like knowledge-structures to be constructed that can be based on the hierarchies produced in the Ladder Tool (if desired). Templates are used to define the structure, style and contents of annotation pages. These can include special commands to automatically insert information from the knowledgebase into the annotation page. Figure 5. a diagram of an accident case (example) coustructed by the PC PACK diagram tool Stage 6: Validation and publishing of the developed knowledge This step is to help ontology developers to look back on previous stages in order to check missed, wrong concepts or relations and then to revisit previous steps in order to modify inappropriate results. Annotated documents were published on the Web sites and the concepts and relations were checked several times. This process should continue for further examples and the authors expect the ontology to be continually refined because the ontology is not exhausted. Experts in human error and user (e.g. designers) can participate in further validation process. The PCPACK Publisher Tool is used to publish the knowledgebase of design- induced error on a website as shown in Figure 6. As it was published in this way, PCPACK is no longer required to access the knowledge-base. Therefore, it is possible for the knowledgebase contents developed to be sent to other people and viewed by them without the need for PCPACK. This web browser can help to search and index relating concepts and their instances. This study provides 63 concepts, 16 relations, and more than 100 instances of the ontology in the knowledge-base. Figure 6. The published ontology browser by the PC PCACK publisher tool 4. Conclusion This paper has presented the results of developing an ontology for DIE. The ontology was developed to model the concepts and relations related to DIE explicitly. According to the ‘meta-theory’, Design and Human error are related but it is difficult to notice the relationship in texts because their relations are often described indirectly and implicitly. It means that the interpretation of design functions and features that affect human cognition and performance need to be described more clearly. This research was an extension of the previous research that developed “meta-theory of design induced error” in order to address the relations between Design and Human Error. The PCPACK used in this research is easy to use and effective in annotating texts with ontology concepts. However, it does not provide a learning capability that learns annotation patterns from examples. Manual annotation is time-consuming and error- prone, so automatic acquisition is required. We plan to evaluate the usefulness of the DIE ontology with designers. In particular, we are interested in identifying whether or not the ontology helps better understanding the relationships between Design and Human Error. Acknowledgments. The authors acknowledge the support of the Korean Government (MOL), EPSRC, and Rolls-Royce plc through the UTP for Design. References [1] Johnson C.W, "`Proving properties of accidents”, Reliability Engineering and System Safety, vol. 67, pp. 175–191, 2000. [2] Busby, J.S., Hibberd, R.E., “Mutual misconceptions between designers and operators of hazardous systems”, Research in Engineering Design, 13 (2002), 132-138. 2002. [3] McMahon, C., Lowe, A., Culley, S., “Knowledge management in engineering design: personalization and codification”, Journal of Engineering Design, vol. 15, No. 4, p307-325, 2004. [4] Gruber, T. R., “A Translation Approach to Portable Ontology Specification”, Knowledge Acquisitio”, Knowledge Acquisition, 5, 199-220, 1993. [5] Shin I J, Busby J S, Hibberd R E and McMahon C A, “A Theory-based Ontology of design induced error”, International Conference on Engineering Design, ICED 05 Melbourne, August 15-18, 2005. [6] ATSB, “Australian Aviation Accident Report System”, available at (http://www.atsb.gov.au/publications/investigation_reports/) [7] Noy, N F. and McGuinness, D L., “Ontology Development 101: A Guide to Creating Your First Ontology”, Stanford Knowledge Systems Laboratory Technical Report KSL-01-05 and Stanford Medical Informatics Technical Report SMI-2001-0880, March 2001. [8] PCPACK Manual, http://www.epistemics.co.uk/ [9] Reason, J. “Human error”. Cambridge, UK: Cambridge University Press. 1990. [10]Gruninger, M., and Fox, M.S., (1994), “The Role of Competency Questions in Enterprise Engineering”, Proceedings of the IFIP WG5.7 Workshop on Benchmarking - Theory and Practice, Trondheim, Norway. June 94. [11] Milton, N., Shadbolt, N. R., Cottam, H., and Hammersley, M. (1999), “Towards a Knowledge Technology for Knowledge Management”, International Journal of Human- Computer Studies, 53(3), 615-641.