FixIt: An Approach towards Assisting Workers in Diagnosing Machine Malfunctions Gudrun Klinker, Hesam Najafi, Tobias Sielhorst, Fabian Sturm, Florian Echtler, Mustafa Isik, Wolfgang Wein, and Christian Trübswetter Technische Universität München Institut für Informatik Boltzmannstr. 3 85748 Garching +49 89 289 18215 {klinker, najafi, sielhorst, sturmf, echtler, isikm, wein, truebswe}@in.tum.de ABSTRACT and has been evaluated [3] or used [8] in real industrial settings. Reasons for this are manifold. On the one hand, AR involves many very hard research problems, such as Augmented Reality (AR) is a newly emerging user interface real-time tracking at very high precision and robustness, paradigm that is currently under rapid development. AR is that haven’t been solved yet. Furthermore, AR has to come still in its infancy. Only very few cases exist, in which AR out of the lab and face software engineering issues such as technology has come out of the laboratories and has been reusability, extensibility and scalability from prototypical evaluated [2] or used [8] in real industrial settings. demonstrators to deployable systems. At the Technical University of Munich, we address some of In particular, the adaptation of local tracking systems that such open questions annually in three-month laboratory are currently confined to the corner of a lab have to be classes for graduate students in their junior years. By generalized toward being usable in a real world where many posing the problems within industrial settings (e.g., by objects and people are moving about in unpredictable ways. involving an industrial sponsor) we try to ensure a realistic In this respect, the layout of a partially wearable, partially setting within which AR solutions are considered. Students ubiquitous system infrastructure that supports generating investigate the posed problem, suggest a solution and build augmented views to individual users has to be defined. a very rough, prototypical demonstrator that illustrates the key ideas. Furthermore, it is still unclear how virtual information should be structured and automatically generated to be The result is a prototypical system, illustrating the key easily available for real-time inclusion in an AR system. points of using AR in a specific problem context. At the Thus, authoring tools for AR systems need to be same time, the system design process has also laid out (and developed. Such authoring also involves live visualization partially addressed) both system building and very general of real-time control data to diagnose and repair system design issues, which will be led back onto research malfunctioning machines. agenda of the university and picked up in due time. At the Technical University of Munich, we address some of such open questions anually in three-month laboratory Introduction classes for graduate students in their junior years. By posing the problems within industrial settings (e.g., by involving an industrial sponsor) we try to ensure a realistic Augmented Reality (AR) is a newly emerging user interface setting within which AR solutions are considered. Students paradigm that is currently under rapid development. Coined investigate the posed problem, suggest a solution and build by Caudell and Mizell [1, 2] in the early nineties, the term a very rough, prototypical demonstrator that illustrates the “Augmented Reality” describes the concept of providing key ideas. users with three-dimensional virtual information embedded within their real environment.[4]. Within only one decade, The purpose of such courses is threefold: the concept has gained significant importance, resulting in - For the students: In addition to attending theoretical an annual international symposium alternating between classes on Augmented Reality, students here obtain three continents [5] and major industry-supported projects hands-on experience regarding the difficulties and [6,7]. subtleties of building a real AR application. By Yet, AR is still in its infancy. Only very few cases exist, following a rigorous software engineering approach in which AR technology has come out of the laboratories towards designing and building the system [9], students get actively involved in determining the result is an intricate new, highly immersive net of customer’s requirements for the application, laying out interactions and relationships in a man-computer-machine a generally suitable system architecture, seeking triangle. currently available solutions for required components from industry or research labs, and identifying and prioritizing open issues. System Development Process - For the customer (when available): Industrial sponsors obtain a feasibility study, analyzing options for using Arranged as one of our collaborative, team-oriented student AR technology in their envisioned scenario. Due to the lab courses, we have built a first prototype of the FixIt involvement of a rather large group of students, the system. It assists workers in diagnosing malfunctions of feasibility study tends to be rather creative and complex machines, such as robots, by superimposing the visionary, yet realistic. current control status of the machine onto the currently active machine parts – thereby indicating which - For the AR research group: By thoroughly exploring components are expected to be active at what time. the requirements posed by a particular application Discrepancies between the expected behavior and the real domain with a customer, we are able to identify robot behavior help workers understand the causes of research issues in a wider context than our laboratory malfunctions. setting – with the intention to classify them and further pursue them at a more abstract level across Students were asked to first set up an overall system design several applications. and then to form smaller teams to each address the issues involved in building one of the components. Furthermore, each course builds upon our growing software base, embedded within the Distributed As a first prototype system, a toy robot by Fischer Technik Wearable Augmented Reality Framework (DWARF) [13] was built and connected to a computer running the [10]. Thus, we are able to test and extend this base FixIt demonstration. The current control state was overlaid with every student project. on the robot by highlighting motors and sensors while they were active. Within such courses, we have been able to explore a number of problems in recent years: The authoring and augmented use of interactive electronic technical manuals (IETMS) for nuclear power plant maintenance (STARS project) [11], the AR-based visualization and evaluation of automotive designs (Fatamorgana) [12], and, most recently, the live diagnosis of malfunctions of machines (FixIt). Using a similar approach with senior graduate students after they had attended the course, we have been able to set up an AR-assisted welding system for automotive use which is now being used continuously in the early technical integration phases for new cars [8]. The FixIt Problem Every once in a while, machinery in industrial settings does not function as intended. Reasons for this are Figure 1: Real Robot with Computer manifold: a cable controlling one of the motors or sensors might be loose, thereby not conveying the correct control or status information between the machine an the control System Design system. The machine might be hindered by an unforeseen physical obstacle. Or the control program might be in an The FixIt system was designed to consist of three major unplanned state. One of the main problems in diagnosing components: the problem is to establish a mapping between the internal - A robot welding application controlling the robot. It control state of the computer and the actual physical state sends commands to the robot to move its arm forward of the machine. To this end, the maintenance person needs and backward, rotate around its base, rotate the arm up to have a clear understanding of both, as well as their and down, and perform a welding action. The same expected/actual dynamic changes over time. commands are forwarded to the visualization component. By overlaying virtual information of the control system - A tracking component, determining the current directly onto the machine while it is in operation, physical state of the robot, as well as the current Augmented Reality has the potential to help workers obtain viewing position of a camera representing the a better understanding of the reasons for malfunctions. The maintenance person. All position data is forwarded to - Suitable visualization schemes are required which the visualization component. transform control state information into minimal - A visualization component, overlaying the control augmentations – just enough to help workers state information onto the robot, according to the understand which machine parts are involved without camera’s current point of view. covering the machine up more than necessary. At the same time, visualizations for conveying the program context need to be developed, indicating to workers the reasons for malfunctions, e.g. that a motor doesn’t stop because it is waiting for a tactile sensor to return a signal – which it doesn’t return due to a malfunction. First Solutions Within the course of a three-month practical class, students have been able to implement only a rudimentary, first prototype of an augmented robot maintenance system. Using the Distributed Wearable Augmented Reality Framework (DWARF) [10] that is available in the AR research lab, the students were able to develop first versions of the three individual system components and combine them into a peer-to-peer-based system prototype to demonstrate the overall concepts. Future work will involve Figure 2: Schematic follow-up classes and student diploma theses to improve upon the individual components or add new components addressing the raised issues at greater depth. Research Issues The system requires a number of novel components and The Control Component concepts that go beyond the currently typical augmented presentation of static graphical visualizations within a real The FixIt control component focuses thus far on environment. accumulating the current system state from individual robot - The FixIt system requires that relevant information be control commands. It does not yet send program context derived from the current system context – in particular information to the visualization component. from the current control state of the machine. To this According to the physically possible motions and sensory end, control commands issued to the robot need to be evaluations of the robot at hand, the component currently accumulated and transferred into meaningful state focuses on sending the following, very specific status information. Furthermore, the dynamics and information to the visualization component: dependencies of the control program, such as conditional executions depending on sensor input, - Motor state for each of three existing motors: looping constructs waiting for an exit condition to {stopped, moving right, moving left} become true, and parallel or alternative execution, need - Status of the welding tool (a light at the tip of the to be formulated as program context primitives and robot arm): {off, on} forwarded to the visualization component. - Status of 5 switches, associated with tactile sensors on - Since both the worker and the robot move, tracking is the robot: 2 stop-switches that can be on or off, and required both for determining the current worker three click switches. position and for determining the current physical shape For testing the robot and for analyzing malfunctions, of a machine such that augmentations are actually workers can request that the tip of the robot move in any placed correctly onto individual machine parts while physically reachable three-dimensional position. The these are moving. control program internally divides the request into a As a consequence, many assumptions built into current sequence of control commands steering the individual tracking algorithms break down. In the case of a motors. marker-based system, markers have to be divided in A general authoring question with respect to generalizing different groups according to their respective mobility. this very specific communication scheme for this particular Going beyond marker-based tracking towards marker- robot is raising its head here: what would be a general less optical tracking, this means that complex syntax for an arbitrary robot control program to mappings have to be established between image communicate with a standardized visualization component? changes due to user motion and image changes due to How many, and what kinds of primitives are required? robot motions. The Tracking Component Tracking is required both for determining the current worker position and for determining the current physical shape of the machine. For their first implementation, the students used the AR-Toolkit [14] because it was easily available and provided them with a jump start. The scene was augmented with two sets of markers: - Scene-stabilized “base” markers that would not move. These markers formed the basis for determining the current viewer position. - Object-stabilized “robot” markers at the tip of the robot Figure 4: Virtual Robot arm (at the welding tool). These markers were used to determine the current robot pose. In a sense, these markers were redundant, since the object pose could The visualization component receives input both from the also be determined from the sequence of control robot control component and from the tracking component. commands issued by the control program. The Currently, it uses the robot control input to steer the discrepancy between the virtual, computable position animation of the virtual model (assuming that every issued of the robot tip and its actually measured position due command of the steering system has actually been executed to marker locations is one of the cues that students correctly by the robot). It uses the tracking input to expected to be critical to help workers diagnose determine the current worker position and to render the malfunctions. augmentations accordingly. Thus, the tool does not yet exploit the redundancy between issued control commands and visually tracked robot positions to automatically identify malfunctions. The stream of robot control commands also determines which parts of the virtual robot model are rendered transparently (i.e., are invisible to the worker), and which parts are actually rendered opaquely – and are thus pointed out to the worker. Figure 3: Marker Placement The Visualization Component Suitable visualization schemes are required to transform Figure 5: Information Flow control state information into scene augmentations. As a first requirement, the system needs to have an internal model of the real robot, animated to account for any potential robot movement. To this end, a VRML model of the Fischer Technik robot was built (see Figure 4). The model was composed from a library of basic models describing individual Fischer Technik parts. In addition to visualizing the entire robot model, the visualization component is also able to - overlay the model transparently onto a picture of the real robot (assisting the tracking state), - light up active robot parts according to recently issued robot control commands (assisting the malfunction Figure 6: Model Updates diagnosis state). Results: A Demonstration System The formalization of such program context, as well as the standardized indication of a program control state (for an Using the Distributed Wearable Augmented Reality arbitrary machine) and the visualization of discrepancies Framework (DWARF) [10], students were able develop and between the virtual and real robot state are important build a first demonstration system within one semester to visualization issues that need to be addressed in order to overlay the control state of a Fischer Technik robot onto its generate authoring systems that will be suitable as actual physical incarnation. augmented debugging aides for larger sets of robots. The system uses a camera to view the robot scene. Since Within the current three-month student project, we have the scene is carefully set up to contain a number of markers been able to lay the ground work towards exploring these on the base plate, the camera can move freely within the and other exciting issues related to an online diagnosis of scene: camera pictures are continually updated to malfunctions of machines while they are in operation. superimpose the current robot view onto the video image. According to recently issued robot control commands, Acknowledgments currently active units of the robot (e.g. some motors or tactile sensors) are highlighted to indicate their active status Many thanks go to all students who attended this class and to the worker. Such highlighting works well even when the contributed to the design and implementation of the robot moves, since the system tracks mobile robot parts demonstrator. We are also grateful to Asa MacWilliams, independently of the scene-stabilized markers on the base Martin Wagner, Martin Bauer, Christian Sandor and plate. Thomas Reicher for making the DWARF system available to the class and for continually supporting the class in using it. The entire Lehrstuhl for Applied Software Engineering (Prof. Brügge) was involved in supporting the successful execution of the lab project, providing us with needed utilities and advice – as well as the Fischer Technik robot. References 1. T.Caudell and D. Mizell. Augmented Reality: An Application of Heads-up Display Technology to Manual Manufacturing Processes. Proc. Hawaiian International Conference on System Sciences (HICSS'92), pp. 659-669. 2. A. Janin, D. Mizell, and T. Caudell. Calibration of head-mounted displays for augmented reality Discussion applications. Proc. of the Virtual Reality Annual International Symposium (VRAIS’93), 1993, pp. 246-255. The augmentation of a robot in action provides a lot of potential. This is often alluded to in machine maintenance 3. D. Curtis, D. Mizell, P. Gruenbaum, and A. and repair scenarios. Janin. Several Devils in the Details: Making an AR App Work in the Airplane Factory. Proc. 1st Yet, the actual consequences of implementing a system International Workshop on Augmented Reality within which physical components really do move (IWAR’98), San Francisco, Nov. 1998, pp. 47-60. according to a control program and thus need to be followed, have not yet been explored much. 4. R.T. Azuma. A survey of augmented reality. Presence, Special Issue on Augmented Reality, Especially in the case of diagnosing malfunctions, there is 6(4):355-385, August 1997. an interesting discrepancy between the internal (virtual) state of a machine and its real (physical) state. If due care is 5. International Workshop/Symposium on being taken to formalize this discrepancy and to determine (Mixed)Augmented Reality (IWAR, ISAR, it from physical measurements (tracking data) and control ISMAR), 1998-2003. status information, it can form the basis for very powerful 6. Friedrichs W (2002) ARVIKA Augmented Reality diagnosis aids for machine repair personnel. for Development, Production, and Service. In: Yet, the discrepancy alone will not suffice. It needs to be Proc Int Symp on Mixed and Augmented Reality supplemented with program context data which indicates (ISMAR’02), Darmstadt, Germany, IEEE. See what the robot control program is trying to achieve while also: http://www.arvika.de/www/pdf/IVIP Forum issuing a certain sequence of robot control commands. e.pdf. 7. Tamura H (2000) What happens at the border 11. G. Klinker, O. Creighton, A. Dutoit, R. between real and virtual worlds – the MR project Kobylinski, C. Vilsmeier and B. Brügge: and other research activities in Japan. In: Proc Int Augmented maintenance of powerplants: A Symp on Augmented Reality (ISAR’00), Munich, prototyping case study of a mobile AR system. IEEE. Proc. IEEE International Symposium on 8. F. Echtler, F. Sturm, K. Kindermann, G. Klinker, Augmented Reality (ISAR'01), New York, 2001. J. Stillla, J. Trilk, and H. Najafi. (2003). The 12. G. Klinker, A. Dutoit, M. Bauer, J. Bayer, V. Intelligent Welding Gun: Augmented Reality for Novak and D. Matzke, “Fata Morgana -- A Experimental Vehicle Construction. In: Virtual Presentation System for Product Design”, Proc. and Augmented Reality Applications in IEEE International Symposium on Mixed and Manufacturing. S.K. Ong and A.Y.C. Nee, eds. Augmented Reality (ISMAR'02), Darmstadt, Springer Verlag 2004. 2002. 9. B. Bruegge and A. Dutoit. Model-based Software 13. Fischer Technik. http://www.fischertechnik.de/ Engineering: A Project-Oriented Approach, 14. H. Kato, M. Billinghurst, I. Puopyrev, K. Prentice Hall 1999. Imamoto and K. Tachibana. Virtual Object 10. M. Bauer, B. Brügge, G. Klinker, A. Manipulation on a Table-Top AR Environment. MacWilliams, T. Reicher, S. Riß, C. Sandor, and Proc. 1rst International Symposium on Augmented M. Wagner. Design of a Component-Based Reality (ISAR’00), Munich, Oct. 2000, pp. 111- Augmented Reality Framework. ISAR ’01. 119.