Surgeons receive training in diferent skills. One of them are procedural skills, which are key to perform surgical operations. However, there is no standardized and structured way in medical education to teach the sequence of steps of surgical procedures, a key aspect of procedural skills. In order to support procedural skills instructors in teaching the sequence of steps, we developed diferent artifacts to quantify the errors that residents make in simulation-based courses. With this, instructors can incorporate the sequence of steps as a learning objective, allowing residents to advance from a novice to an advanced novice level of proficiency. Using percutaneous tracheostomy and central venous catheterization procedures as case studies, we developed an instrument for instructors, a sequence of steps learning curve, and metrics to describe resident performance. After systematically implementing the results of this thesis in procedural skills simulation courses, it is expected that the residents will pass from the novice level of competence to the advanced novice level, in order to positively impact the health of the population.
Surgeons receive training in many skills. One of them are procedural skills, which allow doctors
to perform surgical procedures [
Process mining is a discipline coming from the business process management, whose
algorithms allow to generate insights to support the processes that are executed in organizations,
using as data source the footprints they leave in the information systems [
In the literature it has been seen that a surgical procedure can be understood as a process [
Additionally, surgical procedure instructors need information on the performance of residents regarding the sequence of steps, to support them throughout the teaching-learning process [8]. In particular, Aydin et al. [9] proposed a cycle of five stages (see Figure 1), focused on the teaching of procedural skills. We adopted such cycle in this thesis to understand the sequence of steps of residents performing surgical procedures.
This thesis seeks to provide tools to facilitate the educational task of procedural skills instructors, helping them to incorporate the sequence of steps as a learning objective. In particular, the focus is on residents who are moving from a novice level of proficiency (needs direct supervision, views procedure steps in isolation) to advanced novice (performs routine tasks with indirect supervision, views procedure as a sequence of steps) [10]. The POME (Process-Oriented Medical Education) approach is proposed for teaching the sequence of steps, which is made up of POME artifacts developed at the intersection between medicine, education and engineering. The proposed POME artifacts are the contributions presented below, which together allow the teaching of the sequence of steps with solutions that consider the process perspective of procedural skills. The case studies used in the thesis are the installation of the central venous catheter guided by ultrasonography and the percutaneous tracheostomy guided by bronchoscope.
The thesis shows five contributions:
The first contribution [ 11] is the identification of studies that explicitly report the incorporation of the sequence of steps in teaching strategies and evaluation of procedural skills, through a systematic review of the literature. Recommendations are also proposed to include the sequence of steps in procedural skills training. 4326 articles were reviewed, of which only 9 met the inclusion criteria. The results show that the most common teaching strategies are videos that show the procedure and the delivery of immediate feedback on the sequence of steps. To assess the learning of the sequence of steps, it was found that adherence to a predefined order and the omission of steps are usually measured with diferent types of scales, all of which are diferent from each other. For this reason, it is concluded that the sequence of steps is rarely mentioned in studies of teaching and evaluation of procedural skills, for which the use of process models and the use of process mining algorithms are proposed to include the sequence of steps. steps as a specific training goal.
The second contribution [12] is a method to develop POME artifacts. The method consists of three stages: (1) a process model is designed after a bibliographic review, in which the steps that a surgical procedure must have are identified, and then changes are made to the process model based on the opinion of the experts involved. Through a Delphi panel, a consensus is reached on the ideal process model; (2) videos of the residents performing the surgical procedure are recorded, to later label each video with the steps performed by the surgeon based on the consensus process model generated in the previous stage, and thus obtain the event logs with the sequence steps taken by each resident; (3) the process mining analysis is done considering the stage of the procedural skills teaching cycle in which the instructor needs support.
The third contribution [13] is a POME instrument to identify deficiencies in the sequence of steps learning. The instrument provides instructors with information about deficiencies committed after a training session, which can be useful for making changes to the teaching strategy and identifying parts that need to be reinforced. A taxonomy was generated to classify the steps of ultrasonography-guided central venous catheter installation, and based on this classification, an instrument for instructors was generated. Both developments were validated by experts (physicians with experience in the instruction and execution of this procedure), who found it useful for their educational tasks. Both POME artifacts (the taxonomy and the instrument) were generated based on the data of ten residents in training. In addition, the artifacts were validated with three instructors who have 5.7 years of experience teaching the procedure through a test to assess their understanding of the information contained in the instrument, a usability test, and the collection of instructor feedback on both artifacts. The results showed that the instructors understand the information presented in the instrument, that it had an acceptable level of usability, and that the instructors considered that both the taxonomy and the instrument were useful for the educational task.
The fourth contribution [14] is a learning curve of the sequence of steps to know the performance of the residents throughout the training. The curve was constructed based on data from 8 residents who participated in a 7-session training course to learn the percutaneous tracheostomy procedure. A similarity metric based on Levenshtein’s normalized distance was used to compare the sequence of steps performed by the residents with the ideal sequence proposed in the process model. The results showed that the residents improve their learning as the sessions progress and the curve reaches a plateau at the fith session. It was also found that, when analyzing each stage of the procedure separately, there are stages that the residents learned well and others that need to be reinforced at the end of the course (tracheal puncture and tracheal dilation stages).
The fith contribution [ 15] corresponds to POME metrics to determine the performance of its residents throughout the course in a more detailed way. Using data from eight residents in a 7-session percutaneous tracheostomy course, the number of omissions, deviations, and repetitions of activities was determined, making a detailed analysis by stage and by activity. The results showed that the errors decreased as the course progressed when the procedure was analyzed as a whole. However, the analysis by stages showed that only one stage (preparation) did not present errors at the end of the training. The POME metrics were also validated with classic medical education metrics (time and global scales) through the calculation of correlations, obtaining statistically significant and confirmatory values of the expected behavior. In particular, there was a positive correlation with total time duration, and a negative correlation with global rating scores.
After developing POME artifacts and shaping the POME approach, three statements can be concluded. First, the POME approach provides useful artifacts to teach the sequence of steps. Second, developing POME devices requires process knowledge, but also medical sense. Third, residents do not perform the procedural skills used in the case study the way they should, even at the end of the course. As future work, it is proposed to build and validate POME artifacts for the training, evaluation, and feedback stages, as well as to demonstrate the impact of the use of POME artifacts on residents’ learning. [6] R. Lira, J. Salas-Morales, L. Leiva, R. D. Fuente, R. Fuentes, A. Delfino, C. H. Nazal, M. Sepúlveda, M. Arias, V. Herskovic, J. Munoz-Gama, Process-Oriented Feedback through Process Mining for Surgical Procedures in Medical Training: The Ultrasound-Guided Central Venous Catheter Placement Case, International Journal of Environmental Research and Public Health 16 (2019). [7] R. de la Fuente, R. Fuentes, J. Munoz-Gama, J. Dagnino, M. Sepúlveda, Delphi Method to Achieve Clinical Consensus for a BPMN Representation of the Central Venous Access Placement for Training Purposes, International Journal of Environmental Research and Public Health 17 (2020). [8] A. Bogarín, R. Cerezo, C. Romero, A survey on educational process mining, WIREs Data
Mining and Knowledge Discovery 8 (2018) e1230. [9] A. Aydin, R. Fisher, M. S. Khan, P. Dasgupta, K. Ahmed, Training, assessment and accreditation in surgery, Postgraduate Medical Journal 93 (2017) 441. [10] K. Khan, S. Ramachandran, Conceptual framework for performance assessment: Competency, competence and performance in the context of assessments in healthcare – deciphering the terminology, Medical Teacher 34 (2012) 920–928. [11] V. Galvez-Yanjari, R. de la Fuente, J. Munoz-Gama, M. Sepúlveda, The sequence of steps: A key concept missing in surgical training—a systematic review and recommendations to include it, International Journal of Environmental Research and Public Health 20 (2023) 1436. [12] J. Munoz-Gama, V. Galvez, R. de la Fuente, M. Sepúlveda, R. Fuentes, Interactive Process Mining for Medical Training, in: Interactive Process Mining in Healthcare, 2021, pp. 233–242. [13] V. Galvez, R. de la Fuente, C. Meneses, L. Leiva, G. Fagalde, V. Herskovic, R. Fuentes, J. Munoz-Gama, M. Sepúlveda, Process-Oriented Instrument and Taxonomy for Teaching Surgical Procedures in Medical Training: The Ultrasound-Guided Insertion of Central Venous Catheter, International Journal of Environmental Research and Public Health 17 (2020). [14] V. Galvez-Yanjari, E. Kattan, R. de la Fuente, C. Kychenthal, J. Munoz-Gama, M. Sepúlveda, Percutaneous Dilatational Tracheostomy Training Progress from a Control-Flow Perspective, To be Published (2023). [15] J. J. Martínez, V. Galvez-Yanjari, R. de la Fuente, C. Kychenthal, E. Kattan, S. Bravo, J. MunozGama, M. Sepúlveda, Process-oriented metrics to provide feedback and assess the performance of students who are learning surgical procedures: The Percutaneous Dilatational Tracheostomy case, Medical Teacher 11 (2022) 1–9.