Fracture detection has been a long-standing paradigm on the medical imaging community. Many algorithms and systems have been presented to accurately detect and classify images in terms of the presence and absence of fractures in di erent parts of the body. While these solutions are capable of obtaining results which even surpass human scores, few e orts have been dedicated to evaluate how these systems can be embedded in the clinicians and radiologists working pipeline. Moreover, the reports that are included with the radiography could also provide key information regarding the nature and the severity of the fracture. In this paper, we present our rst ndings towards assessing how computer vision, natural language processing and other systems could be correctly embedded in the clinicians' pathway to better aid on the fracture detection task. We present some initial experimental results using publicly available fracture datasets along with a handful of data provided by the National Healthcare System from the United Kingdom in a research initiative call. Results show that there is a high likelihood of applying transfer learning from di erent existing and pre-trained models to the new records provided in the challenge, and that there are various ways in which these techniques can be embedded along the clinicians' pathway.
In recent years, fracture detection has been one of the most
cited challenges in medical imaging analysis, evidenced both
by public competitions [
In addition, a key contribution of this work is the modelling of the clinician's pathway, exploring the current process of radiology imaging for fracture treatment. This was built through a series of co-creation sessions with a reporting radiographer, then veri ed by two clinicians and two other reporting radiographers. By designing this pathway, we identi ed three key stakeholders (clinician, radiologist, patient) and four sub-processes: (1) requesting radiology images; (2) acquiring radiology images; (3) reporting on radiology images; and (4) decision support. By understanding the current approach to radiology imaging for fracture treatment, we consider that we are capable of o ering a valuable contribution that will allow research groups to pinpoint opportunities for smart automation of this process in future.
The contributions of this paper are as follows: 1. Identify the current processes involved with the various procedures a ected by radiology imaging for fractures. 2. Explore where these processes can be improved through the implementation of AI. 3. Identify what form of AI would be most applicable in order to maximise the obtained bene ts by the a ected stakeholder. 4. Given the limited amount of samples provided by the challenge proposer, perform initial proof of concept tests using baseline methods to identify the potential of transfer learning in this domain. 2 2.1
Only a handful of demonstrations of machine learning,
computer vision and natural language processing for bone
fracture detection appear in scienti c literature. Lindsey et al. [
Three technical frames have been described as applicable
for ML in radiology: image segmentation, registration and
computer-aided detection and diagnosis [
Modelling a Clinician's Pathway Any arti cially intelligent solution which is suggested to resolve some problem within the eld of radiology should be rooted in deep understanding of the domain and user requirements. With this in mind, we have received input from domain-experts to develop a clinician's pathway, detailing current process of radiology imaging for fracture treatment. In this paper, we aim to demonstrate the opportunities which offer potential for smart automation in this eld. In particular, we aim to highlight the areas where the application of arti cial intelligence would be impactful for increasing e ciency, improving patient experience and decreasing cost. 3.1
Co-Creation of Clinician's Pathway We performed a multi-stage co-creation process to model the clinician's pathway which was indicative of real-world radiology practice. These stages were divided into a design phase, where we aimed to understand and model the current processes for the acquisition and reporting of radiology images, and a validation phase, in which we obtained unbiased feedback on our model from personnel external to our co-creation. The result is a clinician's work ow which has been developed alongside a reporting radiographer, and veri ed by two clinicians (one consultant radiologist and one senior accident & emergency doctor) and two reporting radiographers from within the National Health Service (NHS) Scotland. We are therefore con dent in its accuracy and its suitability to describe real radiology processes. Although this pathway has been built with input from British radiologists, we suggest it can be generalised to wider radiology practices (within reason).
During the design phase, we organised two separate cocreation sessions. In the rst session, we met with a reporting radiographer to discuss the complete journey of a patient who was given an appointment for radiology imaging for a suspected fracture. This session was useful to establish the process start-points and end-points. In the second session, we observed a reporting radiographer reporting on a series of xray images for suspected fractures. This session was intended to identify relevant technologies and the role they played in reporting on radiology images. The outcome of this two stage design phase was an initial draft of the clinician's pathway which could be validated by domain experts.
We then organised three sessions for the validation of the developed work ow. In the rst session, we obtained feedback upon the pathway from the reporting radiographer who was directly involved in its formation. This allowed any errors which had arisen due to misunderstanding aspects of the design phase to be corrected. In the second session, a member of the research team met with a clinician and a reporting radiographer to explain and discuss the draft pathway and obtain feedback. This session was designed to ensure that the pathway could be generalised to more than just the single radiographer with whom it had been co-designed. In particular, the session resulted in a number of updates to the role of the clinician as an actor in the process. Finally, we used the third session as an opportunity to obtain blind feedback on the developed pathway as a form of litmus test regarding its accuracy to radiology practice. We presented the pathway to a new clinician and reporting radiographer, and requested feedback on any areas where they felt (a) that the pathway was not indicative of real-world practice and (b) that there were opportunities for arti cial intelligence to make the process more e cient.
The results of the validation sessions were very valuable for the design process. As a methodology, by performing our co-creation of the clinician's pathway in this manner, we are con dent it is accurate to real-world practice, and generalisable beyond simply an individual's viewpoint. In the nal validation session, the clinician did not highlight any areas of the work ow which were not indicative of real-world practice, while the reporting radiologist suggested only a minor amendment to terminology. Furthermore, the areas which both of the participants suggested were suitable for arti cial intelligence to make a process improvement very closely overlapped with our own ndings as researchers. We discuss this in more detail in Section 3.3. In the following subsection, we will introduce and discuss the developed clinician's pathway in detail. 3.2
Resulting Clinician's Pathway In modelling the process of radiology imaging for fracture treatment, we identi ed three key stakeholders (clinician, radiologist, patient) and four sub-processes: (1) requesting radiology images; (2) acquiring radiology images; (3) reporting on radiology images; and (4) decision support. The complete gure can be accessed via this link4 and can be seen in Figure 1. We summarise our pathway using a work ow diagram which we will break down into respective processes in Figures 2, 3 and 4. 4 https://www.dropbox.com/s/3gx7bicf43bn0lx/wrk w comp c. pdf?dl=0 s e g a m I y g o l o i d a R g n i t s e u q e R s e g a m I y g o l o i d a R g n i r i u q c A s e g a m I y g o l o i d a R n o g n i t r o p e R t r o p p u S n o i s i c e D
information entered and scheduled on RIS and ordering info sent to PACs.
On day of appointment
(or attendance)
commences appointment.
1. Requesting Radiology Images: When requesting radiography images for a patient, a clinician sends an imaging request to the radiology department. This request contains information on the patient's demographic (including a medical history), information about the clinician making the request and the examinations requested. It allows an appointment to be scheduled on the Radiology Information System (RIS), and the ordering information is dispatched to Picture Archiving and Communications System (PACS). PACS is an end-to-end system that supports the process of acquiring radiography images of a patient from referral of the patient until diagnosis and subsequent treatment are agreed. Contained within PACS is the RIS, where the textual components of patient information is stored.
2. Acquiring Radiology Images: Each day, the RIS automatically generates a work list for each imaging modality. This work list gives the radiologist (or the reporting radiographer) access to the request information created by the original referring clinician, helping them to understand the imaging requirements. This enables the radiologist to perform the requested imaging. After the imaging has been performed, the medical equipment will generate images in Digital Imaging and Communication in Medicine (DICOM) standard, and load them into a graphic user interface where they are made available for the radiologist to annotate and report. A graphical representation of this is displayed in Figure 2. DICOM de nes an image standard and format for medical images. Images are high resolution and are linked directly with other patient data (such as name, gender and age). It was developed by the National Electrical Manufacturers Association (NEMA) as part of a set of standards that de ne best practice and inform the international standard for the capture, retrieval, storage and transmission of medical imaging data. DICOM is currently the most commonly used standard across the world for medical imaging, and is implemented in most radiology, cardiology and radiotherapy devices, as well as devices in other medical domains such as dentistry.
3. Reporting on Radiology Images: Radiologists can access these images by calling up a list of unreported examinations. For each patient, previous images and reports are also available to support diagnosis. PACS enables radiologists to annotate the images to highlight areas of interest or identify supporting evidence for their diagnosis. These annotations include the ability to perform simple measurements (length of objects, angles of intersections, etc) and to mark a Region of Interest (ROI) on the image. This allows the radiologist to use tools to capture metadata about the ROI, including its area, average pixel values, standard deviation, and range of pixel values.
The radiologist will then generate a textual report to summarise and describe their ndings. These reports have no set template or length, but generally include a statement of whether a fracture has been detected, what type of fracture it is, where it is located, and the seriousness of the breakage. Furthermore, the reports may be appended to existing documentation on the patient (if previous radiology records exist) or may be used to begin a radiology record (if no previous visits have been recorded). Many countries then require the reports to be authorised by a radiologist before being released to a clinician. For example, within the United Kingdom the standards for fail-safe communication of radiology reports are governed by the Royal College of Radiologists (RCR)5. The result of these factors is that the reports are a complex textual data source with limited uniformity and describing a broad range of diagnosis and observations. We represent this information as part of the work ow displayed in Figure 3.
4. Decision Support In the existing pathway, decision 5 http://bit.ly/rcr-standards support occurs after the radiology reports are generated. This is non-optimal; often for accident and emergency fracture cases (which make up the majority of fractures in a hospital) the clinician will attempt to read and comprehend the generated radiology images without any input from an expert radiologist. This can be seen in the current work ow in Figures 2 and 4. This occurs because experts can be unavailable - not all radiology sta are su ciently trained to report on acquired images. As a result the clinician is forced to make a diagnosis and organise follow up treatment on the basis of their individual knowledge. This can lead to misdiagnosis, if the clinician's ndings are not consistent with the radiologist's, which can have an impact on the patient's health as well as nancial consequences for the hospital involved.
Having developed an understanding of the current procedure of radiology imaging for fracture treatment, we are motivated to make some recommendations where arti cial intelligence could make improvements to this process. 3.3
Opportunities for Arti cial Intelligence The key outcome of this work is highlighting the applicability of arti cial intelligence in two places: to reduce burden on radiologists by (1) autonomously classifying radiology images and (2) generating understandable and accurate medical reports to describe the intelligent system's ndings. Applications of arti cial intelligence to ll these gaps presents an opportunity to improve decision-support for clinicians by giving them access to the information immediately. This is a key factor that is missing from much of the research literature on this topic; although an arti cial intelligence method for fracture recognition should enhance the e ciency of radiologists, it should also improve the decision-making of clinicians. Therefore, it should be of a suitable form to be absorbed by that user group.
This outcome is supported by the veri cation obtained from our test group of clinicians and radiologists. Based on their feedback, we have highlighted the most impacted area of the current clinical pathway in Figure 5.
As seen in our discussion of related work, there has been
much exploration of autonomous classi cation of fracture
images throughout the literature [
Experiments on Image Classi cation The main purpose of the experimental framework was to test the learning capabilities of di erent baseline algorithms and settings to classify the images provided in the challenge as fracture/no fracture. To do so, the rst task consisted of having a specialist re-annotate the data provided by splitting it into fracture and no fracture labels, based both on the visual aspect of the radiography and on the information provided by the text reports. It was discovered that while most of the
Start
Patient
Report sent to t clinician. images corresponrded to "regular" scenarios (where the o p No pose is to assess wphether the patient has su ered a fracture or u New report not), some other ScasesRaepRloesrptosertnct otontained fFoolllolwouwp -up reports (ide nap-pended tofracture) ti ed as POP) wiiscnotrpohere dtmccilhaliainngtineicochoiiaaesnnisis.ssueNo is n oaoprtgpaonintistoemdetnoitdentify ence/absence of aeu fractur?e, but rathceorrrect treagtmievnet a follow p to DiiscSeno YedmcsiRlaiaengtpi?nccohoiarestniss No coraorperFgpocaotllniotnirwsetemadutepmtnoetnt D ow Nfoor reporting on radiology images.
Nerewporertp?ort appended to puexris-ting reportas. patient
which already has had the fracture identi ed in a previous visit. We labelled 73 images as positive (i.e. with and 138 negative (i.e. no fracture). Moreover, six existing reports. the pres- examples were POP fractures, and thus were not included in up for our experiments.
Yes End
End To demonstrate the potential of transfer learning capabilities of the selected algorithms towards the NHS provided records, we used the following publicly available dataset.
MURA: The MURA (MUsculoskeletal RAdiographs) dataset is a large dataset of bone X-rays. Algorithms are tasked with determining whether an X-ray study is normal or abnormal. It consists of X-ray scans on elbow, nger, forearm, hand, humerus, shoulder and wrist. The training set consists of 14'873 positive cases and 21'939 negative cases while the validation set has 1'530 positive examples and 1'667 negative examples. Among them, forearm and wrist are close to our problem, which consists of 7'443 negative and 5'094 positive examples. Images from this dataset can be accessed here6. 6 https://stanfordmlgroup.github.io/competitions/mura/ 4.2
To increase the likelihood of classi cation and the training
sample size, We applied the following preprocessing
techniques, which are the most commonly used in related
literature [
These followings baseline architectures were used in our
experiments:
A VGG 16 [
B Resnet 50 [
To have di erent points of comparison, we tested the results
of using the three aforementioned classi ers to classify
images from the MURA dataset. We distinguished between the
following four con gurations:
1. When the networks were pre-trained with ImageNet [
The accuracy result can be seen in Table 4.4 and the run time in Table 4.4:
VGG16 Resnet50 InceptionV3 regardless of its origin. Moreover, case 2 with these same architectures also showed good performance, (79% and 78% respectively), but even with the retraining on wrist and arm images, results were slightly worse than training only with ImageNet images. This may be due to the fact that some wrist/arm images had to be used for such retraining instead of testing. In terms of run time, we also found out that case 1 overall is faster to train and test.
After this initial validation, we tested the transfer learning capability from MURA to the newly acquired images. We tested the following three cases: 5. When networks were pre-trained on ImageNet, trained on
MURA and tested on the new dataset. 6. Mixing MURA and new images to generate both training and test sets (70% train, 30% test). 7. Same as the previous case, however the test set was composed of 70% of MURA images and 30% from the new dataset.
The accuracy results are shown in Table 4.4:
In contrast to what was expected from the previous test, we observed that for case 5, all CNNs were unable to learn how to classify the new images. In contrast, it was more likely to obtain higher accuracy rates for case 6 and VGG 16 (81%), although this is a direct result of images from the MURA dataset being mixed within the test set. Meanwhile, case 7 and Inception V3 obtained 75% accuracy, but keeping in mind that the test set is only composed of new images, this was a clear indication that it is possible to transfer a model using a larger amount of images. In terms of run time, we discovered that it was faster to train networks through case 6, followed by case 6 and case 7 respectively. The complete run time results can be seen in Table 4.4.
The results showed that case 1 with VGG 16 and ResNet 50 delivered the best accuracy overall (82% and 80% respectively), implying that it is possible to obtain good accuracy provided that we can train the systems with su cient data, 5
In this paper, we have presented a rst step towards assessing the most proper way to embed machine learning, computer vision and natural language processing into the clinicians' pathway to improve assisted diagnostics of fracture detection. We have reviewed the most signi cant literature and designed a pipeline where we have annotated the most relevant action points where arti cial intelligence can be used to improve the current practices. In addition, we have carried out some initial experiments to verify how current methods and transfer learning perform on identifying fractures in a reduced dataset provided by the British public health service. Results show that there is a great likelihood of being able to apply transfer learning for these purposes, and in the case that more images are provided by the challenge setter, then the accuracy can vastly improve.
We will continue this partnership to explore more ways in which we can further improve our ndings and including other technologies to enhance the existing results. Finally, we will keep working with clinicians and radiographers to correctly assess their pathways and e ectively applying these technologies in commercial settings.
We would like to acknowledge the Scottish Government's Small Business Research Initiative (SBRI) and Opportunity North East (ONE) for supporting this work, and to the Data Safe Haven (DaSH) from the National Health Service (NHS) for granting access to the test data.