Deep neural networks (DNNs) have good overall performance in medical imaging, but they are susceptible to obvious mistakes that violate common sense concepts. Unexplained errors have reduced trust and prevented widespread adoption in real-world clinical practice. We introduce SimpleMind, an open-source Cognitive AI software environment for medical image understanding. It uses a hybrid Neurosymbolic AI approach that integrates both DNNs and machine reasoning from a knowledge base. We demonstrate its use in building trustworthy AI for checking endotracheal tube (ETT) placement on chest X-rays (CXRs). The AI was integrated into clinical practice and the correctness of the ETT misplacement alerts were compared with radiology reports as the reference. 214 CXRs were ordered by ICU physicians to check ETT placement with AI assistance. ETT alert messages had a positive predictive value (PPV) of 42% and a negative predictive value (NPV) of 98%. Physicians indicated that they agreed with the AI outputs, had increased confidence in their decisions, and were more effective with AI assistance.
2020 Copyright for this paper by its authors. for intervention if the ETT is misplaced, ICU physicians often take a preliminary look at the CXR at the bedside and immediately adjust a misplaced tube. However, assessment of tube placement can be challenging, especially for non-radiologists.
In this paper we introduce SimpleMind, an open-source Cognitive AI software environment for
medical image understanding [
A SimpleMind application was developed to automatically identify the ETT, trachea, and carina in CXRs. In SimpleMind, an application is built by specifying a knowledge base that describes expected characteristics and relationships between image objects. To check the ETT tip placement, a “safe zone” is defined in the knowledge base as the region inside the trachea and 3 - 7 cm above the carina. SimpleMind computes this region using spatial inferencing for explainable decisions regarding ETT placement. During image understanding, SimpleMind uses the knowledge base to guide DNN segmentation agents and machine reasoning agents that evaluate the results. It enables reasoning on multiple detected objects to ensure consistency, providing cross-checking between DNN outputs. This machine reasoning improves the reliability and trustworthiness of DNNs through an interpretable model and explainable decisions. The CXR application was integrated and evaluated in the clinical imaging workflow at our institution. 2.1.
The knowledge base for a SimpleMind application is created as a semantic network (SN) [
The semantic network attributes can also represent procedural knowledge used by processing agents, including DNN architectures (e.g., U-Net, ResNet, or any user-defined architectures), learning hyper parameters, and image pre and post processing parameters. Crucially, all attribute parameters in the semantic network are exposed (separate from the processing code) and human readable, so they can be both specified by a human and auto optimized by SimpleMind. The SimpleMind environment allows a DNN agent to train weights using the above attribute parameters from a given SN node. The DNN weights are then stored with the node, embedding the DNN within the semantic network. Thus, a SimpleMind knowledge base can include both declared knowledge (that it is “taught”) and learned knowledge from examples (acquired through machine learning), i.e., we can actively teach the Cognitive AI as well as have it learn passively from data.
The CXR knowledge base is derived from the medical literature that states that the tip of the endotracheal tube should be 5 ± 2 cm above the carina, where the trachea bifurcates into the two main stem bronchi. The semantic network shown in Fig 1 includes DNNs for the trachea (trachea_cnn), carina (carina_cnn), and ETT (et_tube_1_cnn and et_tube_2_cnn). It defines a “safe zone” for the ETT tip using spatial concepts: part of the trachea: Line 3 of the et_zone_1 node (Fig 1B) 3 - 7 cm above the carina: Line 4 of the et_zone_1 node - based on the y-coordinate of the centroid (Fig 1B) ETT tip must be inside the safe zone: et_tip_correct node describes this relative to the et_zone node and represents the state of the ETT tip (Fig 1C) the ETT path must be within the trachea (and thus not going into the esophagus): et_path_incorrect node describes this relative to the trachea node and represents the state of the ETT path for the ETT position to be correct the two criteria above must be met - this requirement is defined in the et_tube_correct node which represents the final decision of the system based on its machine reasoning (Fig 1D)
The knowledge base also demonstrates checking of DNN outputs for consistency. The carina_cnn outputs a coordinate for the position of the carina, represented by the carina_1 node. The carina location can also be derived from the inferior portion of the trachea where it branches into the two main stem bronchi, represented by the carina_2 node. The carina_3 node indicates that these two should correspond and refines the final result. Accurate detection the carina is necessary for ETT position checking. Crucially, if the alternate carina locations do not correspond, then the system will report that it is unable to reliably identify the carina rather than outputting an incorrect result. Using knowledge to identify interpretation errors is an important benefit of machine reasoning that allows the system to determine when it is likely to be wrong rather than failing silently.
SimpleMind provides a Think Module for computer vision, i.e., recognizing objects (nodes) from
the knowledge base in a given image. Multiple software agents work together to segment the image into
candidate regions, then select the best candidate based on object attributes described in the knowledge
base. Software agents collaborate to solve the vision problem by reading from, and writing to, a global
Blackboard data structure [
For each node in the semantic network, a data structure called a Solution Element is created on the Blackboard, corresponding to an object to be recognized in the image. The Solution Element stores all agent contributions while recognizing the object. Knowledge base attributes and candidate image regions are transformed into a common feature space for selection of the best candidate. A Knowledge Agent accesses the knowledge base and creates each Solution Element, initializing it with prior expectations for object feature values. Thus, the objects and their relationships represented in the semantic network are transformed into a directed graph of Solution Elements on the Blackboard, with the direction of the link reflecting the dependency of an object’s attribute upon another object.
Solution Elements are processed sequentially by agents. Objects are recognized in order based on the directed links between their Solution Elements. When a particular Solution Element is scheduled for processing (by a Scheduling Agent), a Reasoning Agent computes an image search area using spatial inferencing from the relationships to previously recognized objects. This search area is provided as a mask to guide a Segmentation Agent that generates candidate image regions for the object. Segmentation is typically performed by a DNN agent that generates multiple connected components as candidate regions. Feature values are computed for each candidate region and compared against the expected values by a Reasoning Agent. Feature values are computed for each candidate according to the attributes provided in the knowledge base and the corresponding fuzzy membership function yields a confidence value for that attribute. The overall confidence for a candidate region is then computed as the minimum confidence of any attribute. Thus, by pattern classification, the candidate that best matches these expectations from the knowledge base is selected.
Agents are activated iteratively, one at a time. At each iteration an activation score is computed for each registered agent. The agent with the highest score is activated and can contribute to the solution on the Blackboard. Each agent provides a function to compute its activation score based on the contents of the Blackboard, in particular whether the Solution Element being processed has the relevant attributes and necessary data required by the agent. The process repeats until all activation scores are zero and no further agents activate. The system control is simple, yet highly flexible, with agent priorities determined through their activation functions.
The contents of the Blackboard reflect what SimpleMind is thinking at any point in time and its current understanding of the image. Once all Solution Elements have been processed, the Blackboard contains an instantiation of the general knowledge base to a particular image. The object attributes from the general knowledge base are now instantiated with actual numerical feature values from the image, enabling further high-level reasoning. 2.3.
SimpleMind provides a Learn Module for machine learning within a knowledge base, in particular
for training the weights of embedded DNNs. For the CXR AI, we used 2000 images collected
retrospectively from ICU patients between April 2018 and September 2019. All of the DNN nodes are
trained with 1488 images, and the application was initially tested experimentally on 512 images [
Although not used in this application, the SimpleMind environment also includes a Knowledge
Network Learning and Optimization (KNoLO) method. It comprehensively co-optimizes all attribute
parameters from all nodes simultaneously, including: object expected characteristics, DNN input
channels and image preprocessing options, and DNN learning hyper parameters. The parameter
optimization is performed using a genetic algorithm and details can be found in [
The CXR AI is currently deployed within our institution’s clinical workflow for investigational use only as a quality improvement (QI) tool. A new cloud-based computing infrastructure was designed to integrate the CXR AI system with the clinical Picture Archiving and Communications System (PACS). An image router was configured to push CXRs to an on-premise Azure AI/ML platform where the AI system is deployed. The CXR AI processes the image, detects tubes and anatomic landmarks on the image, and generates an enhanced CXR image with overlays and an alert/informational message that is pushed back to the PACS. Both the original and AI CXR images are available in PACS viewers for the radiologist or ICU physician. The total turnaround time from CXRs reaching PACS to the AI output being available in PACS is within 3 to 4 minutes, ensuring that AI outputs are available to ICU physicians at the point of care during their CXR review. A specific order code was set up for CXR with AI processing, providing a limited deployment on identifiable cases to be reviewed by a selected pool of ICU physicians and radiologists. From June 11, 2021 to November 3, 2022, 214 CXRs were ordered by ICU physicians through this specific order code for checking ETT placement with AI assistance.
The AI displays one of the three possible ETT messages: (1) “Found” (ETT tip was determined to
be in the safe zone), (2) “Position Alert” (ETT tip was not in the safe zone or the AI could not determine
the safe zone), (3) “Not Found” (no ETT was detected by the AI). The AI alerts were evaluated against
the findings in the radiology report in which the radiologists were asked to include the following
statement: “An investigational endotracheal tube AI overlay was available and was/was not consistent
with my interpretation”. We evaluated the AI performance by defining a positive output (alert) as
messages (2) and (3), and a negative output (no alert) as message (1). When the AI output was positive,
a true positive (TP) required that the ETT be misplaced per the radiology report or that the ETT was
missing (since cases being routed to AI were expected to have an ETT), otherwise it was a false positive
(FP). When the output was negative, a false negative (FN) required the ETT to be misplaced, otherwise
it was a true negative (TN). When alerts were issued, follow-up CXRs were reviewed and radiology
reports checked to confirm repositioning of the tube. Positive predictive value (PPV = TP/(TP+FP))
and negative predictive value (NPV=TN/(TN+FN)) metrics were computed to give a sense of
trustworthiness of the AI from a physician perspective. In previous experimental testing, the PPV, NPV,
and sensitivity to misplaced tubes were 42%, 99%, and 95%, respectively [
A survey was conducted to qualitatively evaluate the ICU physicians and radiologists experience in using the CXR AI in their clinical workflow. They were asked to provide ratings for usefulness and satisfaction with the AI clinical application.
For the 214 CXR images ordered to check ETT placement with AI assistance by ICU physicians, a confusion matrix is shown in Table 1. The AI alert messages had a positive predictive value (PPV) of 42% (21 / (21 + 29)) and a negative predictive value NPV of 98% (161 / (161 + 3)) based on the radiology reports. These performance metrics were consistent with clinical requirements and previous experimental testing.
The AI generates CXR overlays, showing the ETT path and distance from the ETT tip to the carina as shown in Figs 3 and 4. Fig 3 is a case with correct ETT position showing the internal results of SimpleMind (Fig 3b,c) that explain why it thinks the position is correct (tip inside the safe zone) and the final output of the system as presented to the ICU physician (Fig 3D). Fig 4 shows an example of incorrect ETT placement, with the tip too low relative to the carina (outside of the safe zone).
Seven clinicians completed the user survey: three were radiologists with 9 - 26 years of experience, four were physicians with 1 - 10 years of experience in critical care medicine. Five of the seven clinicians had reviewed over 20 CXRs with AI, and two had reviewed over 50. Table 2 summarizes the frequency and median ratings. Users indicated that they agreed with the AI outputs, had increased confidence in their decisions, and were more effective with AI assistance. The trust and willingness to adopt the system was further confirmed in weekly user group meetings. Usefulness and Satisfaction It helps me be more effective It helps me be more productive It works the way I want it to work I am satisfied with it
SimpleMind brings explainability and trustworthiness to ETT placement checking on CXRs using a knowledge base that describes not only the ETT but also relevant anatomic landmarks and includes relational attributes to cross-check multiple DNNs and ensure consistency and overall reliability of the system. Rather than attempting to learn misplacement of the ETT indirectly from examples, the SimpleMind knowledge base can directly describe when an alert should be given based on the tip location relative to the carina.
SimpleMind is a Cognitive AI software environment that enables users to build applications for
image understanding by specifying a knowledge base in the human-readable language of SimpleMind
and then tuning its parameters. Developing a SimpleMind application is like teaching or instructing a
human at a cognitive level, it allows non-programmers to build a medical application directly and
completely using their domain knowledge without knowing the details of the processing code. At
runtime, the knowledge base is applied to recognize objects. SimpleMind is open source
(https://gitlab.com/sm-ai-team/simplemind) and the environment can be extended through application
programming interfaces (APIs) whereby developers can expand the vocabulary and implement new
processing algorithms as agents. SimpleMind automatically handles the aggregation and chaining of
many processing agents, enabling a multi-DNN Cognitive AI system. It has also been applied to
segmentation of the kidney on CT [
SimpleMind can be considered a “hybrid learning system” that brings together features from
connectionism and symbolic AI. Four key advantages have been suggested as arising from this
combined approach [
The knowledge base makes explicit the knowledge that was previously implicit in pre and post processing code and makes it easier to apply more knowledge intuitively. The thinking of SimpleMind as it processes an image is captured in the Blackboard. A human can know what it was thinking by reviewing the Blackboard contents. Using a human-provided knowledge base, SimpleMind can perform object recognition with little or no training data. When there is insufficient data to train a DNN, other segmentation agents (e.g., intensity thresholding or edge detection) can use the knowledge base to generate initial segmentation results. Little or no training data is needed since the initial semantic network can be constructed using declarative knowledge rather than machine learning. These initial results can be used with manual editing to generate training sets for DNN learning. When an OOD situation arises it can be added to the knowledge base and handled without training data being initially available.
These benefits are also consistent with goals of trustworthy AI according to the High-Level Expert
Group on AI from the European Commission [
SimpleMind is a Neurosymbolic AI environment for medical imaging that supports DNNs with a knowledge base and machine reasoning. It was used to build an AI for checking ETT placement on CXR that was adopted and evaluated as trustworthy in real-world clinical practice. We believe that there is strong potential utility for broader research and commercial applications in building trustworthy AI. The open source software allows for knowledge base expansion and agent aggregation by a community of developers.
The authors wish to thank their friends and colleagues at the UCLA Center for Computer Vision and Imaging Biomarkers.