What if the trust in the output of a predictive model could be acted upon in richer ways than a simple binary decision of accept or reject? Designing assistive AI tools for medical specialists entails supporting a complex but safety-critical decision process. It is common that decisions in this domain can be decomposed to a combination of many smaller decisions. In this paper, we present Verification Staircase - a design strategy that can be used for such scenarios. The verification staircase is when multiple interactive assistive tools are combined to allow for a nuanced amount of automation to aid the user. This can support a wide range of prediction quality scenarios, spanning from unproblematic minor mistakes to misleading major failures. By presenting the information in a hierarchical way, the user is able to learn how underlying predictions are connected to overall case predictions, and over time, calibrate their trust so that they can choose the appropriate level of automatic support.
• Human-centered computing → Human computer interaction (HCI); Interface design prototyping; User interface design; User centered design; • Computing methodologies → Machine learning; • Social and professional topics → Automation.
human-in-the-loop systems, human-ML collaboration, explanations, interaction design ∗Also with Sectra AB.
In our suggested design strategy, multiple characteristics combine to enable answers to such questions, including inthe-loop correction, decomposition to allow explanations through causal inference and designing to aford use with both high performing predictions as well as border-case accuracies.
Our insights are from ongoing human-centered design
explorations. The presented perspective is rooted in our
experience as UX practitioners within the field of digital
pathology, with a strong emphasis on practical relevance.
Typically, the goal of our design efort is to make systems
where the resulting value is co-created between artifacts
and humans in the context of use [
The layout of this paper is as follows; first we present and motivate the strategy of verification staircase. Second, we illustrate the concept by an explorative design case for assisted quantification in digital pathology. Finally, we discuss our concept in the context of explainable intelligent user interfaces and outline our proposed continuation of the research.
Consider a predictive model trained to assess whether a patient is eligible to receive some cancer-inhibiting drug. In the context of digital pathology, where tissues are viewed at high magnification, the result might be visualized in the context of the area of interest as depicted in Figure 1.
PD-L1 Positivity: 3%
For such an interaction, the user is supposed to look at the visualization and if everything looks fine, accept the overall result. An appropriate strategy might be to trust and accept the result if the underlying accuracy is good enough for this particular case and reject it otherwise. If the user rejects the result, they will need to resort to performing the task manually. If the user interface afords no other means of judging the underlying accuracy than the manual approach, chances are that unless there exist very strong guarantees that the model performs well on all possible cases, they will always reject the result and be forced to perform their manual method.
We call this kind of human-ML interaction a verification clif, as depicted in Figure 2
What if there instead were multiple levels at which
humanML collaboration could be performed? Having modes of
human operation corresponding to nuanced levels of control
have long been recognized as important factors for
interaction with automation [
We argue that the performance characteristics of many ML applications make them suitable for splitting collaboration into several levels, in a similar manner to the hierarchies of
ecological interface design [
Many diagnostics tasks within pathology can be divided into multiple sub tasks, e.g. an overall case-level score is derived from a formula combining the detection and classification of many individual cells. Consequently, it is possible to measure the accuracy per diagnostic case. When predictive algorithms are evaluated, it is common that an overall accuracy across cases in the form of an AUC, F1-score or Cohen’s kappa is presented. However, in a scenario with case-level sub tasks, we can also characterize the distribution of per case accuracies over a large number of cases, see Figure 3
Accuracy distribution per case seHigh csoaebuN fr m Low 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00
F1 Score
The shape that is seen in the figure is typical and has been observed for many applications in our research. There is usually a peak in the distribution corresponding to the average accuracy and then a long tail of cases, with some cases almost always completely failing.
A better design strategy would be to think about how we can help our users when predictions fall within diferent intervals on the distribution. It can in many cases be possible to divide the design into multiple interactions, such as: (1) A good result visualization that can be used to quickly verify predictions on the 0.9-1.0 span (2) A correction tool for small modifications of predictions that updates the overall result on the 0.7-0.9 span (3) A semi-automatic aid not even based on the original predictions on the 0.4-0.7 span etc.
This way we could attempt to create multiple user interfaces aimed at helping the user when predictions happen to fall in diferent positions on the accuracy distribution.
The decision of whether to trust or not trust the prediction would now be a question of degree - the placed trust could guide the choice to an interaction with an appropriate level of automatic support. The question then becomes: How would the user learn in which level to place their trust?
We suggest that requiring that levels are connected, correctable and composable together with visualizations that make errors apparent, could be enough. In such a design, users should be able to dynamically move between interaction levels and perform corrections. Actions at one level should immediately be reflected in the others. We argue that this combination of actionable and composable levels will enable users to calibrate their trust over time, through learning to correlate top-level observations with the suitable amount of drill-down behavior. We call this strategy a verification staircase, as depicted in Figure 4.
In the following part of this paper we will describe an ongoing case study where we have instantiated this design strategy for a tool that aids quantification in digital pathology.
We followed an iterative user-centered design (UCD) methodology combining sketching, high fidelity (hi-fi) prototyping, data collection, model debugging, user observations and interviews. Pathologists and clinical experts were consulted throughout the process. Compared to traditional UCD, we used hi-fi prototyping earlier and more frequently. This is motivated by the dificulty of eliciting how the predictive output will be experienced and behave through sketches and other low fidelity methods. Our account of the design process selectively highlights those insights we believe are important for appropriation and adaptation of the concept of verification staircase to other domains. tr High o p p u s c it a m o t u a lf o e v e L Low Few s e s a c fr o e b m uNMany
The assisted quantification task targeted in our case study is to determine the ratio of two types of cells. Some cancers hide from the immune system by a kind-of cloaking mechanism and can efectively be treated by disabling the cancerous cells’ ability to do this. However, not all cancers hide by this mechanism. In order to determine whether a patient shall receive this expensive treatment, cells are stained such that the cell membrane of cells having the cloaking ability becomes brown. According to the diagnostic protocol, for treatment to be efective more than 50% of the cancerous cells in the tissue should have a stained membrane. If the tissue has more than 1% stained cells, the treatment might be efective. If stained cells are below 1%, the treatment will likely not work, and the patient should not be ofered the treatment.
Thus, the diagnostic decision is based on estimating or counting this ratio in a possibly large tissue area. This task can be time-demanding and error-prone. Pathologists can use two basic strategies; they can look at the overall impression of the image and use their experience and tacit knowledge to “intuitively” determine the percentage right away. This is a very fast decision but can be error-prone. The second strategy involves manually counting tumor cells both with and without stained membrane, and then deriving the ratio of the two. All things being equal, this second method will result in a more accurate decision but is orders of magnitude more time-demanding. As a middle ground, pathologists sometimes choose a much smaller area as a “representative sample”, and only count within that area.
A machine learning-based predictive model has the potential to always use the second strategy, classifying at the cell level and reporting the exact ratio deriving from the two counts.
We interviewed and observed the working processes of pathologists performing the task manually. We also reviewed the available diagnostic protocols, where available. We collected and manually annotated cases and then trained a convolutional deep neural network to perform the predictions.
In one possible interaction, the user can delineate an area and receive the final result of the model as a percentage, as was depicted in Figure 1. The type of this interaction is the verification clif – the user has two options; either they accept the result blindly or they reject it and perform their usual manual procedure. Based on the notions of a verification staircase, we sought another interaction where, if the user does not accept the top-most level of automation, they could step down to a lower level of automatic support, that is still easier and faster than manual work.
We designed our first intermediate level for the case when most cells have received the correct classification, but a few need to be corrected for a satisfactory overall result. In the devised interface, the user can explore the top-level prediction by viewing and verifying a systematic subset of decisions of the underlying model, as depicted in Figure 5.
At this level, the user is presented with a gallery of patches, sampled in a systematic spatial grid, where the patches are visually grouped according to whether they are considered to represent stained cells or not. For verification, the user can click a patch to review it in full magnification. The user can reclassify a patch by buttons in the magnified view or by drag-and-drop in the gallery. As soon as the user changes a patch, the final (top-level) ratio is updated (e.g. 31.4% [CI 30.0 - 32.8]).
We considered showing the decision of the model for each and every pixel point (a “heatmap”), but this does not fulfill our criteria for the composability of levels. Verifying and correcting every pixel-level decision would be unfeasible for most humans. In order to not create a barrier to the higher level of the summative cell ratio, we thus limit the output of the model to grid-sampled patches. The percentage is always reported with a calculated confidence interval, reflecting the uncertainty derived from only making decisions on a subset of the tissue’s cells. Click to zoom in
Systematic spatial grid
Drag-and-drop to reclassify There are one group per label Reclassify by pressing the buttons
To support cases where the ratio is very close to a decision cut-of, the user can increase the certainty of their decisions by adding patches, making the sampling grid denser.
In evaluation with pathologists, we found that while this design was useful for a large subset of clinical cases where the diagnosis was far from a decision cut-of, there existed cases where the needed precision created a grid so dense that the amount of verification overwhelmed the user, and again they had to resort to the manual approach. Usually, not being able to reach the needed certainty for the case was only realized after extensive verification of many cell-level decisions.
We sought to remedy this by finding another intermediary level, that had more automatic support than the one above, but less than only getting a final percentage. To find opportunities for automatic support we analyzed the bias-based error in our underlying model. We found that most errors are somewhat systematic; visually similar patches might all be assigned the “wrong” classification. For instance, the threshold for brown staining intensity to be considered positive may difer between cases.
Based on this, we added an algorithm for unsupervised visual similarity clustering to our system and sought to design the interaction such that the user could work by only making decisions on a cluster level. The user interface for this mode of interaction is depicted in 6.
In this prototype, the user can choose to look at the resulting percentage (e.g. 72.6% [69-76] N=3601), or to view the first few patches of each cluster, or to expand clusters and inspect their constituent patches. Additionally, clusters are ordered by uncertainty, and patches within the cluster are also ordered by uncertainty. The intent is that the user hopefully can detect errors in only the first few clusters and then accept the rest.
A typical, multi-level workflow when using this would be as follows: (1) Open the case and initiate the use of the tool (2) (top level) Review the overall assigned percentage. Is it reasonable given the overall look of the tissue? If the confidence interval is far from a treatment cut-of, accept the result. Otherwise continue. (3) (individual corrections) Is the grid dense? If not, start reviewing and correcting the patches of the top-most clusters. Observe the updated percentage and the conifdence. Stop when you’re making fewer corrections per cluster. (4) (batch correction) If the grid is dense, and there are over 500 patches, start reviewing the top-most clusters; based on its first patch, does it have the correct classification? If not, correct the classification for the entire cluster. Observe the updated percentage and the confidence. (5) (individual correction) Check the patches in the cluster; does any patch “stand out” as not belonging to the cluster? Correct the patches by dragging them to the correct category, they will automatically be assigned another cluster of that type. (6) (batch correction) Proceed through a few clusters, once no or few errors are detected, the rest is probably correct.
We presented this multi-level version of the tool to three pathologists that had not been part of the design process in a small qualitative assessment. The three pathologists were presented the tool for the first time. We wanted to know whether the prototype could be clinically useful and more specifically, whether it seemed the pathologists could learn multi-level strategies that allowed them to balance detailed control, spent time and diagnostic quality. Our goal was primarily to assess the concept’s viability for further empirical eforts.
We found a recurring theme of initially wanting to drilldown to cell level. Pathologists reported that they would need some “alone time” to learn what kind of systematic errors the prediction was making, and correlate this to the overall appearance of the case. When asked whether they thought they would be able to learn when to work at which level of detail, they were tentatively positive, but stating that time would tell for certain.
To us, it seemed the design had potential in allowing them to work with sometimes inaccurate models, but also, by moving between levels. Through drill-down we hope that they might learn to calibrate their trust towards working at the right level as appropriate. It could be that a more global, model-level understanding can be achieved by interacting with local justifications like ours, over time. By contrast, user interfaces where human-ML collaboration becomes a veriifcation clif does not as readily aford this, as the manual approach and the assisted are completely disjunct.
While the results from such a small user study are mostly anecdotal at this point, we are planning to evaluate this aspect more extensively in future research.
While our concept of verification staircases is early work, we believe it has connections to many of the same issues that research on explainable and transparent intelligent tools seek to address.
For instance, many of the principles outlined for
Explanatory Debugging [
The need for enabling user feedback for explanations [
Enabling global model understanding through repeated
exposure with local justifications is similar to the strategy
employed by the LIME technique [
Our current design aids the user in detecting errors, e.g., by
sorting patches and clusters on confidence. We then rely on
that the user will be able to learn which end of the model’s
accuracy distribution they are in, or at least, the suitable
amount of validation efort to spend. There exist other
approaches to facilitating error detection and determining the
accuracy of classifiers [
A limitation of our current prototype is that a user’s correction of single patches or clusters afect only the directly involved patches, clusters and the overall ratio. We have experimented with versions where the model is fine-tuned using this input and the predictive output is updated, in an interactive machine learning manner. However, this kind of global updates creates a lack of control for which we are yet to find good interaction design solutions that suit our safety-critical domain. We believe this is an interesting area of future research.
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP)