Feedback loops are vital for decision-making and behavior change in health systems, but not all feedback is of equal value. Clinical performance feedback to healthcare professionals and teams has potential for large effects on clinical practice, but evidence suggests that low-value performance feedback is widespread. A primary barrier to understanding the value of feedback loops in health systems may be a lack of a well-defined model and shared semantics for the information that they carry. An ontology for audit and feedback research may be used to address these issues by standardizing feedback intervention metadata. Research describing feedback interventions recognizes differences between the content of the feedback and its delivery process. However, terms describing feedback intervention content are inconsistent, and appear to vary considerably between audit and feedback frameworks, which can result in confusion around what is being delivered in a performance summary. Our objective was to develop an ontology of a performance summary in a clinical performance feedback intervention for the purposes of standardizing metadata. We developed the Performance Summary Display Ontology (PSDO) iteratively by 1) identifying terms for classes from behavior change theories relating to feedback interventions and cognitive theories of visualization, 2) searching for relevant existing ontologies and classes, and 3) using the terms to specify information content and visual displays in published examples of dashboard displays and feedback reports. PSDO is a lightweight application ontology that specifies performance information content and its representations for the purpose of feedback intervention research and evaluation. PSDO contains 3 primary domains: 1) Performance information content, based on constructs from behavior change theories, 2) Marks and their qualities, based on constructs from visualization theories, and 3) roles that link marks, information content, and other emergent characteristics, as interpreted information. PSDO may enable standardization of metadata for the study of feedback interventions.
Feedback loops are vital for decision-making
and behavior change in health systems, but not
all feedback is of equal value. Clinical
performance feedback to healthcare professionals
and teams, also known as audit and feedback,
has potential for large effects on clinical
practice[
A primary barrier to understanding the value
of feedback loops in health systems may be a
lack of a well-defined model and shared
semantics for the information that they carry. In
the audit and feedback research community,
models of feedback are developed from varied
disciplines, including psychology, sociology, and
informatics[
An important advance to standardize the
description of these elements and processes is the
Clinical Performance Feedback Intervention
Theory, (CP-FIT), which incorporates constructs
from more than 30 behavior change theories into
a single theory, using evidence synthesized from
qualitative studies of feedback interventions in
healthcare[9]. CP-FIT is a valuable theory that
can be used to interpret evidence about audit and
feedback, and to guide future research. To our
knowledge, CP-FIT lacks ontologically
consistent definitions, resulting in potential
semantic ambiguity that prevents the effective
organization of research data and evidence
synthesis[
Audit and feedback is widely understood as a
process of delivering a summary of clinical
performance to healthcare professionals and
teams[
Research describing feedback interventions recognizes differences between the content of the feedback and its delivery process[7,9,12]. However, terms describing feedback intervention content are inconsistent, and appear to vary considerably across audit and feedback frameworks (Table 1). This ambiguity can result in confusion around what is being delivered in a performance summary.
Differentiating the information content from its process of delivery is also an issue for visualizations, which are described inconsistently as part of the delivery process of the performance summary (e.g. feedback delivered graphically[12]) and as part of the content of the performance summary (graph presented[7], graphical elements[9]). Visualization theories have a potentially significant role to play in the modeling of feedback content and delivery aspects, due to the potential for visualizations to strongly influence performance information interpretation[17] and to negatively moderate the effect of feedback interventions[12].
Visualization theories offer constructs to clarify relationships between the physical marks (made of ink or pixels) and perceived entities such as areas, points, and lines in a visualization[18]. A cognitive theory of visualizations called relational information displays further specifies relationships between information content and visualizations, by recognizing alignment between the physical characteristics of marks (e.g. length, slope) and the properties of entities that they represent (e.g. performance, rate of change) [19,20]. Visualization theories also offer terms and relationships for understanding visualization success, including characteristics of the person using the visualization, and their task[17].
The processes through which feedback is delivered as an intervention is an important area of research inquiry. Feedback to healthcare professionals and teams can be foundationally understood as a communication process. A communication model[21] offers the constructs of source, transmitter, channel, receiver, many of which have been used in feedback theory[9,22] (Figure 1).
Feedback can be understood as a kind of
communication that influences decisions and
behavior. Two approaches for modeling feedback
in this interventional context are the Behavior
Change Intervention Ontology[23], and
information value chain theory[
Information value chain theory was
developed for the evaluation of information
systems in health-related contexts[
Precision feedback aims to deliver high-value
performance summaries by prioritizing
performance information in short, actionable and
motivating messages to healthcare professionals
and teams[
To develop an ontology of a performance summary in a clinical performance feedback intervention for the purposes of standardizing metadata.
A research team of faculty, graduate students, and staff collaborated to create the Performance Summary Display Ontology (PSDO). We chose to use Basic Formal Ontology as an upper-level ontology, and to participate in the Open Biological and Biomedical Ontology (OBO) Foundry[29] because of the potential for semantic interoperability with BCIO and classes from other related ontologies using BFO.
We developed PSDO iteratively by 1) identifying terms for classes from behavior change theories relating to feedback interventions and cognitive theories of visualization, 2) searching for relevant existing ontologies and classes, and 3) using the terms to specify information content and visual displays in published examples of dashboard displays and feedback reports. We repeated these steps as our specifications revealed issues for refinement of terms, classes, and properties, and the need to identify additional terms.
Toward the goal of supporting a use case for precision feedback, as we refined the ontology, we developed a knowledge base and a prototype feedback system that used classes from the ontology, which enabled us to differentiate information content elements in performance data from perceived information in a feedback message.
In an additional step to assess the feasibility of PSDO to support audit and feedback research broadly, we used its terms and definitions to specify the content of published feedback reports and dashboard displays in studies of audit and feedback from a range of clinical settings[13].
PSDO is a lightweight application ontology that specifies performance information content and its representations for the purpose of feedback intervention research and evaluation. PSDO contains 3 primary domains: 1) Performance information content, based on constructs from behavior change theories, 2) Marks and their qualities, based on constructs from visualization theories, and 3) roles that link marks, information content, and other emergent characteristics, as interpreted information.
Constructs from behavior change theory are
based on Feedback Intervention Theory’s
construct of a feedback-standard gap[
Regarding interpreted information, PSDO models the theory of ‘distributed representation’ in displays that contain two types of representational components: 1) ‘internal’ or cognitive, and 2) ‘external’ or physical-graphical entities. By representing ‘internal’ and ‘external’ entities separately we can study the alignment-of and differences-between sets of features in visualizations of performance, in order to make and test hypotheses about viewer cognition and intervention success. By making these characteristics computable PSDO enables the use of theories such as ‘information match’ between displays and the cognitive tasks they support in order to test theories of behavior change against actual changes in both cognitive processing and clinical performance.
PSDO is designed according to the principles of The Open Biological and Biomedical Ontology (OBO) Foundry[29]. We developed PSDO publicly under an open source software license and deposited the ontology in BioPortal[34]. Formal definitions for selected classes of PSDO are provided in Table 2.
PSDO models information content and its representations of performance summaries in the context of clinical performance communication (Figure 1). The information source is represented as performance information content that is further represented by marks, as both transmitter and receiver, sent via some channel to a destination. The destination is interpreted information of the feedback recipient. Thus, performance information content is carried by marks and interpreted by feedback recipients.
Performance summaries must necessarily relate performance information to a feedback recipient for a specified time interval. Performance information contains performance levels (i.e. high, low, 83%, 22), that are the output of a performance measure. Thus, there are four necessary variables that a performance summary must relate in some way: A recipient, time interval, performance level, and performance measure.
Depending on the design of the performance summary, it may compare the recipient’s performance level to that of a peer-based comparator or an organizational goal. The potential to have multiple entities with attributed performance levels (e.g. recipient and peer group) creates the need to generalize the recipient entity to a dimension of entities with performance ascribed to them. We name this dimension ascribees, including the recipient and any type of comparator (e.g. goal, benchmark).
Performance summaries may contain performance levels from multiple time intervals, with multiple comparators. When these features are interpreted as information, higher-level features may be perceived to represent events such as performance trends, goal achievement, and the gain or loss of performance status among peers.
5.3. Marks
Following Munzner, we informally define a ‘mark’ as a graphical element that is part of a performance summary display. This can include: a single bar, a line, an axis or a cell value in a table. Further, we define a dimension as a multiset of individual marks in a performance summary display. By defining graphical elements at three levels of granularity (1) marks, (2) interpreted information, (3) and the entire performance summary display (the multiset of interpreted information) we can utilize theoretical perspectives from multiple disciplines to inform our design choices. More importantly, using this tiered understanding of graphical construction we can meaningfully describe interaction between elements in any tier, or elements across tiers.
Munzner describes marks and their ‘channels’, or qualities that control aspects of the mark’s physical manifestation, such as color[18]. Based on Munzner’s definition of a ‘channel’ as ‘a way to control the appearance of marks, independent of the dimensionality of the geometric primitive’ we hypothesize that any visual aspect of a performance summary display that can be perceived as contiguous is important to consider when designing, testing, or implementing feedback interventions. By defining performance summary elements at this level of granularity, we gain the ability to map differences between individual marks and their interpreted information to potential theoretical mechanisms of feedback interventions. For example, Feedback Intervention Theory (incorporating constructs from Control Theory and Goal-Setting Theory) asserts that a comparison between a recipient’s performance level and that of a goal or peer benchmark influences the feedback recipient’s motivation. Using Munzner’s theory of marks and channels we can informally define this comparison or ‘gap’ as a perceived distance between two marks.
Above the level of marks, we adopt the theoretical framework of Zhang and Norman[20] to represent interpreted information that are collections of marks. Using this approach, we gain the ability to describe theoretical mechanisms in association with emergent properties, both cognitive and physical, that govern our cognitive and potentially reaction to different visualizations. Perhaps most importantly this allows us to map particular multisets of marks to cognitive tasks. For example, the achievement of a goal (represented by a recipient’s performance level improving to reach the level of the goal) is an emergent property of multiple dimensions. Cognitive analysis of goal achievement depends on properties of the lower level: marks.
To our knowledge, none of the OBO Foundry ontologies represent visual information artifacts in a way that accounts for distributed representation. PSDO represents the physical graphical components (‘marks’), the informational entities, and what those marks may come to represent to the viewer through processes of human perception, cognition or design.
We developed a prototype precision feedback system that operates as a software pipeline with performance data as a primary input. The system uses a knowledge base composed of knowledge about feedback loops (represented as causal pathway models) with the preconditions for their availability expressed as basic characteristics of performance information content. The system first processes performance data to identify and annotate these characteristics, such as a loss of social status, or the achievement of a goal. Next, candidate email messages are created from email message templates that can be populated with the recipient’s performance data. The system assesses each candidate message to determine if it satisfies the preconditions of each feedback loop, given the performance data for the recipient. Finally, candidate messages that have one or more feedback loops available are scored to select the highest-value message (Figure 3).
PSDO enables standardization of metadata for the study of feedback interventions in healthcare organizations and quality improvement networks. PSDO has high potential to support research and evaluation for these widely used interventions, especially for those delivered digitally via email and web-based dashboards that use visualizations to highlight important care quality gaps[5,6].
A key problem for the modeling of feedback
intervention elements is the inadequate
granularity of description of graphical displays of
clinical performance information. Visualization
studies of clinical feedback interventions
typically describe visual displays at the level of a
whole visual display, such as bar and line
charts[
Based on PSDO, performance summary information can be understood to necessarily contain data about performance levels related to a feedback recipient for one or more time intervals and performance measures[13]. Comparators can be modeled as an optional kind of ascribee within a performance summary that, when included in a visual display, share a dimension with the feedback recipient. Describing feedback intervention elements without this level of granularity may limit our ability to learn about the success of feedback interventions.
By using Basic Formal Ontology (BFO) as its upper-level ontology, PSDO allows for semantic interoperability with a wide net of formal scientific ontologies and serves as an easily extendable framework for future work on representation of feedback intervention elements and information visualization artifacts.
In developing PSDO, we chose to use the label content for terms about performance data and information, and the label element (and set where there are multiple elements) for terms about the interpreted information that recipients perceive. For example, PSDO has the terms ‘performance gap content’ and ‘performance gap set’(Table 2). These performance terms are analogous but necessary to differentiate so that a precision feedback system can reason about alternate messages with alternate kinds of interpreted information, based on the same performance data.
A primary limitation of PSDO is that it has yet to undergo a formal evaluation. We plan to initiate evaluation of PSDO in multiple phases using an ontology life cycle model[37] which recognizes differences in requirements for multiple purposes of ontology use. Our development of the precision feedback system that uses PSDO provides a primary use case with requirements that the ontology has satisfied for our prototype system.
We developed PSDO as an application ontology without the close involvement of the audit and feedback research community, which may present challenges for its broader uptake. PSDO does not yet contain important terms for metadata about performance summaries, such as the author (source) and date of delivery, as its scope has been limited to performance information, its delivery process, and information that is interpreted by recipients.
PSDO is an ontology for the content and delivery of performance summaries in feedback interventions. It defines performance information content, visual display elements, and interpreted information for recipients of clinical performance feedback. PSDO may enable standardization of metadata for the study of feedback interventions.
Funding for this study was provided by the National Library of Medicine at the National Institutes of Health through 1K01LM012528-01. We thank Charles Friedman, Anne Sales, Allen Flynn, Dahee Lee, and Veena Panicker for their contributions to this work.
1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev Online.
2012;6:CD000259. 2. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med. 2014 Nov;29(11):1534–41. 3. Grimshaw JM, Ivers N, Linklater S, Foy R, Francis JJ, Gude WT, et al. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019
May;28(5):416–23. 4. McClure RC, Macumber CL, Skapik JL, Smith AM. Igniting Harmonized Digital Clinical Quality Measurement through Terminology, CQL, and FHIR. Appl Clin Inform. 2020;11(1):23–33. 5. Tuti T, Nzinga J, Njoroge M, Brown B, Peek N, English M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci. 2017;12:61. 6. Dowding D, Randell R, Gardner P,
Fitzpatrick G, Dykes P, Favela J, et al.
Dashboards for improving patient care: