Performance Summary Display Ontology: Feedback intervention content, delivery, and interpreted information Zach Landis-Lewis 1, Cooper Stansbury 1, John Rincón 2, and Colin Gross 1 1 University of Michigan Medical School, Ann Arbor, Michigan, USA 2 Harvard Medical School , Cambridge, Massachusetts, USA Abstract Feedback loops are vital for decision-making and behavior change in health systems, but not all feedback is of equal value. Clinical performance feedback to healthcare professionals and teams has potential for large effects on clinical practice, but evidence suggests that low-value performance feedback is widespread. A primary barrier to understanding the value of feedback loops in health systems may be a lack of a well-defined model and shared semantics for the information that they carry. An ontology for audit and feedback research may be used to address these issues by standardizing feedback intervention metadata. Research describing feedback interventions recognizes differences between the content of the feedback and its delivery process. However, terms describing feedback intervention content are inconsistent, and appear to vary considerably between audit and feedback frameworks, which can result in confusion around what is being delivered in a performance summary. Our objective was to develop an ontology of a performance summary in a clinical performance feedback intervention for the purposes of standardizing metadata. We developed the Performance Summary Display Ontology (PSDO) iteratively by 1) identifying terms for classes from behavior change theories relating to feedback interventions and cognitive theories of visualization, 2) searching for relevant existing ontologies and classes, and 3) using the terms to specify information content and visual displays in published examples of dashboard displays and feedback reports. PSDO is a lightweight application ontology that specifies performance information content and its representations for the purpose of feedback intervention research and evaluation. PSDO contains 3 primary domains: 1) Performance information content, based on constructs from behavior change theories, 2) Marks and their qualities, based on constructs from visualization theories, and 3) roles that link marks, information content, and other emergent characteristics, as interpreted information. PSDO may enable standardization of metadata for the study of feedback interventions. Keywords 1 feedback intervention, audit and feedback, visualization, quality improvement, behavior change 1 ICBO 2022, September 25–28, 2022, Ann Arbor, MI, USA EMAIL: zachll@umich.edu(A. 1) ORCID: 0000-0002-9117-9338 (A. 1); 0000-0003-2413-8314 (A. 2); 0000-0003-1584-3943 (A. 3); 0000-0002-7061-5073 (A. 4) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 1. Introduction synthesis[15]. An ontology for audit and Feedback loops are vital for decision-making feedback research may be used to address these and behavior change in health systems, but not issues by standardizing feedback intervention all feedback is of equal value. Clinical metadata, and for the refinement of CP-FIT and performance feedback to healthcare professionals other theories contributing to our knowledge of and teams, also known as audit and feedback, the value of feedback in healthcare. has potential for large effects on clinical 2. Background practice[1]. Unfortunately, decades of evidence 2.1.Feedback content vs delivery from hundreds of trials demonstrates persistent Audit and feedback is widely understood as a mixed effects[2,3], suggesting that low-value process of delivering a summary of clinical performance feedback is widespread. The need to performance to healthcare professionals and better understand the value of feedback gains teams[1,16]. Thus, audit and feedback has an importance as healthcare organizations expand audit component, in which a performance digital infrastructure for performance feedback in summary is developed, followed by a feedback the form of dashboards and email component, in which the performance summary communication for population health and quality is delivered. Given its central role in audit and improvement[4–6]. feedback, the performance summary may be an A primary barrier to understanding the value important starting place for ontology of feedback loops in health systems may be a development. lack of a well-defined model and shared Research describing feedback interventions semantics for the information that they carry. In recognizes differences between the content of the the audit and feedback research community, feedback and its delivery process[7,9,12]. models of feedback are developed from varied However, terms describing feedback intervention disciplines, including psychology, sociology, and content are inconsistent, and appear to vary informatics[7–12]. The resulting set of considerably across audit and feedback foundational theoretical constructs contribute frameworks (Table 1). This ambiguity can result various terms and definitions for the content of in confusion around what is being delivered in a feedback. For example, differing terms are used performance summary. to refer to feedback content about social Differentiating the information content from comparison, including benchmarks[8], normative its process of delivery is also an issue for information[12], and others’ previous visualizations, which are described inconsistently performance[7]. The same terms are used with as part of the delivery process of the performance differing definitions, leading to confusion, such summary (e.g. feedback delivered as trend, which has been alternately defined as a graphically[12]) and as part of the content of the comparison to one’s past performance[8,9] and a performance summary (graph presented[7], change in performance over time[13], and graphical elements[9]). Visualization theories velocity, defined as feedback intervention have a potentially significant role to play in the frequency[14] and the amount of change in modeling of feedback content and delivery performance since the previous feedback aspects, due to the potential for visualizations to intervention[12]. strongly influence performance information An important advance to standardize the interpretation[17] and to negatively moderate the description of these elements and processes is the effect of feedback interventions[12]. Clinical Performance Feedback Intervention Visualization theories offer constructs to Theory, (CP-FIT), which incorporates constructs clarify relationships between the physical marks from more than 30 behavior change theories into (made of ink or pixels) and perceived entities a single theory, using evidence synthesized from such as areas, points, and lines in a qualitative studies of feedback interventions in visualization[18]. A cognitive theory of healthcare[9]. CP-FIT is a valuable theory that visualizations called relational information can be used to interpret evidence about audit and displays further specifies relationships between feedback, and to guide future research. To our information content and visualizations, by knowledge, CP-FIT lacks ontologically recognizing alignment between the physical consistent definitions, resulting in potential characteristics of marks (e.g. length, slope) and semantic ambiguity that prevents the effective the properties of entities that they represent (e.g. organization of research data and evidence performance, rate of change) [19,20]. Table 1 success, including characteristics of the person Terms for the content of feedback from selected using the visualization, and their task[17]. A&F frameworks 2.2.Feedback intervention Content type Term The processes through which feedback is delivered as an intervention is an important area Processes of care of research inquiry. Feedback to healthcare Patient outcomes professionals and teams can be foundationally Performance of individual understood as a communication process. A provider communication model[21] offers the constructs Performance of provider group of source, transmitter, channel, receiver, many of Individual patient cases which have been used in feedback theory[9,22] Aggregate of patient cases (Figure 1). Information Specific behavior to be Feedback can be understood as a kind of delivered[7] changed communication that influences decisions and Comparison provided behavior. Two approaches for modeling feedback Others previous in this interventional context are the Behavior performance Change Intervention Ontology[23], and Standardized guideline information value chain theory[24–26]. The Behavior Change Intervention Ontology (BCIO) Own previous models planned processes that deliver some performance content with an aim to influence human behavior Graph presented as an outcome[23]. Mechanisms of action are the intermediate steps in a process through which the Correct solution information content influences behavior. In the case of Attainment level feedback interventions, a performance summary Velocity may function as an information container for Feedback Normative information various kinds of content that are known to content[12] Goal setting type influence behavior, with each type of content Difficult and specific potentially relating to a unique theoretical Do your best mechanism of action. BCIO is developed as an domain-specific ontology for intervention-related Benchmarking ontologies, using Basic Formal Ontology as their Framing upper-level ontology[23]. Graphical elements Information value chain theory was Number of metrics developed for the evaluation of information Patient lists systems in health-related contexts[24,26,27]. Performance level Like BCIO, information value chain theory Feedback Prioritization models steps in a sequence of events that are display Specificity necessary for the success of interventions. These variables[9] Target steps represent interaction with a system and the Timeliness cognitive processing of information received, Trend followed by changes in decisions, behavior, and Usability health-related outcomes. Information value chain Detailed patient-level theory has been used to understand the information limitations of audit and feedback interventions, Qualitative data as well as other forms of decision support[24,25]. BCIO and information value Measures chain theory offer constructs that may be useful Information Ascribees for representing parts of the feedback delivery content[13] Performance levels process, and the context of a performance Time intervals summary. 2.3.Precision Feedback Visualization theories also offer terms and Precision feedback aims to deliver high-value relationships for understanding visualization performance summaries by prioritizing performance information in short, actionable and Toward the goal of supporting a use case for motivating messages to healthcare professionals precision feedback, as we refined the ontology, and teams[28]. For example, a message may alert we developed a knowledge base and a prototype a feedback recipient about a significant drop in feedback system that used classes from the performance, or about the achievement of a goal. ontology, which enabled us to differentiate A precision feedback system requires formal information content elements in performance description of not only the information content data from perceived information in a feedback that is available to deliver to a feedback message. recipient, but also a range of possible messages In an additional step to assess the feasibility that could be motivating and actionable, given of PSDO to support audit and feedback research the recipient’s performance data. broadly, we used its terms and definitions to 3. Objective specify the content of published feedback reports To develop an ontology of a performance and dashboard displays in studies of audit and summary in a clinical performance feedback feedback from a range of clinical settings[13]. intervention for the purposes of standardizing 5. Results metadata. PSDO is a lightweight application ontology 4. Methods that specifies performance information content A research team of faculty, graduate students, and its representations for the purpose of and staff collaborated to create the Performance feedback intervention research and evaluation. Summary Display Ontology (PSDO). We chose PSDO contains 3 primary domains: 1) to use Basic Formal Ontology as an upper-level Performance information content, based on ontology, and to participate in the Open constructs from behavior change theories, 2) Biological and Biomedical Ontology (OBO) Marks and their qualities, based on constructs Foundry[29] because of the potential for from visualization theories, and 3) roles that link semantic interoperability with BCIO and classes marks, information content, and other emergent from other related ontologies using BFO. characteristics, as interpreted information. We developed PSDO iteratively by 1) Constructs from behavior change theory are identifying terms for classes from behavior based on Feedback Intervention Theory’s change theories relating to feedback construct of a feedback-standard gap[30], which interventions and cognitive theories of represents the discrepancy between current visualization, 2) searching for relevant existing performance and a comparator that is a shared ontologies and classes, and 3) using the terms to focus of Control Theory[31] and Goal-Setting specify information content and visual displays Theory[32]. Constructs from visualization theory in published examples of dashboard displays and build on Zhang and Norman’s representational feedback reports. We repeated these steps as our analysis of charts and other visual specifications revealed issues for refinement of displays[19,20], and Munzner’s framework for terms, classes, and properties, and the need to analysis of visualizations[18,33] to describe identify additional terms. information visualization artifacts and cognitive tasks of the viewer. Regarding interpreted information, PSDO further represented by marks, as both transmitter models the theory of ‘distributed representation’ and receiver, sent via some channel to a in displays that contain two types of destination. The destination is interpreted representational components: 1) ‘internal’ or information of the feedback recipient. Thus, cognitive, and 2) ‘external’ or physical-graphical performance information content is carried by entities. By representing ‘internal’ and ‘external’ marks and interpreted by feedback recipients. entities separately we can study the alignment-of 5.2. Performance information and differences-between sets of features in visualizations of performance, in order to make content and test hypotheses about viewer cognition and Performance summaries must necessarily intervention success. By making these relate performance information to a feedback characteristics computable PSDO enables the use recipient for a specified time interval. of theories such as ‘information match’ between Performance information contains performance displays and the cognitive tasks they support in levels (i.e. high, low, 83%, 22), that are the order to test theories of behavior change against output of a performance measure. Thus, there are actual changes in both cognitive processing and four necessary variables that a performance clinical performance. summary must relate in some way: A recipient, PSDO is designed according to the principles time interval, performance level, and of The Open Biological and Biomedical performance measure. Ontology (OBO) Foundry[29]. We developed Depending on the design of the performance PSDO publicly under an open source software summary, it may compare the recipient’s license and deposited the ontology in performance level to that of a peer-based BioPortal[34]. Formal definitions for selected comparator or an organizational goal. The classes of PSDO are provided in Table 2. potential to have multiple entities with attributed performance levels (e.g. recipient and peer 5.1. Communication context group) creates the need to generalize the PSDO models information content and its recipient entity to a dimension of entities with representations of performance summaries in the performance ascribed to them. We name this context of clinical performance communication dimension ascribees, including the recipient and (Figure 1). The information source is represented any type of comparator (e.g. goal, benchmark). as performance information content that is Table 2 Table 2 (continued) Selected formal definitions from PSDO Term Definition Term Definition A material information An information content bearer that is a basic visual Mark entity that is about a element of a relational Achievement change from a negative information display. content performance gap to a An information content positive performance gap. Performance entity that is an aggregate A represented set where content of performance level the mark attributes are content entities. Achievement about a change from a An information content set negative performance gap Performance entity that is about a to a positive performance gap content discrepancy between gap. performance levels. An information content A represented set where Ascribee entity that is about an the mark attributes are content entity that is ascribed a Performance about a discrepancy performance value. gap set between performance A represented set where levels. the mark attributes are An information content Ascribee set about entities that have Performance entity that is about the attributed performance level content output value of a method data. of measuring performance. Ascribee content that is Performance An information content used to identify a measure entity about a method of Comparator discrepancy with the content measuring performance. content performance level of the A relational information recipient of an intervention. display whose display A role inhering in a mark Performance components bear some that is disjoint with a focal summary combination of sets (roles): Comparative element and that has the display ascribee set, performance element same scale type as the focal measure set, performance element. set, or time set. Goal Comparator content that An information content comparator has been ascribed a desired Performance entity that is about a content future performance level. trend content change in performance A generically dependent A realizable entity whose Information continuant that is about Role manifestation brings about content entity some thing. some result or end. An information content Comparator content about entity that is about a Social living systems that are Loss content change from a positive comparator ascribed a performance performance gap to a content level. negative performance gap. An information content A represented set where Time interval entity that is about a unit of the mark attributes are content time. about a change from a Loss set positive performance gap Performance summaries may contain to a negative performance performance levels from multiple time intervals, gap. with multiple comparators. When these features are interpreted as information, higher-level mechanisms in association with emergent features may be perceived to represent events properties, both cognitive and physical, that such as performance trends, goal achievement, govern our cognitive and potentially reaction to and the gain or loss of performance status among different visualizations. Perhaps most peers. importantly this allows us to map particular 5.3. Marks multisets of marks to cognitive tasks. For Following Munzner, we informally define a example, the achievement of a goal (represented ‘mark’ as a graphical element that is part of a by a recipient’s performance level improving to performance summary display. This can include: reach the level of the goal) is an emergent a single bar, a line, an axis or a cell value in a property of multiple dimensions. Cognitive table. Further, we define a dimension as a analysis of goal achievement depends on multiset of individual marks in a performance properties of the lower level: marks. summary display. By defining graphical elements To our knowledge, none of the OBO Foundry at three levels of granularity (1) marks, (2) ontologies represent visual information artifacts interpreted information, (3) and the entire in a way that accounts for distributed performance summary display (the multiset of representation. PSDO represents the physical interpreted information) we can utilize graphical components (‘marks’), the theoretical perspectives from multiple disciplines informational entities, and what those marks may to inform our design choices. More importantly, come to represent to the viewer through using this tiered understanding of graphical processes of human perception, cognition or construction we can meaningfully describe design. interaction between elements in any tier, or 5.5. Use Case: Precision Feedback elements across tiers. We developed a prototype precision feedback Munzner describes marks and their system that operates as a software pipeline with ‘channels’, or qualities that control aspects of the performance data as a primary input. The system mark’s physical manifestation, such as color[18]. uses a knowledge base composed of knowledge Based on Munzner’s definition of a ‘channel’ as about feedback loops (represented as causal ‘a way to control the appearance of marks, pathway models) with the preconditions for their independent of the dimensionality of the availability expressed as basic characteristics of geometric primitive’ we hypothesize that any performance information content. The system visual aspect of a performance summary display first processes performance data to identify and that can be perceived as contiguous is important annotate these characteristics, such as a loss of to consider when designing, testing, or social status, or the achievement of a goal. Next, implementing feedback interventions. By candidate email messages are created from email defining performance summary elements at this message templates that can be populated with the level of granularity, we gain the ability to map recipient’s performance data. The system differences between individual marks and their assesses each candidate message to determine if interpreted information to potential theoretical it satisfies the preconditions of each feedback mechanisms of feedback interventions. For loop, given the performance data for the example, Feedback Intervention Theory recipient. Finally, candidate messages that have (incorporating constructs from Control Theory one or more feedback loops available are scored and Goal-Setting Theory) asserts that a to select the highest-value message (Figure 3). comparison between a recipient’s performance level and that of a goal or peer benchmark 6. Discussion influences the feedback recipient’s motivation. PSDO enables standardization of metadata for Using Munzner’s theory of marks and channels the study of feedback interventions in healthcare we can informally define this comparison or organizations and quality improvement networks. ‘gap’ as a perceived distance between two marks. PSDO has high potential to support research and 5.4. Interpreted information evaluation for these widely used interventions, Above the level of marks, we adopt the especially for those delivered digitally via email theoretical framework of Zhang and Norman[20] and web-based dashboards that use visualizations to represent interpreted information that are to highlight important care quality gaps[5,6]. collections of marks. Using this approach, we A key problem for the modeling of feedback gain the ability to describe theoretical intervention elements is the inadequate granularity of description of graphical displays of extendable framework for future work on clinical performance information. Visualization representation of feedback intervention elements studies of clinical feedback interventions and information visualization artifacts. typically describe visual displays at the level of a In developing PSDO, we chose to use the whole visual display, such as bar and line label content for terms about performance data charts[35,36]. While display-level description and information, and the label element (and set may be adequate for the evaluation of one size where there are multiple elements) for terms fits most feedback interventions, it is insufficient about the interpreted information that recipients for understanding relationships between perceive. For example, PSDO has the terms information content, the visual elements that can ‘performance gap content’ and ‘performance gap strongly influence their successful set’(Table 2). These performance terms are interpretation[17,19], and the mechanisms of analogous but necessary to differentiate so that a feedback interventions that are critical for their precision feedback system can reason about success. alternate messages with alternate kinds of Based on PSDO, performance summary interpreted information, based on the same information can be understood to necessarily performance data. contain data about performance levels related to A primary limitation of PSDO is that it has a feedback recipient for one or more time yet to undergo a formal evaluation. We plan to intervals and performance measures[13]. initiate evaluation of PSDO in multiple phases Comparators can be modeled as an optional kind using an ontology life cycle model[37] which of ascribee within a performance summary that, recognizes differences in requirements for when included in a visual display, share a multiple purposes of ontology use. Our dimension with the feedback recipient. development of the precision feedback system Describing feedback intervention elements that uses PSDO provides a primary use case with without this level of granularity may limit our requirements that the ontology has satisfied for ability to learn about the success of feedback our prototype system. interventions. We developed PSDO as an application By using Basic Formal Ontology (BFO) as its ontology without the close involvement of the upper-level ontology, PSDO allows for semantic audit and feedback research community, which interoperability with a wide net of formal may present challenges for its broader uptake. scientific ontologies and serves as an easily PSDO does not yet contain important terms for metadata about performance summaries, such as review of the literature. Int J Med Inf. 2015 the author (source) and date of delivery, as its Feb;84(2):87–100. scope has been limited to performance 7. Colquhoun H, Michie S, Sales A, Ivers N, information, its delivery process, and Grimshaw JM, Carroll K, et al. Reporting information that is interpreted by recipients. and design elements of audit and feedback 7. Conclusions interventions: a secondary review. BMJ Qual PSDO is an ontology for the content and Saf. 2016 Jan 25; delivery of performance summaries in feedback 8. Gude WT, Brown B, van der Veer SN, interventions. It defines performance information Colquhoun HL, Ivers NM, Brehaut JC, et al. content, visual display elements, and interpreted Clinical performance comparators in audit information for recipients of clinical performance and feedback: a review of theory and feedback. PSDO may enable standardization of evidence. Implement Sci IS. 2019 metadata for the study of feedback interventions. 24;14(1):39. 9. Brown B, Gude WT, Blakeman T, van der 8. Acknowledgements Veer SN, Ivers N, Francis JJ, et al. Clinical Funding for this study was provided by the Performance Feedback Intervention Theory National Library of Medicine at the National (CP-FIT): a new theory for designing, Institutes of Health through 1K01LM012528-01. implementing, and evaluating feedback in We thank Charles Friedman, Anne Sales, Allen health care based on a systematic review and Flynn, Dahee Lee, and Veena Panicker for their meta-synthesis of qualitative research. contributions to this work. Implement Sci. 2019 Apr 26;14(1):40. 9. References 10. Michie S, Richardson M, Johnston M, 1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Abraham C, Francis J, Hardeman W, et al. Odgaard-Jensen J, French SD, et al. Audit The Behavior Change Technique Taxonomy and feedback: effects on professional (v1) of 93 Hierarchically Clustered practice and healthcare outcomes. Cochrane Techniques: Building an International Database Syst Rev Online. Consensus for the Reporting of Behavior 2012;6:CD000259. Change Interventions. Ann Behav Med Publ 2. Ivers NM, Grimshaw JM, Jamtvedt G, Soc Behav Med. 2013 Mar 20; Flottorp S, O’Brien MA, French SD, et al. 11. Brown B, Balatsoukas P, Williams R, Growing literature, stagnant science? Sperrin M, Buchan I. Interface design Systematic review, meta-regression and recommendations for computerised clinical cumulative analysis of audit and feedback audit and feedback: Hybrid usability interventions in health care. J Gen Intern evidence from a research-led system. Int J Med. 2014 Nov;29(11):1534–41. Med Inf. 2016 Oct;94:191–206. 3. Grimshaw JM, Ivers N, Linklater S, Foy R, 12. Hysong SJ. Meta-analysis: audit and Francis JJ, Gude WT, et al. Reinvigorating feedback features impact effectiveness on stagnant science: implementation care quality. Med Care. 2009 laboratories and a meta-laboratory to Mar;47(3):356–63. efficiently advance the science of audit and 13. Lee D, Panicker V, Gross C, Zhang J, feedback. BMJ Qual Saf. 2019 Landis-Lewis Z. What was visualized? A May;28(5):416–23. method for describing content of 4. McClure RC, Macumber CL, Skapik JL, performance summary displays in feedback Smith AM. Igniting Harmonized Digital interventions. BMC Med Res Methodol. Clinical Quality Measurement through 2020 Apr 23;20(1):90. Terminology, CQL, and FHIR. Appl Clin 14. Michie S, West R, Campbell R, Brown J, Inform. 2020;11(1):23–33. Gainforth H. ABC of Behaviour Change 5. Tuti T, Nzinga J, Njoroge M, Brown B, Peek Theories (ABC of Behavior Change): An N, English M, et al. A systematic review of Essential Resource for Researchers, Policy electronic audit and feedback: intervention Makers and Practitioners. Silverback effectiveness and use of behaviour change Publishing (Silverback IS); 2014. theory. Implement Sci. 2017;12:61. 15. Arp R, Smith B, Spear AD. Building 6. Dowding D, Randell R, Gardner P, Ontologies with Basic Formal Ontology. Fitzpatrick G, Dykes P, Favela J, et al. MIT Press; 2015. 245 p. Dashboards for improving patient care: 16. Brehaut JC, Eva KW. Building theories of knowledge translation interventions: Use the Foundry: coordinated evolution of entire menu of constructs. Implement Sci. ontologies to support biomedical data 2012 Nov 22;7(1):114. integration. Nat Biotechnol. 2007 17. Hegarty M. Advances in Cognitive Science Nov;25(11):1251–5. and Information Visualization. In: Score 30. Kluger AN, DeNisi A. The Effects of Reporting Research and Applications Feedback Interventions on Performance: A [Internet]. Routledge; 2018 [cited 2019 Apr Historical Review, a Meta-Analysis, and a 30]. Available from: Preliminary Feedback Intervention Theory. https://www.taylorfrancis.com/ Psychol Bull March 1996. 18. Munzner T. Visualization Analysis and 1996;119(2):254–84. Design. 1 edition. Boca Raton: A K 31. Carver CS, Scheier MF. Control theory: A Peters/CRC Press; 2014. 428 p. useful conceptual framework for personality 19. Zhang J, Norman DA. Representations in -- social, clinical, and health psychology. Distributed Cognitive Tasks. Cogn Sci. 1994 Psychol Bull Vol 921. 1982 Jan 1;18(1):87–122. Jul;92(1):111–35. 20. Zhang J. A representational analysis of 32. Locke EA, Latham GP. Building a relational information displays. Int J Hum practically useful theory of goal setting and Comput Stud. 1996;45(1):59–74. task motivation: A 35-year odyssey. Am 21. Shannon CE, Weaver W. The Mathematical Psychol Vol 579. 2002 Sep;57(9):705–17. Theory of Communication. University of 33. Munzner T. A Nested Model for Illinois Press; 1963. 148 p. Visualization Design and Validation. IEEE 22. Ilgen DR, Fisher CD, Taylor MS. Trans Vis Comput Graph. 2009 Consequences of individual feedback on Nov;15(6):921–8. behavior in organizations. J Appl Psychol. 34. Performance Summary Display Ontology - 1979;64(4):349–71. Summary | NCBO BioPortal [Internet]. 23. Michie S, West R, Finnerty AN, Norris E, [cited 2022 Feb 10]. Available from: Wright AJ, Marques MM, et al. https://bioportal.bioontology.org/ontologies/ Representation of behaviour change PSDO interventions and their evaluation: 35. Dowding D, Merrill JA, Onorato N, Barrón Development of the Upper Level of the Y, Rosati RJ, Russell D. The impact of home Behaviour Change Intervention Ontology. care nurses’ numeracy and graph literacy on Wellcome Open Res. 2021 Jan 6;5:123. comprehension of visual display 24. Coiera E. Assessing Technology Success information: implications for dashboard and Failure Using Information Value Chain design. J Am Med Inform Assoc JAMIA. Theory. Stud Health Technol Inform. 2019 2018 Feb 1;25(2):175–82. Jul 30;263:35–48. 36. Petit-Monéger A, Saillour-Glénisson F, 25. Gude WT, van der Veer SN, de Keizer NF, Nouette-Gaulain K, Jouhet V, Salmi LR. Coiera E, Peek N. Optimizing Digital Health Comparing Graphical Formats for Feedback Informatics Interventions Through of Clinical Practice Data. Methods Inf Med. Unobtrusive Quantitative Process 2017;56(1):28–36. Evaluations. Stud Health Technol Inform. 37. Neuhaus F, Ray S, Sriram RD. Toward 2016;228:594–8. ontology evaluation across the life cycle. US 26. Coiera E. A New Informatics Geography. Department of Commerce, National Institute Yearb Med Inform. 2016 Nov 10;(1):251–5. of Standards and Technology; 2014. 27. Coiera E. Guide to Health Informatics. 3rd edition. Boca Raton: CRC Press; 2015. 710 p. 28. Landis-Lewis Z, Flynn A, Janda A, Shah N. A Scalable Service to Improve Health Care Quality Through Precision Audit and Feedback: Proposal for a Randomized Controlled Trial. JMIR Res Protoc. 2022 May 10;11(5):e34990. 29. Smith B, Ashburner M, Rosse C, Bard J, Bug W, Ceusters W, et al. The OBO