=Paper=
{{Paper
|id=Vol-3805/ICBO-2022_paper_2172
|storemode=property
|title=Performance Summary Display Ontology: Feedback Intervention Content, Delivery, and Interpreted Information
|pdfUrl=https://ceur-ws.org/Vol-3805/ICBO-2022_paper_2172.pdf
|volume=Vol-3805
|authors=Zach Landis-Lewis,Cooper Stansbury,John Rincon,Colin Gross
|dblpUrl=https://dblp.org/rec/conf/icbo/Landis-LewisSRG22
}}
==Performance Summary Display Ontology: Feedback Intervention Content, Delivery, and Interpreted Information==
https://ceur-ws.org/Vol-3805/ICBO-2022_paper_2172.pdf
Performance Summary Display Ontology: Feedback intervention
content, delivery, and interpreted information
Zach Landis-Lewis 1, Cooper Stansbury 1, John Rincón 2, and Colin Gross 1
1
University of Michigan Medical School, Ann Arbor, Michigan, USA
2
Harvard Medical School , Cambridge, Massachusetts, USA
Abstract
Feedback loops are vital for decision-making and behavior change in health systems, but not
all feedback is of equal value. Clinical performance feedback to healthcare professionals and
teams has potential for large effects on clinical practice, but evidence suggests that low-value
performance feedback is widespread. A primary barrier to understanding the value of
feedback loops in health systems may be a lack of a well-defined model and shared semantics
for the information that they carry. An ontology for audit and feedback research may be used
to address these issues by standardizing feedback intervention metadata. Research describing
feedback interventions recognizes differences between the content of the feedback and its
delivery process. However, terms describing feedback intervention content are inconsistent,
and appear to vary considerably between audit and feedback frameworks, which can result in
confusion around what is being delivered in a performance summary. Our objective was to
develop an ontology of a performance summary in a clinical performance feedback
intervention for the purposes of standardizing metadata. We developed the Performance
Summary Display Ontology (PSDO) iteratively by 1) identifying terms for classes from
behavior change theories relating to feedback interventions and cognitive theories of
visualization, 2) searching for relevant existing ontologies and classes, and 3) using the terms
to specify information content and visual displays in published examples of dashboard
displays and feedback reports. PSDO is a lightweight application ontology that specifies
performance information content and its representations for the purpose of feedback
intervention research and evaluation. PSDO contains 3 primary domains: 1) Performance
information content, based on constructs from behavior change theories, 2) Marks and their
qualities, based on constructs from visualization theories, and 3) roles that link marks,
information content, and other emergent characteristics, as interpreted information. PSDO
may enable standardization of metadata for the study of feedback interventions.
Keywords 1
feedback intervention, audit and feedback, visualization, quality improvement, behavior
change
1
ICBO 2022, September 25–28, 2022, Ann Arbor, MI, USA
EMAIL: zachll@umich.edu(A. 1)
ORCID: 0000-0002-9117-9338 (A. 1); 0000-0003-2413-8314 (A.
2); 0000-0003-1584-3943 (A. 3); 0000-0002-7061-5073 (A. 4)
© 2022 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
1. Introduction synthesis[15]. An ontology for audit and
Feedback loops are vital for decision-making feedback research may be used to address these
and behavior change in health systems, but not issues by standardizing feedback intervention
all feedback is of equal value. Clinical metadata, and for the refinement of CP-FIT and
performance feedback to healthcare professionals other theories contributing to our knowledge of
and teams, also known as audit and feedback, the value of feedback in healthcare.
has potential for large effects on clinical 2. Background
practice[1]. Unfortunately, decades of evidence 2.1.Feedback content vs delivery
from hundreds of trials demonstrates persistent Audit and feedback is widely understood as a
mixed effects[2,3], suggesting that low-value process of delivering a summary of clinical
performance feedback is widespread. The need to performance to healthcare professionals and
better understand the value of feedback gains teams[1,16]. Thus, audit and feedback has an
importance as healthcare organizations expand audit component, in which a performance
digital infrastructure for performance feedback in summary is developed, followed by a feedback
the form of dashboards and email component, in which the performance summary
communication for population health and quality is delivered. Given its central role in audit and
improvement[4–6]. feedback, the performance summary may be an
A primary barrier to understanding the value important starting place for ontology
of feedback loops in health systems may be a development.
lack of a well-defined model and shared Research describing feedback interventions
semantics for the information that they carry. In recognizes differences between the content of the
the audit and feedback research community, feedback and its delivery process[7,9,12].
models of feedback are developed from varied However, terms describing feedback intervention
disciplines, including psychology, sociology, and content are inconsistent, and appear to vary
informatics[7–12]. The resulting set of considerably across audit and feedback
foundational theoretical constructs contribute frameworks (Table 1). This ambiguity can result
various terms and definitions for the content of in confusion around what is being delivered in a
feedback. For example, differing terms are used performance summary.
to refer to feedback content about social Differentiating the information content from
comparison, including benchmarks[8], normative its process of delivery is also an issue for
information[12], and others’ previous visualizations, which are described inconsistently
performance[7]. The same terms are used with as part of the delivery process of the performance
differing definitions, leading to confusion, such summary (e.g. feedback delivered
as trend, which has been alternately defined as a graphically[12]) and as part of the content of the
comparison to one’s past performance[8,9] and a performance summary (graph presented[7],
change in performance over time[13], and graphical elements[9]). Visualization theories
velocity, defined as feedback intervention have a potentially significant role to play in the
frequency[14] and the amount of change in modeling of feedback content and delivery
performance since the previous feedback aspects, due to the potential for visualizations to
intervention[12]. strongly influence performance information
An important advance to standardize the interpretation[17] and to negatively moderate the
description of these elements and processes is the effect of feedback interventions[12].
Clinical Performance Feedback Intervention Visualization theories offer constructs to
Theory, (CP-FIT), which incorporates constructs clarify relationships between the physical marks
from more than 30 behavior change theories into (made of ink or pixels) and perceived entities
a single theory, using evidence synthesized from such as areas, points, and lines in a
qualitative studies of feedback interventions in visualization[18]. A cognitive theory of
healthcare[9]. CP-FIT is a valuable theory that visualizations called relational information
can be used to interpret evidence about audit and displays further specifies relationships between
feedback, and to guide future research. To our information content and visualizations, by
knowledge, CP-FIT lacks ontologically recognizing alignment between the physical
consistent definitions, resulting in potential characteristics of marks (e.g. length, slope) and
semantic ambiguity that prevents the effective the properties of entities that they represent (e.g.
organization of research data and evidence performance, rate of change) [19,20].
Table 1 success, including characteristics of the person
Terms for the content of feedback from selected using the visualization, and their task[17].
A&F frameworks 2.2.Feedback intervention
Content type Term The processes through which feedback is
delivered as an intervention is an important area
Processes of care of research inquiry. Feedback to healthcare
Patient outcomes professionals and teams can be foundationally
Performance of individual understood as a communication process. A
provider communication model[21] offers the constructs
Performance of provider group of source, transmitter, channel, receiver, many of
Individual patient cases which have been used in feedback theory[9,22]
Aggregate of patient cases (Figure 1).
Information Specific behavior to be Feedback can be understood as a kind of
delivered[7] changed communication that influences decisions and
Comparison provided behavior. Two approaches for modeling feedback
Others previous in this interventional context are the Behavior
performance Change Intervention Ontology[23], and
Standardized guideline information value chain theory[24–26]. The
Behavior Change Intervention Ontology (BCIO)
Own previous
models planned processes that deliver some
performance
content with an aim to influence human behavior
Graph presented
as an outcome[23]. Mechanisms of action are the
intermediate steps in a process through which the
Correct solution information content influences behavior. In the case of
Attainment level feedback interventions, a performance summary
Velocity may function as an information container for
Feedback
Normative information various kinds of content that are known to
content[12]
Goal setting type influence behavior, with each type of content
Difficult and specific potentially relating to a unique theoretical
Do your best mechanism of action. BCIO is developed as an
domain-specific ontology for intervention-related
Benchmarking
ontologies, using Basic Formal Ontology as their
Framing upper-level ontology[23].
Graphical elements Information value chain theory was
Number of metrics developed for the evaluation of information
Patient lists systems in health-related contexts[24,26,27].
Performance level Like BCIO, information value chain theory
Feedback Prioritization models steps in a sequence of events that are
display Specificity necessary for the success of interventions. These
variables[9] Target steps represent interaction with a system and the
Timeliness cognitive processing of information received,
Trend followed by changes in decisions, behavior, and
Usability health-related outcomes. Information value chain
Detailed patient-level theory has been used to understand the
information limitations of audit and feedback interventions,
Qualitative data as well as other forms of decision
support[24,25]. BCIO and information value
Measures chain theory offer constructs that may be useful
Information Ascribees for representing parts of the feedback delivery
content[13] Performance levels process, and the context of a performance
Time intervals summary.
2.3.Precision Feedback
Visualization theories also offer terms and Precision feedback aims to deliver high-value
relationships for understanding visualization performance summaries by prioritizing
performance information in short, actionable and Toward the goal of supporting a use case for
motivating messages to healthcare professionals precision feedback, as we refined the ontology,
and teams[28]. For example, a message may alert we developed a knowledge base and a prototype
a feedback recipient about a significant drop in feedback system that used classes from the
performance, or about the achievement of a goal. ontology, which enabled us to differentiate
A precision feedback system requires formal information content elements in performance
description of not only the information content data from perceived information in a feedback
that is available to deliver to a feedback message.
recipient, but also a range of possible messages In an additional step to assess the feasibility
that could be motivating and actionable, given of PSDO to support audit and feedback research
the recipient’s performance data. broadly, we used its terms and definitions to
3. Objective specify the content of published feedback reports
To develop an ontology of a performance and dashboard displays in studies of audit and
summary in a clinical performance feedback feedback from a range of clinical settings[13].
intervention for the purposes of standardizing 5. Results
metadata. PSDO is a lightweight application ontology
4. Methods that specifies performance information content
A research team of faculty, graduate students, and its representations for the purpose of
and staff collaborated to create the Performance feedback intervention research and evaluation.
Summary Display Ontology (PSDO). We chose PSDO contains 3 primary domains: 1)
to use Basic Formal Ontology as an upper-level Performance information content, based on
ontology, and to participate in the Open constructs from behavior change theories, 2)
Biological and Biomedical Ontology (OBO) Marks and their qualities, based on constructs
Foundry[29] because of the potential for from visualization theories, and 3) roles that link
semantic interoperability with BCIO and classes marks, information content, and other emergent
from other related ontologies using BFO. characteristics, as interpreted information.
We developed PSDO iteratively by 1) Constructs from behavior change theory are
identifying terms for classes from behavior based on Feedback Intervention Theory’s
change theories relating to feedback construct of a feedback-standard gap[30], which
interventions and cognitive theories of represents the discrepancy between current
visualization, 2) searching for relevant existing performance and a comparator that is a shared
ontologies and classes, and 3) using the terms to focus of Control Theory[31] and Goal-Setting
specify information content and visual displays Theory[32]. Constructs from visualization theory
in published examples of dashboard displays and build on Zhang and Norman’s representational
feedback reports. We repeated these steps as our analysis of charts and other visual
specifications revealed issues for refinement of displays[19,20], and Munzner’s framework for
terms, classes, and properties, and the need to analysis of visualizations[18,33] to describe
identify additional terms. information visualization artifacts and cognitive
tasks of the viewer.
Regarding interpreted information, PSDO further represented by marks, as both transmitter
models the theory of ‘distributed representation’ and receiver, sent via some channel to a
in displays that contain two types of destination. The destination is interpreted
representational components: 1) ‘internal’ or information of the feedback recipient. Thus,
cognitive, and 2) ‘external’ or physical-graphical performance information content is carried by
entities. By representing ‘internal’ and ‘external’ marks and interpreted by feedback recipients.
entities separately we can study the alignment-of 5.2. Performance information
and differences-between sets of features in
visualizations of performance, in order to make content
and test hypotheses about viewer cognition and Performance summaries must necessarily
intervention success. By making these relate performance information to a feedback
characteristics computable PSDO enables the use recipient for a specified time interval.
of theories such as ‘information match’ between Performance information contains performance
displays and the cognitive tasks they support in levels (i.e. high, low, 83%, 22), that are the
order to test theories of behavior change against output of a performance measure. Thus, there are
actual changes in both cognitive processing and four necessary variables that a performance
clinical performance. summary must relate in some way: A recipient,
PSDO is designed according to the principles time interval, performance level, and
of The Open Biological and Biomedical performance measure.
Ontology (OBO) Foundry[29]. We developed Depending on the design of the performance
PSDO publicly under an open source software summary, it may compare the recipient’s
license and deposited the ontology in performance level to that of a peer-based
BioPortal[34]. Formal definitions for selected comparator or an organizational goal. The
classes of PSDO are provided in Table 2. potential to have multiple entities with attributed
performance levels (e.g. recipient and peer
5.1. Communication context group) creates the need to generalize the
PSDO models information content and its recipient entity to a dimension of entities with
representations of performance summaries in the performance ascribed to them. We name this
context of clinical performance communication dimension ascribees, including the recipient and
(Figure 1). The information source is represented any type of comparator (e.g. goal, benchmark).
as performance information content that is
Table 2 Table 2 (continued)
Selected formal definitions from PSDO Term Definition
Term Definition A material information
An information content bearer that is a basic visual
Mark
entity that is about a element of a relational
Achievement
change from a negative information display.
content
performance gap to a An information content
positive performance gap. Performance entity that is an aggregate
A represented set where content of performance level
the mark attributes are content entities.
Achievement about a change from a An information content
set negative performance gap Performance entity that is about a
to a positive performance gap content discrepancy between
gap. performance levels.
An information content A represented set where
Ascribee entity that is about an the mark attributes are
content entity that is ascribed a Performance
about a discrepancy
performance value. gap set
between performance
A represented set where levels.
the mark attributes are An information content
Ascribee set about entities that have Performance entity that is about the
attributed performance level content output value of a method
data. of measuring performance.
Ascribee content that is Performance An information content
used to identify a measure entity about a method of
Comparator
discrepancy with the content measuring performance.
content
performance level of the A relational information
recipient of an intervention. display whose display
A role inhering in a mark Performance components bear some
that is disjoint with a focal summary combination of sets (roles):
Comparative
element and that has the display ascribee set, performance
element
same scale type as the focal measure set, performance
element. set, or time set.
Goal Comparator content that An information content
comparator has been ascribed a desired Performance
entity that is about a
content future performance level. trend content
change in performance
A generically dependent A realizable entity whose
Information
continuant that is about Role manifestation brings about
content entity
some thing. some result or end.
An information content Comparator content about
entity that is about a Social
living systems that are
Loss content change from a positive comparator
ascribed a performance
performance gap to a content
level.
negative performance gap. An information content
A represented set where Time interval
entity that is about a unit of
the mark attributes are content
time.
about a change from a
Loss set
positive performance gap Performance summaries may contain
to a negative performance performance levels from multiple time intervals,
gap. with multiple comparators. When these features
are interpreted as information, higher-level mechanisms in association with emergent
features may be perceived to represent events properties, both cognitive and physical, that
such as performance trends, goal achievement, govern our cognitive and potentially reaction to
and the gain or loss of performance status among different visualizations. Perhaps most
peers. importantly this allows us to map particular
5.3. Marks multisets of marks to cognitive tasks. For
Following Munzner, we informally define a example, the achievement of a goal (represented
‘mark’ as a graphical element that is part of a by a recipient’s performance level improving to
performance summary display. This can include: reach the level of the goal) is an emergent
a single bar, a line, an axis or a cell value in a property of multiple dimensions. Cognitive
table. Further, we define a dimension as a analysis of goal achievement depends on
multiset of individual marks in a performance properties of the lower level: marks.
summary display. By defining graphical elements To our knowledge, none of the OBO Foundry
at three levels of granularity (1) marks, (2) ontologies represent visual information artifacts
interpreted information, (3) and the entire in a way that accounts for distributed
performance summary display (the multiset of representation. PSDO represents the physical
interpreted information) we can utilize graphical components (‘marks’), the
theoretical perspectives from multiple disciplines informational entities, and what those marks may
to inform our design choices. More importantly, come to represent to the viewer through
using this tiered understanding of graphical processes of human perception, cognition or
construction we can meaningfully describe design.
interaction between elements in any tier, or 5.5. Use Case: Precision Feedback
elements across tiers. We developed a prototype precision feedback
Munzner describes marks and their system that operates as a software pipeline with
‘channels’, or qualities that control aspects of the performance data as a primary input. The system
mark’s physical manifestation, such as color[18]. uses a knowledge base composed of knowledge
Based on Munzner’s definition of a ‘channel’ as about feedback loops (represented as causal
‘a way to control the appearance of marks, pathway models) with the preconditions for their
independent of the dimensionality of the availability expressed as basic characteristics of
geometric primitive’ we hypothesize that any performance information content. The system
visual aspect of a performance summary display first processes performance data to identify and
that can be perceived as contiguous is important annotate these characteristics, such as a loss of
to consider when designing, testing, or social status, or the achievement of a goal. Next,
implementing feedback interventions. By candidate email messages are created from email
defining performance summary elements at this message templates that can be populated with the
level of granularity, we gain the ability to map recipient’s performance data. The system
differences between individual marks and their assesses each candidate message to determine if
interpreted information to potential theoretical it satisfies the preconditions of each feedback
mechanisms of feedback interventions. For loop, given the performance data for the
example, Feedback Intervention Theory recipient. Finally, candidate messages that have
(incorporating constructs from Control Theory one or more feedback loops available are scored
and Goal-Setting Theory) asserts that a to select the highest-value message (Figure 3).
comparison between a recipient’s performance
level and that of a goal or peer benchmark 6. Discussion
influences the feedback recipient’s motivation. PSDO enables standardization of metadata for
Using Munzner’s theory of marks and channels the study of feedback interventions in healthcare
we can informally define this comparison or organizations and quality improvement networks.
‘gap’ as a perceived distance between two marks. PSDO has high potential to support research and
5.4. Interpreted information evaluation for these widely used interventions,
Above the level of marks, we adopt the especially for those delivered digitally via email
theoretical framework of Zhang and Norman[20] and web-based dashboards that use visualizations
to represent interpreted information that are to highlight important care quality gaps[5,6].
collections of marks. Using this approach, we A key problem for the modeling of feedback
gain the ability to describe theoretical intervention elements is the inadequate
granularity of description of graphical displays of extendable framework for future work on
clinical performance information. Visualization representation of feedback intervention elements
studies of clinical feedback interventions and information visualization artifacts.
typically describe visual displays at the level of a In developing PSDO, we chose to use the
whole visual display, such as bar and line label content for terms about performance data
charts[35,36]. While display-level description and information, and the label element (and set
may be adequate for the evaluation of one size where there are multiple elements) for terms
fits most feedback interventions, it is insufficient about the interpreted information that recipients
for understanding relationships between perceive. For example, PSDO has the terms
information content, the visual elements that can ‘performance gap content’ and ‘performance gap
strongly influence their successful set’(Table 2). These performance terms are
interpretation[17,19], and the mechanisms of analogous but necessary to differentiate so that a
feedback interventions that are critical for their precision feedback system can reason about
success. alternate messages with alternate kinds of
Based on PSDO, performance summary interpreted information, based on the same
information can be understood to necessarily performance data.
contain data about performance levels related to A primary limitation of PSDO is that it has
a feedback recipient for one or more time yet to undergo a formal evaluation. We plan to
intervals and performance measures[13]. initiate evaluation of PSDO in multiple phases
Comparators can be modeled as an optional kind using an ontology life cycle model[37] which
of ascribee within a performance summary that, recognizes differences in requirements for
when included in a visual display, share a multiple purposes of ontology use. Our
dimension with the feedback recipient. development of the precision feedback system
Describing feedback intervention elements that uses PSDO provides a primary use case with
without this level of granularity may limit our requirements that the ontology has satisfied for
ability to learn about the success of feedback our prototype system.
interventions. We developed PSDO as an application
By using Basic Formal Ontology (BFO) as its ontology without the close involvement of the
upper-level ontology, PSDO allows for semantic audit and feedback research community, which
interoperability with a wide net of formal may present challenges for its broader uptake.
scientific ontologies and serves as an easily PSDO does not yet contain important terms for
metadata about performance summaries, such as review of the literature. Int J Med Inf. 2015
the author (source) and date of delivery, as its Feb;84(2):87–100.
scope has been limited to performance 7. Colquhoun H, Michie S, Sales A, Ivers N,
information, its delivery process, and Grimshaw JM, Carroll K, et al. Reporting
information that is interpreted by recipients. and design elements of audit and feedback
7. Conclusions interventions: a secondary review. BMJ Qual
PSDO is an ontology for the content and Saf. 2016 Jan 25;
delivery of performance summaries in feedback 8. Gude WT, Brown B, van der Veer SN,
interventions. It defines performance information Colquhoun HL, Ivers NM, Brehaut JC, et al.
content, visual display elements, and interpreted Clinical performance comparators in audit
information for recipients of clinical performance and feedback: a review of theory and
feedback. PSDO may enable standardization of evidence. Implement Sci IS. 2019
metadata for the study of feedback interventions. 24;14(1):39.
9. Brown B, Gude WT, Blakeman T, van der
8. Acknowledgements Veer SN, Ivers N, Francis JJ, et al. Clinical
Funding for this study was provided by the Performance Feedback Intervention Theory
National Library of Medicine at the National (CP-FIT): a new theory for designing,
Institutes of Health through 1K01LM012528-01. implementing, and evaluating feedback in
We thank Charles Friedman, Anne Sales, Allen health care based on a systematic review and
Flynn, Dahee Lee, and Veena Panicker for their meta-synthesis of qualitative research.
contributions to this work. Implement Sci. 2019 Apr 26;14(1):40.
9. References 10. Michie S, Richardson M, Johnston M,
1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Abraham C, Francis J, Hardeman W, et al.
Odgaard-Jensen J, French SD, et al. Audit The Behavior Change Technique Taxonomy
and feedback: effects on professional (v1) of 93 Hierarchically Clustered
practice and healthcare outcomes. Cochrane Techniques: Building an International
Database Syst Rev Online. Consensus for the Reporting of Behavior
2012;6:CD000259. Change Interventions. Ann Behav Med Publ
2. Ivers NM, Grimshaw JM, Jamtvedt G, Soc Behav Med. 2013 Mar 20;
Flottorp S, O’Brien MA, French SD, et al. 11. Brown B, Balatsoukas P, Williams R,
Growing literature, stagnant science? Sperrin M, Buchan I. Interface design
Systematic review, meta-regression and recommendations for computerised clinical
cumulative analysis of audit and feedback audit and feedback: Hybrid usability
interventions in health care. J Gen Intern evidence from a research-led system. Int J
Med. 2014 Nov;29(11):1534–41. Med Inf. 2016 Oct;94:191–206.
3. Grimshaw JM, Ivers N, Linklater S, Foy R, 12. Hysong SJ. Meta-analysis: audit and
Francis JJ, Gude WT, et al. Reinvigorating feedback features impact effectiveness on
stagnant science: implementation care quality. Med Care. 2009
laboratories and a meta-laboratory to Mar;47(3):356–63.
efficiently advance the science of audit and 13. Lee D, Panicker V, Gross C, Zhang J,
feedback. BMJ Qual Saf. 2019 Landis-Lewis Z. What was visualized? A
May;28(5):416–23. method for describing content of
4. McClure RC, Macumber CL, Skapik JL, performance summary displays in feedback
Smith AM. Igniting Harmonized Digital interventions. BMC Med Res Methodol.
Clinical Quality Measurement through 2020 Apr 23;20(1):90.
Terminology, CQL, and FHIR. Appl Clin 14. Michie S, West R, Campbell R, Brown J,
Inform. 2020;11(1):23–33. Gainforth H. ABC of Behaviour Change
5. Tuti T, Nzinga J, Njoroge M, Brown B, Peek Theories (ABC of Behavior Change): An
N, English M, et al. A systematic review of Essential Resource for Researchers, Policy
electronic audit and feedback: intervention Makers and Practitioners. Silverback
effectiveness and use of behaviour change Publishing (Silverback IS); 2014.
theory. Implement Sci. 2017;12:61. 15. Arp R, Smith B, Spear AD. Building
6. Dowding D, Randell R, Gardner P, Ontologies with Basic Formal Ontology.
Fitzpatrick G, Dykes P, Favela J, et al. MIT Press; 2015. 245 p.
Dashboards for improving patient care: 16. Brehaut JC, Eva KW. Building theories of
knowledge translation interventions: Use the Foundry: coordinated evolution of
entire menu of constructs. Implement Sci. ontologies to support biomedical data
2012 Nov 22;7(1):114. integration. Nat Biotechnol. 2007
17. Hegarty M. Advances in Cognitive Science Nov;25(11):1251–5.
and Information Visualization. In: Score 30. Kluger AN, DeNisi A. The Effects of
Reporting Research and Applications Feedback Interventions on Performance: A
[Internet]. Routledge; 2018 [cited 2019 Apr Historical Review, a Meta-Analysis, and a
30]. Available from: Preliminary Feedback Intervention Theory.
https://www.taylorfrancis.com/ Psychol Bull March 1996.
18. Munzner T. Visualization Analysis and 1996;119(2):254–84.
Design. 1 edition. Boca Raton: A K 31. Carver CS, Scheier MF. Control theory: A
Peters/CRC Press; 2014. 428 p. useful conceptual framework for personality
19. Zhang J, Norman DA. Representations in -- social, clinical, and health psychology.
Distributed Cognitive Tasks. Cogn Sci. 1994 Psychol Bull Vol 921. 1982
Jan 1;18(1):87–122. Jul;92(1):111–35.
20. Zhang J. A representational analysis of 32. Locke EA, Latham GP. Building a
relational information displays. Int J Hum practically useful theory of goal setting and
Comput Stud. 1996;45(1):59–74. task motivation: A 35-year odyssey. Am
21. Shannon CE, Weaver W. The Mathematical Psychol Vol 579. 2002 Sep;57(9):705–17.
Theory of Communication. University of 33. Munzner T. A Nested Model for
Illinois Press; 1963. 148 p. Visualization Design and Validation. IEEE
22. Ilgen DR, Fisher CD, Taylor MS. Trans Vis Comput Graph. 2009
Consequences of individual feedback on Nov;15(6):921–8.
behavior in organizations. J Appl Psychol. 34. Performance Summary Display Ontology -
1979;64(4):349–71. Summary | NCBO BioPortal [Internet].
23. Michie S, West R, Finnerty AN, Norris E, [cited 2022 Feb 10]. Available from:
Wright AJ, Marques MM, et al. https://bioportal.bioontology.org/ontologies/
Representation of behaviour change PSDO
interventions and their evaluation: 35. Dowding D, Merrill JA, Onorato N, Barrón
Development of the Upper Level of the Y, Rosati RJ, Russell D. The impact of home
Behaviour Change Intervention Ontology. care nurses’ numeracy and graph literacy on
Wellcome Open Res. 2021 Jan 6;5:123. comprehension of visual display
24. Coiera E. Assessing Technology Success information: implications for dashboard
and Failure Using Information Value Chain design. J Am Med Inform Assoc JAMIA.
Theory. Stud Health Technol Inform. 2019 2018 Feb 1;25(2):175–82.
Jul 30;263:35–48. 36. Petit-Monéger A, Saillour-Glénisson F,
25. Gude WT, van der Veer SN, de Keizer NF, Nouette-Gaulain K, Jouhet V, Salmi LR.
Coiera E, Peek N. Optimizing Digital Health Comparing Graphical Formats for Feedback
Informatics Interventions Through of Clinical Practice Data. Methods Inf Med.
Unobtrusive Quantitative Process 2017;56(1):28–36.
Evaluations. Stud Health Technol Inform. 37. Neuhaus F, Ray S, Sriram RD. Toward
2016;228:594–8. ontology evaluation across the life cycle. US
26. Coiera E. A New Informatics Geography. Department of Commerce, National Institute
Yearb Med Inform. 2016 Nov 10;(1):251–5. of Standards and Technology; 2014.
27. Coiera E. Guide to Health Informatics. 3rd
edition. Boca Raton: CRC Press; 2015. 710
p.
28. Landis-Lewis Z, Flynn A, Janda A, Shah N.
A Scalable Service to Improve Health Care
Quality Through Precision Audit and
Feedback: Proposal for a Randomized
Controlled Trial. JMIR Res Protoc. 2022
May 10;11(5):e34990.
29. Smith B, Ashburner M, Rosse C, Bard J,
Bug W, Ceusters W, et al. The OBO