=Paper=
{{Paper
|id=Vol-3227/Stockdill-et-al.PP5
|storemode=property
|title=Cognitive Analysis for Representation Change
|pdfUrl=https://ceur-ws.org/Vol-3227/Stockdill-et-al.PP5.pdf
|volume=Vol-3227
|authors=Aaron Stockdill,Grecia Garcia Garcia,Peter Cheng,Daniel Raggi,Mateja Jamnik
|dblpUrl=https://dblp.org/rec/conf/hlc/StockdillGCRJ22
}}
==Cognitive Analysis for Representation Change==
https://ceur-ws.org/Vol-3227/Stockdill-et-al.PP5.pdf
Cognitive Analysis for Representation Change
Aaron Stockdill1,∗ , Grecia Garcia Garcia1 , Peter C.-H. Cheng1 , Daniel Raggi2 and
Mateja Jamnik2
1
University of Sussex, Brighton, UK
2
University of Cambridge, Cambridge, UK
Abstract
The rep2rep project is developing an AI tool to automatically select an appropriate representation to solve
a particular problem for a particular person. A prerequisite of this tool is to understand (i.e., model) how
a reader interprets a representation. But interpretations can vary wildly between novices and experts,
readers of similar ability, or even the same reader in different tasks. We present a theory and notation
(RIST and RISN) for analysing the cognitive features of a representation’s interpretation, and introduce
a web app to construct RISN models. These models provide information about cognitive properties of
representations to guide automated representation selection to support human problem solving.
Keywords
Cognition, Representations, Interpretation, Schemas
1. Introduction
When solving problems, representation choice can be critical [1]. Unfortunately, choosing the
right representation is difficult: for example, in a classroom context, students struggle to change
representation, and teachers can be inconsistent in what they recommend [2, 3]. Where one
student (correctly) understands slope in a velocity-time plot to be acceleration, another may
read a steeper slope to be ‘faster’ – incorrectly projecting their knowledge from distance-time
plots onto the similar representation. Or a student might draw a line chart for discrete data,
incorrectly suggesting interpolation is possible, before switching to a bar chart to remove this
potential source of misunderstanding.
The rep2rep project is working to support people in their representation selection by building
AI tools that can analyse the problem being solved, the representational systems available
to represent the problem, and the person solving the problem. The person matters – what
is trivial for you might be difficult for me. But to accurately assess how suitable a specific
representation is for a specific person, we must analyse their interpretation of the representation.
In previous work, we identified cognitive properties of representations that influence its cost
HLC 2022: 3rd International Workshop on Human-Like Computing, September 28–30, 2022, Cumberland Lodge, UK
∗
Corresponding author.
Envelope-Open a.a.stockdill@sussex.ac.uk (A. Stockdill); g.garcia-garcia@sussex.ac.uk (G. Garcia Garcia);
p.c.h.cheng@sussex.ac.uk (P. C.-H. Cheng); daniel.raggi@cl.cam.ac.uk (D. Raggi); mateja.jamnik@cl.cam.ac.uk
(M. Jamnik)
Orcid 0000-0003-3312-5267 (A. Stockdill); 0000-0002-7327-7225 (G. Garcia Garcia); 0000-0002-0355-5955
(P. C.-H. Cheng); 0000-0002-9207-6621 (D. Raggi); 0000-0003-2772-2532 (M. Jamnik)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
upon a person [4, 5]; however, evaluating these properties requires a structured methodology.
So we are developing Representational Interpretive Structure Theory and Notation (RIST and
RISN, respectively), first presented by Cheng [6], then refined [7]. Section 2 provides a summary.
RIST and RISN serve as the foundation of cognitive property analysis [4]. The rep2rep frame-
work [5] uses libraries that describe representational systems, meaning our automated tools can
analyse new representations and compare them with alternatives within those representational
systems. To support analysts – people who teach with and design representations – in building
these libraries for cognitive properties, we have created the RISN Editor described in Section 3.
2. Representation Interpretation: Theory and Notation
RIST and RISN provide means to modelling the interpretations of representations across many
modalities and domains. At its core, there are four ‘schemas’, and three ‘connections’ between
them. We also identify ‘idioms’ – commonly occurring arrangements of schemas and links. We
Undo Redo Representation R-Scheme R-Dimension R-Symbol Placeholder Duplicate Connect Anchor Equivalence Overlap Disjoint Generic Unlink Delete Grid Manual
〉 〈
provide a brief sketch of the theory and notation; details are available in previous work [6, 7].
RIST has four schemas: Representation, R-Scheme, R-Dimension, and R-Symbol.
Representation R-Scheme R-Dimension, Q R-Symbol
Graphic Graphic Graphic, Q Graphic
A schema associates a concept with a graphic, mapping an aspect of the interpretation to a
feature of the representation. The R-Dimension schema also associates each with a quantity
scale (one of Nominal, Ordinal, Interval, and Ratio [8]; the bold letter replaces Q).
Schemas may be connected in three ways: they are part of a hierarchy, e.g., an R-Symbol
is one element of an R-Dimension; one concept is anchored beneath another, e.g., a region is
14 errors 1 warning Show
anchored by bounding curves; or they are equivalent, requiring mental bookkeeping, e.g., there
are two derivations of one quantity. Each connection has rules on which schemas it may join.
After building many RISN models, we observed recurring patterns; we call these idioms. To
date, the idioms we have collected fit into three classes: collections of elements, how dimensions
are composed and decomposed, and how coordinate systems can be read.
3. Analysis in Practice
Having briefly described the theory and notation, we now consider how RISN models get built.
An analysis consists of three steps in two phases: first, we decide on the interpretation; second,
we (a) identify representation features and how they are modelled in RIST schemas, and (b) build
the RISN model by connecting these schemas. We emphasise the importance of phase one – if
the interpretation is not settled, we risk building an incoherent model that mixes interpretations.
To facilitate building RISN models, we have created a web app, Figure 1. This web app
provides a structured environment in which schemas and links between them are easily created
and updated; analysts do not need to remember every one of the schema’s slots, as the inspector
panel on the right lists them. We provide extensive help and shortcuts to make building RISN
models as effortless as possible, freeing the analyst to focus on the interpretation, not RISN
syntax. The help and shortcuts are shown when the analyst hovers over labels and buttons in
the interface; we also include these in a manual, which describes how to use the editor.
Figure 1: A partial RISN model inside our online editor (left), showing an interpretation of a timetable
(right): a lecture for a subject occurs at a given time and day. D1, etc., are annotations the analyst added.
At the bottom of the window in Figure 1, we have the ‘intelligence’ panel. The RISN editor is
continually analysing the model, checking for both ‘Errors’ – e.g., illegal connections – and
for ‘Warnings’ – features of the model that may be correct, but are unexpected. These support
analysts like static analysers support programmers: the editor can highlight features of the
model so the analyst can catch mistakes, iterate more quickly, and produce high quality models.
We are developing a third intelligence category, ‘Insights’ – primarily for identifying idioms.
The RISN model in Figure 1 is for a student’s interpretation of a timetable, with days presented
horizontally, hours vertically, and subject names in the cells. This seemingly-trivial example
has its depth made apparent by the RISN model: we consider day and time separately from
subject, but bring them together into a new ‘product’: lectures. Each original dimension – day,
time, subject – has concepts distinct from the those arising from the product. Without RIST
and RISN, these subtleties are easy to miss; in Figure 1, these interactions are apparent.
The rep2rep project aims to be accessible to all potential analysts, so we have run two
preliminary workshops on using RIST and RISN. In the first workshop, we taught the basics
over two hours to five participants who are familiar with representation analysis, but not our
framework. During the workshop, participants constructed simple RISN models for alternative
interpretations, but we believe it was too short for them to ingest the subtleties of the theory
and the features of the editor.
For the second workshop, we extended the time to four hours: we spent more time discussing
the similarities and differences between the schemas, particularly addressing the differences
between R-Dimensions, R-Schemes, and class R-Symbols. The participants were more engaged,
and the models they built had fewer errors than the previous participants. Based on our
experiences from these two workshops we are planning a future workshop with teachers, who
are a fantastic audience for representation change, as they work with diverse representations
for many readers. We believe that RIST, RISN, and our editor will be valuable to their practice,
and their feedback valuable to our research.
4. Future work
Ongoing work with RIST and RISN involves analysing representations in the literature. We
shall use our theory to determine the cognitive features of representations, from which we can
estimate their efficacy; this estimate will be compared to the empirical results in the literature.
Making falsifiable, verified claims based on RIST modelling will provide evidence in support of
RIST as a cognitive modelling framework.
Finally, we return to the rep2rep project, and the need to adapt representation selection to
the person using the representation. RIST lets us explore how people interpret representations,
giving us data to design algorithms that consider both the structure of a representation, and
how it will be understood by its reader, to produce appropriate recommendations for problem
solving. Through RIST we gain insight into both the parameters to consider, and the potential
range of values those parameters may take for different people; e.g., idiom use at different
ability levels signals both a meaningful discriminant (the idiom) and its impact (the use), or the
relationship between concepts and graphics (the same graphics used for multiple concepts, or
one concept repeated for multiple graphics).
Acknowledgments
Supported by EPSRC grants EP/R030650/1, EP/T019603/1, EP/R030642/1, and EP/T019034/1.
References
[1] R. Cox, Representation construction, externalised cognition and individual differences,
Learning and Instruction 9 (1999) 343–363. doi:10.1016/S0959- 4752(98)00051- 6 .
[2] Y. Uesaka, E. Manalo, S. Ichikawa, The effects of perception of efficacy and diagram
construction skills on students’ spontaneous use of diagrams when solving math word
problems, in: A. K. Goel, M. Jamnik, N. H. Narayanan (Eds.), Diagrammatic Representation
and Inference, Diagrams 2010, Lecture Notes in Computer Science, Springer, 2010, pp.
197–211. doi:10.1007/978- 3- 642- 14600- 8_19 .
[3] A. Stockdill, G. Stapleton, D. Raggi, M. Jamnik, G. Garcia Garcia, P. C.-H. Cheng, Examining
experts’ recommendations of representational systems for problem solving, in: IEEE
Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022, Accepted
for publication, 2022.
[4] P. C.-H. Cheng, G. Garcia Garcia, D. Raggi, A. Stockdill, M. Jamnik, Cognitive prop-
erties of representations: A framework, in: A. Basu, G. Stapleton, S. Linker, C. Legg,
E. Manalo, P. Viana (Eds.), Diagrammatic Representation and Inference, Diagrams 2021,
volume 12909 of Lecture Notes in Computer Science, Springer, 2021, pp. 1–16. doi:10.1007/
978- 3- 030- 86062- 2_43 .
[5] D. Raggi, G. Stapleton, A. Stockdill, M. Jamnik, G. Garcia Garcia, P. Cheng, How to
(re)represent it?, in: 32nd International Conference on Tools with Artificial Intelligence
(ICTAI), IEEE, 2020, pp. 1224–1232. doi:10.1109/ICTAI50040.2020.00185 .
[6] P. C.-H. Cheng, A sketch of a theory and modelling notation for elucidating the structure of
representations, in: A.-V. Pietarinen, P. Chapman, L. Bosveld-de Smet, V. Giardino, J. Corter,
S. Linker (Eds.), Diagrammatic Representation and Inference, Diagrams 2020, Lecture Notes
in Computer Science, Springer, 2020, pp. 93–109. doi:10.1007/978- 3- 030- 54249- 8_8 .
[7] P. C.-H. Cheng, A. Stockdill, G. Garcia Garcia, D. Raggi, M. Jamnik, Representational Inter-
pretive Structure: Theory and Notation, in: Diagrammatic Representation and Inference,
Diagrams 2022, Accepted for publication, 2022.
[8] S. S. Stevens, On the theory of scales of measurement, Science 103 (1946) 677–680. doi:10.
1126/science.103.2684.677 .