Cognitive Analysis for Representation Change Aaron Stockdill1,∗ , Grecia Garcia Garcia1 , Peter C.-H. Cheng1 , Daniel Raggi2 and Mateja Jamnik2 1 University of Sussex, Brighton, UK 2 University of Cambridge, Cambridge, UK Abstract The rep2rep project is developing an AI tool to automatically select an appropriate representation to solve a particular problem for a particular person. A prerequisite of this tool is to understand (i.e., model) how a reader interprets a representation. But interpretations can vary wildly between novices and experts, readers of similar ability, or even the same reader in different tasks. We present a theory and notation (RIST and RISN) for analysing the cognitive features of a representation’s interpretation, and introduce a web app to construct RISN models. These models provide information about cognitive properties of representations to guide automated representation selection to support human problem solving. Keywords Cognition, Representations, Interpretation, Schemas 1. Introduction When solving problems, representation choice can be critical [1]. Unfortunately, choosing the right representation is difficult: for example, in a classroom context, students struggle to change representation, and teachers can be inconsistent in what they recommend [2, 3]. Where one student (correctly) understands slope in a velocity-time plot to be acceleration, another may read a steeper slope to be ‘faster’ – incorrectly projecting their knowledge from distance-time plots onto the similar representation. Or a student might draw a line chart for discrete data, incorrectly suggesting interpolation is possible, before switching to a bar chart to remove this potential source of misunderstanding. The rep2rep project is working to support people in their representation selection by building AI tools that can analyse the problem being solved, the representational systems available to represent the problem, and the person solving the problem. The person matters – what is trivial for you might be difficult for me. But to accurately assess how suitable a specific representation is for a specific person, we must analyse their interpretation of the representation. In previous work, we identified cognitive properties of representations that influence its cost HLC 2022: 3rd International Workshop on Human-Like Computing, September 28–30, 2022, Cumberland Lodge, UK ∗ Corresponding author. Envelope-Open a.a.stockdill@sussex.ac.uk (A. Stockdill); g.garcia-garcia@sussex.ac.uk (G. Garcia Garcia); p.c.h.cheng@sussex.ac.uk (P. C.-H. Cheng); daniel.raggi@cl.cam.ac.uk (D. Raggi); mateja.jamnik@cl.cam.ac.uk (M. Jamnik) Orcid 0000-0003-3312-5267 (A. Stockdill); 0000-0002-7327-7225 (G. Garcia Garcia); 0000-0002-0355-5955 (P. C.-H. Cheng); 0000-0002-9207-6621 (D. Raggi); 0000-0003-2772-2532 (M. Jamnik) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) upon a person [4, 5]; however, evaluating these properties requires a structured methodology. So we are developing Representational Interpretive Structure Theory and Notation (RIST and RISN, respectively), first presented by Cheng [6], then refined [7]. Section 2 provides a summary. RIST and RISN serve as the foundation of cognitive property analysis [4]. The rep2rep frame- work [5] uses libraries that describe representational systems, meaning our automated tools can analyse new representations and compare them with alternatives within those representational systems. To support analysts – people who teach with and design representations – in building these libraries for cognitive properties, we have created the RISN Editor described in Section 3. 2. Representation Interpretation: Theory and Notation RIST and RISN provide means to modelling the interpretations of representations across many modalities and domains. At its core, there are four ‘schemas’, and three ‘connections’ between them. We also identify ‘idioms’ – commonly occurring arrangements of schemas and links. We Undo Redo Representation R-Scheme R-Dimension R-Symbol Placeholder Duplicate Connect Anchor Equivalence Overlap Disjoint Generic Unlink Delete Grid Manual 〉 〈 provide a brief sketch of the theory and notation; details are available in previous work [6, 7]. RIST has four schemas: Representation, R-Scheme, R-Dimension, and R-Symbol. Representation R-Scheme R-Dimension, Q R-Symbol Graphic Graphic Graphic, Q Graphic A schema associates a concept with a graphic, mapping an aspect of the interpretation to a feature of the representation. The R-Dimension schema also associates each with a quantity scale (one of Nominal, Ordinal, Interval, and Ratio [8]; the bold letter replaces Q). Schemas may be connected in three ways: they are part of a hierarchy, e.g., an R-Symbol is one element of an R-Dimension; one concept is anchored beneath another, e.g., a region is 14 errors 1 warning Show anchored by bounding curves; or they are equivalent, requiring mental bookkeeping, e.g., there are two derivations of one quantity. Each connection has rules on which schemas it may join. After building many RISN models, we observed recurring patterns; we call these idioms. To date, the idioms we have collected fit into three classes: collections of elements, how dimensions are composed and decomposed, and how coordinate systems can be read. 3. Analysis in Practice Having briefly described the theory and notation, we now consider how RISN models get built. An analysis consists of three steps in two phases: first, we decide on the interpretation; second, we (a) identify representation features and how they are modelled in RIST schemas, and (b) build the RISN model by connecting these schemas. We emphasise the importance of phase one – if the interpretation is not settled, we risk building an incoherent model that mixes interpretations. To facilitate building RISN models, we have created a web app, Figure 1. This web app provides a structured environment in which schemas and links between them are easily created and updated; analysts do not need to remember every one of the schema’s slots, as the inspector panel on the right lists them. We provide extensive help and shortcuts to make building RISN models as effortless as possible, freeing the analyst to focus on the interpretation, not RISN syntax. The help and shortcuts are shown when the analyst hovers over labels and buttons in the interface; we also include these in a manual, which describes how to use the editor. Figure 1: A partial RISN model inside our online editor (left), showing an interpretation of a timetable (right): a lecture for a subject occurs at a given time and day. D1, etc., are annotations the analyst added. At the bottom of the window in Figure 1, we have the ‘intelligence’ panel. The RISN editor is continually analysing the model, checking for both ‘Errors’ – e.g., illegal connections – and for ‘Warnings’ – features of the model that may be correct, but are unexpected. These support analysts like static analysers support programmers: the editor can highlight features of the model so the analyst can catch mistakes, iterate more quickly, and produce high quality models. We are developing a third intelligence category, ‘Insights’ – primarily for identifying idioms. The RISN model in Figure 1 is for a student’s interpretation of a timetable, with days presented horizontally, hours vertically, and subject names in the cells. This seemingly-trivial example has its depth made apparent by the RISN model: we consider day and time separately from subject, but bring them together into a new ‘product’: lectures. Each original dimension – day, time, subject – has concepts distinct from the those arising from the product. Without RIST and RISN, these subtleties are easy to miss; in Figure 1, these interactions are apparent. The rep2rep project aims to be accessible to all potential analysts, so we have run two preliminary workshops on using RIST and RISN. In the first workshop, we taught the basics over two hours to five participants who are familiar with representation analysis, but not our framework. During the workshop, participants constructed simple RISN models for alternative interpretations, but we believe it was too short for them to ingest the subtleties of the theory and the features of the editor. For the second workshop, we extended the time to four hours: we spent more time discussing the similarities and differences between the schemas, particularly addressing the differences between R-Dimensions, R-Schemes, and class R-Symbols. The participants were more engaged, and the models they built had fewer errors than the previous participants. Based on our experiences from these two workshops we are planning a future workshop with teachers, who are a fantastic audience for representation change, as they work with diverse representations for many readers. We believe that RIST, RISN, and our editor will be valuable to their practice, and their feedback valuable to our research. 4. Future work Ongoing work with RIST and RISN involves analysing representations in the literature. We shall use our theory to determine the cognitive features of representations, from which we can estimate their efficacy; this estimate will be compared to the empirical results in the literature. Making falsifiable, verified claims based on RIST modelling will provide evidence in support of RIST as a cognitive modelling framework. Finally, we return to the rep2rep project, and the need to adapt representation selection to the person using the representation. RIST lets us explore how people interpret representations, giving us data to design algorithms that consider both the structure of a representation, and how it will be understood by its reader, to produce appropriate recommendations for problem solving. Through RIST we gain insight into both the parameters to consider, and the potential range of values those parameters may take for different people; e.g., idiom use at different ability levels signals both a meaningful discriminant (the idiom) and its impact (the use), or the relationship between concepts and graphics (the same graphics used for multiple concepts, or one concept repeated for multiple graphics). Acknowledgments Supported by EPSRC grants EP/R030650/1, EP/T019603/1, EP/R030642/1, and EP/T019034/1. References [1] R. Cox, Representation construction, externalised cognition and individual differences, Learning and Instruction 9 (1999) 343–363. doi:10.1016/S0959- 4752(98)00051- 6 . [2] Y. Uesaka, E. Manalo, S. Ichikawa, The effects of perception of efficacy and diagram construction skills on students’ spontaneous use of diagrams when solving math word problems, in: A. K. Goel, M. Jamnik, N. H. Narayanan (Eds.), Diagrammatic Representation and Inference, Diagrams 2010, Lecture Notes in Computer Science, Springer, 2010, pp. 197–211. doi:10.1007/978- 3- 642- 14600- 8_19 . [3] A. Stockdill, G. Stapleton, D. Raggi, M. Jamnik, G. Garcia Garcia, P. C.-H. Cheng, Examining experts’ recommendations of representational systems for problem solving, in: IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022, Accepted for publication, 2022. [4] P. C.-H. Cheng, G. Garcia Garcia, D. Raggi, A. Stockdill, M. Jamnik, Cognitive prop- erties of representations: A framework, in: A. Basu, G. Stapleton, S. Linker, C. Legg, E. Manalo, P. Viana (Eds.), Diagrammatic Representation and Inference, Diagrams 2021, volume 12909 of Lecture Notes in Computer Science, Springer, 2021, pp. 1–16. doi:10.1007/ 978- 3- 030- 86062- 2_43 . [5] D. Raggi, G. Stapleton, A. Stockdill, M. Jamnik, G. Garcia Garcia, P. Cheng, How to (re)represent it?, in: 32nd International Conference on Tools with Artificial Intelligence (ICTAI), IEEE, 2020, pp. 1224–1232. doi:10.1109/ICTAI50040.2020.00185 . [6] P. C.-H. Cheng, A sketch of a theory and modelling notation for elucidating the structure of representations, in: A.-V. Pietarinen, P. Chapman, L. Bosveld-de Smet, V. Giardino, J. Corter, S. Linker (Eds.), Diagrammatic Representation and Inference, Diagrams 2020, Lecture Notes in Computer Science, Springer, 2020, pp. 93–109. doi:10.1007/978- 3- 030- 54249- 8_8 . [7] P. C.-H. Cheng, A. Stockdill, G. Garcia Garcia, D. Raggi, M. Jamnik, Representational Inter- pretive Structure: Theory and Notation, in: Diagrammatic Representation and Inference, Diagrams 2022, Accepted for publication, 2022. [8] S. S. Stevens, On the theory of scales of measurement, Science 103 (1946) 677–680. doi:10. 1126/science.103.2684.677 .