The Evolution of a Tangible for Children’s Conversations: Research Questions and Progress Mehdi Rizvi Faculty of Computer Science, Free University of Bozen-Bolzano, Piazza Domenicani 3, 39100 Bolzano, Italy, srizvi@unibz.it Abstract. Conversations and discussions in classrooms among children and with teachers are an important part of a learning environment. Social Learning (SL) curricula place special emphases on using conversations as learning tools to teach children positive social norms. Smart tangible objects, briefly, tangibles can help scaffold such conversations by pro- moting a multimodal experience. The presented PhD research aims at creating such tangibles. The design process is participatory, based on action-research: prototype solutions are rapidly developed; these are used by users and experts of the SL domain in the field, and their usage is re- flected over to stir design directions or uncover design possibilities. This DC paper reports on the evolution of tangible prototypes: it shows how new design ideas emerged by moving designers into the field and making users adopt prototype solutions in their environment. Keywords: IoT; Tangible; Action Research; Social Learning; Blended Learning 1 Introduction 1.1 Research Problem and Question Social Learning (SL) aims at teaching children positive social behaviours [11]. SL techniques in the education domain also use conversations to teach social skills to children, starting with primary schools. A number of these techniques focus on teaching specific social interaction norms. Maybe the best known is the sharing of turns in a group-based conversation. Mastering this norm means being able to take turns, let others speak and keep the conversation balanced. The turn-sharing norm with groups of children is often regulated in class with physical tokens being passed around. Examples are a stick or a pebble. However such tokens are easily forgotten or not effective in stirring the adoption of the norm. Literature shows that tangible interaction design (ID) solutions could greatly help both learners and educators in conversation-based SL, by adding interactiv- ity and data-logging, which can sustain the adoption of social interaction norms and the reflection about conversation behaviours, e.g., [10, 23]. However, cur- rently, there are no definitive guidelines for designing SL tangibles for children and their learning contexts. The design of such SL tangibles is however complex as it should consider their usability, experience of children, besides how to support the scaffolding of social interaction norms. In view of that, it is not surprising that the existing body of work in interaction design for SL is still in early stages, according to the literature review in [23]. According to Slovak and coauthors, most of the research so far is limited in scope, focusing on specific disadvantaged populations, espe- cially people with autism; this choice crucially leaves out mainstream schools, besides family environments. Moreover, the majority of the existing work is not conducted in ecological settings, or repeatedly so as to assess solutions over time. Therefore the main research question of the PhD is as follows. How to design tangibles for children’s conversations and SL, for their school context, and assess them over time? 1.2 Research Methodology To answer the PhD research question, and considering the issues mentioned in previous section, a meta-design approach is adopted for developing SL tangibles for conversations among children. The design approach crucially involves users, mainly teachers and primary school children, and relevant stakeholders, mainly SL experts, in their learning contexts, over time [5]. The design process takes the form of an action-research process, spiraling through the (1) rapid development of tangibles, (2) and actions with field studies for (3) reflecting about users’ usage of tangibles [14]. Specifically, the process rapidly develops and uses modular vertically-prototyped tangibles. Verticality means that the prototypes come with few critical functionalities to explore in the field. This and modularity help to discard if found unfeasible, or rapidly adapt solutions according to the design possibilities that emerge through actions in the field. In the first PhD year, we developed TurnTalk, a playful tangible for support- ing and enhancing the turn-sharing rule in co-located face-to-face group discus- sions in classroom, by stimulating different learning modalities—visual, auditory, tactile, kinesthetic. Its design and usage in the field are reported in [19, 14]. In particular, its usage by stakeholders and its users in their environment opened up new design possibilities, which are explored in the current PhD year, which is the second, and are reflected over in this paper. 1.3 Outline The paper is organised as follows. Related research is briefly recapped in the opening. Then the paper comes to its core matters and presents the evolution of two novel design solutions, stemming from the usage of TurnTalk in the field. 2 Related Research Interactive solutions for teams of adults. The majority of interactive so- lutions for mediating group conversations have been developed with a focus on adults at work environments. These solutions tend to record the interaction be- tween participants, and provide feedback based on it in order to promote self or group reflection. Examples are Conversation Clock [3] and Second Messen- ger [8], which display different sorts of visual feedback relating to the different conversation statistics or audio patterns. Another well-known solution is pre- sented in [21]: it shows not only visual feedback relating to the conversation but it also shows textual cues to encourage users to participate more actively in the conversation. However group conversations with children in classrooms are different. For instance, they are highly dynamic and, if not structured with, e.g., scripts, they also tend to be highly chaotic: children’s voices tend to overlap, not respect- ing turns in conversations; some children tend to dominate the conversation and leave out less prominent members, e.g., [4, 9]. Moreover, visual metaphors adopted for displaying conversation behaviours in meeting could be difficult for children to grasp. Therefore, interactive solutions developed for adults’ meetings in group at work places cannot be directly used for children in classrooms. Some solutions like Teacher’s Dashboard [18] do exist for educational settings but then again they are either not intended for groups of primary school children or they do not have a particular focus on sustaining their SL. Besides the visual modality, different other modalities for feedback about conversation progress have also been investigated. Grouper, for instance, is meant for coordinating team work [22], acting as group coordinator: through vibration, it alerts its users to pay attention to the leader or current speaker of the group. Interactive solutions for learning contexts. Not many tabletop or screen- based solutions focus on children and learning contexts. Solutions for younger children usually enable them to exchange and manipulate objects through their surface, without addressing specifically conversation. For example, Ely the Ex- plorer [1] is intended for school children for fostering collaboration through the exchange of physical objects. In this solution, each child uses physical objects such as dolls, rotary knobs and RFID cards to interact with the system, which in- teracts back with the children through animations. The animations are designed in such a manner to encourage discussion. Technology-enhanced solutions for conversations among university students have been designed and evaluated in field studies [18]: university students used a shared surface or screen which is touch sensitive, while data, such as ver- bal communication and gestures, is recorded. Students collaborate through the shared work-space, but they are not given any live input or feedback for their collaboration, which is instead given to teachers. Video-conversation solutions. In some instances, a child may be at home or have frequent stays in a hospital due to illness. For such scenarios, video- conversation or remote-presence solutions can be useful. Interactivity seems to be the key for such solutions [15]. Some solutions [2, 17] propose using robotics or surrogates for remote presence for the remote user but the cost and porta- bility of such solutions out-weight their benefits in classrooms. Some tangible solutions [20] provide an option for the user to change the shape or surface of a deformable shape display but the focus is only on communicating in the haptic medium but not in audio or video. With the aim of sharing emotional states, TIE Ground [16] was developed as an interactive carpet to communicate emotions over long distances by tracing footsteps of a remote user and showing them on the carpet at another location. Different prototypes and solutions discussed above focus on different parts rather than the whole solution for supporting conversations for children’s SL. However, each sets the ground for the design of different aspects of the proposed design solution which is discussed in the next sections. 3 The Evolution of TurnTalk TurnTalk was initially developed as a tool to support and promote SL based con- versations for small groups of children of age group 8–10 in classrooms. TurnTalk for co-located conversations has been used with different SL experts and children in learning environments, e.g., [19, 13], and recently with a class of a cooperative primary school in a repeated measure controlled study. The solutions evolving from TurnTalk and proposed in this paper stem from interviews with SL experts or teachers, as well as designers’ observations of children or teachers in their own environment, playfully engaged for design pur- poses like in [7, 6]. The first is TTapp, a mobile app for turning TurnTalk into a blended learning solution and supporting distant learners. The advice of SL ex- perts and work with the class brought in this new design possibility. The second is ClassTalk: TurnTalk adapted for class level conversations. Designers’ observa- tions and interviews with teachers of the primary school class disclosed this new design possibility. 3.1 The TTapp Mobile Solution for Blended SL The original TurnTalk prototype consisted of a pentagon shaped tabletop de- vice for conversations in small groups, of 3 to maximum 5 children [12, 19]. It implements different design features, primarily a play-card mechanism and a progression feedback. The play-card mechanism is enabled as follows. The top cover of the pentagon is split into 5 sectors, one for each child. Children can tangibly play a card and so take or reserve their turn to speak. This mechanism is enforced by a visual feedback of LED pins, which relies on the semaphore metaphor: the current speaker’s pin lights up in green color; another speaker’s LED pin flashes yellow if he or she has reserved the next turn, any other will light up red. The progression feedback is about the conversation behaviour. In the end of the conversation, the progression in conversation is visually displayed as follows: the LED bars on top of the TurnTalk light up for each child proportionally to the number of turns taken by him/her and then to the amount of time taken to talk, with a different color. Other design features are implemented in TurnTalk so as to support educa- tors’ and children’s reflections and assessment of conversation behaviours, but they are not detailed in this paper because they are less relevant here. The newer version of TurnTalk with the TTapp mobile prototype advanced in this paper consists of: (1) the main tangible device, TurnTalk described above, enhanced with audio and video support; (2) a linked mobile app called TTapp. TurnTalk is used in the classroom while the TTapp is used at the remote location of the child, away from the classroom. In TurnTalk and TTapp alike, the interaction behaviours of the play-card mechanism and progression feedback are synchronised as explained and illus- trated in the following. Fig. 1. A conversation session in progress: TurnTalk device in classroom (left) and the remote participant with TTapp mobile app (right) In the novel TurnTalk, there is a ring of 5 microphones, one per side of the pentagon facing children. Moreover, cameras are also placed around the device. In case the speaker is in classroom, the predominant mic and the active camera are those of the child with the speaking turn, that is, the one with the play-card inserted and with the LED pin flashing green in the TurnTalk device. This way the remote user can focus on the voice and facial expression of the person who is talking. In case it is the remote user herself speaking, the non-muffled mic and the active camera in the TurnTalk device are of the user with the reserved turn to talk, e.g., so as to catch visual cues concerning her willingness to continue the conversation. In case there are no reservations, the mic and camera of TurnTalk remain on the most recent speaker in classroom. The interaction behaviour of the mobile TTapp closely resembles that of the TurnTalk device; see Fig. 2. In the device, we have tangible play-cards to book turns. In the app, these are represented by a play-card button which the user can select (e.g, tap) to take a turn to speak or to reserve the next turn. The app screen also shows a pentagon with the same shape as the physical TurnTalk. It also shows the user LEDs to indicate next and current turn and the progression feedback at the end of a conversation. The progression feedback delivered after the conversation is also mirrored on both TurnTalk and TTapp to show conversation distribution in terms of number of turns and turn times. See Fig. 2. Fig. 2. Different aspects of the TTapp mobile app: (1) User#2 speaking with no next turn reserved by any user; (2) User#2 speaking and User#5 (remote user) with next turn reserved by taping the play-Card on the screen; (3) visualisation at the end of conversation showing the progression feedback concerning number of taken turns 4 ClassTalk The repeated-measure study in a cooperative learning primary school opened up a new design possibilities: turning TurnTalk into a device for orchestrating class conversations, after group discussions. During the study, a practice common in the school emerged. When the time for the group conversation is over, all groups have to present the results of their discussion to the class and the teacher, who acts as moderator. Turns in the class conversation are taken by raising hands. A pebble can be given by the teacher to assign speaking turns. However, turns are often not respected, in spite of pebbles. Teachers may not see children out of their line of sight and, again, groups with dominant members get more often speaking turns. ClassTalk evolves TurnTalk into a design solution for these discovered issues. The class has one TurnTalk device per group. During the class conversation, when groups’ results are reported, TurnTalk devices are connected to a mas- ter tangible device. This helps the teacher orchestrate the class conversation. The play-card mechanism of TurnTalk is maintained: a group member can take the turn in speaking on behalf of his group by placing the card in the group’s TurnTalk device. If the LED strips of TurnTalk flash green, then the child has the turn on behalf of his group. If the LED strips flash yellow, then the child has booked the next turn for his group. Otherwise the group has to await for the other groups to finish their interventions. See the example illustration in Fig. 3. A first version of the ClassTalk prototype was brought into the aforemen- tioned primary school classroom and used in the class conversation, reporting on groups’ results. The usage stirred novel design possibilities, related to the role of teachers. During the class conversation, the class teacher wished to stop and reflect on a point made by a group but the ClassTalk prototype at that stage, did not allow for that. The newer version picks up the challenge and im- plements more features through a special teacher card which can be used also for intervention as well as for booking turns in a normal fashion implemented previously. Fig. 3. ClassTalk usage example with three groups and one TurnTalk per group. The group with turn to talk indicated with inner pentagon of LED strips lit up green. More LED strips are lit up green for that group to show turn of the individual member within the group. Similarly yellow for next reserved turn. 5 Conclusions and Future Work The paper reports a PhD research work in the second year, revolving around tan- gibles for children’s conversations and SL. The paper describes the PhD research main question and methodology, with a design process that spirals through rapid prototypes and actions in the field, where users’ use prototypes and designers reflect with users over the usage. It presents the PhD first year’s tangible solution for group conversations in primary schools, TurnTalk, and it shows how the adopted design process led to novel design possibilities, explored in this second year of the PhD: one for blended SL; one for supporting not only group conversations but also class conversations. The meta-design and modular design of TurnTalk, with rapid prototyping, made it possible to rapidly introduce new features according to the discovered design possibilities. The last year of the PhD will cumulate over the design experiences and ac- tions in the field in order to formulate guidelines for the meta-design of tangibles for conversation-based SL for schools. Acknowledgements I thank my Ph.D. advisors G. Dodero and R. Gennari, as well as A. Melonio for their supervision work. References 1. Africano, D., Berg, S., Lindbergh, K., Lundholm, P., Nilbrink, F., Persson, A.: Designing tangible interfaces for children’s collaboration. In: CHI ’04 Ext Abs on Human Factors in Comp Sys. pp. 853–868. CHI EA ’04, ACM, NY, USA (2004) 2. Beer, J.M., Takayama, L.: Mobile remote presence systems for older adults: Accep- tance, benefits, and concerns. In: Proceedings of the 6th International Conference on Human-robot Interaction. pp. 19–26. HRI ’11, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/1957656.1957665 3. Bergstrom, T., Karahalios, K.: Conversation clock: Visualizing audio patterns in co-located groups. In: Proc of 40th Annual Hawaii Int Conf on System Sciences. pp. 78–. HICSS ’07, IEEE Computer Society, Washington, DC, USA (2007) 4. Brondino, M., Dodero, G., Gennari, R., Melonio, A., Pasini, M., Raccanello, D., Torello, S.: Emotions and Inclusion in Co-design at School: Let’s Measure Them!, pp. 1–8. Springer International Publishing, Cham (2015) 5. Cabitza, F., Fogli, D., Piccinno, A.: “Each to His Own”: Distinguishing Activi- ties, Roles and Artifacts in EUD Practices. In: Smart Organizations and Smart Artifacts, pp. 193–205. Springer (2014) 6. Di Mascio, T., Vittorini, P., Gennari, R., Melonio, A., De La Prieta, F., Alrifai, M.: The learners’ user classes in the terence adaptive learning system. pp. 572–576 (2012) 7. Di Mascio, T., Gennari, R., Melonio, A., Tarantino, L.: Engaging “New Users” into Design Activities: The TERENCE Experience with Children, pp. 241–250. Springer International Publishing, Cham (2014) 8. DiMicco, J.M., Hollenbach, K.J., Pandolfo, A., Bender, W.: The Impact of In- creased Awareness While Face-to-face. Hum.-Comput. Interc. 22(1), 47–96 (2007) 9. Dodero, G., Gennari, R., Melonio, A., Torello, S.: Gamified Co-design with Co- operative Learning. In: CHI ’14 Extended Abstracts on Human Factors in Com- puting Systems. pp. 707–718. CHI EA ’14, ACM, New York, NY, USA (2014), http://doi.acm.org/10.1145/2559206.2578870 10. Dourish, P.: Embodied Interaction: Exploring the Founda- tions of a new Approach to HCI. Retrieved online from http://www.ics.uci.edu/ jpd/publications/misc/embodied.pdf (1999) 11. Durlak, J.A., Weissberg, R.P., Dymnicki, A.B., Taylor, R.D., Schellinger, K.B.: The impact of enhancing student’s social and emotional learning: A meta-analysis of school-based universal interventions. Child Development 82(1), 405–432 (2011) 12. Gennari, R., Melonio, A., Torello, S.: Gamified Probes for Cooperative Learning: a Case Study. Multimedia Tools and Applications 76(4), 4925–4949 (2017) 13. Gennari, R., Pavani, F., Rizvi, M.: Tangible Design for Inclusive Conversations with Deaf or Hard-of-Hearing Children. In Proceedings of the 1st International Sym- posium on Emerging Technologies for Education (SETE 2016), LNCS, Springer, Rome (2016) 14. Gennari, R., Melonio, A., Rizvi, M.: Participatory Design of Tangibles for Chil- dren’s Socio-Emotional Learning. Springer International Publishing (2017) 15. Greenberg, A., Colbert, R.: Navigating the sea of research on video conferencing- based distance education. Wainhouse Research, LLC (2004) 16. Hong, Y., Lee, M.: Tie ground: Sense of coexistence. Retrieved online from http://thisiseng.com/en/products/test/ (2016) 17. Hughes, C.E.: Human surrogates: Remote presence for collaboration and education in smart cities. In: Proceedings of the 1st International Workshop on Emerging Multimedia Applications and Services for Smart Cities. pp. 1–2. ACM (2014) 18. Martinez Maldonado, R., Kay, J., Yacef, K., Schwendimann, B.: An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Envi- ronment, pp. 482–492. Springer Berlin Heidelberg, Berlin, Heidelberg (2012) 19. Melonio, A., Rizvi, M.: The Design of TurnTalk for The Scaffolding of Balanced Conversations in Groups of Children. In Proc. of the 1st International Symposium on Emerging Technologies for Education (SETE 2016), Rome (2016) 20. Nakagaki, K., Vink, L., Counts, J., Windham, D., Leithinger, D., Follmer, S., Ishii, H.: Materiable: Rendering dynamic material properties in response to direct physical touch with shape changing interfaces. In: CHI (2016) 21. Schiavo, G., Cappelletti, A., Mencarini, E., Stock, O., Zancanaro, M.: Overt or subtle? supporting group conversations with automatically targeted directives. In: Proc. of the 19th Int. Conf. on Intelligent User Interfaces. pp. 225–234. IUI ’14, ACM, New York, NY, USA (2014) 22. Shaw, F.W., Klavins, E.: Grouper: a Proof-of-Concept Wearable Wireless Group Coordinator. In: Proceedings of the 12th ACM International Conference Adjunct Papers on Ubiquitous Computing. pp. 379–380. ACM (2010) 23. Slovák, P., Fitzpatrick, G.: Teaching and developing social and emotional skills with technology. ACM Trans. Comput.-Hum. Interact. 22(4), 19:1–19:34 (Jun 2015)