12 IoT for Smart City Learning: Towards Requirements for an Authoring Tool Francesco Gianni Simone Mora Abstract Norwegian Univesity of Science Norwegian Univesity of Science Pervasive information visualization and interaction are fun- and Technology and Technology damental tools to support learning in smart cities (SCL), Department of Computer and Department of Computer and for example to promote sustainable behaviours and social Information Science Information Science interaction. Trondheim, Norway Trondheim, Norway francesco.gianni@idi.ntnu.no simone.mora@idi.ntnu.no Building on a review of existing work, we identify the main limitations of traditional approaches based on large displays and smart-phones apps, first from a technological point of Monica Divitini view then connecting to implications for design, user inter- Norwegian Univesity of Science action and experience. and Technology Department of Computer and In the paper we propose a set of authoring primitives at Information Science different semantic levels, ranging from more generic inter- Trondheim, Norway net of things (IoT) primitives to more domain specific ap- monica.divitini@idi.ntnu.no proaches connected to learning in SCL applications. We focus more in detail on new design opportunities, de- veloping possible scenarios of interest that involve SCL. The use of a toolkit for rapid prototyping is proposed as a valuable support instrument for application design and de- velopment. Copyright is held by the author/owner(s). Author Keywords AVI, June 07–10, 2016, Bari, Italy smart objects; internet of things; authoring; smart city learn- ing. 13 ACM Classification Keywords adopting these types of interfaces are (i) to enable the cre- H.5.2 [Information interfaces and presentation (e.g., HCI)]: ation of rich and unobtrusive user experiences, (ii) to extend User Interfaces; H.5.3 [Information interfaces and presenta- the type of data that can be captured to be used as learning tion (e.g., HCI)]: Group and Organization Interfaces content, including sensor data from the environment and from citizens’ whereabouts. Therefore sensor-based TUIs Introduction could complement traditional approaches based on large Studies demonstrate that social connections in cities stim- screens and smartphones, especially when the learning ulates creativity and improves work quality [6]. This is only environment can be as wide and heterogeneous as a city. one of the reasons why the percentage of people living in Todays’ increasingly adoption of sensors and IoT technolo- urban environments is growing. gies are acting as enabling factors for the development of such interfaces; yet whether a number of studies have re- Smart cities present, by definition, a strong technological ported design guidelines for urban screens, there’s a lack of component. In Technology Enhanced Learning (TEL), the guidelines to help the design of different types of interfaces. role of technology is to direct, foster thinking and facilitate the acquisition of higher order skills [9]. Current research As identified during a systematic mapping of the literature applied to learning in the cities seem to focus on two main on smart city learning [7], novel interaction modalities e.g. technological means for learning contents: situated large interactive objects and IoT, are not fully exploited. Even displays and mobile devices, intended as tablets and smart- when used, the affordances employed are only a limited phones [12]. subset of the available ones. Unexplored opportunities emerged also when considering the learning aspect: rather Traditional technology is a limiting factor: mobile devices than communities of citizens in the urban space, the re- and large screens support a very strict and confined set search scenario usually involves schools or governance. of interaction strategies. It’s often not possible to tailor the user experience to properly fit the specific scenario The need for more SCL research involving applications built because technology is too limiting. Our goal is to design around IoT and smart objects suggest the need to define a aiming at the best possible strategy for the users, build- design space and a set of primitives, lying at different se- ing the technology around this process and avoiding the mantic levels, useful to structure and guide the authoring constraints typically introduced by more general-purpose process. hardware/software combinations. Tangible user interfaces as a tool We claim there is a space of opportunity for SCL applica- Characteristics of Tangible Interfaces and Smart Objects tions in adopting novel ubiquitous computing approaches Tangible user interfaces denote systems that rely on “tangi- like tangible user interfaces (TUIs) and augmented objects ble manipulation, physical representation of data and em- (AOs). These technologies have already been found effec- beddedness in the real space”, allowing for an embodied tive in supporting learning [15], but their applications were interaction with digital information. Embodied interaction, mainly oriented to support learning as it happen in conven- as defined by Dourish [5], is a collection of trends emerged tional schools and classrooms. The principal advantages in 14 in HCI, relying on the common ground to provide a more tive aspects that are often quite different. Big diversities are natural user interaction with digital information. observed on the reasons why different cities are defined as smart. Embodied interaction takes the interaction “off the screen” to the real world by distributing inputs and outputs in space This situation is the consequence of the lack of a clear and rather than in time, desequentialising interaction and re- recognized definition of smart city. ducing the gap between where the information is created and where it is accessed. In this picture TUIs seamlessly Komninos [10], in his attempt to delineate the intelligent integrate both representation and control of computation city, (perhaps the concept most closely related to the smart into physical artifacts: “By treating the body of the device city), sees intelligent (smart) cities as “territories with high as part of the user interface -an embodied user interface- capacity for learning and innovation, which is built–in the we can go beyond the manipulation of a GUI and allow the creativity of their population, their institutions of knowledge user to really directly manipulate an integrated physical- creation, and their digital infrastructure for communication virtual device”. When these artifacts also resemble and and knowledge management”. retain the functionalities of traditional objects, they can be Smart cities are also a powerful ecosystem for learning. called smart or augmented objects. Smart city learning aim to support the improvement of all TUIs and smart objects (SOs) allow interaction designers to key factors contributing to the regional competitiveness: be free to experiment with new types of metaphors, taking mobility, environment, people, quality of life and gover- advantage of the users’ physical skills and providing inter- nance. The approach is aimed at optimizing resource con- faces which exploit people’s knowledge with the everyday sumption and saving time improving flows of people, goods non-digital world. and data1 . End user development can also play an important role in Education in this context is pursued as a bottom-up pro- this scenario [11]. The focus is shifted on empowering cess, where person and places are central. Smartness from users that are not familiar with any programming language, a learning perspective exists both in the ambient data col- allowing them to develop and modify the original behav- lected and among the communities that exists within a city. ior of programmable systems. End user development has The separation between student and teacher will fade out. gained interest even in connection with ubiquitous comput- Their role will be content or situation dependent: everybody ing. Several works have explored the possibilities offered will be a learner and the relation between persons will get a by end-users building applications for IoT and ubiquitous bigger role. computing [2] [3]. Characteristics of Smart City Learning Applications Limitations of current technology Technologies like mobile devices, tags, web based appli- Smart City Learning cations, geographical information and e-learning systems The concept of smart-city has also been used in many dif- 1 ferent context and is associated with distinctive and innova- http://www.mifav.uniroma2.it/inevent/events/sclo/ 15 have already been used to develop smart city learning ap- Multi-axial rotation detects with sufficient precision object plications on the field [14] [4]. rotation and tilt; Smart city learning applications consist in the implementa- Shake and displacement intended as the ability to de- tion of urban informatics techniques and approaches to pro- tect when the object is shaken vigorously or is being mote innovative engagement strategies [1]. Studies found physically moved. that urban informatics provide an innovative opportunity to enrich students’ place of learning within the city [1]. FEEDBACK AND OUTPUT Led light can be used as a low fidelity output, more com- No doubt that among the consequences of such attention plex communication strategies can be implemented, there is an acceleration in supporting the integration and like blinking, color fading and led matrix; embedding of ICT within physical environments to realize what has been defined the everyware [8]. Haptic defined as vibration pattern that differ in intensity and duration; Augmented Objects and TUIs for SCL applications We identified a list of primitives at different semantic levels Sound intended as simple beeping or composition of multi- useful to describe, design and author applications for SCL tone sounds. supported by SOs and TUIs: OBJECT AUGMENTATION Untethered operation augmented object should work in- dependently and autonomously, without being hooked to any external device that provides connectivity or power support; Easily embeddable technology should be easy to inte- grate into objects, without altering the original func- tion and nature of the object; Energy autonomous the objects should be as much au- Figure 1: Semantic layers of the primitives. tonomous as possible, effective energy usage, bat- tery efficiency and energy harvesting can help at this regard. Generic level PHYSICAL MANIPULATION “Generic/Domain Specific” overlapping level Touch as the ability to detect interactions like a simple touch, swipes, multiple taps et simila; DATA Sensor data collection intended as the opportunity to em- ploy data gathered in real time from the surrounding 16 ambient. Domain specific data can include for exam- Domain Specific level ple air pollution, geolocation, temperature; LEARNING Data visualization as the ability to support visualization of Reflection occurs when reflecting on previous experience simple information through low fidelity output. Adapt- and behaviours; ing to the domain implies that only a specific subset Behaviour change the explicit goal is to modify or improve of data visualization strategies are suitable. It is also a specific behaviour; important to trigger the learning process and help the users reflect on their experience; Data enabled knowledge is extracted directly from col- lected data; Data processing involves elaboration of data coming from sensors and/or from third party services. Social occurs when it’s the result of a community process; Game based process gamification and situated games INTERACTION STRATEGIES where smart objects extend and improve the game- Object sharing intended more as a dimension, SOs allow play and the learning experience. to freely move from the private and personal sphere to sharable artifacts and community objects, continu- Challenges ously exploring the space in between; Although the need for a set of authoring primitives is driven Background/foreground interaction should be possible by the possibility to follow a more lightweight design ap- when the object is in the center of attention, but also proach and to better adapt to the context, defining an ap- when in background, providing context information propriate semantic level can be intricate since it is closely possibly through glances and nudging; related to the skills of the target group (end-users, devel- opers, designers, etc). A series of challenges are also con- Distributed interaction with several augmented objects nected to the authoring process: orchestrated as a single user interface; Multimodal object interaction based on more than one • Several semantic levels require several competen- strategy, e.g. touch and speech recognition on the cies, which implies that collaboration among experts same object; is fundamental to address complexity along multiple dimensions; Sensor based it’s an enabler for physical manipulation, since smart objects do not provide a dedicated in- • Proposed primitives can be combined in multiple teraction interface (like a keyboard or buttons), the ways, picking the best alternatives for each appli- interaction happens with the object itself. cation can still be challenging; • Data are a valuable source of knowledge to improve the design process, analytics are important to keep 17 track of the process and spot opportunities for im- composition rules, (ii) hardware modules with sensors and provements; actuators to generate and consume primitives, (iii) a soft- ware framework to allow manipulation and use of primitives • The learning process can follow several approaches. within a specific application logic. TILES toolbox aim at sup- Based on the specific domain, can be challenging to porting the iterative process of building prototypes of in- find the most effective; teractive objects, using abstract primitives developed as a • Promoting creativity is essential when dealing with bridge to gracefully support the transition between design different target groups, it is important to balance and and implementation steps [13]. Prototypes might then be define the primitives in a way that are useful to guide released for user testing. the design without introducing heavy constraints that can hinder creativity and original solutions. TILES is a promising project that fits very well with the generic authoring primitives defined above. It is designed taking into account many of the challenges usually found in Examples of applications prototyping toolkits. TILES’ flexible software framework and Example of SCL applications resulting from this approach event-driven messaging system allow to develop abstrac- can be situated augmented games where players interact tions at different semantic levels for the coding process. with smart pones that are also location aware. Pones can This helps to match more closely the skills of end users, be displaced in the environment, shared between players, developer, designers and possibly other categories. can guide the gameplay, support different interaction modal- ities and provide simple triggers for reflection, supporting Conclusion the learning experience. In this article we proposed a new way to empower technol- ogy in SCL scenarios. Starting from the literature, the lim- Another example can involve the use of augmented objects itations of current technological patterns were highlighted, to facilitate cooperation between communities in the city. then AOs and TUIs were introduced as a viable alternative The process of urban planning in Norway require the mu- to more traditional approaches. TILES is then proposed as nicipalities to gather feedback from communities of citizens. a valuable toolkit for rapid prototyping with SOs. Difficulties has been encountered in providing value in the process, since traditional methods do not fit well when deal- Our claim is that the proposed primitives can be combined ing with children for example. Using smart objects that can and used to author and design SCL applications that em- be physically manipulated and provide an engaging expe- power SOs, IoT and TUIs. Strengths and challenges of the rience can motivate and attract participants to collaborate authoring process were also presented. and at the same time stimulate their creativity. Using this approach we believe it will be possible to suc- TILES toolkit for AOs cessfully address several of the challenges that charac- Tiles2 is a rapid prototyping toolkit for AOs, it is composed terize research and interaction in SCL scenarios. More by: (i) a set of abstracted physical interaction primitives and research and feedback are needed to better ground the 2 http://tilestoolkit.io claims proposed. 18 REFERENCES 9. Peter Goodyear and Symeon Retalis. 2010. 1. Glenda Amayo Caldwell, Marcus Foth, and Mirko Technology-enhanced learning. Sense Publishers. Guaralda. 2013. An urban informatics approach to 10. Nicos Komninos. 2002. Intelligent Cities: Innovation, smart city learning in architecture and urban design Knowledge Systems, and Digital Spaces. Taylor & education. Interaction Design and Architecture(s) Francis. Journal (2013). 11. Henry Lieberman, Fabio Paternò, Markus Klann, and 2. Barbara Rita Barricelli and Stefano Valtolina. 2015. Volker Wulf. 2006. End-user development: An Designing for End-User Development in the Internet of emerging paradigm. Springer. Things. In End-User Development. Springer, 9–24. 12. Paul Luff and Christian Heath. 1998. Mobility in 3. Andrea Bellucci, Giulio Jacucci, Veera Kotkavuori, Collaboration. In Proceedings of the 1998 ACM Baris Serim, Imtiaj Ahmed, and Salu Ylirisku. 2015. Conference on Computer Supported Cooperative Work Extreme Co-design: Prototyping with and by the User (CSCW ’98). ACM, New York, NY, USA, 305–314. DOI: for Appropriation of Web-connected Tags. In End-User http://dx.doi.org/10.1145/289444.289505 Development. Springer, 109–124. 13. Simone Mora, Monica Divitini, and Francesco Gianni. 4. Vincenzo Del Fatto and Gabriella Dodero. 2013. 2016. TILES: an inventor toolkit for interactive objects. Geographic Learning Objects in Smart Cities Context. In Proceedings of AVI Conference (Poster). Interaction Design and Architecture(s) Journal (2013). 14. Mar Pérez-Sanagustín, Ilona Buchem, and Carlos 5. Paul Dourish. 2004. Where the action is: the Delgado Kloos. 2013. Multi-channel, multi-objective, foundations of embodied interaction. MIT press. multi-context services: The glue of the smart cities 6. Richard L Florida. 2005. Cities and the creative class. learning ecosystem. Interaction Design and Psychology Press. Architecture(s) Journal (2013). 7. Francesco Gianni and Monica Divitini. 2016. 15. Danae Stanton, Victor Bayon, Helen Neale, Ahmed Technology-enhanced Smart City Learning: a Ghali, Steve Benford, Sue Cobb, Rob Ingram, Claire Systematic Mapping of the Literature. Interaction O’Malley, John Wilson, and Tony Pridmore. 2001. Design and Architecture(s) Journal 27 (March 2016), Classroom Collaboration in the Design of Tangible 28–43. Interfaces for Storytelling. In Proceedings of the SIGCHI Conference on Human Factors in Computing 8. Rudolf Giffinger, Christian Fertner, Hans Kramar, Systems (CHI ’01). ACM, New York, NY, USA, Robert Kalasek, Natasa Pichler-Milanovic, and Evert 482–489. DOI: Meijers. 2007. Smart cities-Ranking of European http://dx.doi.org/10.1145/365024.365322 medium-sized cities. Technical Report. Vienna University of Technology.