Our work focuses on textual and graphical explanations for smart heating systems. We have started to investigate the opportunities for when to provide explanations, how we can design these explanations, and we have started to evaluate these explanations. We argue that explanations need to be carefully crafted to fit with their desired aim, in our case to encourage users' trust and reliance while minimizing user interactions.
Not only are user interfaces for controlling heating systems getting more intelligent, the network and heat sources involved in customers’ heating are also getting smarter. Two such new ways are demand response and hybrid heat pumps. Demand response means fluctuating consumption in the energy network is taken into account to reduce or shift a customer's energy consumption during peak periods. A hybrid heat pump is a heat source which can move energy from outside (either air or the ground) and "pumps" it to a unit where it can be used to provide central heating. In addition to this mode of operation, it can also switch to gas or electricity as a primary heat source. The system can, for © 2018. Copyright for the individual papers remains with the authors.
Copying permitted for private and academic purposes. ExSS '18, March 11, Tokyo, Japan. example, determine to run on gas if the home needs a short temperature "boost" which cannot be achieved with a heat pump that takes longer to heat up. Therefore, hybrid heat pumps are ideally suited for demand response due to their ability to shift to different energy sources without a reduction in comfort.
These modern smart heating systems are highly optimized
to balance comfort and cost. Previous work involving smart
heating systems has called for constrained engagement
between the system and user [
Our work is set in a larger project to understand the overall value and user experience of hybrid heat pump deployment in demand response settings. This project, FREEDOM (Flexible REsidential Energy Demand Optimisation and Management), led by Passiv Systems Ltd. and funded by Western Power Distribution and Wales and West Utilities, is the first of its kind to investigate consumer engagement with these new technologies as part of a smart heating system. We describe a study to understand when and what needs to be explained, how we created designs for a smart system control app, and a study to investigate the impact of aspects of these explanations on user understanding and trust. We conclude with a brief discussion and our plans for future work.
INVESTIGATING OPPORTUNITIES FOR CONSTRAINED
ENGAGEMENT AND CALM TECHNOLOGY
Drawing on previous work in intelligibility, transparency
and explanations [
We found that there are certain points in the system’s behavior when opportunities for constrained engagement present themselves. Participants interacted with the smart heating system when their comfort was compromised or when the system actions were unexpected. Participants expected to be notified when decisions were taken, however, they can be dissuaded from intervening to override system behavior through appropriate explanations.
Participants wanted detailed information about what the system is doing as well as the motivation and benefits for doing them. Textual and very simple graphical visualizations seem to be able to communicate past and future actions but if the content and form are not appropriate it might lead to users to abandon the system and might cause distrust in energy providers (instead of the user interface providers). More complex graphical explanations were often found to be overwhelming and confusing.
DESIGNING THE EXPLANATIONS FOR A SMART
HEATING USER INTERFACE
We then engaged in designing and implementing the user
interfaces for a smart heating system that explains its
behavior to users while trying to minimize follow-on user
interactions. Our premise was that in some instances,
explanations serve as persuasive arguments and can deter
users from correcting system actions [
Next: IN period starts 18:15 Now I1nd6o1.o5N6rso˚.w5˚ 1 indoors
Electricity– heatpump Preheating to 19˚ So you'l be comfortable <in the morning>, your home is preheating. 2 It wil take <X> hrs <Y> mins to reach <19˚> by <18:45>, based on: • The indoor and outdoor temperatures 3 • How wel your home holdsIiNts heat • Your Comfort &Savings settings <Plus, your electricity is currently at a lower rate, so preheating now is better value for you.> Preheating wil continue until the start of the next IN period.
4 5 6 Show graph RemoOveKmessage Edit schedule
Phone is forced landscape 1 Icnotleorufar-cmeaetlcehmtehnotsse on graphical explanation. 2 bCroapcykeintsa<n>gled supplied by system data. 3 Tmeexntutiaolnesxaplllafnaacttioorns on which system relies to make decisions affecting the user. This works as system feedback. 4 oTappepnisngas‘ssohcoiwategdragprha’pbhuicttaoln explanation, for richer customer understanding and more comprehensive system feedback. 5 Tmaepspsiangge‘.OPKa’srseimv/Coviteysto try tracking this interaction for data examination. Message returns at next preheating period etc.
10 7 8 9
Once we had all of this information we began to craft Swyasrtmeimngohno,me lHoewatppouwmerpon Systemof
Preheating tHoegaatsswboitcilehred tHoehaetastwpitucmhepd sCeotminfgosrti&neSfaveicntgs £ rface elements our-match those graphical lanation. py in angled ckets <> plied by system a. tual explanation ntions all factors which system es to make isions affecting user. This works system feedback.
10 soHurecaet TTMeemmoorppe9uieenterrddaaxootptuouoerrrrnsseesi11v111246842e600280 Ah0S2eP611aLr::2tEe44i˚n-E55gP– 00961I–N::944˚55 efficiecnocsyt Lessexpensive 05:00 08:00 £
£ hPerea-ting 18:15I–N21:45
19˚
Screen 1.2 tplnaoisnnmagaetsi‘rossuohnnco,difwaoetrregsrdritacagphnrheda’ripnbhguictataonlnd fbFuaitgcuukr7reteoRd4emt:aurtnGaasiu,rnsaeirnsptochtrteihxectueiaasnllecxe8paxl)aspncealtoiaopnnn.rfaietghiuoernaestidsnhagot.a1w16t)opsEaclaetsuomshreatrntoretadswe,nincwdttiithhnesge‘frygusatptsurhirtaemlei’npamearshetdleea97ssdsh))ed. drebaccokm.prehensive system fsucatuler8de tbUieyssecmrhlceepacsnkeisntorggsaogarlteutugnurcrahrpeehackf1teinae0gtdu)trheerstseoeonlbaooixrtneoisdvff eicavtaelu“e1u2onfchSeeocrnuarlteesaorfufigtntuimrytee.ypCa”xurirsri:1ec1n22titn)himoguteriis1smpa1alwes)ta,y1s2 centre, with graph scrolling past.
with9cutShrcaarnleemonafxttesmettpiImeNrateteumraep,al2xw˚isb:ae2lyo˚wwsarimnerthe middle. ping ‘OK’ removes ssage. Passiv/City to try cking this interaction for a examination. MessageexplanatiopMrenetdsOicftfeifcdoetefremepdae).lralturtehoeutsdoeorsd(seetcbiysion points. Feedback during urns at next preheating iod etc. the previous study had indicated that graphical explanations ping ‘Edit schedule’ takews ere c1o0ngVtarealupnehstdiooofn‘’Huterseafltesacotunarcbdesoclsoustiet mepfrfiicpciinelgne.cy’textual feedback also fit with r to occupancy schedule. Instead, this scale shows relative the notio n‘exopefnsciveanlemss’ fotrethcehcunstoomleor,gy. Hence, our guiding principle was thattbheetahraimnegoiunndmtoienfdhfeathauettchloetsftuoeefl txshoepufrulceael nanadtion would be a simplified textual exgepnelraatens.ation, and every time a new decision was made this was communicated (Figure 3). If the user disagrees with the system, appropriate actions, e.g. to edit the heating schedule, were supported by the explanations.
More complex graphical explanations could be shown on request which visualized the main data underlying the system behavior and its decisions. To provide flexibility, a particular combination of data can be shown in response to a system decision, for example, in pre-heating (Figure 4) a wide variety of data figures into the indoor temperature. However, the user can select (or de-select) which data is presented, i.e. the user can simplify the graphical explanation (or indeed show more data, if desired).
We carried out some limited usability testing on these explanations which showed that there were no major problems in using them. These designs were presented to the project partners to use as a basis for the design of the control app for the hybrid heat pump trial. In addition, we also designed a heating cost calculator that showed the impact of raising or lowering temperature set points in the schedule on heating bills.
SMART HEATING USER INTERFACES AND TRUST More recently, we have undertaken a study to explore the effect of these different types of explanations on users. Recall that our premise was that increased understanding and trust might lead to not correcting the system. We carried out a between-group experiment to assess understanding and trust with 60 participants with one of four explanation conditions (15 participants each): Control (no explanation), Text-only (like Figure 3), Graphical-only (like Figure 4), and Both (explanation consisting of both graphical and written elements). Each participant was 8 Comfort& Savings Temp.outdoors Temp.indoors Preheating Oc upancyperiod Primaryheat source Heatsource costeficiency Electricityhigh demandperiod presented with 5 simulated home heating scenarios: Normal maintaining, Normal preheating, Normal away mode, Preheating due to Demand Response, and Delayed heating due to Demand Response.
We captured participants’ perceived and demonstrated understanding (the latter coded from free text responses), trust ratings after each scenario, and qualitative feedback.
The results showed that explanations, in particular textbased explanations, improved demonstrated understanding, yet there was no significant effect on perceived understanding (Figure 5) based on explanation type. However, perceived understanding had a strong correlation with trust (Figure 6). Giving different types of explanations did not have an effect on trust in the system, however, participants without explanations were more sensitive to predictability of system behavior while participants with any of the explanations focused more on their perception of what the system was doing and whether they agreed with it. In order to improve trust, participants frequently asked for even more information and data to be provided especially about costs.
DISCUSSION AND FUTURE WORK There are many options for providing explanations and controllability in a smart heating system. Here we discuss some of the findings and implications from our work so far.
Our work has explored when to provide explanations for a smart heating system. We believe that this work could be extended and generalized but that points for “constrained engagement” need to be investigated for each system.
We have also started to investigate what should be communicated in these explanations. We have focused on the “input data” and “output data” instead of the details of the algorithmic model as these aspects are more readily explained both in textual as in graphical form. However, we also noted that users expect to see reasons for making decisions and potential benefits to them. This area warrants further investigation, especially as this seems to be very dependent on the systems and its context of use. There have also been suggestions that “what-if” explanations could be very useful but so far our work has not explored this with users, and neither have we had any calls for this clustering of participants in the scatterplots and heatmaps below. Therefore, though it cannot be said that actual understanding increased trust, it is true that the more that people thought they understood the system, the more they trusted it.
FFiigguurrese18 a6n:d 19C. Hoerartemlaaptsioshnowinbgetthwe efreenquenrcyaotifnpgarsticipoanfts fporeeraccehilveevedl of utnrudstearnsdtaeancdhilenvgel,ofapnerdceivcehdaunndgeerstainndintgr.uLesftt to(lreigfhtt): chaanndge infitnruastl, fitnraultsrtust. ratings (right). functionality from participants in our studies.
Our designs have explored how to provide explanations. All evidence seems to suggest that graphical explanations need to be employed cautiously: some people love them, some people hate them. However, textual explanations, even if high-level, seem to be valuable.
We have not yet investigated the effects on perceived controllability of our UI, mainly because this is very difficult to do in a lab study since changes in temperature in response to changes in settings are very hard to simulate. However, we believe that there is a link between intelligibility, controllability and user experience but that this relationship is complex and needs to be further explored. 35 In future work, we will be participating in a field study over the heating season in 2017/18 during which we will have the opportunity to gather feedback from users in a real world setting after the complete smart heating system has been installed. This will provide some answers about how the explanations are perceived, and whether this might have any effect on interacting with the smart heating app. 5. 6. 7. 8.
Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY, USA, Article 74, 10 pages. DOI: https://doi.org/10.1145/2971485.2971525 René F. Kizilcec. 2016. How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16).
ACM, New York, NY, USA, 2390-2395.
Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell me more?: the effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12).
ACM, New York, NY, USA, 1-10.
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI '15).
ACM, New York, NY, USA, 126-137.
Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New
York, NY, USA, 2119-2128.