In this position paper we show how micro-intelligences can be used to remotely collect behavioral, contextual and experiential data. We introduce two micro-intelligences that are built for being the physiological and afect annotation layers of a broader intelligent ecosystem. We describe how we used this system to conduct rapid, autoethnographic experiments to understand how working from home impacts emotional states and physiology of ofice employees. We discuss how our micro-intelligences build on top of one another and how they can be enriched using a qualitative reflection layer. We aim to inspire novel directions for creating situated, adaptive design interventions that can enable afective computing using IoT components.
When designing for afective computing [
We were in the process of setting up the contextual step of a study at a hospital when the Coronavirus pandemic began. The circumstances made it impossible to follow our initial intentions to iteratively develop, test and improve our intelligent ecosystem together with healthcare professionals as study participants. We realized that our initial setup for the research-oriented contextual step too much relied on the possibility to be present in real-life at the hospital context. Therefore, we decided to take a step back, and focused on the improvement of the experience-related and the functionalityrelated aspects of our upcoming contextual step by running small experiments in the working from home context of our study team. In this position paper we show how our team built a physiological and an afect annotation layer for post-hoc reflections, collecting physiological signals and enabling event annotations. We introduce the concept of micro-intelligences for situated afective computing. We discuss our study setups, and the results of our experiments. Finally, we conclude on our plans to combine the separate layers in one intelligent ecosystem that can be used for conducting our contextual DED step. 2
We started our project with the intention to be able to remotely collect behavioral, contextual and experimental data from healthcare professionals (HCP) during their work hours from their hospital context. The topic of our research is to find out what are moments of interest related to the HCP’s work-life experience. During the ifrst, preparation phase of our project we were still able to carry out interviews, co-creation sessions, and a shadowing study with our clinical partners. Based on these activities we have made the following design decisions for our planned contextual DED step at the hospital. 2.1
Firstly we looked for a method to gather post-hoc reflections about the participating healthcare professional’s workdays. We needed to be careful as we did not want to increase their workload. Therefore, we need to limit the number of interactions with our ecosystem during their workday. We decided to focus on enabling them to reflect on their workday when they are finished with all their tasks. In practice, this would require about 5 minutes of their time to answer questions that are asked via a Chatbot that runs on one of the major messaging platforms. By having reflection questions asked on their phone we expect the professionals to be able to recall the interesting moments that happened during their workday. Moreover, many of them have access to their work calendar on their phone and therefore can consult this data stream to remember what they were doing at the specific timeframes they’re interested to share with us about their workday. 2.2
In combination with the post-hoc reflections our team is also
interested whether there are physiological signals that can be measured
at the moments that are of interest for the healthcare professionals.
Therefore, we looked for a way to collect objective behavioral data
from the context. We decided to collect Heart rate variability (HRV)
data. HRV represents non-invasive, unobtrusive information about
modulation of heart rate by the autonomic nervous system in a
variety of dynamic circumstances, including evoked emotions, and
exercise [
Based on our pre-contextual study steps we noted that sharing
the overall explanation of one’s workday with our design team
can wait until the participant is finished with their workday. We
described in our post-hoc reflection section that it is possible to
recall an overall impression about the day and potentially use other
data sources one has access to. However, there are moments that
are important to be noted at the moment they happen. Our team
decided to look into the possibility to make it easily possible in the
contextual step to mark the exact time of such events happening in
a participant’s workday by a physical design probe (e.g. an event
annotation button). Marking these events have two main advantages:
(1) it can function as an additional data stream for the participant
when answering the post-hoc reflection questions, (2) the design
(researchers) can use the annotated events in combination with
the physological signals. Both of these techniques result in a better
understanding of the workday.
Inspired by the micro-services architecture [
The physiological layer is a micro-intelligence that was built
applying our design decisions to (1) enable post-hoc reflections on
one’s workday, and (2) to facilitate the collection of HRV data
remotely. This layer consists of the following components: (1) a
Scosche Rhythm24 Heart Rate Monitor, (2) a messaging platform,
(3) a Bluetooth Low Energy (BLE) enabled device (e.g. a computer,
or a smart phone), (4) software tools enabling HRV analysis
(Kubios [
The results of the data collection are in Table 1. The data suggests
that the highest percentage of missing data occurs while performing
the cleaning activity (58%). That can be explained by the
significant amount of movement during the data collection process and
connectivity to the receiving device that afects the quality of the
signal and subsequently leads to data loss. It can be expected that
devices using the same technology will have the same problem.
In the meantime, the lowest percentage of losing data appears for
resting phase data collection (8%) while the participant is located
close to the signal receiving device and is not involved in any
dynamic activities. This high data loss highlights the limitation of
relying solely on the HRV data. The HRV only indicates arousal,
but not valence. Thus, to meaningfully understand the experience
and clearly distinguish the context between the same HRV results,
the physiological states derived from the HRV need to be linked to
the ground truth to express valence. This can be achieved by
involving additional input sources that provide the ability for collecting
post-hoc reflection, and event annotation data. At the same time,
our micro-intelligence setup has the advantage of being able to
continue this experiment with the aim of determining the possible
improvements (e.g. the location of the data receiving device) that
would facilitate the quality of calculated HRV parameters while
continuing our preparation for the contextual DED step [
Based on the collected data and the insights we gathered via the interview it was clear that our hypothesis that the participant could reflect on their day looking at their daily visualization was conifrmed. The participant reported that the sensor’s measurements were a good indicator of the moments when their heart rate variability was diferent compared to their baseline. When the participant was looking at their visuals it was possible to let our team know about what happened at those moments. Many times the participant could not fully recall what happened in their workday during the highlighted time but in this case they looked at their agenda. We also noted a significant experience-related finding, namely that measuring heart rate variability made the participant pay more attention to the stressful or rather negative moments of their day when sending us their post-hoc reflections. This finding has enabled our team to better scope the afect annotation layer of our DED study. 3.2
The afect annotation layer is a micro-intelligence that was built applying our design decisions to (1) enable post-hoc reflections on one’s workday, and (2) to facilitate event annotations. This layer consists of the following components: (1) two Flic buttons, (2) a messaging platform that can be programmed by Flow.ai, (3) a smartphone, and (4) software tools enabling event annotation analysis (e.g. AWS Lambda function, local Python script). We tested the experience-related and the functionality-related capabilities of this micro-intelligence by collecting data from three members of our research team from their working from home context. 3.2.1 Collecting post-hoc reflections. Our way of working of developing our DED contextual step as micro-intelligences enabled our team to run a study that focused on our hypothesis related to the experience-related aspect of our design decision to enable post-hoc reflection’s on one’s workday. For this experiment we intended to figure out whether a chatbot would be adequate to use as a daily reflection journal. We setup a WhatsApp group with our third participant and asked them to use it as their personal diary to collect their reflections on working from home as a designer. Our design researcher sent a list of generic working from home related topics at the end of every workday. The participant could choose what to reflect on from that list. This provided them a trigger for sending us their post-hoc reflections. The participant was also allowed to reflect on something that was not present on the list. We ran this experiment for seven days. Our main finding was that this way of post-hoc reflection quickly became monotonous and too repetitive and this resulted in less and less nuanced responses from our participant. Therefore, we decided to focus our team’s attention to develop the afect annotation layer further to overcome these shortcomings. 3.2.2 Combining post-hoc reflections with event annotations. Our ifnding about the need to enable participants to be able to report positive events happening during their workday that is described in 3.1.1 highlighted the need to provide an event annotation option for them. Moreover, we wanted to make sure that the way we collect post-hoc reflections at the end of the workday does not become monotonous and repetitive. Flic buttons are Bluetooth Low Energy enabled IoT devices that can be used for this purpose. The buttons are connected to their companion app running on a smartphone. When one presses the buttons it sends this to the Flic app. Inside the Flic application it is possible to program what action such physical press would trigger. We gave two Flic buttons each to our second and fourth participant. We instructed them to press these buttons during their workday when they recognized that either positive or negative moments were happening. Inside the Flic app we setup a POST request that would send JSON data containing the timestamp and whether the positive or the negative button was pressed to our backend. We programmed the Flow.ai Chatbot platform to trigger a WhatsApp message at 17.00 after the end of the participants’ workday. The participants were asked to rate their workday. In response to their reply they were presented with a list of their positive and negative moments. Finally, they were asked to elaborate on their day (See figure 3). We ran this experiment for 7 days with two participants working from home. After the experiment we conducted data-enabled design interviews where the participants were presented with their collected data, and they could reflect on it.
Based on the collected data and the follow-up interviews of this experiment we learnt that the insights we could collect from our participants about their workday difered a lot from the insights we gathered using the physiological layer or during the experiment for collecting post-hoc reflections. We could notice from the replies that the participants were focusing to tell more about the positive aspects of their workday. They reported this was also due to the clear conscious action required to press a physical button. During their end of the day reflection it was mostly possible to recall why they pressed the buttons. Based on this experiment we verified that we are able to execute our design decision to introduce event annotations as a reflection tool on one’s workday. 4
In this position paper we introduced what micro-intelligences are. Based on the evaluation of four experiments carried out in the working from home context of our design research team we reflected on their characteristics when designing and developing them as preparation for a contextual DED step.
Firstly, as each micro-intelligence is a self-contained layer of a broader intelligent ecosystem it is possible to reuse them across several DED studies. We showed that this is the case as the four experiments introduced across the two micro-intelligences introduced in this paper all use common components and infrastructure and are developed in a way that it would be possible to reuse them by other design (research) teams.
Secondly, we showed that when using the designed micro-intelligences it is possible to evaluate modular components of an intelligent ecosystem. In our experiments we were able to adjust whether the design (research) would focus on testing the experience-related or the functionality-related aspect of a component. We evaluated our design decision of collecting post-hoc reflections using a chatbot in three diferent ways (see figure 4). First, we used the chatbot component to learn about the experience-related aspects of collecting post-hoc reflection data. Then we combined the chatbot with heart rate variability data and could focus our attention on both functionality related (can participants provide post-hoc reflections based on their HRV data) and experience-related (what graphs to include in the chatbot messages) aspects. We could also separately test the functionality-related aspect of how to connect our chatbot to the afect-annotation layer that enables event annotations. Lastly designing micro-intelligences enables the testing of the level of automation required to be developed for the modular components of an intelligent ecosystem. In our four experiments presented in this positioning paper we could flexibly decide whether to use the components as fully-automated data trackers (a chatbot that can automatically retrieve data from the event annotation component) or in a Wizard of Oz prototyping style (the design researcher manually generated and sent out the HRV graphs to the participant). As stated earlier, our next step is to combine the separate microintelligence layers in one intelligent ecosystem, that can be used to collect, analyze and visualize behavioral, contextual, and experimental data from healthcare professionals that work at a collaborating hospital partner. Our research will focus on enabling the healthcare professionals to annotate positive and negative moments of their workday. We will also collect their physiological signals to see whether their reported moments could be confirmed by objective data. At the end of their workday we will collect post-hoc reflections using a chatbot.
With the work presented in this paper, we set out to employ a micro-intelligence approach for designing and testing two layers for situated afective computing (the physiological layer and the afect annotation layer). The experiments discussed enabled us to further refine our design decisions and to uncover experiential or functional gaps in our micro-intelligences. With this, we have tried to take a first step towards designing a broader intelligent ecosystem for conducting afective computing in a situated manner.