Improving Search Strategies of Auditors - A Focus Group on Reflection Interventions

Abstract

Financial auditors routinely search internal as well as public knowledge bases as part of the auditing process. Efficient search strategies are crucial for knowledge workers in general and for auditors in particular. Modern search technology quickly evolves; and features beyond keyword search like facetted search or visual overview of knowledge bases like graph visualisations emerge. It is therefore desirable for auditors to learn about new innovations and to explore and experiment with such technologies. In this paper, we present a reflection intervention concept that intends to nudge auditors to reflect on their search behaviour and to trigger informal learning in terms of by trying out new or less frequently used search features. The reflection intervention concept has been tested in a focus group with six auditors using a mockup. Foremost, the discussion centred on the timing of reflection interventions and how to raise motivation to achieve a change in search behaviour.

Keywords

• Reflection Intervention
• Reflective Learning
• Social Comparison
• Search Behaviour Improvement

-

Finding relevant information for work is necessary in most professions, but the search requirements are different to those in leisure time. Typically, auditors are aware of how to use the features of search platforms when looking for changes in laws and regulations relevant to the audit. At a first glance, there is no need for them to change their routinised search behaviour [ 5 ]. However, the question is if this behaviour is sufficient especially in fast changing technology environments and varying information sources? We assume that auditors could benefit from guidance via reflection interventions by adapting faster to changing technological and environmental changes.

Financial auditors conduct audits to provide an opinion whether financial statements are in accordance with law and compliance. To perform these audits, they have to follow the International Standards on Auditing (ISA), like for example the ISA 3151. Typically, auditors are aware of how to use the features of the search platforms or which information sources to access in order to find relevant information to understand the entity and the environment. However, working with outdated information can have serious negative effects [ 1 ] and thus finding the best information is very crucial for the job performance. Further, currently there are no technologically supported reflection interventions available that nudge auditors to reflect on their own search behaviour and to experiment with less frequently used search features. Reflective learning [ 2 ] is a proven learning strategy for self-directed learning at work [ 9 ] that turns past experiences into learning and a behaviour change.

In this sense, the challenge that we tackle in this paper is to support auditors to learn how to use modern search technology efficiently. We are aware that the challenge of adapting and continuously reflecting on the search behaviour is also relevant for other professions. Our approach combines the observation of search behaviour with providing guidance for reflective learning in order to motivate auditors to reflect about their search behaviour and initiate a change in behaviour if necessary. To achieve this, we present a reflection intervention concept that intends to nudge auditors to reflect on their own search behaviour and to experiment with less frequently used search features to improve their search efficiency. 2

Related Work

Reflective learning: Reflective learning can be seen as the re-evaluation of past experiences with the goal to learn from them to guide new behaviour. This is in line with Boud et al. [ 2 ], who define reflective learning as “those intellectual and affective activities in which individuals engage to explore their experience in order to lead to new understanding and appreciations”. In the area of workplace learning, reflective learning has been identified as a crucial method for informal learning, as it does not rely on explicitly available learning material, curricula, or teachers [ 4,9,8 ].

Role of technology: Technologies like reflection interventions that guide reflective learning can consist of prompting approaches, meaningful visuals and social comparison. They can spark respectively foster reflection in work related settings [ 6 ]. By reflection prompts, we understand interventions such as small text messages that try to motivate a user to reflect. Social comparison in form of performance indicators provide a possibility to compare the own behaviour with the behaviour of others to increase the intrinsic motivation to improve [ 3 ] and thus can therefore stimulate reflective learning [ 13 ]. Visuals that summarise actions taken by the user in the past can be worth being reflected on to influence actions in the future [ 6 ].

Challenges: From literature we know that prompting approaches are very successful in formal learning environments, however in working environments several challenges emerge [ 7 ]. First, in working environments the working tasks are not always known 1 https://www.iaasb.org/clarity-center/clarified-standards beforehand, thus it is very challenging to adapt prompts to the working tasks of a user. Second, in working environments it is often not obvious when to present a prompt (timing). Third, most workplaces are very stressful and work-intensive; thus it has to be very well considered, which type of reflection intervention can be applied. 3

Research Goal

Search satisfaction is defined as the fulfilment of a user’s information need [ 5 ]. As long as this is the case, there is little need to change the search behaviour. Therefore, it is very challenging to motivate users to abandon their comfort zone to change routinised and well known working or searching behaviour or trying out new functionalities of a system [ 10 ], because acquiring new skills through learning is often associated with a significant investment of money and time. This is particular challenging for high skilled knowledge workers such as auditors working under extreme time pressure. On this observation, we therefore conceptualise the goal of our research as supporting auditors to learn how to use a newly developed search platform efficiently.

This goal presents multiple challenges: A first challenge is to embed the learning support into the work process, as time is a critical factor in all workplace learning scenarios (see e.g. [ 7 ]). For design, this means that reflection interventions should not disrupt the users’ workflow. Thus, reflection interventions however need to connect well to users’ operative work procedures. A second major challenge is that learning in this case addresses an activity that is already highly routinised and “works” - even if the existing search strategies may not be optimal given search technology features. 4

Reflection Intervention Design

Our solution approach leans on the work described by Malacria et al. [ 10 ], who designed an intervention that helps users to learn how to use “keyboard shortcuts within Apple Keynote”. They developed a Skillometer for hotkey usage consisting of three parts: a bar-chart showing the time taken to select one of the last six used hotkeys, a meter that grades the user’s performance and a motivational text promoting that hotkeys selections are faster. By taking into account the Skillometer and using our own work conducted on reflective learning technologies [ 6,7,11 ], we developed a reflection intervention concept consisting of a tripartite user interface as depicted in Fig. 1:  Visualisation: Feature Usage. This visualisation (see Fig. 1, top) presents the last five used features of the user (e.g. different search filters used, different visualisations to explore the results) with the goal to raise the user's awareness of the feature usage and subsequently on the search behaviour.  Comparison: Performance Indicator. The indicator provides the possibility to compare oneself with other users using the same search platform. As a representative example, it could present the number of clicks used to receive the desired search results in comparison to the average number of clicks used by other users. By providing this direct comparison, the user's extrinsic motivation to improve should be raised (Fig. 1, middle).  Guidance: Reflection and Recommendation. This part presents either a reflective prompt to motivate the user to reflect on past user behaviour, recommend the user to use another search feature or information source. Thus, the prompts ask for reflection and action (Fig. 1, bottom)

This reflection intervention, presenting real-time information, will be integrated into a search platform developed within the MOVING2 project. This search platform will automatically track the user behaviour serving as input for the reflection intervention. Out of this activity click stream data the used features (Visualisation) as well as predefined performance indicators (Comparison) can be automatically extracted. Furthermore, the reflection guidance questions shown (Guidance) are adapted to the user’s search behaviour. 5

Focus Group & Results

Setting of the evaluation: Company A is one of the Big Four accounting companies operating in 140 countries with worldwide 145.000 professionals. A local office in Germany was involved in the set-up and the participated in the focus group. Participants: Six auditors (4 senior auditors and two junior auditors) from the German innovation group of company A participated in the focus group. Usually they have weekly phone conferences and physically meet 3-4 times a year to discuss new and innovative solutions worth having a closer look at with regard to company A's needs. 2 http://moving-project.eu/ Focus group: To investigate the developed reflection intervention concept with regard to the selected use case ISA 315, we conducted an online focus group [ 12 ]. In this focus group, we provided impulses for discussing possible performance indicators for social comparison (meaning which information would be worth being presented) as well as different types of prompts for initiating reflective learning. We also raised the topic of when (at what time) and where (on which device) to present the performance indicators and reflective prompts to be perceive as useful and not as disruptive.

Results: We discussed the different perspectives of the individual auditor vs. the company as a whole with regard to the performance indicators relevant for the reflection on search behaviour. From a company’s point of view, the search performance and the search time are of crucial relevance, as one can imagine, the quicker the better as time is money. Thus performance indicators should cover this information. Individual auditors showed a more diverse picture depending on their preferences, e.g. the number of click’s used to get the desired results or number of search terms used. Auditors are particularly interested in context-related and often used search terms for improving their search. Participants also suggested to present not only one performance indicator at once but two or three in relation to each other to get more detailed insights on the own search behaviour in comparison to others. Hence, the focus group showed that the performance indicators on the individual level really targeted on learning, whereas the company level indicators just point at the economic effects.

A second topic concerned social comparison in the sense of “With whom should I compare myself with?” Suggestions for comparisons were top performers, persons with high search performance or doing a similar job or who are in the same team. Another suggestion was that the users themselves can select with whom they want to compare with. Here, the goal was to learn from the best in order to improve the own behaviour.

Finally, time and timing was, as one would have expected from existing literature, a key concern. Auditors have very tight timelines, so any learning support that “takes away time” from operative work processes was seen as highly critical. The auditors remarked that especially in the “busy season” at year-end, they would most likely not answer open questions that require manual text input. In this time they would prefer “YES/NO” questions or just ratings. Another possibility would be to present such a prompt for reflection decoupled from the working activity. This means that such a prompt can be presented on a mobile device e.g. on the way home. This can have the advantage that the auditor has time to reflect when a prompt pops up, however, it can also be that the auditor does not want to be bothered with work-related questions after work. The discussion ended with the suggestion that the user should be able to choose in the preferences when and on which device s/he wants to receive a prompt. 6

Discussion & Conclusion

For designing successful reflection interventions it is crucial that the interventions do not disrupt the user’s workflow (see e.g. [ 7 ]). This was also highlighted in the focus group. We see two complementary strategies for addressing this time constraint: First, the reflection intervention could be unobtrusively always visible as in [ 10 ] or second, show up from time to time during work as in [ 6 ]. However, the need for decoupling of search and reflection was an important insight and seems to be very important for this type of reflection intervention. In this strategy this means to present the intervention on demand or proactively via regular notifications on a user-selected schedule (e.g., weekly), in a particular situation (e.g. waiting on a bus stop, sitting in an airplane) and on a user-selected channel (e.g. email, SMS).

Additionally, we note that searching is a highly routinised task for auditors; and we therefore suspect that it will be very challenging to motivate auditors to change their current practice or to use a new search platform. One option to address this challenges is to show the users a clear benefit for themselves by answering the question “What is the benefit for me?” right from the beginning [ 6 ]. Social comparison has been discussed in the focus group as potentially helpful, in the sense of providing relevant information about what features very similar users found useful. Social comparison features have also been very well received in past research on reflection interventions [ 13 ]. However, it needs to be carefully considered which kind of performance indicator to present in order to be useful for the auditor and the company. With this regard, privacy as well as protection issues need to be considered as well. Using automatically tracked user data visualising the social comparison might be perceived rather as an observation tool than as a tool to provide guidance and support reflection. Further from a knowledge protection perspective, disclosing information sources can be also seen as a knowledge leakage [ 11 ].

Moreover, we identified two design implications: First, with regard to the timing of reflection, we plan to investigate which type of intervention stimulates reflective learning more effective: unobtrusive always present interventions vs. interventions that show up automatically during work in comparison to interventions that show-up decoupled from work. This research should bring us a step closer to find out when is a good time for reflection [ 6 ]. Nevertheless, we are aware that in some work-related setting, a reflection intervention as presented above might not be perceived as useful and thus would not achieve the desired result. Second, although social comparison is well investigated in psychology [ 3 ], less research is available in the area of TEL on how to influence the user’s intrinsic motivation with technological support. Thus we aim to close this gap and to explore whether comparison should take the form of competitive comparison or of rather being informative communication about others’ search strategies. Finally, we would encourage research on how to adapt search strategies in general and to research this challenge with other professional groups. 7

Acknowledgement

The project “MOVING - TraininG towards a society of data-saVvy inforMation prOfessionals to enable open leadership iNnovation” is funded under the Horizon 2020 of the European Commission (project number 693092). The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.

References

1. Bachlechner , D. , Thalmann , S. , and Manhart , M. : Auditing Service Providers: Supporting Auditor's in Cross-Organizational Settings . Managerial Auditing Journal 29 ( 4 ) ( 2014 )

2. Boud , D. , Keogh , R. , Walker , D. : Reflection: Turning Experience into Learning, chap . Promoting Reflection in Learning: a Model . pp. 18 - 40 . Routledge Falmer,New York ( 1985 )

3. Corcoran , K. , Crusius , J. , Mussweiler , T. : Social comparison: Motives, standards, and mechanisms . In: (Ed.), I.D.C. (ed.) Theories in Social Psychology. pp. 19 - 139 ( 2011 )

4. Eraut , M. : Informal learning in the workplace . Stud. in Conti. Edu., 26 ( 2 ), 247 - 273 ( 2004 ).

5. Feild , H.A. , Allan , J. and Jones , R.: Predicting searcher frustration . In SIGIR'10 . ACM Press, New York, NY, 34 ( 2010 ).

6. Fessl , A. ; Wesiak , G. ; Rivera- Pelayo , V. ; Feyertag , S. & Pammer , In V. Conole, G.; Klobucar, T. ; Rensing , C. ; Konert , J. & Lavouee , E . (Eds.) In-App Reflection Guidance for Workplace Learning . Proceedings of the 10th European Conference on Technology Enhanced Learning (ECTEL '15) , Springer Int. Publ., 2015 , 9307 , 85 - 99 ( 2015 )

7. Fessl , A. , Blunk , O. , Prilla , M. , Pammer , V. : The Known Universe of Reflection Guidance: a Literature Review. Special Issue on: Awareness and Reflection in Technology-Enhanced Learning, Int . Journal of Technology Enhanced Learning, (forthcoming).

8. Hoyrup , S. : Reflection as a core process in organisational learning . In Journal of Workplace Learning , 16 ( 8 ), 442 - 454 ( 2004 )

9. Knipfer , K. , Kump , B. , Wessel , D. , and Cress , U. : Reflection as catalyst for organisational learning . Studies in Continuing Education ,, 35 ( 1 ), 30 - 48 ( 2013 )

10. Malacria , S. , Scarr , J. , Cockburn , A. , Gutwin , C. , Grossman , T. : Skillometers: Reflective widgets that motivate and help users to improve performance . In: Proc. of the UIST '13 , pp. 321 - 330 . ACM, New York, NY, USA ( 2013 )

11. Manhart , M. & Thalmann , S. : Protecting organizational knowledge: a structured literature review . Journal of Knowledge Management , 19 ( 2 ), 190 - 211 ( 2015 )

12. Morgan , D.L. : Focus groups as qualitative research , vol. 16 . Sage publications ( 1996 )

13. Rivera-Pelayo , V. ; Fessl , A. ; Müller , L. & Pammer , V. Introducing Mood Self-Tracking at Work: Empirical Insights from Call Centers . ACM Trans. Comput.-Hum. Interact . 2017 , 24 ( 1 ), 3: 1 - 3 : 28 ( Feb . 2017 )