=Paper=
{{Paper
|id=Vol-1997/paper2
|storemode=property
|title=Improving Search Strategies of Auditors – A Focus Group on Reflection Interventions
|pdfUrl=https://ceur-ws.org/Vol-1997/paper2.pdf
|volume=Vol-1997
|authors=Angela Fessl,Viktoria Pammer,Michael Wiese,Stefan Thalmann
|dblpUrl=https://dblp.org/rec/conf/ectel/FesslPWT17
}}
==Improving Search Strategies of Auditors – A Focus Group on Reflection Interventions ==
https://ceur-ws.org/Vol-1997/paper2.pdf
Improving Search Strategies of Auditors – A Focus
Group on Reflection Interventions
Angela Fessl1, Viktoria Pammer1,2, Michael Wiese3, and Stefan Thalmann1,2
1
Know-Center, Inffeldgasse 13, A - 8010 Graz
{afessl,sthalmann,vpammer}@know-center.at
2
Graz University of Technology, Institute of Interactive Systems and Data Science,
Inffeldgasse 13, 8010 Graz
3 Ernst & Young Wirtschaftsprüfungsgesellschaft, Wittekindstraße 1a, 45131 Essen
michael.wiese@de.ey.com
Abstract. Financial auditors routinely search internal as well as public
knowledge bases as part of the auditing process. Efficient search strategies are
crucial for knowledge workers in general and for auditors in particular. Modern
search technology quickly evolves; and features beyond keyword search like fac-
etted search or visual overview of knowledge bases like graph visualisations
emerge. It is therefore desirable for auditors to learn about new innovations and
to explore and experiment with such technologies. In this paper, we present a
reflection intervention concept that intends to nudge auditors to reflect on their
search behaviour and to trigger informal learning in terms of by trying out new
or less frequently used search features. The reflection intervention concept has
been tested in a focus group with six auditors using a mockup. Foremost, the
discussion centred on the timing of reflection interventions and how to raise mo-
tivation to achieve a change in search behaviour.
Keywords: Reflection Intervention, Reflective Learning, Social Comparison,
Search Behaviour Improvement.
1 Introduction
Finding relevant information for work is necessary in most professions, but the search
requirements are different to those in leisure time. Typically, auditors are aware of how
to use the features of search platforms when looking for changes in laws and regulations
relevant to the audit. At a first glance, there is no need for them to change their rou-
tinised search behaviour [5]. However, the question is if this behaviour is sufficient
especially in fast changing technology environments and varying information sources?
We assume that auditors could benefit from guidance via reflection interventions by
adapting faster to changing technological and environmental changes.
2
Financial auditors conduct audits to provide an opinion whether financial statements
are in accordance with law and compliance. To perform these audits, they have to fol-
low the International Standards on Auditing (ISA), like for example the ISA 315 1. Typ-
ically, auditors are aware of how to use the features of the search platforms or which
information sources to access in order to find relevant information to understand the
entity and the environment. However, working with outdated information can have se-
rious negative effects [1] and thus finding the best information is very crucial for the
job performance. Further, currently there are no technologically supported reflection
interventions available that nudge auditors to reflect on their own search behaviour and
to experiment with less frequently used search features. Reflective learning [2] is a
proven learning strategy for self-directed learning at work [9] that turns past experi-
ences into learning and a behaviour change.
In this sense, the challenge that we tackle in this paper is to support auditors to learn
how to use modern search technology efficiently. We are aware that the challenge of
adapting and continuously reflecting on the search behaviour is also relevant for other
professions. Our approach combines the observation of search behaviour with provid-
ing guidance for reflective learning in order to motivate auditors to reflect about their
search behaviour and initiate a change in behaviour if necessary. To achieve this, we
present a reflection intervention concept that intends to nudge auditors to reflect on
their own search behaviour and to experiment with less frequently used search features
to improve their search efficiency.
2 Related Work
Reflective learning: Reflective learning can be seen as the re-evaluation of past expe-
riences with the goal to learn from them to guide new behaviour. This is in line with
Boud et al. [2], who define reflective learning as “those intellectual and affective activ-
ities in which individuals engage to explore their experience in order to lead to new
understanding and appreciations”. In the area of workplace learning, reflective learn-
ing has been identified as a crucial method for informal learning, as it does not rely on
explicitly available learning material, curricula, or teachers [4,9,8].
Role of technology: Technologies like reflection interventions that guide reflective
learning can consist of prompting approaches, meaningful visuals and social compari-
son. They can spark respectively foster reflection in work related settings [6]. By re-
flection prompts, we understand interventions such as small text messages that try to
motivate a user to reflect. Social comparison in form of performance indicators provide
a possibility to compare the own behaviour with the behaviour of others to increase the
intrinsic motivation to improve [3] and thus can therefore stimulate reflective learning
[13]. Visuals that summarise actions taken by the user in the past can be worth being
reflected on to influence actions in the future [6].
Challenges: From literature we know that prompting approaches are very successful
in formal learning environments, however in working environments several challenges
emerge [7]. First, in working environments the working tasks are not always known
1 https://www.iaasb.org/clarity-center/clarified-standards
3
beforehand, thus it is very challenging to adapt prompts to the working tasks of a user.
Second, in working environments it is often not obvious when to present a prompt (tim-
ing). Third, most workplaces are very stressful and work-intensive; thus it has to be
very well considered, which type of reflection intervention can be applied.
3 Research Goal
Search satisfaction is defined as the fulfilment of a user’s information need [5]. As
long as this is the case, there is little need to change the search behaviour. There-
fore, it is very challenging to motivate users to abandon their comfort zone to change
routinised and well known working or searching behaviour or trying out new function-
alities of a system [10], because acquiring new skills through learning is often associ-
ated with a significant investment of money and time. This is particular challenging for
high skilled knowledge workers such as auditors working under extreme time pressure.
On this observation, we therefore conceptualise the goal of our research as supporting
auditors to learn how to use a newly developed search platform efficiently.
This goal presents multiple challenges: A first challenge is to embed the learning
support into the work process, as time is a critical factor in all workplace learning sce-
narios (see e.g. [7]). For design, this means that reflection interventions should not dis-
rupt the users’ workflow. Thus, reflection interventions however need to connect well
to users’ operative work procedures. A second major challenge is that learning in this
case addresses an activity that is already highly routinised and “works” - even if the
existing search strategies may not be optimal given search technology features.
4 Reflection Intervention Design
Our solution approach leans on the work described by Malacria et al. [10], who de-
signed an intervention that helps users to learn how to use “keyboard shortcuts within
Apple Keynote”. They developed a Skillometer for hotkey usage consisting of three
parts: a bar-chart showing the time taken to select one of the last six used hotkeys, a
meter that grades the user’s performance and a motivational text promoting that hotkeys
selections are faster. By taking into account the Skillometer and using our own work
conducted on reflective learning technologies [6,7,11], we developed a reflection inter-
vention concept consisting of a tripartite user interface as depicted in Fig. 1:
Visualisation: Feature Usage. This visualisation (see Fig. 1, top) presents the last
five used features of the user (e.g. different search filters used, different visualisa-
tions to explore the results) with the goal to raise the user's awareness of the feature
usage and subsequently on the search behaviour.
Comparison: Performance Indicator. The indicator provides the possibility to
compare oneself with other users using the same search platform. As a representative
example, it could present the number of clicks used to receive the desired search
4
results in comparison to the average number of clicks used by other users. By provid-
ing this direct comparison, the user's extrinsic motivation to improve should be
raised (Fig. 1, middle).
Guidance: Reflection and Recommendation. This part presents either a reflective
prompt to motivate the user to reflect on past user behaviour, recommend the user to
use another search feature or information source. Thus, the prompts ask for reflection
and action (Fig. 1, bottom)
Fig. 1. Prototype of the designed reflection intervention
This reflection intervention, presenting real-time information, will be integrated into
a search platform developed within the MOVING 2 project. This search platform will
automatically track the user behaviour serving as input for the reflection intervention.
Out of this activity click stream data the used features (Visualisation) as well as pre-
defined performance indicators (Comparison) can be automatically extracted. Further-
more, the reflection guidance questions shown (Guidance) are adapted to the user’s
search behaviour.
5 Focus Group & Results
Setting of the evaluation: Company A is one of the Big Four accounting companies
operating in 140 countries with worldwide 145.000 professionals. A local office in Ger-
many was involved in the set-up and the participated in the focus group.
Participants: Six auditors (4 senior auditors and two junior auditors) from the German
innovation group of company A participated in the focus group. Usually they have
weekly phone conferences and physically meet 3-4 times a year to discuss new and
innovative solutions worth having a closer look at with regard to company A's needs.
2 http://moving-project.eu/
5
Focus group: To investigate the developed reflection intervention concept with regard
to the selected use case ISA 315, we conducted an online focus group [12]. In this focus
group, we provided impulses for discussing possible performance indicators for social
comparison (meaning which information would be worth being presented) as well as
different types of prompts for initiating reflective learning. We also raised the topic of
when (at what time) and where (on which device) to present the performance indicators
and reflective prompts to be perceive as useful and not as disruptive.
Results: We discussed the different perspectives of the individual auditor vs. the com-
pany as a whole with regard to the performance indicators relevant for the reflection on
search behaviour. From a company’s point of view, the search performance and the
search time are of crucial relevance, as one can imagine, the quicker the better as time
is money. Thus performance indicators should cover this information. Individual audi-
tors showed a more diverse picture depending on their preferences, e.g. the number of
click’s used to get the desired results or number of search terms used. Auditors are
particularly interested in context-related and often used search terms for improving
their search. Participants also suggested to present not only one performance indicator
at once but two or three in relation to each other to get more detailed insights on the
own search behaviour in comparison to others. Hence, the focus group showed that the
performance indicators on the individual level really targeted on learning, whereas the
company level indicators just point at the economic effects.
A second topic concerned social comparison in the sense of “With whom should I com-
pare myself with?” Suggestions for comparisons were top performers, persons with
high search performance or doing a similar job or who are in the same team. Another
suggestion was that the users themselves can select with whom they want to compare
with. Here, the goal was to learn from the best in order to improve the own behaviour.
Finally, time and timing was, as one would have expected from existing literature, a
key concern. Auditors have very tight timelines, so any learning support that “takes
away time” from operative work processes was seen as highly critical. The auditors
remarked that especially in the “busy season” at year-end, they would most likely not
answer open questions that require manual text input. In this time they would prefer
“YES/NO” questions or just ratings. Another possibility would be to present such a
prompt for reflection decoupled from the working activity. This means that such a
prompt can be presented on a mobile device e.g. on the way home. This can have the
advantage that the auditor has time to reflect when a prompt pops up, however, it can
also be that the auditor does not want to be bothered with work-related questions after
work. The discussion ended with the suggestion that the user should be able to choose
in the preferences when and on which device s/he wants to receive a prompt.
6 Discussion & Conclusion
For designing successful reflection interventions it is crucial that the interventions
do not disrupt the user’s workflow (see e.g. [7]). This was also highlighted in the focus
group. We see two complementary strategies for addressing this time constraint: First,
the reflection intervention could be unobtrusively always visible as in [10] or second,
6
show up from time to time during work as in [6]. However, the need for decoupling of
search and reflection was an important insight and seems to be very important for this
type of reflection intervention. In this strategy this means to present the intervention on
demand or proactively via regular notifications on a user-selected schedule (e.g.,
weekly), in a particular situation (e.g. waiting on a bus stop, sitting in an airplane) and
on a user-selected channel (e.g. email, SMS).
Additionally, we note that searching is a highly routinised task for auditors; and we
therefore suspect that it will be very challenging to motivate auditors to change their
current practice or to use a new search platform. One option to address this challenges
is to show the users a clear benefit for themselves by answering the question “What is
the benefit for me?” right from the beginning [6]. Social comparison has been discussed
in the focus group as potentially helpful, in the sense of providing relevant information
about what features very similar users found useful. Social comparison features have
also been very well received in past research on reflection interventions [13]. However,
it needs to be carefully considered which kind of performance indicator to present in
order to be useful for the auditor and the company. With this regard, privacy as well as
protection issues need to be considered as well. Using automatically tracked user data
visualising the social comparison might be perceived rather as an observation tool than
as a tool to provide guidance and support reflection. Further from a knowledge protec-
tion perspective, disclosing information sources can be also seen as a knowledge leak-
age [11].
Moreover, we identified two design implications: First, with regard to the timing of
reflection, we plan to investigate which type of intervention stimulates reflective learn-
ing more effective: unobtrusive always present interventions vs. interventions that show
up automatically during work in comparison to interventions that show-up decoupled
from work. This research should bring us a step closer to find out when is a good time
for reflection [6]. Nevertheless, we are aware that in some work-related setting, a re-
flection intervention as presented above might not be perceived as useful and thus
would not achieve the desired result. Second, although social comparison is well inves-
tigated in psychology [3], less research is available in the area of TEL on how to influ-
ence the user’s intrinsic motivation with technological support. Thus we aim to close
this gap and to explore whether comparison should take the form of competitive com-
parison or of rather being informative communication about others’ search strategies.
Finally, we would encourage research on how to adapt search strategies in general and
to research this challenge with other professional groups.
7 Acknowledgement
The project “MOVING - TraininG towards a society of data-saVvy inforMation prO-
fessionals to enable open leadership iNnovation” is funded under the Horizon 2020 of
the European Commission (project number 693092). The Know-Center is funded
within the Austrian COMET Program - Competence Centers for Excellent Technolo-
gies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and
7
Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the
State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.
References
1. Bachlechner, D., Thalmann, S., and Manhart, M.: Auditing Service Providers: Supporting
Auditor’s in Cross-Organizational Settings. Managerial Auditing Journal 29(4) (2014)
2. Boud, D., Keogh, R., Walker, D.: Reflection: Turning Experience into Learning, chap. Pro-
moting Reflection in Learning: a Model. pp. 18-40. Routledge Falmer,New York (1985)
3. Corcoran, K., Crusius, J., Mussweiler, T.: Social comparison: Motives, standards, and mech-
anisms. In: (Ed.), I.D.C. (ed.) Theories in Social Psychology. pp. 19-139 (2011)
4. Eraut, M.: Informal learning in the workplace. Stud. in Conti. Edu., 26(2), 247–273 (2004).
5. Feild, H.A., Allan, J. and Jones, R.: Predicting searcher frustration. In SIGIR’10. ACM
Press, New York, NY, 34 (2010).
6. Fessl, A.; Wesiak, G.; Rivera-Pelayo, V.; Feyertag, S. & Pammer, In V. Conole, G.;
Klobucar, T.; Rensing, C.; Konert, J. & Lavouee, E. (Eds.) In-App Reflection Guidance for
Workplace Learning. Proceedings of the 10th European Conference on Technology En-
hanced Learning (ECTEL '15), Springer Int. Publ., 2015, 9307, 85-99 (2015)
7. Fessl, A., Blunk, O., Prilla, M., Pammer, V.: The Known Universe of Reflection Guidance:
a Literature Review. Special Issue on: Awareness and Reflection in Technology-Enhanced
Learning, Int. Journal of Technology Enhanced Learning, (forthcoming).
8. Hoyrup, S.: Reflection as a core process in organisational learning. In Journal of Workplace
Learning, 16(8), 442–454 (2004)
9. Knipfer, K., Kump, B., Wessel, D., and Cress, U.: Reflection as catalyst for organisational
learning. Studies in Continuing Education,, 35(1), 30-48 (2013)
10. Malacria, S., Scarr, J., Cockburn, A., Gutwin, C., Grossman, T.: Skillometers: Reflective
widgets that motivate and help users to improve performance. In: Proc. of the UIST ’13, pp.
321-330. ACM, New York, NY, USA (2013)
11. Manhart, M. & Thalmann, S.: Protecting organizational knowledge: a structured literature
review. Journal of Knowledge Management, 19(2), 190-211 (2015)
12. Morgan, D.L.: Focus groups as qualitative research, vol. 16. Sage publications (1996)
13. Rivera-Pelayo, V.; Fessl, A.; Müller, L. & Pammer, V. Introducing Mood Self-Tracking at
Work: Empirical Insights from Call Centers. ACM Trans. Comput.-Hum. Interact. 2017,
24(1), 3:1-3:28 (Feb. 2017)