In recent years, researchers have developed a number of techniques to assist the user in locating a fault within a spreadsheet. The evaluation of these approaches is often based on spreadsheets into which arti cial errors are injected. In this position paper, we summarize di erent shortcomings of these forms of evaluations and sketch possible remedies including the development of a publicly available spreadsheet corpus for benchmarking as well as user and eld studies to assess the true value of the proposed techniques.
Locating the true causes why a given spreadsheet program
does not compute the expected outcomes can be a tedious
task. Over the last years, researchers have developed a
number of methods supporting the user in the fault localization
and correction (debugging) process. The techniques range
from the visualization of suspicious cells or regions of the
spreadsheet, and the application of known practices from
software engineering like spectrum-based fault localization
(SFL) or slicing, to declarative and constraint-based
reasoning techniques [
However, there is a number of challenges common to all
these approaches. Unlike other computer science sub-areas,
such as natural language processing, information retrieval
or automated planning and scheduling, no standard
benchmarks exist for spreadsheet debugging methods. The
absence of commonly used benchmarks prevents the direct
comparison of spreadsheet debugging approaches.
Furthermore, fault localization and debugging for spreadsheets
require the design of a user-debugger interface. An important
question in this context is: what input or interaction can
realistically be expected from the user? Finally, the main
question to be answered is whether or not automated
debugging techniques actually help the developer as discussed
in [
In this position paper, we discuss some limitations of the current research practice in the eld and outline potential ways to improve the research practice in the future. 2.
To demonstrate the usefulness of a new debugging
technique, we need spreadsheets containing faults. Since no
public set of such spreadsheets exists, researchers often
create their own suite of benchmark problems, e.g., by
applying mutation operators to existing correct spreadsheets [
In some papers, spreadsheets from the EUSES corpus1
are used for evaluations. As no information exists about the
intended semantics of these spreadsheets, mutations are
applied in order to obtain faulty versions of the spreadsheets.
The spreadsheets in this corpus are however quite diverse,
e.g., with respect to their size or the types of the used
formulas. Often only a subset of the documents is used in the
evaluations and the selection of the subset is not justi ed well.
Even when the benchmark problems are publicly shared like
the ones used in [
A corpus of diverse benchmark problems is strongly needed
for spreadsheet debugging to make di erent research
approaches better comparable and to be able to identify
shortcomings of existing approaches. Such a corpus could be
incrementally built by researchers sharing their real-world
and arti cial benchmark problems. In addition, since it is
not always clear if typical spreadsheet mutation operators
truly correspond to mistakes developers make, insights and
practices from the Information Systems eld should be
better integrated into our research. This in particular includes
the use of spreadsheet construction exercises in laboratory
settings that help us identify which kinds of mistakes users
make and what their debugging strategies are, see, e.g., [
Spreadsheet debugging research is often based on o ine
experimental designs, e.g., by measuring how many of the
injected faults are successfully located with a given
technique, see, e.g., [
In many cases, it remains unclear, if an average spreadsheet developer will be willing or able to provide these inputs since concepts like test cases do not exist in the spreadsheet paradigm. Therefore, researchers have to ensure that a developer interprets the requests from the debugger correctly and provides appropriate inputs as expected by the debugger. One additional problem in that context is that user inputs, e.g., the test case speci cations, are usually considered to be reliable and most existing approaches have no built-in means to deal with errors in the inputs.
Overall, we argue that o ine experimental evaluations
should be paired with user studies whenever possible as
done, e.g., in [
Again, insights and practices both from the elds of Information Systems and Human Computer Interaction should be the basis for these forms of experiments.
In addition to user studies in laboratory environments,
research on real spreadsheets as suggested in [
A number of proposals have been made in the recent literature to assist the user in the process of locating faults in a given spreadsheet. In this position paper, we have identi ed some limitations of current research practice regarding the comparability and reproducibility of the results. As possible remedies to these shortcomings we advocate the development of a corpus of benchmark problems and the increased adoption of user studies of various types as an evaluation instrument. As experimental settings di er from real-life, we additionally propose to use eld studies to obtain insights on how debugging methods are used in companies. 6.