=Paper=
{{Paper
|id=Vol-1355/paper5
|storemode=property
|title=Assessment for Spreadsheets
|pdfUrl=https://ceur-ws.org/Vol-1355/paper7.pdf
|volume=Vol-1355
|dblpUrl=https://dblp.org/rec/conf/icse/ToaderKK15
}}
==Assessment for Spreadsheets==
https://ceur-ws.org/Vol-1355/paper7.pdf
Assessment for Spreadsheets
Alexandru Toader & Michael Kohlhase Andrea Kohlhase
Jacobs University University of Applied Sciences Neu-Ulm
Germany Germany
Abstract—Spreadsheets are tools to model personal, business is transferred along with the document – a standard situation,
and scientific data and to support decision-making based on when spreadsheets are used as a means of communication:
such models. Moreover, spreadsheets are used for communicating An indication of this is given in a study that categorizes the
data, models, and decisions, which makes the interpretability of
spreadsheet intents an important consideration in the design of help queries of users of a financial controlling spreadsheet in
spreadsheet interaction. a large public/private research institution (see Figure 1). [6]
In this paper we will focus on the problem of model assessment, reports that assessment-related questions make up 28% of the
i.e., the process of passing judgements on a situation based required explanation types.
on modeled spreadsheet data, for instance, the development
of profit over time represented in a controlling spreadsheet.
We introduce a framework for specifying, documenting and
personalizing assessments semi-formally and a set of plugins for
the Semantic Alliance Architecture that can visualize assessments
to assist the (spreadsheet) user in her decision-making.
I. I NTRODUCTION : S PREADSHEETS AS
D ECISION -S UPPORT S YSTEMS
Spreadsheets are used to model quantitative real life sce-
narios and entire decision making processes have been built
around them. In [1] they are even defined as “software products
that help users apply analytical and scientific methods to
decision making”. Here we take up this view and consider
them special decision support systems (DSSs). So far though,
spreadsheet functionality does not directly support this quality, Fig. 1. Types of Help Requests for a Financial Controlling System
it rather assists indirectly by realizing calculations and en-
abling reports necessary for analyzing a situation and making This paper aims to reduce the occurrence of interpretation
decisions built on these. Thus, extending spreadsheet software errors by enhancing the spreadsheet user’s assessment capa-
to directly support its DSS quality would lower the hurdle bilities.
and make DSS available to even more users, businesses, We build on the theoretical analysis in [6] and report on a
and institutions. We call systems that use spreadsheets for concrete implementation by the first author in the scope of his
modeling and support the decision process spreadsheet-based B.Sc thesis [7] based on the “Semantic Alliance Framework”
decision support systems (SSDSS), see [2], [3]. In a nutshell, developed in our research group [8], [9]. This gives us a DSS
an SSDSS assists users with two tasks: i) deciding whether that combines the ease and simplicity of modeling business
a particular aspect of a model is favorable to the business data in spreadsheets with a knowledge-based assessment ser-
goal, and ii) finding remedies if it is not. We call the first vice.
task assessment and we will concentrate on supporting it in In Section II we briefly review the concept “assessment”,
SSDSS in this paper. in Section III the Semantic Alliance Framework, and in
We can see assessment in spreadsheet documents as just Section IV we present the assessment service itself. Section V
another facet of the interpretation process of tabular data evaluates the services via a use case, and Section VI concludes
with respect to the situation modeled by the electronic ledger. the paper and discusses future work.
Interpretation is a source of errors1 that has been often
II. A SSESSMENT IN S PREADSHEETS
neglected although it is of great importance in the case of
long-lived spreadsheets. This is especially virulent in cases Before devising an assessment service for spreadsheets, it is
where the reader and the creator of a spreadsheet are different important to understand what assessment of spreadsheet data is
individuals, the reader is unfamiliar with the domain of the and how it fits into the user’s workflow so that we can derive
spreadsheet, and not all the knowledge of the problem domain requirements for our solution. To ease reading we will not
point out any longer that we are specifically concerned with
1 For general studies of spreadsheet errors see[4], [5]. assessment of spreadsheet data and in a spreadsheet context.
Assessment consists of value and purpose judgments passed really be the case, since if we shift the whole spreadsheet by
on situations modeled by the spreadsheet document or parts one row, then we have to readjust the assessment. So it has to
of it. Indeed, [6] classifies assessment queries in Figure 1 into be about the intended meaning of row [6], i.e., the development
two categories: of revenues over the years. Indeed we can paraphrase i)
• Assessment of Value refers to the ability of a user to make with ii) — another clue that the assessments are really about
judgments on the spreadsheet data after having interpreted situations modeled by a functional block in the spreadsheet.
the concrete values. But assessments are not restricted to functional blocks as
• Assessment of Purpose refers to the ability of a user statements iii) and iv) only refer to individual cells. Note again
to draw the proper conclusions or perform the correct that the statements are not about the numbers 65, 480 and
actions based on the content of a spreadsheet document. −71, 600 (numbers in themselves are not good or bad, they
just are). Here the assessment seems to be intensional, i.e.,
A. Running Example about the intension “the profit in 2014 is 65,480e” rather than
To fortify our intuition about SSDSS consider the simple the extension. Another way to view this is that the latter two
spreadsheet in Figure 2, which will serve as a running example assessments are about the argument-value pairs {2014;65,480}
for this paper. It is used to model the evolution of a small resp. {2016;-71,600}.
department of a multinational company over a number of We will make this view the basis of our treatment of
years. It acts as a controlling tool on which all present and assessment: We need to have a model of spreadsheets, which
future decisions are based, thereby qualifying as a simple can capture the intensional level, so that we can assess it. In
SSDSS. the next section (part III-A) we will present spreadsheets with
a structured background ontology that captures the spreadsheet
intention, and in IV-A we show how we can extend that for
assessing spreadsheet values and functional properties.
C. Assessment is Personal
But there is another aspect of assessment we need to
model: assessments depend on context and who assesses the
situation. Indeed, example v) below shows a statement relating
assessments by three stakeholders – arguably enabling the
addressee of the statement to form her own assessment from
that.
v) “Upper Management will be happy about the leftover
funds in [nn] that they can now use elsewhere, but the PI
of the project will be angry that he got less work out of the
project than expected. Not to mention the funding agency;
Fig. 2. Running Example: A Simple Controlling Tool for Project Management they cannot be told of this at all, because it violates their
subsistence policy.”
Let us use Figure 2 to introduce an important concept: In particular, we cannot tie assessments rigidly to particular
We call a rectangular fragment of a spreadsheet a functional cells and functional blocks, but need a setup, where assess-
block, if it corresponds to a function in the modeled situation. ments are objects in their own right that can be “applied” to
For instance, cells [B6:D6] model the revenues as a function cells and functional blocks. We consider these “assessment
on the years 2012-2014. Cells and functional blocks are the objects” as reified background knowledge about values and
basic building blocks of spreadsheet models and, therefore, the functions of the model expressed in the spreadsheet.
natural domains of assessment.
B. Assessment is Intensional D. Assessment is Computational
The following can be considered typical assessment state- Finally, we note the obvious: An assessment is computed
ments in the context of Figure 2: on the values in the respective cell or functional block –
i) “Row [6] looks good.” according to the specification in the “assessment object” we
ii) “The revenues look good.” have stipulated above.
iii) “I like this (points to cell [D17]) but that (points to cell All of these requirements do not fit with the conventional
[F17]) is a disaster.” approach to extending spreadsheets by macros in the scripting
iv) “I like the profit/loss in 2014 but of course not the expected language of the application (e.g., .Net for MS Excel or Java
one in 2016.” for LibreOffice). Instead we make use of a semantic framework
On the surface, the first statement refers to a row in the technology we have previously developed and which we will
spreadsheet, but if we look closer, then we see that this cannot present now.
III. T HE S EMANTIC A LLIANCE F RAMEWORK the information stored in the ontology. Enhancing the spread-
sheet application with functionality to use a local ontology is
The entry point for the Semantic Alliance Framework we
prohibitively expensive. Therefore, an external ontology that
use as a basis for our Assessment Service is the realization,
can be used by multiple services and documents seems more
that spreadsheets per se only represent the data and compu-
sensible and is the approach taken by the Semantic Alliance
tation layers of a model, and leave the intended meaning and
Framework which will support the Assessment Service. This
the background knowledge about the situation modeled in the
approach was materialized as the Semantic Illustration archi-
spreadsheet implicit – usually hinted at in the “legend” cells
tecture in [13].
– see [6] for details.
We will now give a brief overview of the Semantic Alliance
A. Semantic Illustration Framework and explain how it supports semantic services like
the Assessment Service.
To remedy this shortcoming, we give the spreadsheet cre-
ators the opportunity to document the background knowledge B. Semantic Alliance Framework
– not in a human-readable “manual”, but in a machine-
actionable structured background ontology2 . The back-
ground ontology concepts themselves can be linked to the
meaning-carrying fragments of the spreadsheets. Figure 3
shows the approach.
Application Background Ontology
· Projected revenues
· Prognosis function
· Lagrange extrapolation
· Projected expenses
Fig. 4. The Semantic Alliance Framework
Illustration Mapping The Semantic Alliance Framework is an architecture and
Fig. 3. The Semantic Illustration Architecture software framework for semantic allies3 . It combines an
invasive approach that allows users to profit from seman-
In our system the background ontology is represented in tic technology without leaving their accustomed workflows
the OMDoc [10] format, a markup format and data model and tools with an application-independent way of extending
for semi-formal, structured document collections. OMDoc applications with knowledge management technologies. The
provides a strong, logically sound module system based on Semantic Alliance Framework provides the foundation for the
structured theory graphs [11]. In contrast to other ontology development of the Assessment Service. As we will see later
modeling languages like OWL, the OMDoc format does not on, it offers functionality to tap into the user’s interactions
commit to a formal logical language. Therefore, it lacks with the document and to provide meaningful services in the
a native concept of inference, but also does not force the appropriate context while giving the user the impression of
author to fully formalize the ontology and to work around (almost) perfect integration with the host application.
the expressivity limitations of the underlying logical system. The Semantic Alliance Framework works by mashing up
Instead, OMDoc allows to locally formalize elements - and the GUIs of the knowledge management systems and applica-
thus provide partial inference - with whatever formal system tions themselves. The Semantic Alliance Framework has three
is most suitable. components (see Figure 4):
After choosing a representation format for the ontology, the • Sally: The main component of the mashup enabler,
biggest challenge left is how to integrate it with the docu- integrates the functionality of the application A (e.g.,
ment and make it usable by the associated services. In [12] spreadsheet application) and the semantic services S
E RIKSSON suggested to combine documents and ontologies (e.g., the Assessment Service) into a joint user interface
by “adding annotations to electronic-documents formats and and interaction model.
including the ontologies in electronic documents” yielding • Alex: The application A is extended by a slim API
semantic documents. This approach limits the reusability of (Application Programming Interface) “Alex” that reports
the ontology for multiple documents that contain the same and executes relevant user interactions within A (e.g., cell
concepts and poses a problem for maintaining and enhancing clicks in spreadsheets) to and from Sally.
2 We will use the term “ontology” in its broad meaning, where it stands 3 Semantic allies are semantic systems that complement existing software
for “a representation of the objects in a given situation, and their relations to applications with semantic services and interactions based on a background
each other”. ontology.
• Theo: A screen-area manager that enables invasive de- illustration mapping as the Sally component that operational-
sign. Content supplied by S is embedded as a pop-up izes it also contains an “abstract spreadsheet model” that can
into the GUI of A. store concrete parameters.
• Semantic Services: are independent components which Let us make this concrete with an example from Figure 2.
merge and process information from multiple sources in- Say we want to assess the projected profits with the intuition
cluding the machine-actionable ontology, the application that profits are good as long as they are positive. As we want
A and external sources. They return the result to the to assess a value in a functional block, we use a general
user through Sally (e.g., The Definition Service, which is assessment theory that takes a function f and an argument
offered through the Semantic Alliance Framework, when p as parameters. For the assessment of the value in cell
invoked returns the definition of a concept associated with [F17] we instantiate f to the profit function π and p to the
a spreadsheet cell. For instance, when the user asks for year 2016. As π is the target concept in the background
the definition of cell [E15], he gets the definition of the ontology for the functional block [B17:F17] under the illustra-
concept Projected Expenses.) tion mapping and 2016 the argument of the functional block
Note that the Semantic Alliance Framework relies on the [B17:F17]. These instantiations were already in effect in the
Semantic Illustration architecture – the machine-actionable abstract spreadsheet model. For the particular assessment in
background ontology is the basis for the semantic services and our example, we make use of the “positive-balance-is-good”
the illustration mapping (the green dashed arrows in Figure 3) theory and bind it to the functional block [B17:F17]. This
show which parts apply at a given point in the spreadsheet. But theory can be constructed by simply instantiating the general
from the point of view of the Semantic Alliance Framework, theory of “monetary quantities”, which inherits from “real
the background ontology is encapsulated into the semantic numbers” (from elementary math). We obtain a comparison
(web) services reducing the knowledge integration problem to operator for the general theory of “partially ordered sets”
a mashup of (web) services (this is what the Sally component (also from “general math”) and instantiate the base of the
does). comparison (a parameter in our assessment model) to the
real number zero. Given this illustration binding, the MMT
IV. T HE VALUE A SSESSMENT S ERVICE API can instantiate all parameters in the formulae in the
various theories and compute4 the assessment value. Here,
In this section we will give a brief overview of how the the computation amounts to π(2016) = −71.600 e which
Assessment Service is integrated into the Semantic Alliance is negative, leading to a negative assessment.
Framework, describe the interaction between the user and the Note that we only had to provide the “positive-balance-is-
service, and discuss about the representation and computation good” theory5 and extend the illustration mapping to get the
of assessments with theory graphs. An Assessment Service assessment for the full functional block – all other theories
differs from – say – a Definition Lookup Service which only were so general that we can assume them to pre-exist in some
relays and aggregates parts of the background ontology, in Semantic Alliance ontology.
that it has to compute a verdict about possibly changing
spreadsheet data based on the background ontology and feed
it back to the user on the fly in an intuitive way. B. Integration into the Semantic Alliance Framework
The Assessment Service is just one of the many semantic
A. Representing/Computing Assessments in Theory Graphs services [8] offered through the Semantic Alliance Framework.
The Semantic Alliance Framework organizes the back- For the purpose of this example, we will use LibreOffice
ground ontology into theories – collections of concept def- Calc [15] as our host spreadsheet application and the document
initions together with statements of properties of and relations illustrated in Figure 2 as our running example.
between the objects denoted by these concepts. Theories are The service registers itself with the framework at initializa-
connected by views, that is, meaning-preserving mappings tion and is available to the user each time he tries to assess a
that allow to interpret objects from the source theory as semantic object that is linked to an assessment rule.
objects of the target theory. The theory-graph paradigm in When the Assessment Service is invoked on a cell range, it
OMDoc/MMT [10], [11] gives us a powerful, modular/object- collects information from the spreadsheet application regard-
oriented representation framework for formal and informal ing the selected object, merges it with knowledge from the
mathematical knowledge. Crucially, OMDoc/MMT allows for ontology, and it runs the assessment for the concrete values in
parametric theories and embedded computation in the MMT the document as described above.
web service [14]: Representing assessment knowledge in
(collections of) theories that can be connected to cells and 4 In the declarative OMDoc/MMT framework nothing restricts the compu-
functional blocks in spreadsheets meets the requirement of tation to two-value “good/bad” assessments: Instead of a predicate, we can
“object-like” assessment knowledge from subsection II-C. The specify functions into any assessment scale without changing the framework.
5 Arguably, this theory is rather general as well. In our (admittedly limited)
object-oriented/parametric nature allows us to make general experience assessment is governed by a tractable number of theories that
theories of “assessment patterns” and specialize them in the correspond to a limited number of assessment stereotypes.
C. User Interaction of the Assessment Service (see Figure 5). After clearing all
The Assessment Service addresses both assessment needs: the results, the user can edit the document as before the
to be able to assess a value and the purpose. The former Assessment Service was invoked.
refers to an interaction on cell level, the latter requires an Our Assessment Service provides a simple, intuitive inter-
interaction concerning a functional block. If the service visu- face that hides the formal theory structure and the evaluation
alizes the assessment outcome, then the original layout should of arbitrarily complex assessment rules. The theory graph that
be regathered with a simple click. From the user’s point of is the basis of the service can be easily extended with new
view, therefore, the Assessment Service has to support three assessment rules with a minimal implementation cost while
operations: incrementally increasing the value of the entire system.
• Assess a value, V. U SE C ASES
• Assess a domain (a group of cells), and
• Clear assessment results (i.e., revive the original spread-
In this section we use our running example to understand in
sheet state). what ways the Assessment Service will be of value in a real
life scenario simple as it may be. While this is no substitute
In order to assess the value of cell [F17], the user right-clicks
to a usability evaluation, it already gives us an indication of
on the cell and selects the Assessment Service “Assess value
the potential of the new technology.
of F17” from the list of services offered by the framework
A strong point of our theory-graph-based approach is
for that particular object (see Figure 5). At this point, the
that it can handle different assessments – even adversarial
service mashes-up information about the selected object from
ones – simply by associating different theories and views;
the spreadsheet and the knowledge base, and returns the result
composition and computation are supplied uniformly by the
of the associated assessment theory evaluated for the object.
OMDoc/MMT framework. We expose this by assuming two
The result of an assessment is shown to the user by
users of the same spreadsheet document with distinct junior
highlighting the assessed cell or cell ranges in bright green
and senior perspectives and responsibilities, John and Robert.
for a positive result, and in bright red for a negative result. At
Neither of them is the author of the document, but both rely
this point, LibreOffice locks down the sheets containing the
on it for assessing the past, present and future evolution of the
colored cells so that no modification is possible. We can see
company department modeled by the spreadsheet.
the result of assessing the projected profit for the year 2016
John is a young manager who has just taken over the
in Figure 5.
department. He is in charge of managing the project’s team,
schedule, and budget. On his first day on the job, John received
the spreadsheet document in Figure 2. His challenge lies in
understanding the data contained in the spreadsheet document
and determining if there are changes needed to the division’s
budget structure and business plan. Robert, John’s boss, is
in charge of managing multiple divisions. He uses the same
spreadsheet document, created by John’s predecessor, to obtain
an overview of the project’s expenses state and to decide if an
intervention in the management of the department is necessary.
So, both users have to be able to grasp very quickly financial
risks based on the information summarized in the spreadsheet
to make decisions and act fast, but neither of them has a full
Fig. 5. Assessed Projected Profit understanding of the spreadsheet.
The process of assessing an entire functional block is almost
identical. After, for example, the range [B17:D17] was selected
and the Assessment Service “Assess Domain of B17:D17” was
invoked, the system checks if the selected domain represents
one homogeneous functional block. If there are cells in the Fig. 6. Assessed Actual Profit Block
block that are associated with a different concept, i.e., a
different intended function, we cannot assess the block. If this John wants to maximize resource consumption in order to
sanity check passes, the process continues as in the case of maximize growth and the division’s potential market value,
the assessment of value scenario. while maintaining a profit margin. The electronic ledger offers
After performing an assessment the spreadsheet cannot be the Assessment Service maintained before by John’s predeces-
edited by the user. This locking is enforced in order to sor. To get a quick impression on how the department has been
offer the user the possibility of returning the spreadsheet doing up to current year, he selects the cell range [B17:D17]
document to a pre-assessment state. In particular, this enables and the “Assess Domain of B17:D17” service. In turn, the
the “Clear Assessment” and “Clear All Assessments” options range is colored in bright green (see Figure 6) reflecting the
fact that the profit has increased over time, which is reassuring Assessment of Dependencies: Recall the two tasks of a DSS
to John. He could have chosen the service “Assess Value of mentioned in the introduction. With the assessment service
B17:D17” showing the same result, this time caused by all presented in this paper, we have only addressed the first
values in the range being positive. Turning his attention to one. Finding remedies based on the assessments still remains
the Projected Profit running the Assessment Service “Assess unaddressed.
value of E17” on the cell representing the Projected Profit
for the year 2016, he directly perceives a problem by the
subsequent red coloring of cell [E17] in Figure 5. This requires
John to immediately take a closer look at the business plan of
the division to determine whether the decisions taken by his
predecessor were sound.
Robert starts by assessing the Actual Profit expressed in
percentages of the Revenues. He considers that a division is
reaching its target if the profit grows faster than the Dow Jones
Industrial Average and he assigned the according assessment
theory some time ago. Now, when he selects the entire Actual
Profit range and chooses the “Assess Domain of B17:D17”
service, the block’s color turns red and he has to take according
counter measures.
Fig. 7. Assessment of Dependencies
Robert and John are both worried that the division is not
reaching its full potential and they now know that something It is conceivable that the information in the assessment can
has to be done about that, but they are grateful that the be used for pinpointing causes of negative assessments and
Assessment Service has helped them in taking a fast decision thus problems the remedies might attack.
on what problem area requires their interest. Consider for instance the cell [F17] in the running example,
VI. C ONCLUSIONS & F UTURE W ORK which represents the Projected Profit for the year 2016. The
Projected Profit is defined as “Projected Revenues - Projected
We have presented a knowledge-based approach to assess- Total Expenses”. It makes sense to trace the assessments
ment in spreadsheets and reported on our experiences of through the dependency tree in an attempt to gather more
implementing an assessment service in the Semantic Alliance insight about the assessment’s result.
Framework. Given a small set of carefully constructed “assess- In Figure 7 we can see the definition graph for the negatively
ment theories” (constructed by OMDoc/MMT specialists) and assessed cell [F17] for the Projected Profit in year 2016. The
a larger set of theories that specify the background knowledge node associated with Projected Revenue for the year 2016 is
about the domain modeled by the spreadsheet (created by colored in red, meaning that it was also negatively assessed,
domain specialists), individual assessment views can be set so the problem might be with the revenues.
by linking to specific theories via the illustration mapping The JavaScript InfoVis Toolkit [16] provides all the nec-
(set up by each user). From these the MMT web service can essary functionality to visualize the dependency graph and
compute assessments and visualize them to the spreadsheet the Assessment Service already provides functionality for
user. Extended by assessment knowledge, spreadsheet-based assessing the individual cells and functional blocks. At this
models can in turn act as decision support systems. point, the only missing functionality to implement this service
Future Services is a Formula Parser that would transform spreadsheet formulae
into a tree structure, which is left for further research.
In Section V we hid the underlying multifaceted question
“Why is this assessment positive/negative?”. On the one hand, ACKNOWLEDGMENT
it is concerned with how the assessment theory is defined, We gratefully acknowledge fruitful discussions with Mihnea
but on the other hand, it is concerned with the factors that Iancu: his extensions of the MMT API made the modular
determined the assessment to have a positive or negative result. computation of assessments possible. This work has partially
The following two envisioned services will attempt to tackle been supported by the German Research Council under Grants
both issues. KO 2428/11-1 and KO 2428/13-1.
Assessment Explanation: To explain an assessment result
the user has to understand the assessment theory behind it. As R EFERENCES
the computation in the MMT web service is inference-based, [1] H. K. Bhargava, S. Sridhar, and C. Herrick, “Beyond
we can generate explanations of every step of the computation spreadsheets: tools for building decision support sys-
and the assessment. In fact, it should be relatively simple to tems.,” IEEE Computer, vol. 32, no. 3, pp. 31–39,
generate assessment explanations specific to the particular cell Jan. 3, 2002, [Online]. Available: http : / / dblp . uni -
or functional block, since the MMT API has access to all trier . de / db / journals / computer / computer32 . html #
parameter instances. BhargavaSH99.
[2] M. M. Şeref and W. L. Ahuja Ravindra K. Winston, Available: http://kwarc.info/kohlhase/papers/mkm12-
“Spreadsheet-based decision support systems,” English, SAlly.pdf.
in Handbook on Decision Support Systems 1, Springer, [9] C. Jucovschi, “Towards an interaction-based integra-
2008, pp. 277–298, DOI: 10.1007/978- 3- 540- 48713- tion of mkm services into end-user applications,” in
5 14. Intelligent Computer Mathematics 2014, S. Watt, J.
[3] V. Toader and V. R. Rus. (2014). Spreadsheet-based Davenport, A. Sexton, P. Sojka, and J. Urban, Eds.,
decision support systems, [Online]. Available: http : ser. Lecture Notes in Computer Science, Springer, 2014,
/ / steconomiceuoradea . ro / anale / volume / 2008 / pp. 344–356, [Online]. Available: http://arxiv.org/abs/
v4 - management - marketing / 275 . pdf (visited on 1403.5058.
01/26/2014). [10] M. Kohlhase, OMD OC – An open markup format for
[4] T. SH Teo and M. Tan, “Quantitative and qualitative mathematical documents [Version 1.2], ser. LNAI 4180.
errors in spreadsheet development,” in Proceedings Springer Verlag, Aug. 2006, [Online]. Available: http:
of the 30th Hawaii International Conference on Sys- //omdoc.org/pubs/omdoc1.2.pdf.
tem Sciences: Information System Track-Organizational [11] F. Rabe and M. Kohlhase, “A scalable module system,”
Systems and Technology - Volume 3, ser. HICSS ’97, Information & Computation, vol. 0, no. 230, pp. 1–
Washington, DC, USA: IEEE Computer Society, 1997, 54, 2013, [Online]. Available: http://kwarc.info/frabe/
pp. 149–, [Online]. Available: http://dl.acm.org/citation. Research/mmt.pdf.
cfm?id=938435.938681. [12] H. Eriksson, “The semantic-document approach to com-
[5] K. Rajalingham, D. R. Chadwick, and B. Knight, bining documents and ontologies,” International Jour-
“Classification of spreadsheet errors,” CoRR, vol. nal of Human-Computer Studies, vol. 65, no. 7, pp. 624
abs/0805.4224, 2008. –639, 2007, Knowledge representation with ontologies:
[6] A. Kohlhase and M. Kohlhase, “Spreadsheets with Present challenges - Future possibilities, DOI: http://dx.
a semantic layer,” Electronic Communications of the doi.org/10.1016/j.ijhcs.2007.03.008.
EASST: Specification, Transformation, Navigation – [13] A. E. Kohlhase and M. Kohlhase, “Semantic trans-
Special Issue dedicated to Bernd Krieg-Brückner on the parency in user assistance systems,” in Proceedings
Occasion of his 60th Birthday, vol. 62, T. Mossakowski, of the 27th ACM International Conference on Design
M. Roggenbach, and L. Schröder, Eds., pp. 1–20, 2013, of Communication, ser. SIGDOC ’09, Bloomington,
[Online]. Available: http : / / journal . ub . tu - berlin . de / Indiana, USA: ACM, 2009, pp. 89–96, DOI: 10.1145/
eceasst/article/view/870. 1621995.1622013.
[7] A. V. Toader, Assessment service for spreadsheet doc- [14] F. Rabe, “The MMT API: A Generic MKM System,”
uments, Bachelor’s Thesis, 2014. in Intelligent Computer Mathematics, J. Carette, D.
[8] C. David, C. Jucovschi, A. Kohlhase, and M. Kohlhase, Aspinall, C. Lange, P. Sojka, and W. Windsteiger, Eds.,
“Semantic Alliance: a framework for semantic ser. Lecture Notes in Computer Science, Springer, 2013,
allies,” in Intelligent Computer Mathematics, J. Jeuring, pp. 339–343, DOI: 10.1007/978-3-642-39320-4.
J. A. Campbell, J. Carette, G. Dos Reis, P. Sojka, [15] Libreoffice, http://www.libreoffice.org/, [Accessed on
M. Wenzel, and V. Sorge, Eds., ser. LNAI, Berlin and 21-April-2014].
Heidelberg: Springer Verlag, 2012, pp. 49–64, [Online]. [16] Javascript infovis toolkit, http://philogb.github.io/jit/,
[Accessed on 21-April-2014].