This paper describes the evaluation of coursework set for nal year degree students designed to teach Formal Concept Analysis (FCA). The usefulness of this approach is discussed with respect to its application in future iterations of the coursework. The source data was the result of a simulation between competing student teams undertaken on a mainstream ERP system provided by the business software vendor SAP A.G. and using the ERPsim software provided by ERPsim Lab at HEC Montreal. The simulation generated data on which Business Intelligence (BI) is typically based and is representative of business activity. The data generated by the simulation exercise was not speci cally for FCA, thus it provides a meaningful test of FCA in BI.
This paper describes the evaluation of coursework set for nal year degree students designed to teach Formal Concept Analysis (FCA). The assessment applied a set of FCA tools and conventional Business Intelligence (BI) using graphical or statistical methods in Microsoft O ce Software. There were to distinct objectives to this activity, rstly to ful l the students learning objectives and secondly to support an action research project about the application of FCA within Enterprise Resource Planning (ERP) systems. The ful lment of the learning activity was assessed in two ways, rstly as a comparison between the use of conventional BI analysis and FCA tools and secondly as a comparison of FCA against established theory. Topics for the action research project are highlighted in the conclusions and further work sections, however, this is not the primary focus of this paper.
FCA is mathematical theory of data analysis using formal contents and
concept lattices [
The rationale for selecting this research is due to the demands being placed
on BI systems to improve and the di culties in identifying semantic data. A
simple de nition is "semantics = data + behaviour" [
The coursework is described in more detail later, however the principle is
to introduce frameworks and techniques for representing and reasoning with
knowledge for smart applications [
The need for analysis and decision making within enterprises is not new but
competition and complexity do combine to make the task vast and di cult to
execute e ciently or accurately. Business Intelligence (BI) is frequently used to
support analysis and decision making and can be traced back at least as far as
1958 [
ERP systems are essentially transactional systems that support a vast array of business functions within the majority of organisations that exist today. They are designed to be explicit and accurate in terms of control and data but often lack the analysis tools and communication methods to support all of an organisation's functions. This is where complimentary tools have a role to play.
ERP systems support integration and control across various functional areas
of a company, therefore supporting the achievement of the company's plans [
CUBIST [
In this speci c example the source data was the result of a game between
competing student teams undertaken on a mainstream ERP system provided
by the business software vendor SAP A.G. [
ERPsim is based on SAP ECC 6.0 which is an ERP system capable of supporting in this example logistics and nancial activities for a number of competing companies. All sales, procurement, master data, inventory, marketing and nancial transactions are captured real time in additional to a limited number of reports to show sales, inventory, balance sheet and pro t and loss. These are transaction based reports and o er no analysis without the application of further tools.
As an ERP system is e ectively a relational database with data held in joined tables it is possible to extract data that contributed towards a goal via a query. Therefore a query using the table relationships was able to extract all the transactional data available that contributed towards the outcome. For example all sales transactions within the time period could be found via the connection from billing through the outbound shipments to the sales orders. Correspondingly individual sales order pro t based on the materials cost price could also be extracted.
The chart in gure 1 provides an example of the input and output variables plotted to highlight the relationships that can exist in the simulation game. On the right hand side cumulative pro t and percentage pro t above cost per sale are shown. Cost is indexed at 100%, therefore 105 equates to 5% pro t over cost. On the left hand side days inventory cover and the percentage of sales price attributable to marketing spend is shown. In summary the data could represent a number of relationships including: { Increasing cumulative pro t has an inverse relationship with decreasing days of inventory cover (how many days the stock will last given the sales forecast). [Holding less stock will result in more pro t] { Increasing pro t has a direct relationship with increasing marketing spend.
[Spending on marketing leads to more sales, therefore more pro t] { Increasing pro t has a direct relationship with increasing pro t per sale.
[More pro table individual sales leads to higher overall pro t] 2
The primary problem is how to analyse data and identify semantic data or
relationships from a generic transactional data set. The coursework addressed
three of the learning outcomes from the course [
The scenario presented to the students was:
Day
Cost % Profit Days_cover Marketing Spend %
You are performing the role of a business analyst who has been tasked with analysing the performance of your ERP Water Company by understanding how a) your decision-making and that of others has impacted the organisation and b) identifying rules that could be used to help this decision making in the future. You are also evaluating the method of analysis in order to re ne the approach employed for future iterations of this process. It is therefore less the intention to learn ERP; rather through this experience you will explore business intelligence and the role that Formal Concept Analysis (FCA) might play in this context.
The coursework had three main sections consisting of conventional BI, FCA and Evaluation / Conclusions. The BI analysis would use MS Access and Excel in order to familiarise the students with the data using tools that would already be familiar and o er graphical analysis techniques that are common and taught at a school level of mathematics. Secondly FCA tools are applied based on essentially the same data with any calculated values added to support the analysis. This is expected to be an iterative process in order to produce the best results possible but the core section of the data extract should be stable and reusable. The nal section is an evaluation of the two approaches and conclusions.
It is acknowledged that the goal of both the BI and FCA approach is to identify potentially the same relationships, this is deliberate in order to encourage an understanding of the data using tools and techniques in applications such as Line chart (2 variables) 100 Graph on graph comparisons 69 Cumulative and actual data charts 62 Detailed Focus with annotation. 46 Line chart (3 variables) 23 Pie chart 23 Data table 15 Pivot table 15 Summary table (annotated) 8 Use of trend lines 0
Table 1. Methods Applied under BI Excel that will be familiar and well supported with documentation and guides. An understanding of the data and relationships was deemed necessary given the students had no prior knowledge of ERP systems or an understanding of the processes in operation.
The tools set consists of ve key software packages: MS Access as a
mechanism for extracting data from the ERPSim SAP system and creating the initial
data le (CSV) for analysis, MS Excel and FCA tools including: FcaBedrock [
The method selected was generally an experimental and iterative approach in order to extract and analyse key data, gradually re ning the method to explore the anticipated relationships and evaluate the capabilities of the tool set. The aim was to supply a consistent set of data to the FCA tool set making it a repeatable process. 3
The basic analysis methods applied across all the course work are shown in Table 1 and 2 with a percentage occurrences. It is noted that the marking of the coursework did take into account more than the range of techniques applied.
A minority of students also attempted to identify rules that explicitly stated relationships and could be reused in future iterations of the simulation.
The average mark achieved was 57 % with a standard deviation of 15.3.
Tables 3 and 4 contains a summary list of points made within the Evaluations and Conclusions section of the coursework for BI and FCA respectively. 4
The initial reaction of the students was one of confusion in how to tackle the coursework, this is re ected to a degree in Tables 1 and 2, these show that less complex forms of analysis were prevalent in all work, for example line charts with two variables, but relatively few progressed onto considering more complex selections such as line charts with three variables. A little trial and error coupled with con dence could have eliminated most problems, this could also be supported better with guided examples attempting the coursework.
The marking of the coursework produced a normal distribution of marks with an unexpected enthusiasm for FCA although this was tempered by the di culties in using the tool set. It is not surprising that they experienced di culties given the di erence between the development e ort behind the FCA tool set and BI tools from providers such as Microsoft. It could be surmised that the students understood the advantages of analysing large and relatively unstructured data without expert knowledge or time consuming analysis. It would have been nice to see the students experimenting more with the data and discovering or at least looking for less obvious relationships.
A consistent criticism of the FCA tool set, see table 4, was the di culty level involved in data preparation and use of the tools. It would have been nice to eliminate some of the repetitive tasks required by the exercise as the students struggled to grasp and achieve a reusable data extraction mechanism, therefore consuming time that could have been spent more productively on the analysis. A problem that is not uncommon in real life applications.
The presentations produced for assessment made it relatively easy to mark however it was sometimes di cult to understand what was trying to be communicated especially where annotations or additional notes were not present or of low quality. The graphical nature of the presentation medium did form a good basis for presenting the analysis and forced a summary rather than lengthy descriptions of the process and mechanisms involved.
It was clear from the conclusions in Tables 3 and 4 that an appreciation about the di culties involved in delivering BI was achieved even from this relatively small data set.
The students really failed to identify data or relationships outside of the key parameters, this is partially due to the data available as it was only a partial extract of ERP systems. Even so there are many factors that could have been o ered for consideration even if they could not directly be included in the analysis. Examples of this could include the team structure or the decision making of certain individuals being categorically better or worse in outcome to others.
Graph on graph comparisons featured highly in the BI analysis, essentially this included graphical comparisons that were either overlaid or annotated to illustrate an event or relationship. Considering the frequency of this type of analysis when it came down to the FCA tools set it was hardly applied, even though the concept lattices are primarily a visual tool. The reasons for this were not clear and possibly related to the di culty experienced in using the tool set. This feature was not supported in the tool set but it was easily possible to capture and present images side by side within the presentation.
Discrete values proved much easier to understand than ranges, in order for ranges to be understood manual input is required in order to create meaningful sub ranges. Progressive scaling was applied but the de nition of the discrete values was not appropriate to take advantage of this. With this in mind a biordinal scale would be more useful when representing such values but this will require a di erent approach when extracting the data or within FCA.
As soon as the analysis required calculations to be performed it started to face many of the challenges also faced by BI. Firstly there may be di erences in the calculations between analysts, regions or indeed of interpretation. Secondly, calculated gures and performance measures can lack scale. The analysis was more successful when focus was given to a speci c attribute, this was achieved by restricting the data being analysed. The down side of this was that it was a manual process with relatively long iterations even though the source data set did not alter. This limits the scope of data available and potentially the results obtained which could be a signi cant disadvantage.
It was clearly di cult to analyse the lattices unless a speci c feature was
chosen as the focus for the analysis, primarily due to their size and complexity.
A possible side e ect of focussing would be the accidental exclusion of data that
could highlight unknown or unexpected relationships which should have been
a major bene t for this type of analysis. The whole problem of visualising and
exploring or "concept exploration" as termed by Stumme [
The analysis was limited as it only included attributes that could be attributed to a strategic goal within the ERP system. Making the link within relational database is relatively straight forwards however a far greater challenge would be including data from sources with less well de ned relationships. This maybe possible using tentative links such as times and dates but further work is required. This could be achieved within the data extract query as applied in MS Access for this approach.
Learning objective 1 - Describe the notion of representing and reasoning with knowledge for smart applications. This was visible in the coursework by the use of techniques such as performance measures / key performance indicators (KPI) within the data extraction on graphical interrogation of the outcome.
Learning objective 2 - Draw on one or more frameworks and techniques for representing and reasoning with knowledge for smart applications. This was visible in the coursework by the application of the tools and presentation of the analysis in the form of the coursework. The range of techniques applied further demonstrated the depth of analysis. There are a wide range techniques available and a reasonable range have been applied but only the minority of students have applied them.
Learning objective 3 - Identify the practical use of software tools for developing smart applications. This was visible in the coursework clearly by the conclusions where the ability to interact with the analysis and discover relationships was a clear advantage for FCA tools.
An emergent learning outcome was with regard to a developed appreciation of how the application of relatively simple analysis can highlight major aws in the decision making processes employed during the game therefore resulting in poor performance. A number of teams indicated this and identi ed where mistakes had been made due to a lack of analysis or assumptions based on incomplete knowledge. 6
The learning outcomes have been achieved with all students appreciating the value and di culties associated with analysing ERP data. The results did re ect a reasonable range of marks being awarded with all students able to perform both BI and FCA over the data set provided.
The di culty involved in data preparation had a signi cant impact on the analysis performed, particularly with respect to the application of more complex analysis techniques and semantic discovery. This was the main factor that detracted from the learning outcomes.
The coursework would bene t from more focus on the analysis and less e ort required for the preparation of data. It is expected that signi cant manual input will still be required in terms of de ning any calculations and manipulation of graphical outputs.
A structured criteria for the analysis techniques expected could lead to an improvement in the marks awarded. This could include a pre-con gured solution containing the basic forms of analysis, therefore forcing the use of more advanced analysis methods as a minimum criteria for the coursework. This could be achievable by reducing in the amount of data preparation activity required, however this must not place a constraint on the experimental aspect of this coursework and the ability to perform an open analysis.
There is a continued value in applying two methods of analysis, the BI approach is already familiar to the audience and clearly help understanding of the data set. Applying purely an FCA approach would be very challenging at this point in time.
As part of the action research aspect a number of factors should be changed for the next deployment of this coursework in order to permit the students to progress towards more advanced use of FCA, this is detailed in Further Work.
It was clear from the conclusions that the notion of applying BI and FCA was
understood and the value it has in real life applications. The value of good
analysis and the ability to evaluation unknown relationships was imparted. Equally
the potential for error, misunderstanding and potential lack of uptake because
of the complexity was clear and echoed the comments from Gartner in the
introduction with respect to what how analysis will be controlled by business units
and not technical experts [
Further Work The further work section will contribute towards the action research agenda and includes ideas or approaches to be included in the next iteration of the coursework.
A solution to reduce the amount of data preparation is required in order to support a focus of more advanced FCA. The rst area for consideration is providing a starting point that already supports the simpler forms of analysis.
More advanced forms of analysis should be directly supported, this includes
utilising qualitative data, better visualisations such as lattice on lattice
comparisons and concept clustering with iceberg lattices [
Utilising a solution that integrates directly to the data instead of the restricted data set contained in the MS Access extract is de nitely a requirement in order to support the forms of analysis highlighted above while providing a mechanism for an iterative and experimental approach to nding relationships within the data.