This paper examines and identi es issues associated with the applicability of FCA on sample data provided by a CUBIST use-case partner. The paper explains the various steps related to the transformation of these data to formal contexts, such as preprocessing, cleansing and simpli cation, as well as preprocessing and limitation issues, by using two FCA tools currently being developed in CUBIST, FcaBedrock and InClose. The paper demonstrates what is achievable to date, using the above-mentioned tools and what issues need to be considered to achieve more meaningful and intuitive FCA analyses. The paper concludes by suggesting and explaining techniques and features that should be implemented in later iterations of these tools, to deal with the identied barriers. This work has been carried out as a part of the European CUBIST FP7 Project: http://www.cubist-project.eu
It has been shown that a variety of datasets can be converted into formal
contexts [
This paper attempts to identify such issues, by conducting FCA on a dataset provided by Innovantage, a CUBIST use-case partner, providing market and competitive intelligence in the United Kingdom. The paper concludes on further work and explains what techniques will be deployed, in later iterations of the tools, to cater for the issues identi ed while analysing the speci c dataset.
The speci c dataset consists of job vacancies advertised on the United Kingdom's leading job boards, as well as employers' own websites, tracked in real-time using Innovantage's proprietary software. The dataset is in XML format and has been extracted from a MySQL RDBMS. The dataset comprises of 900 jobs accompanied by their details: { Title: The job's title. { Description: A brief description outlining the requirements of the job. { Date Found: The exact time of when the job was tracked. { URL: The website where the job was found at. { Raw Location: The location of the employer. { Raw Salary: The advertised salary, sometimes also including information about bonuses and bene ts.
An example of a job entry is shown below (File 1). <?xml version="1.0" encoding="UTF-8" standalone="no"?> <jobs> <job> <title>Data Centre Developer</title> <description>Data centers Developer opportunity based in Amsterdam on a 6 months rolling contract. This is a very senior position and requires the candidate to have at least 7 years experience and have extensive knowledge and expertise in the construction of a data centers facilities including electrical systems, cooling plants etc and familiar with EU regulations and best practices.</description> <date_found>2011-01-12 17:01:58.0</date_found> <url>http://www.itjobspost.com/JobSeeker/</url> <raw_location>Amsterdam, Other Countries, UK</raw_location> <raw_salary>400-600 Per Day</raw_salary> </job> </jobs> File 1 innovantage sample.xml, XML le. 3 3.1
Preprocessing Some issues surfaced during preprocessing, mainly due to the XML le failing to render properly because of illegal, non-UTF-8 characters contained in the data, possibly related to the automated process in which jobs are tracked and recorded. This was worked-around by parsing the XML le using a custommade algorithm and removing all illegal characters and symbols, without loss of information and without a ecting the quality of the data. The XML le was then loaded in MS Excel and converted to CSV (Figure 1), as FcaBedrock does not currently support XML as input.
Another issue that a ected the analysis was the fact that free-text attributes, as their name implies, are inconsistent, mostly due to the fact that the recorded data originate from various sources. Taking the `Raw Location' attribute as an example, a job's location can be recorded as \Manchester" for one job, \Manchester, United Kingdom" for another job and \Greater Manchester" for another job. The same problem applied for the `Raw Salary' attribute, as some employers use ranges (e.g \15000-20000"), some use nite values and also include currencies (e.g. \18000 GBP") and others also include additional information (e.g. \25000 per annum, negotiable"). For this type of free-text attribute to be succesfully and meaningfully converted, some kind of Semantic Extract Transform Load (SETL) or Natural Language Processing (NLP) process would be required rst, in order to identify values. As such, the data had to be manually modi ed; 100 jobs were randomly selected from the lot and had the above-mentioned attributes recon gured for consistency. In addition, some attributes were excluded from the analysis; in particular, `Description' was excluded as it is not an attribute, but rather the title (or descriptive annotation) of the object, although it could be useful as part of the case-study in terms of adding meaning to an analysis. The `Date Found' attribute was also excluded, as all of the jobs in the dataset were tracked on the same date, thus adding no speci c value to the analysis. The `URL' attribute was excluded, as URLs are unique for each job posted (thus considered free-text data). In terms of recon guring attributes, the `Raw Location' attribute was con gured to hold only city names and the `Raw Salary' attribute was con gured to be purely numeric. An extra attribute was created to hold additional information originally contained in the `Raw Salary' attribute, such as whether the salary is negotiable or not. This resulted in four attributes remaining: `Title', `Raw Location', `Raw Salary' and `Negotiable Salary' (the new attribute that resulted during preprocessing). A screenshot showing how the dataset looks after preprocessing is shown at Figure 2 below. The dataset was loaded in FcaBedrock and the `Title' attribute was excluded from the analysis, using the attribute exclusion feature. The metadata autodetection feature of FcaBedrock was used to avoid entering metadata manually (Figure 3).
Converting the dataset with FcaBedrock resulted in a formal context with
67 formal attributes. Feeding the formal context in InClose resulted in 110
formal concepts; although not quite a large amount, the concepts had to be
reduced to an amount where the concept lattice would be readable and
manageable. Over a trial-and-error process, using InClose and the well-known idea
of minimum-support (a semi-automated form of lattice `iceberging' [
Even with a small amount of objects and attributes, interesting information can be extracted from the lattice. For example, all non-negotiable salaries are the ones that fall in the £40000-60000 range, while the negotiable salaries fall in the £20000-40000 range. For salaries where it is unde ned (or unknown) if they are negotiable, there seems to be no distinct indication as to why this is the case. As for salaries in the £9240-20000 range, they all fall under the `sal negotiablen' and `sal negotiable-?' attributes, with a 50-50 ratio. The overall conclusion indicates that for high-end salaries negotiation is not an option, while negotiation is possible for mid-end salaries. Interesting is the fact that low-end salary jobs tend not to specify whether their salaries are negotiable or not, though when they do they are not negotiable. Why is that the case? Questions of these nature require further investigation.
While the insights provided by the resulting lattice might not be groundbreaking, a close collaboration of the FCA analyst with the domain expert would help in re ning the requirements, to produce meaningful business questions that would be more suitable for analysis of such data. For example, the domain expert might want to investigate why employers tend to not specify or negotiate jobs with low-end salaries. Could it have something to do with their geographic location or the domain of the job? Such kind of analysis is perfectly feasible in FCA, by restricting the context to speci c attributes (and attribute values) of interest. Figure 5 shows how this can be done in FcaBedrock, where the location was restricted to London, Manchester and Birmingham, the raw salary was restricted to low-end only and the negotiable salary attribute was set to unknown. As such, the business question has been rede ned to \display jobs in London, Manchester or Birmingham with low-end salaries, where salary negotiation is unknown or unspeci ed". It is evident that as new data sources and data types are introduced, more preprocessing issues arise. With regards to the tools used in this analysis (FcaBedrock and InClose), further development is currently in process and various issues, mentioned below, are already being considered.
In terms of data sources, XML should be added as a default data source, to avoid XML-to-CSV transformations. Pulling data directly from an RDBMS source would be quite useful as well, by selecting speci c database tables, or even speci c columns from each table, to use in the analysis. Manipulation of RDF data are of high importance as well, given the fact that CUBIST revolves around semantic technologies.
Free-text data have proven to be not suitable for FCA, unless some kind of
Semantic ETL or NLP process, in order to identify values, is deployed rst. Use
of thesauri, such as the approach described in [
Another feature that would be particularly useful would be to embed
additional functionality in the autodetection features of FcaBedrock, particularly for
selecting appropriate scales and intervals for continuous attributes.
Understanding the true nature of a continuous attribute at the moment, using FcaBedrock,
is only feasible when datasets include documentation, such as the ones in the
UCI Machine Learning Repository [
The paper has explored the application of FCA within a market intelligence scenario, using real-life data from a CUBIST use-case partner, deploying freelyavailable and open-source FCA tools, currently being developed in CUBIST, for the analysis. Several preprocessing issues have been identi ed and suggestions, techniques and features have been proposed for further work.
Although the work presented in this paper is still at an early stage, it demonstrates how the market data and FCA communities can bene t from each other. The market data community has provided new challenges that FCA has to consider, mostly in terms of usability and user-friendliness. Within the context of CUBIST, we envisage that the market data analysts will be able to conduct FCA analysis on their data, without collaborating with FCA experts. Acknowledgement This work is part of the CUBIST project (\Combining and Uniting Business Intelligence with Semantic Technologies"), funded by the European Commission's 7th Framework Programme of ICT, under topic 4.3: Intelligent Information Management. More information on the project can be found at http://www.cubist-project.eu