In this vision paper we structure a vision for the Business Intelligence of the near future in terms of a model with novel concepts and operators. We envision systems where the end-user requests information at a very high level, expressed as his intention to discover information, and the system transforms this request to the concrete execution of algorithms in order to compute, visualize and comment data and important highlights among them as an answer to the information request made by the end-user.
How will BI look like 10 years from now? What foundations should academia build in order to rigorously support the building of tools, the optimization of OLAP sessions, and the training of new scientists around a logical Paradigm? In this vision paper, we make a first attempt to revisit the foundations of OLAP and BI in an attempt to address the aforementioned questions.
To start with, it is worth to shortly revisit the evolution of OLAP and BI modeling so far.
• At the beginning of time, people would be working with
relational queries and recordsets, returned by these queries.
This treatment of BI was very DBMS-oriented, as the focus
of attention was on what the DBMS can do for the users
[
What is the vision for the new generation of BI tools? Broadly speaking, we envision the redefinition of what an OLAP query is, both with respect to what users ask the system and with respect to what the answer entails.
• Concerning the aspect of what users can ask, we wish to further simplify the operations and choices that the users face and add an extra layer of abstraction. Specifically, we propose to replace cube operators with intentional operators over the data – in other words, with an "algebra" of operators closely to the user’s intentions. Instead of the users operating in terms of roll-up and drill-down with the cube, we wish to empower them to state their intentions with respect to an operation. For example, instead of saying "drill down this cell one level", the user might ask "explain the drop in the sales of this product", or "is the main drop in sales observed at the best selling city also observed in similar cities?"; as another example, instead of issuing a roll-up operator, the user can state "verify whether a trend that I observe in a certain context still holds in general". • Concerning the aspect of what the answer to a query is, we believe we can no longer remain with "sets of tuples" as the answers to queries. We can envision BI tools where the answer to a query includes (i) a set of tuples accompanied with appropriate visualizations, (ii) the automatic mining of models and patterns, (iii) the extraction of important "jewels" hidden in the result (which we call highlights), and naturally, (iv) advanced, intuitive reporting.
Contributions and Benefits . The main contribution of this paper is that it structures a vision for the Business Intelligence of the near future in terms of a model with novel concepts and operators. We aim our definitions to be broad enough, yet as precise as possible; at the same time, we want to link them as much as possible to the intentional nature of the next generation of BI tools, where the end-user requests information at a very high level and the system transforms these requests to concrete execution of algorithms in order to compute, visualize and comment data and important highlights among them as an answer to the information request made by the end-user.
Why bother? We are convinced that after 50 years of naive query answering, it is now time to replace it with efortless, automatic insight gaining from the user. Instead of making the end user dig into sets of records, we can increase productivity and the understanding of the essence of the data by using two pillars, one devoted to querying and another devoted to answering, specifically, firstly, by allowing the user to focus on high-level goals of information acquisition, rather than details of what data to bring in, and secondly, by providing focus-points in the answers that will move the user efort from manual "jewel mining" to addressing the insights gained.
Several other practical benefits can be envisioned. Our
motivation was partially triggered from the problem of generating
meaningful, artificial session logs for experimental reasons (e.g.,
like [
An OLAP session is a sequence of dashboards that the analyst sees, each with its own information, including data, charts and informative summaries of KPI performance. The sequence is produced by the actions of the analyst that changes the contents of the dashboard by requesting more information on the basis of a set of operations made available to him by the tool. 2.1
For lack of space, we do not go to the details of data modeling.
We closely follow the model of [
We model an OLAP session as a directed graph G(V ,E). The set of vertices of the graph represent states of the user’s session –practically the dashboard screens the user is receiving by the OLAP tool– and the arcs represent transitions among states. 2.2
Before describing in detail our vision on dashboards and their internals, we start with the deeper essence of the intentional nature of our proposal: user operations, or, equivalently transitions between the states of an OLAP session. The main idea behind transitions is that we move from a concrete model of logical operators like roll-up’s and drill down’s, to an intentional model where the user expresses in terms of operators for high-level requirements like "explain a certain phenomenon", "predict the future values" and this high-level requirement has to be automatically translated to specific OLAP and Data Mining operators/algorithms that will carry the answer. This can also facilitate greatly the extraction of highlights, as the user’s goal is explicitly stated to the system.
We organize the transition operators that can express the users’
requirements for extra information, in families, i.e., a taxonomy
of types for our transitions, which we call transition types. In
contrast to other models where a transition would practically be a
query (relational or multidimensional), in our model, a transition
characterizes the intention of the user with respect to her
information need. Each family is a collection of transition operators
with the same high-level intention, but of course, diferent
semantics at the details. This set of families intends to cover (a)
traditional OLAP operators, (b) various contributions of the
literature, in particular discovery driven analysis operators [
Compare. A transition of type compare is used to retrieve more
data to be compared with what the current dashboard displays.
Intuitively, transitions in this family ask for bringing in more data
to the ongoing analysis. The family includes OLAP operators
pivot, drill-across, the augment operator of [
FocusOn. A transition of type FocusOn is used to exclude data
from the current analysis. Transitions in this family focus on a
particular cell, or group of cells. The family includes the OLAP
slice/dice operation, the take operator of [
Abstract. The transitions of this family are used to reduce the information load of the dashboard, by abstracting them in a more compact form. The family includes the traditional roll-up operator, as well as data mining operations like clustering, frequent itemset mining, etc. For instance, in Figure 1, sales results for years 2011 (not all shown in previous dashboards) to 2016 in Argentina are abstracted in 2 classes based on a result threshold (indicated as ❹).
Analyze and Explain. A transition of type Analyze and Explain
is used to understand the cause of a phenomenon displayed in the
current dashboard and to explore what is displayed at nfier levels
of detail. With respect to the analysis part, the family includes
the drill-down OLAP operation, and the details act of Cinecubes
[
Verify. A transition of type Verify is used to check that a
potential trend or pattern currently observed can be generalized to
broader contexts, e.g., we can check whether it occurs at coarser
levels of detail. The family includes Relax operator[
Predict. A transition of type Predict is used for predictive
analysis of the displayed data. Typically this involves the entire set of
timeseries forecasting methods, as well as classification methods
used for assessing future events [
Suggest. A transition of type Suggest is used to ask for
guidance with respect to where to navigate next in the
multidimensional data space. The family includes Sarawagi’s Inform [
Concluding with transitions, we emphasize that an open research challenge is to define an algebraically closed language of intentional operators, each of which can be translated automatically to the execution of concrete query and data mining operators. The synthesis of these operators can then create entire stories. Assume the user, being at state ❷, would like to: "verify whether the distribution of sales for mfgr#5 in Argentina from 2011 to 2016 still holds in general for all parts, and build a 2 classes model for it, then backtrack to parts and compare with sibling countries, and finally explain the highest country-wise diference." This request would correspond to the following statement in the intentional language: explainhiдhliдht :Max Dif f er ence ( comparesiblinдCount r ies ( backtrackshowT r end ( abstract2Cl asses (
veri f yall P ar t s,showT r end (❷))))).
An optimizer would then automatically translate this request into the sequence of operations: roll up, cluster, backtrack, Cinecube’s put in context, Dif and optimize it to produce a data story using dashboards ❷ to ❻. 2.3
A state is a dashboard that the user sees. In principle, each dashboard is ultimately based on the generating data provided by as a finite collection of queries, posed to the underlying database. A sharp distinction of our approach compared to previous models is that we do not restrict ourselves to data for each state but accompany them with a set of interesting findings, which come in two flavors, specifically, (a) in terms of models, i.e., results of data mining or machine learning algorithms applied over the data of a dashboard’s state, and, (b) important findings that accompany the dashboard, to which we refer as the highlights of the state.
Take for example, the small scenario of Fig. 2. The dashboard is based on a simple cube, as its generating data, involving Product , Reдion and Y ear as dimension levels and Sales and Cost as (aggregate) measures. Then, the dashboard automatically computes the Bene f it as the application of a simple arithmetic function, specifically, the diference of the two measures. A second step involves the building of a model, and specifically the classification of sales with respect to their stability of the benefit measure. Finally, highlights are extracted on the basis of this model, and specifically, these highlight are the cube cells pertaining to the class stable bene f its. Observe how the final data are extended with the class attribute linking them to their specific model counterpart.
Steps. In order to construct and visualize a dashboard, we envision several computations taking place. Here is the sequence of the performed actions: (1) First, the queries of the state’s dashboard are issued and their results, the generating data of the dashboard, are computed. Any straightforward computations for extra, derived columns of the dashboard (e.g., дain=price-cost ) are performed too. (2) Then, the available data are fed to model extraction algorithms for the computation of models abstract, summarize and provide patterns and insights for the data. (3) The potentially large amount of data and models computed has to be ranked and assessed on their interestingness for the analyst; the most important findings are classified as the dashboards highlights to be used for providing the main insights and the main directions for future transitions by the analyst. (4) The above are accompanied by visualization, text construction and reporting tasks that aim the process of understanding and communicating the main findings.
The generating data of the dashboard. The results of the underlying queries are the basis for the subsequent computations that take place for the construction of the dashboard: this is why we refer to them as the generating data of the dashboard.
Models. In diference from the state of the art in OLAP modeling, in our approach, we believe that the static results of aggregate queries, and their visualizations with charts and speedometers will simply be not enough in the near future. The automatic assessment and critical characterization of the presented data will be part of the BI of the near-future. See some simple cases based on the example of observing sales data of an international company: • Sales data will be automatically characterized with respect to a decision tree that classifies them (e.g., as "successful", "risky", "potentially hazardous" etc) • Sales per country will be automatically clustered to reveal similarities and diferences, as a first step towards understanding outliers and non-expected behavior • Aggregate sales over significant periods will be fed into time series analysis and forecasting methods to automatically detect trends, seasonalities and to deduce future values We consider the plugging of data analysis algorithms in the backstage of a dashboard as an indispensable part of BI. These algorithms can range from very simple ones (e.g., finding the top values of a cuboid, or detecting whether a dimension value is systematically related to top or bottom sales) to very complicated ones (like, for example, outlier detection, dimensionality reduction, etc). Most importantly, as the operation of the algorithms will likely be as transparent as possible to the end user, their execution will require an almost automatic tuning of their parameters. The findings of these algorithms will be models of the data that are typically (not always) used to annotate the existing data with characterizations and ofer focus points to the visualization of the dashboard (forecasts, outliers, dimension values that dominate top or bottom measures, . . .). The models themselves give a multitude of results. However, some of these results indicate that a part of a dashboard’s data are of important interestingness value to the end user. Due to that, we collectively refer to the important results of the execution of these algorithms as highlights, in an attempt to show that the aim is to enrich the current data-intensive dashboards with knowledge that is worth exploring or using for decision making.
We are going to treat model extraction algorithms as "blackbox" algorithm without probing into their internals, and, most importantly, without assuming any particular properties for their output. What does a model extraction algorithm do? Basically, the algorithm receives as input (a) a set of input data, and, (b) a set of execution parameters that have to be fixed for the algorithm’s execution. Without loss of generality, we can assume that a subset of these parameters will be bound to string or numerical values and the rest will be mapped to attributes of the input data. The output of a model extraction algorithm is a model of the input data. Depending on the algorithm, the result difers. For instance, a descriptive model built using unsupervised clustering is basically just a labeling of each cube’s cells, while a predictive one allows enriching the cube with predictions and comes with an accuracy score. In summary, the main properties of a model extraction algorithm is outlined as follows: (1) Input: a set of input data, which is the result of an extended cube query set of the dashboard, along with a fixing of the algorithm’s input parameters (2) Output: a (possibly complex) result composed of (a) a model of the input data, and, (b) several characterizations of it (precision, strength, p-value, . . . )
Each model type can be arbitrarily complex and consists of a set of model components of versatile nature. Examples: • A time series splits each of its points to 3 measurements, specifically error, trend and seasonality (practically creating 3 times series in the place of one, whose sum reconstructs the original one). • A clustering scheme includes a set of clusters, each with a set of tuples that constitute it, along with a centroid. • A classification decision tree includes a tree structure, best expressed as the composition of a set of paths, leading to characterization classes; again, each class comprises a set of generating tuples that pertain to it.
Typically, each such component, as well as the model on its
entirety is accompanied with a set of metrics of its statistical
power. Given a model type T , we denote its components with
TC = {tmc1 . . . tmcm }. We avoid resolving the internals of the
composition of the tmc parts and treat TC as a set of components.
This is realistic, as we can always mask a part-of relation as a set
of constituting components, even a recursive one, by allowing
an extra composition relation to provide the semantics of the
part of-relationship. Apart from this simple modeling, attempts
to relationally code mining results already exist [
But, then, how do we handle the heterogeneity of diferent model types? Clearly, a cluster is inherently diferent from a decision tree or the formula for a trend. Is there a unifying model to cover them all? The unifying essence of all the plethora of diverse model types is that, at the end of the day, all of them are annotations of the original data. At the end of the day, every component of a complex model type (be it a cluster id, a path in a decision tree and a resulting class, a characterization of the top-k tuples, or a trend formula): (a) refers to a subset of the input data and vice-versa, and, (b) refers to the overall model via a part-of relationship. So, once a model of the underlying data is available, our solution to the problem is to provide a distinct identity (an id) to the components of a complex type T , retain the membership/part-of relationships of T separately and annotate or characterize the data with respect to the part of T that pertains to them. In fact, this step can be blended within the model extraction itself. Examples of such annotation follow: • Assuming a time series model that splits a time series to trend, seasonality and noise, these attributes can be appended to the generating data set. • Assuming a cluster model, the generating data can be annotated with the id of the cluster to which they belong. • Assuming a classification model, the input data can be labeled via an extra attribute with respect to the class(es) of the model to which they belong. • Assuming a model of top-k values of a measure, the input data can be annotated with their rank, if they belong in the top-k set and they have been ranked.
A notable property of our modeling is that we require model components to be directly mapped and linked to their generating data in a bidirectional mapping, so that the end-user can navigate back and forth between cube cells and their models.
Highlight Production. As already mentioned, the set of highlights of the dashboard is a set of important findings that accompany the dashboard. These can be findings of any nature, e.g., important outliers in the contents of the dashboard’s data, all the tuples belonging to a certain class of a classification scheme, the top or bottom values of a measure, etc. It would be straightforward (and doable with our modeling) to treat all the contents of an extended query as highlights. However, we would like to stress that we want to restrict our attention to the ones that are really important for the end-user.
Other "local’ operations. Once all computations are done, there are several tasks to be taken care of before automatically constructing the dashboard. We envision that the development of principled methods for (a) the visualization of the dashboard data in various ways and (b) the creation of reports, via data storytelling methods will present major challenges for the future.
Visualize. Operations in this family compute the contents of visual representations (bar charts, speedometers, scatterplots, ...) on the basis of the current contents of the dashboard. We consider them to be local operations as they are performed without any further interaction with data external to the ones already retrieved from the dashboard.
Data storytelling. Data storytelling is a novel trend that seems
to carry significant weight in the way the future BI will look like.
The main idea involves the automatic generation of a textual
report from the data of a particular dashboard or of an entire
OLAP sessions. There are already some academic eforts, like
Cinecubes [
This paper is a vision paper describing, in broad terms, a potential future for OLAP, to strengthen its place as the corner stone of BI. Our call to arms to the research community can be structured along several lines. Population of the families of the transition operators with concrete operators. Each new operator (or each new formalization of existing operators) should hopefully carry (a) clear semantics, (b) execution algorithms as well as their optimizations, (c) fine tuning algorithms for the parameter fixing of the introduced algorithms and, equally importantly, (d) a graceful linkage to the overall model proposed here (in an attempt to be able to gracefully plug it in the respective BI tools). Automation of transitions and tuning. We believe this to be the most important piece of the puzzle. How do we fully automate what model and highlight extraction methods should be employed? This includes the possibility of predicting interesting results for the end-user, which in turn, requires appropriate models of interestingness. Along with the necessary run-time optimizations, all these tasks provide challenging and open problems of significance practical importance. Key Performance Indicators (at least in the way they are used today) are examples of simple model extraction, and by definition, KPI’s are highlights (or else they wouldn’t be "Key" indicators). Linking them explicitly to a model for OLAP is a necessary add-on for a comprehensive view of what OLAP does. Benchmarking and tools. A reference free, open-source tool and a reference benchmark for the future BI (involving data, model and highlight extraction requests and sessions) can be a really handy tool for the research community (otherwise, each new paper will need to improvise on its experimental assessment). A tool will also trigger other research directions, like e.g., the incorporation of research results on natural language processing to accept the user requests, visualizations to depict the models and highlights, etc.