Data is not static, and attribute value changes often trigger changes in another set of attributes. Traditional methods for analyzing data changes often treat these changes in isolation, failing to consider the broader context in which they occur. This lack of contextual awareness limits the ability to capture relationships between attributes or interpret their significance, especially when distinguishing between normal variations and potential anomalies. In this paper, we discuss the importance of context-awareness and the need to identify normal change behaviour. To achieve this, we introduce a new data quality rule, called change rule, capable of capturing changes in both antecedent and consequent attributes within ordered tuples of a relational instance.
In real-world datasets, values rarely remain static as data continuously changes over time. These changes often carry critical information, revealing patterns, trends, and triggers that are essential for understanding environmental conditions, system, user behaviour and trends. Existing database systems have limited functionality to manage changes, and to identify abnormal changes, often relying on triggers to recognize out-of-bound changes. In this work, we consider changes to relational attributes for an entity. To simplify our setting, attribute changes are modeled as a sequence of ordered tuples, implicitly with respect to time. Hence, a tuple represents the value each attribute holds for an entity at a specific point in time.
Data changes occur in numeric and non-numeric
attributes. Changes to numeric attributes are often measured
using absolute diference, percentage change, rate of change,
rolling average [
For non-numeric attributes, changes are often measured
using edit distances (Levenshtein, Jaro-Winkler, Hamming)
[
Example 1. Table 1 shows two employees (Emp) E1 and E2 and their Position, Salary and number of employees managed (EmpMng) as of a specific Year. Consider the following changes and the need for greater context: Numeric attribute value changes: As observed in tuples 1 − 3 of Table 1, after only two years as a Software Developer, E1 was promoted to the position of Senior Software Developer accompanied by a significant increase in salary ($68,400 to $82,000). Whereas tuples 9 − 13 show that E2 spent four years as a Software Developer before being pro
Year
Emp EmpMng 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Position
Salary moted to Senior Software Developer with a similar salary increase as E1’s (from $71,500 to $80,000). Changes in salary are typically quantified using percentage change (+19.9% for E1 and +11.9% for E2). While this provides a numerical summary of the change, it fails to account for the broader context. For instance, E1 received a larger raise after a shorter tenure and took on the responsibility of managing four employees, whereas E2 had to wait twice as long for a similar promotion and gained the responsibility of managing two fewer employees compared to E1.
Non-numeric changes within and between classes: Traditional edit distance metrics such as Levenshtein distance (LD) quantify changes based on character modifications. The transition from Software Developer to Senior Software Developer has an LD = 7, whereas for Senior Software Developer to Lead Developer, LD = 13. These values suggest that the latter change is almost twice as significant as the former despite both changes being promotions to the next position within the same class (development roles), as shown in Figure 1.
The implications of a change can be much greater between diferent classes. For instance, the LD between Lead Developer and Manager is 11 which suggests that this transition is smaller than the transition from Senior Software Developer to Lead Developer (LD = 13). However, this interpretation is misleading. The change from Lead Developer to Manager represents a more significant career shift as compare to the change from Senior Software Developer to Lead Developer (where both positions are in the same class), as it involves moving from a development role to a managerial position (2 levels up as per Figure 1) which is accompanied by a significant increase in the number of people managed. Existing distance measures fail to capture semantic interpretations of the data.
Problem 1: The need for context. The example highlights that not all changes are equally significant. Context is often needed to interpret data change, and there is a need to augment existing distance metrics with context.
While identifying (significant) changes is important, it is
equally critical to diferentiate between normal changes vs
abnormal changes. Traditional methods have used
declarative methods such as data dependencies of the form → ,
where , are attribute sets, representing antecedent and
consequent attributes. Order Dependencies (OD) [
Example 2. Consider a sequential dependency (SD) stating that when ordered by Position, the change in Salary between consecutive tuples should be between 5% and 20%. For E1, the SD is violated between 3 and 4 with a salary change of 2.6% falling below the range. It is also violated between 7 and 8 , where the salary change (23.8%) exceeds the upper bound. These violations help in identifying abnormal changes. However, we also want to identify patterns where diferent changes in the antecedent attributes, such as changes within Position will elicit diferent changes in the consequent (Salary). For instance, with no change in position, salary still changes annually by 2% to 10%. Whenever there is a promotion (change in position > 0), the salary always changes by 10% to 25%. The existing dependencies do not capture relationships of this form.
To address this, we define a data quality rule called change rule. The change rule captures relationships between changes in attribute values of an ordered relational instance.
Problem 2: Diferentiating normal vs. abnormal data change. Existing dependencies do not capture the dependence between changes from antecedent attributes to consequent attributes on ordered tuples. A declarative specification is needed that models the expected range of value change between attribute sets. We propose change rules to address this problem. 1.1. Challenges • Context representation: Context helps to interpret the significance of data changes. Which attributes, and which subset of values are used to provide this context? Is this context time-dependent? How are existing distance measures augmented to consider this context? • Eficient rule mining: Manual specification of change rules is not practically feasible, and automated solutions are needed. Determining dependent sets of attributes is important towards identifying meaningful data change. Exhaustive enumeration of all attribute sets and their values is not feasible, and eficient methods to evaluate the large space of attribute sets are needed. • Filtering spurious changes. Rule mining is known to produce spurious rules. Determining which changes are most relevant and defining (support) measures that filter less meaningful changes is necessary.
We expect to make the following contributions.
• Context-aware change metric: A metric for
quantifying changes in both numeric and non-numeric attributes,
augmenting them with contextual information from
related attributes.
• Change rules: A new rule that captures the relationship
of changes from one attribute set to another attribute
set across ordered tuples.
• Change rule discovery algorithm: An eficient
discovery algorithm for change rules over ordered datasets. The
algorithm adapts the FastDD method to handle ordered
data [
We discuss the relationship of our work to existing metrics, data dependencies, association rules and statistical/ML approaches.
Traditional numeric metrics analyze individual attributes in
isolation, missing contextual relationships between changes
in diferent attributes. Measures of central tendency (mean,
median, mode) summarize values but can be skewed by
outliers. Dispersion metrics (variance, standard deviation, IQR)
capture data spread, but also prone to outlier sensitivity.
Shape distribution measures (skewness, kurtosis, CV)
describe asymmetry and variability but can be biased when
data is highly skewed or sparse [
For non-numeric (categorical, text) data, cosine similarity
is commonly used. Cosine similarity measures the cosine
of the angle between vectors [
While these metrics are widely used, they do not capture semantic distances. For example, "Software Developer" and "Senior Software Developer" have a high edit distance despite being closely related in meaning. Embedding-based approaches (e.g., BERT) address this by capturing contextual meaning but require pre-trained models and domain-specific tuning. An efective approach for measuring semantic similarity between non-numeric values is to compute cosine similarity on BERT embeddings, which allows for a contextaware representation of the data.
Order Dependencies (ODs) extend functional dependencies
by enforcing ordering relationships [
Sequential Dependencies (SDs) declaratively specify the
change in consequent attributes [
Diferential Dependencies (DDs) model diferences
between any two tuples in a relation independent of the
tuple ordering, i.e., if the antecedent attribute diferences lie
within a range , then the consequence attribute value
diferences must lie within a range [
TSDDs [
Association rules identify co-occurrences of items within
a dataset, typically expressed in the form of {, } → ,
stating that if items and appear together, then is
likely to appear as well [
Statistical and machine learning approaches leverage patterns in historical data to identify deviations that fall outside expected behavior. Statistical methods rely on predeifned thresholds and assumptions about data distribution, while machine learning approaches adapt to complex, highdimensional datasets.
Statistical and machine learning approaches ofer
complementary techniques for identifying and diferentiating
normal and abnormal changes in data. Statistical techniques
include rule-based thresholds and hypothesis testing. For
example, Z-scores and modified Z-scores are commonly
used to detect anomalies by measuring how far a data point
deviates from the mean, relative to the standard deviation
[
Machine learning provides various techniques for
distinguishing normal from abnormal changes in data,
particularly through anomaly detection algorithms. Isolation
Forest [
While numerous anomaly detection methods exist, our approach specifically targets anomalies in the change of attribute values. We achieve this by defining a change rule that not only identifies abnormal behavior but also captures the relationships between changes across multiple attributes.
Let be a relational schema on attributes 1, 2, ..., , and and be sets of attributes such that ⊆ and ⊆ . Let = {1, 2, ..., } be a relational instance of with tuples, ordered on X (implicitly ordered on time). The distance between consecutive tuples in for an attribute is given via a context-aware distance measure: ([], +1[]). We define a permissible range for as = (, ), where , are real values, i.e., if ([], +1[]) ∈ , then ≤ ([], +1[]) ≤ .
We define a support function (, )) that measures the relative strength of a change rule in . Naturally, we seek high-support rules to ensure that they have suficient evidence in the instance. We introduce change rules in the next section, and focus on their discovery (as part of Problem 2).
Problem Definition: Given a minimum support threshold , find all change rules Σ such that satisfies Σ ( |= Σ) , such that for all ∈ Σ , (, ) ≥ . A change rule is a novel data quality rule which describes a relationship between the changes in attributes within . It states that when the change in the antecedent is within some range = (, ), then the corresponding change in the consequent will also be within a defined range = (, ).
DEFINITION 1. Let be the permutation of tuples of increasing on (that is, (1)[] < (2)[] < . . . < ()[]). Change rule : → holds over if for all such that 1 ≤ ≤ − 1, when ( ()[], (+1)[]) ∈ then ( ()[ ], (+1)[ ]) ∈ .
When ordered on X, if the between any two consecutive -values is within the range then the between the corresponding -values must be within . A change rule with a minimum support threshold holds when at least % pairs of consecutive tuples in the instance satisfy the conditions of the change rule.
Example 3. Consider the change rule over Table 1: : (5,15) → (0.1,0.25)
This rule states that if the change in Position is between 5 and 15, then the change in Salary will be between 10% to 25%. This holds true for most of the table except when E2 is promoted from Senior Software Developer to Lead Developer. In this case, the salary increase is only 8.7%, which is below the expected 10% to 25% increase. This deviation from the rule highlights that the employee received a smaller-than-normal raise with their promotion.
We build upon the Diferential Dependency discovery
algorithm, FastDD [
Data changes over time, however, we want to capture relationships between these changes. In this paper, we discussed the importance of context-awareness when capturing these changes and the relevance of identifying normal change behaviour. We introduced a new data rule, called change rule, that captures the relationship between the changes in antecedent and the changes in the consequent.
As next steps, we plan to address the aforementioned problems and challenges: • Exploring transformer-based embeddings (e.g., BERT), to quantify and accurately capture contextaware changes in numeric and non-numeric data without compromising semantic information. • Optimize Set Cover Enumeration by developing an eficient method to minimize the search space when identifying minimal subsets of antecedent changes that explain consequent violations. • Consider the lagged efects of earlier changes on subsequent changes, i.e., the change in an attribute at one time step influences changes at a later time step.