In real-world datasets, values rarely remain static as data continuously changes over time. These changes often carry critical information, revealing patterns, trends, and triggers that are essential for understanding environmental conditions, system, user behaviour and trends. Existing database systems have limited functionality to manage changes, and to identify abnormal changes, often relying on triggers to recognize out-of-bound changes. In this work, we consider changes to relational attributes for an entity. To simplify our setting, attribute changes are modeled as a sequence of ordered tuples, implicitly with respect to time. Hence, a tuple represents the value each attribute holds for an entity at a specific point in time.
Data changes occur in numeric and non-numeric attributes. Changes to numeric attributes are often measured using absolute difference, percentage change, rate of change, rolling average [1]. While these metrics are easy to compute, they fail to capture the broader context of the change, such as the influence of related attributes or the significance of the change.
For non-numeric attributes, changes are often measured using edit distances (Levenshtein, Jaro-Winkler, Hamming) [2] or set-based coefficients (Overlap, Jaccard, Dice) [3]. However, these metrics are insufficient because they ignore the semantic meaning of the changes and the context in which they occur. Context is critical because it provides the necessary information to interpret the significance of a change. Without context, changes are reduced to isolated events, which can lead to misleading interpretations of the data change.
Example 1. Table 1 shows two employees (Emp) E1 and E2 and their Position, Salary and number of employees managed (EmpMng) as of a specific Year. Consider the following changes and the need for greater context: Numeric attribute value changes: As observed in tuples π‘1 β π‘3 of Table 1, after only two years as a Software Developer, E1 was promoted to the position of Senior Software Developer accompanied by a significant increase in salary ($68,400 to $82,000). Whereas tuples π‘9 β π‘13 show that E2 spent four years as a Software Developer before being pro- moted to Senior Software Developer with a similar salary increase as E1's (from $71,500 to $80,000). Changes in salary are typically quantified using percentage change (+19.9% for E1 and +11.9% for E2). While this provides a numerical summary of the change, it fails to account for the broader context. For instance, E1 received a larger raise after a shorter tenure and took on the responsibility of managing four employees, whereas E2 had to wait twice as long for a similar promotion and gained the responsibility of managing two fewer employees compared to E1.
Non-numeric changes within and between classes: Traditional edit distance metrics such as Levenshtein distance (LD) quantify changes based on character modifications. The transition from Software Developer to Senior Software Developer has an LD = 7, whereas for Senior Software Developer to Lead Developer, LD = 13. These values suggest that the latter change is almost twice as significant as the former despite both changes being promotions to the next position within the same class (development roles), as shown in Figure 1.
The implications of a change can be much greater between different classes. For instance, the LD between Lead Developer and Manager is 11 which suggests that this transition is smaller than the transition from Senior Software Developer to Lead Developer (LD = 13). However, this inter- 1 pretation is misleading. The change from Lead Developer to Manager represents a more significant career shift as compare to the change from Senior Software Developer to Lead Developer (where both positions are in the same class), as it involves moving from a development role to a managerial position (2 levels up as per Figure 1) which is accompanied by a significant increase in the number of people managed. Existing distance measures fail to capture semantic interpretations of the data.
The need for context. The example highlights that not all changes are equally significant. Context is often needed to interpret data change, and there is a need to augment existing distance metrics with context.
While identifying (significant) changes is important, it is equally critical to differentiate between normal changes vs abnormal changes. Traditional methods have used declarative methods such as data dependencies of the form π β π , where π, π are attribute sets, representing antecedent and consequent attributes. Order Dependencies (OD) [4], Sequential Dependencies (SD) [5], and Differential Dependencies (DD) [6] specify expected relationships between attribute sets. ODs introduce ordering relationships but do not explicitly quantify changes in attribute values. SDs model consequent attribute changes but do not account for variations in the antecedent attributes. DDs, while addressing changes in both antecedent and consequent attributes, apply to unordered data.
Example 2. Consider a sequential dependency (SD) stating that when ordered by Position, the change in Salary between consecutive tuples should be between 5% and 20%. For E1, the SD is violated between π‘3 and π‘4 with a salary change of 2.6% falling below the range. It is also violated between π‘7 and π‘8 , where the salary change (23.8%) exceeds the upper bound. These violations help in identifying abnormal changes. However, we also want to identify patterns where different changes in the antecedent attributes, such as changes within Position will elicit different changes in the consequent (Salary). For instance, with no change in position, salary still changes annually by 2% to 10%. Whenever there is a promotion (change in position > 0), the salary always changes by 10% to 25%. The existing dependencies do not capture relationships of this form.
To address this, we define a data quality rule called change rule. The change rule captures relationships between changes in attribute values of an ordered relational instance. Exhaustive enumeration of all attribute sets and their values is not feasible, and efficient methods to evaluate the large space of attribute sets are needed.
β’ Filtering spurious changes. Rule mining is known to produce spurious rules. Determining which changes are most relevant and defining (support) measures that filter less meaningful changes is necessary.
We expect to make the following contributions.
β’ Context-aware change metric: A metric for quantifying changes in both numeric and non-numeric attributes, augmenting them with contextual information from related attributes.
β’ Change rules: A new rule that captures the relationship of changes from one attribute set π to another attribute set π across ordered tuples.
β’ Change rule discovery algorithm: An efficient discovery algorithm for change rules over ordered datasets. The algorithm adapts the FastDD method to handle ordered data [7], and identifies changes in sequential attribute values using context-aware metrics.
We discuss the relationship of our work to existing metrics, data dependencies, association rules and statistical/ML approaches.
Traditional numeric metrics analyze individual attributes in isolation, missing contextual relationships between changes in different attributes. Measures of central tendency (mean, median, mode) summarize values but can be skewed by outliers. Dispersion metrics (variance, standard deviation, IQR) capture data spread, but also prone to outlier sensitivity. Shape distribution measures (skewness, kurtosis, CV) describe asymmetry and variability but can be biased when data is highly skewed or sparse [1].
For non-numeric (categorical, text) data, cosine similarity is commonly used. Cosine similarity measures the cosine of the angle between vectors [8] and is often used with embeddings to capture semantic similarity. Overlap, Jaccard, and Dice Coefficients [3] are used to quantify the similarity and diversity of sets. Edit distance such as Levenshtein, Jaro-Winkler, Hamming quantifies the number of operations needed to transform one string into another [2].
While these metrics are widely used, they do not capture semantic distances. For example, "Software Developer" and "Senior Software Developer" have a high edit distance despite being closely related in meaning. Embedding-based approaches (e.g., BERT) address this by capturing contextual meaning but require pre-trained models and domain-specific tuning. An effective approach for measuring semantic similarity between non-numeric values is to compute cosine similarity on BERT embeddings, which allows for a contextaware representation of the data.
Order Dependencies (ODs) extend functional dependencies by enforcing ordering relationships [4]. They ensure that a positive change in the antecedent corresponds to a positive change in the consequent. However, the semantics of ODs do not declaratively capture the change in any attribute values.
Sequential Dependencies (SDs) declaratively specify the change in consequent attributes [5]. They enforce constraints on how the consequent changes in response to an instance ordered on the antecedent, i.e. when the instance is ordered on π, the changes in the consecutive π -values will be within a range π. However, they fail to capture the change in the antecedent. Conditional SDs (CSDs) focus on identifying intervals within ordered data that satisfy a given SD. They prefer larger, contiguous intervals that capture a substantial portion of the data satisfying the embedded SD. However, the continuity of these intervals requires a trade-off with the specificity of the bound π, which is not addressed in the paper.
Differential Dependencies (DDs) model differences between any two tuples in a relation independent of the tuple ordering, i.e., if the antecedent attribute differences lie within a range ππ₯, then the consequence attribute value differences must lie within a range ππ¦ [6]. By not capturing order, DDs miss critical contextual information like trends or patterns across consecutive tuples.
TSDDs [9] designed for time-series data capture temporal relationships by treating data within a given time window as an ordered set and supporting real-valued function operations. However, similar to SDs, they do not account for changes in the antecedent attributes over time. Additionally, selecting an optimal time window remains a challenge, as an overly narrow window may overlook significant trends, while a broader one risks diluting the relevance of dependencies.
Association rules identify co-occurrences of items within a dataset, typically expressed in the form of {π΄, π΅} β πΆ, stating that if items π΄ and π΅ appear together, then πΆ is likely to appear as well [10]. Unlike data dependencies, which enforce constraints that all instances must satisfy, association rules identify probabilistic relationships without guaranteeing consistency. Dependencies ensure structural integrity, while association rules uncover patterns that may not hold universally.
Statistical and machine learning approaches leverage patterns in historical data to identify deviations that fall outside expected behavior. Statistical methods rely on predefined thresholds and assumptions about data distribution, while machine learning approaches adapt to complex, highdimensional datasets.
Statistical and machine learning approaches offer complementary techniques for identifying and differentiating normal and abnormal changes in data. Statistical techniques include rule-based thresholds and hypothesis testing. For example, Z-scores and modified Z-scores are commonly used to detect anomalies by measuring how far a data point deviates from the mean, relative to the standard deviation [1]. For example, if a data point's z-score exceeds a certain threshold (e.g., 3), it may be flagged as abnormal. Similarly, control charts and statistical process control (SPC) methods monitor data streams over time, flagging points that fall outside control limits as potential anomalies [11].
Machine learning provides various techniques for distinguishing normal from abnormal changes in data, particularly through anomaly detection algorithms. Isolation Forest [12] isolates anomalies by partitioning the dataset into smaller subsets. Points that require fewer partitions to be isolated are identified as anomalies. This method works well in high-dimensional data but may struggle with datasets containing overlapping clusters or anomalies that are close to the decision boundary.
While numerous anomaly detection methods exist, our approach specifically targets anomalies in the change of attribute values. We achieve this by defining a change rule that not only identifies abnormal behavior but also captures the relationships between changes across multiple attributes.
Let π be a relational schema on attributes π΄1, π΄2, ..., π΄π , and π and π be sets of attributes such that π β π and π β π . Let πΌ = {π‘1, π‘2, ..., π‘π } be a relational instance of π with π tuples, ordered on X (implicitly ordered on time). The distance between consecutive tuples in πΌ for an attribute π΄ is given via a context-aware distance measure: πππ π‘(π‘π[π΄], π‘π+1[π΄]). We define a permissible range for πππ π‘ as ππ΄ = (π, π), where π, π are real values, i.e., if
We define a support function π π’πππππ‘(π, πΌ)) that measures the relative strength of a change rule π in πΌ. Naturally, we seek high-support rules to ensure that they have sufficient evidence in the instance. We introduce change rules in the next section, and focus on their discovery (as part of Problem 2). Problem Definition: Given a minimum support threshold π, find all change rules Ξ£ such that πΌ satisfies Ξ£ (πΌ |= Ξ£), such that for all π β Ξ£, π π’πππππ‘(π, πΌ) β₯ π.
A change rule is a novel data quality rule which describes a relationship between the changes in attributes within πΌ. It states that when the change in the antecedent is within some range ππ₯ = (π π₯π , ππ₯π’), then the corresponding change in the consequent will also be within a defined range ππ¦ = (π π¦π , ππ¦π’). DEFINITION 1. Let π be the permutation of tuples of πΌ increasing on π (that is, π‘ π (1) [π] < π‘ π (2) [π] < . . . < π‘ π(π ) [π]).
Change rule π :
When ordered on X, if the πππ π‘ between any two consecutive π-values is within the range ππ₯ then the πππ π‘ between the corresponding π -values must be within ππ¦. A change rule with a minimum support threshold π holds when at least π% pairs of consecutive tuples in the instance satisfy the conditions of the change rule. This rule states that if the change in Position is between 5 and 15, then the change in Salary will be between 10% to 25%. This holds true for most of the table except when E2 is promoted from Senior Software Developer to Lead Developer. In this case, the salary increase is only 8.7%, which is below the expected 10% to 25% increase. This deviation from the rule highlights that the employee received a smaller-than-normal raise with their promotion.
We build upon the Differential Dependency discovery algorithm, FastDD [7] over unordered data.
β’ Diff-Set Construction: Encodes pairwise differences between all tuples into a diff-set, where each element represents a differential constraint violation (e.g., π‘π[π΄] β π‘π[π΄] > π), where π‘π and π‘π are any two tuples in a relational instance and π is a numerical value. For change rules, we modify this step by using a sorted instance πΌ on the antecedent attributes π to compute the πππ π‘ between consecutive pairs of tuples. This eliminates the redundant comparisons by restricting diff-set construction to adjacent tuple pairs in the sorted instance πΌ.
β’ Set Cover Enumeration: Finds minimal subsets of differential functions (antecedent) that cover all violations of the consequent. For change rules, instead of fixed thresholds, use intervals ππ₯ and ππ¦ for antecedent and consequent gaps. That is, we find the minimal subsets of πππ π‘(π‘ π(π) [π], π‘ π(π+1) [π]) β ππ₯ that cover all violations of πππ π‘(π‘ π(π) [π ], π‘ π(π+1) [π ]) β ππ¦.
Data changes over time, however, we want to capture relationships between these changes. In this paper, we discussed the importance of context-awareness when capturing these changes and the relevance of identifying normal change behaviour. We introduced a new data rule, called change rule, that captures the relationship between the changes in antecedent and the changes in the consequent.
As next steps, we plan to address the aforementioned problems and challenges:
β’ Exploring transformer-based embeddings (e.g., BERT), to quantify and accurately capture contextaware changes in numeric and non-numeric data without compromising semantic information.
β’ Optimize Set Cover Enumeration by developing an efficient method to minimize the search space when identifying minimal subsets of antecedent changes that explain consequent violations.
β’ Consider the lagged effects of earlier changes on subsequent changes, i.e., the change in an attribute at one time step influences changes at a later time step.