Within the dynamic world of Big Data, traditional systems typically operate in a passive mode, processing and responding to user queries by returning the requested data. However, this methodology falls short of meeting the evolving demands of users who not only wish to analyze data but also to receive proactive updates on topics of interest. To bridge this gap, Big Active Data (BAD) frameworks have been proposed to support extensive data subscriptions and analytics for millions of subscribers. As data volumes and the number of interested users continue to increase, it is imperative to optimize BAD systems for enhanced scalability, performance, and eficiency. To this end, this paper introduces three main optimizations, namely: strategic aggregation, intelligent modifications to the query plan, and early result filtering, all aimed at reinforcing a BAD platform's capability to actively manage and eficiently process soaring rates of incoming data and distribute notifications to larger numbers of subscribers.
Management Systems⋆
with other data, and eveennrich them with additional infor- Tapestry [4] first introduced Continuous Queries as queries mation existing in diferent datasets. These demands have that are issued once and then return results continuously led to the creation of Big Active Data (BAD) frameworksas they become available. Tapestry also defined continuous [1, 2, 3] that aim to support extensive data subscriptionsquery semantics and created rewrite rules for transforming and analytics for millions of subscribers.
The BAD framework is designed to address several es- Subsequent research has primarily concentrated on queries sential needs: (i) It ensures that data is not examined ininvolving streaming data. NiagaraC5Q] [enhanced the isolation but rather in the context of related informatiosnca,lability and eficiency of continuous queries by breakenhancing its overall significance. (ii) It enables data en- ing them down into smaller, manageable components and richment by enhancing newly arriving data with data fromclustering similar queries based on their expression signaexisting datasets, allowing for responses that enable person-tures. It organized signature constants in a specialized table alized and actionable insights. (iii) It supports both real-timeand used joins to process similar queries as a group. Furprocessing and retrospective Big Data analytics, facilitatintghermore, to enhance computational eficiency, the system deeper exploration and long-term analysis of stored data. Byemployed delta files that allow for incremental updates and integrating these capabilities, a BAD framework eliminatesevaluations of the data. STREAM represents a research the ineficiencies of cobbling together multiple independent systems (each dealing with a part of the needed processing,across both data streams and persistent sto6r]a. gIet [proi.e., accessing Big Data, managing incoming streaming data,vided a Continuous Query Language (CQL) for constructing matching subscribers to information, etc). As we confrontcontinuous queries against streams and updatable relations the relentless expansion of data volumes and the burgeon-[7]. Most continuous query projects have struggled with ing number of users interested in this data, the challenge toscalability, making them less suitable for Big Data use cases, prototype that was designed to handle continuous queries user-provided queries into incremental database queries. manage ever-larger datasets becomes increasingly acute.
This work focuses on optimizing a BAD platform to better handle soaring rates of incoming data and a growingdata customizing pipelines and to provide real-time analytguages and Analytical Processing of Big Data, co-located with EDBT/ICDT (V. J. Tsotras) tion. We recommend using the latest version of the ceurart style. ⋆You can use this document as the template for preparing your publica- Cloud Dataflow [14] are specialized for stream processing as they often fail to scale efectively in such environments.
Streaming engines can be used for data processing and
ics. Apache Kafka [ 8, 9], Apache NiFi [
ceur-ws.org © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License data storage solutions. Instead, these systems typically require integration with external storage solutions to persist In AsterixDB, data is stored by creating a Datatype, which processed data for later use or analysis. This “glued-systems” describes known aspects of the data being stored, and then approach has been shown to have performance disadvan- a Dataset, which is a collection of records with a given tages in 3[]. datatype. As an example, Figu2reshows the DDLs that cre
Traditional publication/subscription (pub/sub) systems ate an open datatype nam"eEdnrichedTweet" and a dataset
[
The systems mentioned above generally face challenges continuous query semantics. in scaling, persisting data, or handling complex subscrip- CREATE TYPE EnrichedTweet AS { tion queries, which then requires integrating them with tid:int, additional systems for efective data processing. The BAD text:string, platform, summarized in the next section, was designed retweet_count:int, and built to address these constraints. In this current work,threatening_rate:int, we focus on further optimizing BAD. One of the key steps hate_speech_rate:int,
retweeted_status:string, involves creating the BAD index, an idea that is conceptu-weapon_mentioned:boolean, ally similar to partial indexin2g1][; it focuses on indexing drug_activity:string, only the most frequently accessed or queried segments of about_country:string, a dataset, yet is distinct from partial indexing in its imples-tate:string, mentation as explained in Sect4io.3n. aldodciattiioonn:aplo_iinnft,o:string}; 3. BAD Preliminaries CREATE ACTIVE DATASET
EnrichedTweets(EnrichedTweet) PRIMARY KEY tid; The BAD platform can enable millions of users to subscribe Figure 2: DDLs for creating a Datatype and its Dataset. to data of interest and receive updates continuously. It is diferent than continuous queries, streaming engines and 3.1. A BAD Application Example pub/sub systems as it also supports Big Data analytics with Tweets can ofer valuable insights into public opinions and a declarative language, SQL++ (a SQL-inspired query lan- behaviors, with many people relying on them to stay inguage for semi-structured dat2a2,[23]). formed about important topics. However, given the volume
An overview of the BAD platform is shown in Figure and velocity of tweets generated every second, it can be 1. BAD has five basic blocks [24]: (i) Data Feeds, which very challenging to extract useful information from them. manage the ingestion of rapidly arriving data; (ii) PersistentTherefore, providing a way for users to focus on tweets that Storage, responsible for storing the data; and an (iii) Analytia-re specifically relevant to their interests is highly beneficial. cal Engine, which enables analytic queries on the stored and People may be particularly interested in tweets related to incoming data to reveal useful information. Additionallys,pecific topics such as sports, politics, or various types of BAD includes (iv) Data Channels and (v) Brokers, which are crime. The BAD platform addresses this need by creating important for this paper and are described in detail later inchannels that users can subscribe to, delivering relevant this section using an example application. data based on their preferences, and leveraging a broker network to ensure that tweets are delivered to subscribers in real time. This system enables diferent applications to efectively manage the vast flow of information, allowing users to focus on the content that matters most to them.
To facilitate the distribution of results to millions of subscribers, the BAD platform integrates a broker sub-system to handle both subscription communication and result delivery [2]. Brokers can range from individual servers dedicated to relaying custom data to subscribers, to complex Figure 1: An overview of the BAD platform. networks ofering features like load balancing, subscription handover, and varied caching methods. Diferent brokers
There are 3 diferent types of users for the BAD system; can be registered as HTTP endpoints in the BAD platform, Subscribers who subscribe to channels so they can get the and subscribing end users have the flexibility to select a data of interesDte,velopers who create the BAD channels, broker that aligns with their specific requirements. Once and Analysts who run queries on the data stored in the results are prepared for subscribers, the BAD platform efsystem. ifciently dispatches the relevant updates to all subscribed
For this work we obtained a copy of the BAD open source individuals (end users) through the designated brokers. platform and have extended it further. The existing implementation of the BAD platfo2rm,25[, 3] was built as an ex- 3.3. Data Channels tension of Apache AsterixD2B6[], a Big Data Management The BAD platform user model exploits the shared structure System (BDMS) that provides distributed data managementamong subscriptions and ofers it as a service, namely as a for large-scale, semi-structured data. data channel. Data channels allow developers to activate parameterized queries as services for users to subscribe to and continuously receive their data of interest. In practice4,. Optimizing BAD the need for similar information among users would likely To address the challenges posed by escalating data volumes result in the creation of comparable queries. and a rising count of subscriptions, it is imperative to im
Consider a scenario where users might want the sys- plement optimization strategies for the channels within the tem to “send them threatening tweets that relate to the UBAS,D platform. These optimizations are crucial for ensuring are widely retweeted, and the sender’s location is close tothat the system can manage data efectively, accommodate their location”. ThTeweetsAboutCrime channel, depicted all subscriber requests promptly, expedite query processing, in Figure 3, allows its subscribers to receive nearby en- and consistently meet designated delivery deadlines as its riched tweets that concern the United States, possess uasers and data scale. When examining the processing perretweet_count greater than 10,000, and have a threatenf-ormed by the original implementation of the BAD platform, ing_rate exceeding 5 (on a scale of 0 to 10). The tPerEm- we discovered three patterns which ofer opportunities for RIOD refers to how often the channel is executed, which optimization. In particul(a1r), Duplicate Processing: the will be described later. To ensure that channels only process original BAD platform processes results for subscriptions and deliver newly incoming tweets between executions, the that ask for parameters that are identical as if they were disis_new function is employed in the channel definition to tinct subscriptions, resulting in redundant computa(t2i)ons. exclude already processed tweets and thus implement con- Overprocessing: subscription queries are executed on the tinuous query semantics. The channel’s parameters, such entire dataset that has accumulated since the last execution, as MyUserName in theTweetsAboutCrime channel, allow even when new records do not match existing subscriptions, the system to personalize results for each subscriber. Users leading to superfluous processing.(3) Late Data Filtering: can subscribe to channels of interest by providing param- in the origial BAD platform, although the query and all eters through DDL commands, as demonstrated in Figure predicates of a channel are defined during channel creation, 4. In this example, the user selects “user123” as the parame- which occurs prior to any channel execution, the system ter. The system retrieves the subscriber’s location from thepostpones processing and identifying the relevant data until UserLocations dataset by matching the username and will execution time. For each of these cases, below we discuss provide results for the user based on their location. the proposed solutions and their benefits, which are then
BAD data channels provide two modes for delivering data: showcased in the experimental section. More details about push andpull. In push mode, the data of interest is pushed the proposed optimizations are available30in]. [ to brokers directly. In pull mode, the broker receives a no- SUBSCRIBE TO tification from the channel whenever new data of interestTweetsAboutCrime("user123") ON BrokerA; becomes available for its subscribers; the subscribers can Figure 4: DDL for subscribing to the channel then request (pull) their data at any time. However, sinceTweetsAboutCrime. many applications prefer immediate access to complete information rather than making requests to retrieve resultsS,ELECT result, current_datetime() this paper focuses exclusively on push channels. In this as deliveryTime, sub.subscriptionId as sId approach, the broker receives data as soon as it is produced FROM Metadata.`Broker` b, TweetsAboutCrimeSubscriptions sub, after each channel execution and immediately disseminates TweetsAboutCrime(sub.param0) result (pushes) it to subscribers. We defer the exploration of opti-WHERE result.BrokerName=b.BrokerName; mizing pull channels to future work. Figure 5: The query running under the hood for the channel
Considering the need to promptly inform subscribers, the TweetsAboutCrime. update interval for tThweeetsAboutCrime channel (“period” in Figure3) has been set to every 10 minutes. Upon channel 4.1. Duplicate Processing: Aggregating creation, a dataset namTewdeetsAboutCrimeSubscriptions Subscriptions is created for storing subscriptions. Each subscription is In systems designed to serve a large user base, multiple users identified by a unique ID with an associated broker and sub- often create similar queries, leading to redundant processing scription parameters. Every 10 minutes, a recurring query, when retrieving results separately for each user. The BAD as shown in Figure5, generates channel results and matches platform introduced data channels to group common user them with the relevant subscriptions for distribution viaquery patterns into a single parameterized query, allowbrokers. The broker information is stored in Btrhoeker ing users to select their own parameters. However, further metadata dataset. If results are not ready to be delivered taonalysis has revealed additional areas where we can go a the broker before the deadline, the channel is halted. step further by implementing even more eficient sharing // Users can subscribe to the channel using mechanisms. For instance, in use cases where users sub// their usernames to get the threatening tweets scribe to specific content categories like trending topics or // posted near their location. news alerts, the limited number of categories can result in CREATE CONTINUOUS PUSH CHANNEL numerous subscriptions requesting the same data, diferentiTweetsAboutCrime(MyUserName) ated only by subscription IDs. This redundancy can burden PERIOD duration ("PT10M") { the system with multiple subscriptions that, upon closer SFERLOEMCETnrti.tcheexdtTweets t, UserLocations u inspection, are requesting the same information. A similar WHERE spatial_distance(u.location,t.location)<10 issue is observed in the BAD platform. For example, in the AND u.username=MyUserName AND is_new(t) TweetsAboutDrugs channel, shown in Figur6e, subscribers AND t.about_country="US" must specify their state. Since there are only a finite number AANNDD tt..trhertewaeteetn_icnogu_nrt>a1t0e>0500}; tohfeUs.Sa. msteastteast,te,hwisitsehttuhpeooftennlryesvualrtisatiniomnublteiinplge trehceourdsesrfsoIrDs. Figure 3: DDL for the TweetsAboutCrime channel. To avoid the ineficiencies associated with storing duplicate records, we propose grouping subscriptions based on their parameters and associated brokers. As a result, we create subscription-group records that maintain the group’s pa- After fixing the frame size, if a record is larger in size than rameter and broker name, along with an array that includesthe standard frame size, the particular frame is enlarged to all subscription IDs requesting the respective parameter and include this record; as a result, there may be frames that broker. Since groups may vary in size, these records are of are longer than the fixed size. Consolidating many subvariable length. scriptions into a single, large record afects the distribution
Figure 7 illustrates the transformed subscription datasetof processing tasks across diferent operators. Lfetdefor the new optimized system. To minimize additional over- note the fixed frame size in bytes ansd_i the record size head when grouping subscriptions, the groups are created, in bytes of subscription-grouip. If the group consolidates and subscriptions are assigned to the appropriate group many subscriptions ansd_i surpassesf, we must expand as they enter the system. The ID of each new incoming the frame size to fully encompass the subscription-group subscription is either allocated to a pre-existing group, orsince a record cannot be fragmented across frames. This it initiates a new subscription-group if its parameter andmay lead to fewer but larger frames and potentially reduce broker combination is not yet represented. parallelism. In this case dividing subscription-groiuinpto
Although grouping many subscriptions into a single smaller subgroups can be another option to improve parrecord may seem practical, it introduces certain system allelism. However, dividing subscription-gro uipinto too challenges, which are discussed below along with potential many smaller subgroups will lead to significantly increasing solutions. Note that subscription aggregation can be appliedthe computational load, since the system will calculate the in various domains where multiple users share common in- same result multiple times. terests or query parameters. For instance, in a financial mon- Clearly, there is a trade-of involved in optimizing the itoring system, users may subscribe to get updates about number of subscriptions within each group. This trade-of stock market trends based on specific conditions such as will be further examined in the experimental section. price movements, trading volume, or company performance metrics. Grouping these subscriptions by similar thresholds or parameters allows the system to handle multiple users with similar interests more eficiently, reducing duplicate processing. Similarly, in a content recommendation system, users may subscribe to updates about certain genres, authors, or topics. Aggregating these subscriptions based on similar preferences ensures more eficient content delivery Figure 7: Aggregation of subscriptions based on matching paand query execution. rameters and brokers. system performance.
CREATE CONTINUOUS PUSH CHANNEL 4.1.2. Broker Benefits TweetsAboutDrugs(Mystate) PERIOD duration ("PT10M") { Aggregating subscriptions extends advantages beyond SELECT t.text merely improving query execution times( the query execuWFHREORMEEntr.sicthaetde=TMwyesettastte AND is_new(t) tion time is defined as the duration measured from the start AND t.threatening_rate=10 to the end of the query execution). It also ofers significant AND t.drug_activity="Manufacturing Drugs" }; advantages on the broker side by notably decreasing both Figure 6: DDL for the TweetsAboutDrugs channel. the communication time between the BAD platform and the brokers, as well as the processing time required by brokers to 4.1.1. Parallelism in AsterixDB manage the results prior to dispatching them to subscribers. In all scalable database management systems, when dataConsider a scenario where a neEwnrichedTweets instance is overly aggregated, the ability to leverage the system’s (which is around 32 KB) pertains to drug-related activities in parallel architecture can be compromised, as fewer tasks California, and suppose there are one million subscriptions can be distributed simultaneously, leading to potential botf-or this state on tThweeetsAboutDrugs channel. Previously, tlenecks and decreased performance eficiency. AsterixDB this would create 1 million individual but similar results, faces similar challenges. To delve deeper into the opera- each corresponding to a subscription, thereby burdening tional mechanics, we first note that the unit of data whichthe broker with the management of redundant outcomes. is consumed and produced by diferent tasks in AsterixDB By aggregating the subscriptions according to channel pais called aframe. It is a fixed-size chunk of contiguous bytes rameters we substantially decrease the volume of results which always contains complete records, ensuring that athat need to be transmitted and processed by the broker. record is not split across multiple frames. An operator thatInstead of sending results for each individual subscription, produces data packs a frame with a sequence of completeresults are sent out per group, decreasing the data volume records and sends it to the consumer operator who thenfrom 32 GB to just 0.07756 GB. interprets the records.
Overall, the frame size in AsterixDB is selected to bal- 4.2. Overprocessing: Augmenting the Query ance memory eficiency, data movement, and task execution Plan to Align with User Preferences performance within the system. The optimal frame size is typically chosen based on the characteristics of the datCaonsider theMostThreateningTweets channel, shown in Figbeing processed, the available memory resources, and the ure 8, which allows users to learn about the most threatspecific requirements of the workload. Larger frame sizes ening (level 10) tweets in their state. The query plan are often preferred for workloads involving complex data for implementing this channel appears in Fig9ure(a). processing or large records, as they can reduce the number Originally, the BAD platform would completely scan the of I/O operations and enhance network eficiency. However, EnrichedTweets dataset, the channel’s subscription dataset this choice must be carefully managed to avoid memory con- called MostThreateningTweetsSubscriptions, and the tention or excessive garbage collection, which can degrade Brokers datasets, and eventually perform several steps leading up to a join between the selectEenrdichedTweets and theMostThreateningTweetsSubscriptions dataset. This ap- to occur during the initial data scan, reducing proproach becomes wasteful if most of the incoming data rarelycessing overhead. To further enrich channel results meets the subscription criteria. This is the case when onlywith detailed subscription information, such as each suba tiny fraction of tweets pertain to criminal activities anscdriber’s broker, an index nested loop join is applied berequire law enforcement’s attention. Therefore, failing totween the outcomes derived from the above join and the consider subscription parameters early in the query execu- MostThreateningTweetsSubscriptions dataset. The revised tion process can lead to creating a large volume of resultsquery plan is depicted in Fi9g (b). which will eventually be discarded due to not matching any When a query is known in the system before execution, it subscription criteria. Additionally, it is important to notcereates the possibility for the system to filter out irrelevant that some subscriptions might have similar parameters, andincoming records based on the query’s specific conditions, thus it is also crucial to structure the query process to avoidpotentially improving execution eficiency. We can observe redundant computations. This scenario can arise in anya similar scenario in the BAD system, where channels incorsystem that tries to align users’ interests with incoming oprorate fixed selection criteria to ensure only relevant data stored data. Overloading operators with irrelevant data thaits processed and delivered to subscribers. doesn’t contribute to the desired results is ineficient and un- For example, theTweetsAboutCrime channel appearing desirable. We propose an approach that selects only the datain Figure 3 has three predicates that involve the incomsatisfying at least one subscription. An initial solution mighitng data (combined with ANDt).:about_country="US", involve identifying relevant results by performing a join be- t.retweet_count>10000, t.threatening_rate>5. Knowing tween the dataset containing the required parameters, e.gt.,hese fixed predicates in advance allows for the integration the Enriched tweets dataset, and the subscription dataset oinf an additional pre-execution filter, streamlining the query the first step. However this would result in a massive join evaluation process by focusing only on relevant data from operation between two datasets with millions of records but he outset before executing the main logic of the query. This with a small number of results. For instance, in the case filter identifies an incoming record that meets all fixed seof theMostThreateningTweets example, this solution would lection predicates specified in the channel query’s WHERE involve joiningEnrichedTweets with the subscriptions, lead- clause and adds this record’s id in a dedicated secondary ing to a large join that produces a small number of results. index, known as the channeBl’sAD index. The method for creating the BAD index is described below.
CREATE CONTINUOUS PUSH CHANNEL TweetsAboutCrime(MyUserName) PERIOD duration ("PT10M") {
SELECT t.text FROM UserLocations u, EnrichedTweets t WHERE spatial_distance(u.location,t.location)<10
AND u.username=MyUserName AND is_new(t) AND t.about_country="US" //(I) AND t.retweet_count>10000 //(II) AND t.hate_speech_rate>5 //(III) AND t.threatening_rate>5 //(IV)
AND t.weapon_Mentioned=true //(V) }; Figure 10: TweetsAboutCrime channel DDL with extra conditions.
When a channel query is submitted to the system, a BAD index is created for each active dataset involved in the channel that has fixed selection predicates applied to it. The fixed conditions for each dataset are grouped together and added to an ordered list, caclolendditionsList, created for that active dataset, and when the channel is deleted from the system its conditions are also deleted.
Conditions lists help determining whether incoming data satisfies the conditions for any channel. Each new incoming record will be checked against all groups of conditions in this list, and if it satisfies all conditions for a given channel, it will be added to that channel’s BAD index.
Figure 9: (a) Original vs. (b) Optimized channel plan. For example, when theTweetsAboutCrime channel is created (Figure3), it includes three fixed conditions on the 4.3. Late Data Filtering: BAD Index activeEnrichedTweets dataset. Consequently, the chanTo address efectively, we introduce a “UserParameters” nel’s conditions are added to tEnhreichedTweets dataset’s dataset (a dataset which will be created by the system whenconditionsList and a BAD index calleTdweetsAboutCrimeBa channel is created), integrated into each channel’s planA,Dindex is created for the channel. Any new tweet that replacing the direct use of subscription parameters. This meets all three predicates in tThweeetsAboutCrime channel dataset includes fields for the channel’s parameter(s) and will be added to thTeweetsAboutCrimeBADindex. Algorithm the number of subscriptions interested in each. These fields 1 demonstrates the procedure for adding entries to BAD facilitate the dynamic addition or removal of parameters aisndexes for an active dataset. subscriber interests evolve. Unlike general-purpose indexes, which store data for all
The updated query advances the join operation be- records, the BAD index focuses solely on records that meet tween the UserParameters anEdnrichedTweets datasets the fixed predicates of a channel’s query. The BAD index ofers a major improvement over traditional indexes by filtering and retrieving only the data relevant to a specific query of all optimizations provides comprehensive benefits. It (we call this‘early result filtering ’), thus avoiding the inefi- should be highlighted that, previous research has already ciencies of indexing all records in a dataset. This selectiveestablished the superiority of the BAD platform over a tradiindexing reduces storage overhead and eliminates the need tional, pieced-together solution with similar functionalities to scan irrelevant data, resulting in faster query execution[.3]. Given this, we can assert that an optimized version of This indexing approach facilitates the application of timtehe BAD platform would also outperform other comparable iflters [ 31] to the BAD indexes, enabling eficient retrieval systems. Therefore, we have chosen not to repeat those of only the most recent tweets when the is_new function iesarlier experiments here in this study. present the query. The time filter employs the timestamp For the following experiments, we used the example apfrom the channel’s most recent execution to guarantee thaptlication that has been discussed throughout this paper, the index search includes only records that have arrived atincluding its data model, datasets, and data channels. For or since the time of the last execution. As a result, a BADall experiments we deployed the BAD platform on a 4-node index boosts channel execution by quickly locating datacluster. Each node has an Intel(R) Xeon(R) CPU E5-2603 v4 that meets the channel criteria, eliminating the need to sca@n1.70GHz processor, with 64 GB of RAM, 10 TB of HDD, the entire dataseEtnr(ichedTweets in the above example). and 2×6-core processors.
Algorithm 1 BAD Index Record Insertion
5.1. Data 1: Input Variables: We initialize our experiments by loading the BAD platform 2: ∶ the incoming record with an initiaEnlrichedTweets dataset that contains 2 mil3: ← .() ▷ Dataset in which the record is being lion synthetic tweets. This preloading was performed to 4: iena←scehr.tce(h)dan▷nel List of fixed conditions for iemnspuarcettthhaetptehrfeosrimzeanocfetEohnfrecichhaendnTweeleetxsecduattiaosne.tFodloloeswninogtthe 5: ℎ ← .ℎ () ▷ Names of system’s initiation, the BAD platform consistently receives channels with fixed predicates on the dataset 2000 EnrichedTweets per second, with eacEhnrichedTweet 6: ← . () ▷ List of BAD indexes for being approximately 30 KB in size. For each channel, we utithe dataset lized datasets containing 1 million subscribers. Detailed 7: Algorithm: descriptions of these subscriptions will be provided for 8: for each ← 1 to .() do ▷ Check if the record each experiment. We have also created a dataset called satisfies all conditions for chann el UserLocations which is being used in thTeweetsAboutCrime 9: if checkConditions( , [] ) then channel and includes people’s usernames and their loca10: ← .(ℎ []) tions; each such record is 38 bytes. We assume that this 11: . () ▷ Add the record to the dataset is continuously updated as the data is received as corresponding index subscribers change their device’s location. Only the most 12: function checkConditions( , ) recent location for each user is maintained in the system. 13: for each ← 1 to .() do In this study, synthetic tweets were used to assess sys14: if not satisfy( , [] ) then tem performance, allowing precise manipulation of field 15: return False attributes to evaluate their impact on channel and query op16: return True erations. By altering these parameters, we gained insights
Note that this pre-processing is reminiscent of, but difer- into system behavior under varying conditions, analyzing ent from, Partial Indexin2g1[], which focuses on indexing the eficiency and scalability of the indexing mechanism. only the most frequently accessed or queried segments of To demonstrate the applicability of our optimizations, exa dataset. A BAD index does not index any attributes ofperiments were also conducted with a real-world Twitter the dataset; instead it consolidates the primary keys of adllataset, detailed in Sect5io.6n. records that satisfy all the fixed predicates of a channel. This5.2. Subscription Aggregation Experiments enables quicker access to the relevant records as opposed Firstly, we examine the efectiveness of subscription groupto scanning the entire dataset or using a traditional indeinxg in theTweetsAboutDrugs channel (Figure6). We begin which includes entries for all records, not just the ones thatby examining the trade-of presented in Secti4o.1n.1 regardmeet the specific criteria of the channel. To ensure that ing the optimal size for subscription groups. We conducted each channel query efectively utilizes the BAD index, the an experiment with a specially created dataset of 1 milquery plan must be adjusted to replace full dataset scans orlion subscriptions, all of which are interested in “CA”. This the use of other secondary indexes with the BAD index. As setup allows us to examine the impact of large and small explained earlier, regular secondary indexes contain both subscription-group sizes on system performance. In this relevant and irrelevant records, making the BAD index a experiment the AsterixDB frame sifzeis set to 40 KB while more eficient choice. Additionally, since the fixed predi- each original subscription is around 40 bytes. cates are already processed when data enters the system, Initially, we consolidate all “CA” subscriptions into a they should be removed from the query plan to avoid re- single subscription-group (that is 1024 times bigger than dundant computations. the size of the frame), which we then systematically halve 5. Experimental Evaluation to evaluate smaller subgroup sizes. Execution times for these various configurations are depicted in Figu1r1e. Each We proceed with outlining a series of experiments designed execution time is the time involved in running the chanto assess the performance improvements of the optimizednel after receiving 10 minutes of incoming tweets (at 2K BAD platform as compared to its original configuration. tweets/sec). The left side (denoted as 102)4corresponds Further experiments are detailed i3n0][. We first evalu- to having a single subscription-group of 1M subscriptions, ate the performance of each individual optimization andwhile the right sid e/(1024) corresponds to splitting this subsequently examine how the collective implementation subscription-group into subgroups with 1 subscription each. As it is shown in the figure, the execution time decreases when we have more than a single group, benefiting from enhanced parallelism, but the performance begins to sufer when we create many smaller subgroups, due to increased computational demands. The channel achieves the shortest execution times when the size of each subgroup matches the frame size, efectively balancing computational load and parallelism eficiency. As a result, we see that larger subscription-groups should be split into smaller ones to match the frame size.
In this section, we undertake an experiment utilizing the MostThreateningTweets channel (Figure8) to demonstrate the impact of augmenting the query plan to integrate the UserParameters dataset. In particular we focus on varying the proportion of relevaEnntrichedTweets and examine the system’s responsiveness and eficiency while adjusting the percentage oEfnrichedTweets that align with subscribers’ Figure 13: The execution time of the channel TweetsAboutCrime preferences. We use three subscription datasets that areunder varying conditions. specifically designed to match a certain percentage of relevant tweets: in set 1 10% of the subscriptions are matching To ensure an equitable comparison, we assess the chantweets, in set 2 15%, and in set 3 20% of the subscriptions nel’s execution times using a traditional index crafted on the are matching tweets. attribute that is most selective under the given conditions.
As depicted in Figure12, re-configuring the query plan As an example, for the scenario with 2 conditions (I+II), this to address user interests early on significantly decreases index is based on threetweet_count and when having 4 conchannel execution times for all sets. This optimization isditions, the index will be based on tthhreeatening_rate as particularly crucial for set 3 due to the higher proportion ofthese are the most selective conditions respectively. Figure relevant data, which necessitates producing a larger number13 shows the channel execution time measured with a tradiof results. This need for eficiency is much higher when tional index as well as the BAD index. The creation of the running the original BAD plan. For instance, with the thirdBAD index significantly enhances the data retrieval capaset of subscriptions, the optimized plan is essential as itbilities. As illustrated in the figure, it reduces the execution allows the channel to process data eficiently and meet time- time across various channel selectivities. Notably, the benesensitive deadlines, capabilities that are not as achievablfites of the BAD index increase as the channels become more with the original query plan. selective, enabling more eficient filtering of unsatisfactory records prior to channel execution. 5.5. Comprehensive Performance of this figure, we compare the performance of the original
Optimized BAD BAD system with a traditional index on the most selecWe now proceed with an experiment that synthesizes all tive field, retweet_count, along with various optimizations. three distinct optimization strategies that were proposeAds depicted, each optimization reduces execution time for to enhance the BAD channel performance. It is importantboth channels. Notably, the BAD index ofers greater time to note that each optimization may be particularly bener-eduction when thPeortugueseTrendingTweetsInACountry ifcial for specific scenarios. For instance, the BAD index channel. This occurs because most tweets are in English, proves most advantageous in channels which have very making Portuguese queries more selective and leading to a selective fixed conditions, the subscription aggregation ex- significant diference in the number of records stored in a cels in channels with a limited range of possible parameter traditional index compared to the BAD index. It is imporvalues, while customizing the query plan is most efective tant to note that channels can be created with varying levels when the user parameters significantly restrict the datasetof complexity, from simple to highly sophisticated, dependthat needs to be processed. An important comparison met-ing on specific needs. Diferent channels may benefit from ric is the maximum number of subscriptions that the opti-particular optimizations based on their characteristics. Howmized channel can support; more supported subscriptions ever, the experiment in Figur1e6 demonstrates that even for reflects the system’s improved functionality to process and simpler channels likeEnglishTrendingTweetsInACountry deliver timely and accurate results to an expanded number and PortugueseTrendingTweetsInACountry, that use nonof subscribers within set deadlines (in our environment, the enriched tweets, the execution time is reduced by 62% and maximum number of subscribers that can be supported in 70%, highlighting the benefits of our optimizations. the 10 minutes between subsequent channel invocations). CREATE CONTINUOUS PUSH CHANNEL
Figure 14 showcases the enhanced capacity of three spe- EnglishTrendingTweetsInACountry(countryName) cific channels to manage subscriptions using diferent opti- PERIOD duration ("PT10M") { mizations; the original BAD, each optimization alone, and SFERLOEMCTTwet.ettesxtt the fully optimized BAD as shown. As illustrated in the WHERE t.country=countryName AND is_new(t) ifgure, implementing any combination of the proposed op- AND t.retweet_count>100000 AND t.lang="en" }; timizations, or all of them, increases the capacity of eachFigure 15: TrendingTweetsInACountry channel DDL. channel to support more subscriptions in the BAD platform.
To demonstrate the real-world applicability of our methods,ferent conditions. this section utilizes actual tweets collected from Tw32it].ter [ 6. Conclusions And Future Work According to the stud3y3[], the most common languages in the overall dataset of real tweets are English, JapaneseI,n this paper, we concentrated on enhancing scalability, Spanish, Arabic, and Portuguese. In the subset of the datasetperformance, and eficiency of a Big Active Data (BAD) platused for our experiments, English was the dominant lan- form. We discussed diferent example use cases where users guage, followed by Portuguese, with the other languages use BAD services to monitor a high speed incoming data being less represented. As a result, we focused on the two source like tweets. In order to reduce the execution time and most prevalent languages English and Portuguese. The first increase the supportable number of users, we introduced channel, calledEnglishTrendingTweetsInACountry, shown three diferent approaches including: (i) strategically consolin Figure15, sends subscribers trending tweets in English idating subscriptions, (ii) revising (augmenting) query plans, (those withretweet_counts greater than 100,000). Similarly, and (iii) implementing the BAD index (for early result filterthe second channePlo,rtugueseTrendingTweetsInACountry, ing). Our findings demonstrate a significant enhancement in targets trending tweets in Portuguese. system performance. In this paper, our optimization eforts
Users can subscribe to either channel based on their coun- have focused solely on individual channels. Looking ahead, try of interest, and every 10 minutes, they receive trending our future work will focus on strategies for optimizing multweets from that location. For this experiment, the tweettiple channels concurrently, which holds great potential. inflow rate was set to 6,000 tweets per second, which aligns For example, we will explore grouping channels and refinwith the average tweet generation rate reporte3d4]i.n [ ing the BAD index to synchronize indexing activities across Each tweet averages about 3.5 KB, including details like channels. user data, retweet counts, and location. In total, 1 milA-cknowledgements lion subscriptions were generated, with the distributionThis work was supported in part by NSF awards CNSproportional to each country’s population, meaning more 1925610, CNS-1924694, NIFA award 2024-67022-43695, an populous countries had a larger number of subscribers. Fig- industrial gift from Google, and funding from the Donald ure 16 shows the execution times for both channels. In Bren Foundation (via a Bren Chair at UC Irvine). [30] S. H. A. Shirazi, X. Wang, M. J. Carey, V. J. Tso- [33] T. M. Luu, S. Ram, F. Rangel, P. Rosso, F. Menczer, tras, Optimizing big active data management sys- On the persistence of linguistic inequality: English tems, arXiv preprint arXiv:2412.14519 (2024). URL: and the global balance of languages in wikipedia, https://arxiv.org/abs/2412.1451.9 EPJ Data Science 10 (2021). URLh:ttps://doi.org/10. [31] S. Alsubaiee, M. J. Carey, C. Li, LSM-based stor- 1140/epjds/s13688-021-00271-0. doi:10.1140/epjds/ age and indexing: An old idea with timely bene- s13688-021-00271-0. ifts, GeoRich’15, Association for Computing Ma- [34] X Corp. Engineering, New Tweets per chinery, New York, NY, USA, 2015. URL:https://doi. Second Record, and How, 2013. URL: org/10.1145/2786006.2786007. doi:10.1145/2786006. https://blog.x.com/engineering/en_us/a/2013/ 2786007. new-tweets-per-second-record-and-h,o[wAccessed: [32] Twitter, Twitter Platform, 2024. URhLt:tps://twitter. 2024-09-05].
com, [Accessed: 2024-09-05].