This study investigates the performance of Continuous Processing mode in Apache Spark Structured Streaming, with a focus on its application in scenarios where low latency is a key requirement (such as traffic incident reporting or stock price prediction). Unlike the traditional micro-batching method, which processes data in intervals, the Continuous Processing mode allows for near-real-time data analysis by handling streams as they arrive. This approach significantly reduces latency, making it ideal for time-sensitive applications, but it does come with trade-offs in terms of throughput and fault tolerance. We conducted a comparative analysis of Continuous and micro-batching modes using various configurations and benchmarks, with a focus on latency and throughput metrics. Our findings indicate that while the Continuous Processing mode offers significantly lower latencies while using Rate source (2 ms instead of 528 ms in micro-batch mode), its performance in high-throughput scenarios using Kafka source may be less consistent (260 ms in contrast to 197 ms in micro-batch mode). The study also explores the practical implications of deploying Continuous Processing mode in real-world applications, assessing its compatibility with different data sources and sinks, predominantly Apache Kafka. These findings have practical implications for optimizing text data flow strategies in big data analytics, providing insights that can guide the selection of processing modes based on specific operational needs.
Apache Spark 2.3 marked a significant evolution in the capabilities of Spark Structured Streaming by introducing a new mode known as Continuous Processing. Traditional Spark streaming had relied predominantly on micro-batching techniques, which process data in discrete intervals, combining streaming-like throughput with batch processing's fault tolerance and
manageability benefits [
Continuous Processing in Apache Spark Structured Streaming represents a paradigm shift from the traditional micro-batching execution strategy. While micro-batching processes data in discrete chunks, Continuous Processing treats streams as proper continuous flows, reducing latency substantially. This is achieved by utilizing long-lived operators that maintain their state across data batches rather than resetting for each batch as in micro-batching. This approach minimizes the overhead caused by frequent stopping and starting task executions, which is typical in micro-batch processing.
The execution strategy of Continuous Processing is designed around the concept of pipelines akin to traditional database streaming models. These pipelines utilize long-lived tasks that continuously receive and process data as it arrives without the predefined intervals that characterize micro-batching. This model allows Continuous Processing to deliver lower latency by reducing the delays associated with batching.
As of the latest release of Apache Spark, version 3.5.1, Continuous Processing is still
labeled as experimental [
The limited support for diverse sources and sinks means that deploying Continuous Processing in a production environment requires careful consideration of the input and output interfaces. For organizations heavily using Kafka, Continuous Processing offers a promising opportunity to enhance the responsiveness of their streaming applications. However, for use cases involving other sources or more complex transactional needs, the experimental nature and the current limitations might pose significant challenges.
The ongoing development and incremental improvements in Continuous Processing suggest that future releases of Spark may expand its capabilities and integrations. Developers and organizations interested in cutting-edge streaming technologies should monitor updates to Spark’s Continuous Processing mode, as it will likely evolve significantly.
This paper analyzes these two processing modes within Apache Spark Structured Streaming, focusing on their performance implications across various setups and deployments. By examining several workloads, this study aims to delineate the conditions under which Continuous Processing significantly outperforms micro-batching and, conversely, when micro-batching remains preferable.
Armbrust et al. [
Other works primarily focus on using micro-batching mode [
In the paper [
The proposed methodology uses micro-batching and Continuous Processing to compare latency and throughput under different configurations. The benchmarking process uses a pipeline with either a rate source or a Kafka source as input, transforming the timestamp and value into a JSON string, and a Kafka sink as the output.
Key components of the method include input sources, which involve testing the rate source at 10 and 100 records per second (rps) and using a Kafka source that utilizes text data from a Ukrainian Wikipedia corpus. The output sink is configured with a Kafka sink, which maps the message’s timestamp into the payload as a “new_timestamp” field to measure each record's latency. Spark configuration tests are conducted in local mode with 3 and 16 workers, and each benchmark is executed for 2 minutes. Additionally, the number of partitions for Kafka topics (both source and sink) is adjusted to evaluate its impact on throughput and latency.
The proposed methodology aims to identify the optimal configuration for both microbatching and Continuous Processing modes by systematically varying the number of records per second, worker nodes, and Kafka partitions. This approach helps understand each processing mode's limitations and performance characteristics under different loads. The final configurations are presented in Table 1.
After running the initial benchmarking and analyzing the results, it was suggested that the default number of partitions in the Kafka topic might influence Spark's throughput and latency. It was decided to include parametrization of this number to exclude the possibility of Kafka being the bottleneck in the benchmarking process. The configurations for benchmarking different Kafka topic partition count are presented in Table 2.
To develop an understanding of at what rates continuous processing becomes unreliable, another set of benchmarks focused on how continuous and micro-batch processing handle different loads in test conditions, which could provide a controlled throughput rate, was decided to perform.
The resulting configurations for performing sustainable throughput benchmarks included comparing micro-batching and continuous modes, using rate source at 10 - 10 000 rows per second, 3 and 16 worker nodes, and writing into Kafka topics with 3 – 32 partitions, with the number of Spark partitions equal to Kafka’s.
The total number of configurations was 64, but some configuration executions were canceled during the benchmarking process to make readjustments, as clear patterns have emerged, rendering some of the unperformed configurations worthless. The list of configurations in the stage 2 benchmark is presented in Table 3.
After observing the results of sustainable throughput benchmarks, it was decided to expand the research by decoupling the Spark partition count from the Kafka partition count to see if further improvements could be made. The benchmarking configurations are provided in Table 4.
After setting up and running initial benchmarks on modern versions of Kafka and Spark (Kafka 3.7.0, released February 27, 2024; and Spark 3.5.1, released February 23, 2024), the results showed that continuous processing mode showed significantly lower latency when using the rate source while maintaining the same throughput. However, when using the Kafka source, continuous processing mode showed worse latency and throughput rates than micro-batch mode.
This surfaced a flaw in initial benchmarking configurations, as Kafka produced much higher throughput than expected, so rate source benchmarks should have been adjusted to match Kafka throughput rates.
Also, continuous processing mode's loss in latency to micro-batching indicated that continuous mode is not running optimally and suggested that high volumes of Kafka throughput may overflow the Spark application’s ability to process the data promptly, disrupting latencies. The initial benchmarking results are presented in Table 5. 55 496 81 386 104 553
Three additional configurations for Kafka source were performed to validate if the number of Kafka topic partitions impacted bad continuous processing performance, the results of which are shown in Table 6.
None of those configurations performed significantly better than the initial ones, proving that despite working with a possibly non-optimal number of Kafka topic partitions, the reason for bad continuous processing performance may be related to other issues, such as throughput overflow.
When analyzing the sustainable throughput benchmarking results of the micro-batching mode shown in Figure 1, benchmarks of 3-worker Spark are similar to 16-worker Spark, which infers that Structured Streaming micro-batch mode works well as long as number of Spark’s partitions matches the number of Kafka’s partitions. Also, there is a noticeable correlation between micro-batch mode latency, which gets slightly better with an increase in throughput but remains in the range from 510 to 570 ms.
On the other hand, results presented in Figure 2 clearly show that the continuous processing mode is much more sensitive to Spark’s configuration. Despite generally having sub-millisecond latency (except the configuration with 10 rps and 3 partitions, where the average latency reaches over 2 ms), its throughput suffers greatly from configurations where the number of partitions exceeds the number of workers.
As a result, it is important to analyze existing worker count and partition count configurations when transitioning from micro-batching mode to continuous processing, as continuous processing launches long-running tasks, which continuously work with a single partition. Therefore, not having enough tasks will leave some partitions unprocessed, which may lead to substantial data loss that may be mistaken as a throughput drop.
After observing the benchmark results depicted in Figures 1 and 2, it was decided to expand the research by decoupling the Spark partition count from the Kafka partition count to see if further improvements could be made. Results of micro-batching mode benchmarking are shown in Figure 3, and continuous mode in Figure 4.
Throughput and latencies did not significantly change, and for further benchmarks, 16 Spark partitions and 16 Kafka partitions were decided to be the most suitable configurations for both micro-batching and continuous modes. However, during the benchmark for continuous mode at a rate source set to 10 000 rps, the throughput resulted in 16 000 rps, and after investigating the issue, it was concluded that the rate source might be unreliable starting at 10 000 rps for continuous processing mode. Additional tests in the 1 001 – 9 999 rps range were not performed, so the reliability of the rate source at that throughput level is questionable.
The instability of the rate source for continuous processing was further confirmed by conducting a benchmark for 100 000 rps, where micro-batching mode showed 100 000 rps and continuous mode showed 1 063 758 (more than one million) rps, which exceeds the expected throughput by ten times. The investigation established that the issue does not arise from atleast-once delivery guarantees, as no duplicate record values were present in the Kafka sink, meaning each record was unique. Average latencies concluded 537 milliseconds at 100 000 records/s for micro-batch and 41 297 milliseconds at 1 063 758 records/s for continuous mode.
After doing comprehensive research on the rate source and tuning optimal parameters for continuous processing, the last benchmark was attempted to once again measure continuous mode latencies in a simulation close to a real-world use case. Because of bad continuous processing mode performance on high workload levels, the file source in Kafka Connect was replaced by a custom script that implements rate-limiting data read from a corpus on a row count basis. The benchmark was executed on 1 000, 10 000, and 100 000 rps, with partition count in both Kafka and Spark set to 16, with 16 workers. Benchmarking results are provided in Table 7.
The results have shown that given adjusted parameters for continuous processing mode and rate-limited throughput to 100 000 rps, the throughput got worse by around 20% while improving the mean latency by around 310%. Rate-limiting source to 10 000 rps produced a mean latency of 260 ms, the lowest latency achieved for a Kafka source in continuous mode in the scope of this research.
Spark Structured Streaming’s Continuous Processing mode prioritizes low-latency data processing, offering users a strategic choice between optimizing throughput and minimizing latency. The platform's use of a declarative API simplifies the construction of robust data pipelines and allows for relatively straightforward toggling between micro-batch and continuous modes. This flexibility is advantageous as it requires minimal code changes, making it accessible for users to adjust their processing strategies based on evolving data requirements or operational objectives.
The methodology that was proposed in the paper involved creating a simple pipeline supported by micro-batching and continuous processing modes in Apache Spark, using two data sources – Rate source and Kafka topic containing text data from Ukrainian Wikipedia – and varying sources throughput, number of Kafka partitions, and number of Spark workers in order to measure Spark’s throughput and latency.
The results have shown that transitioning between these modes is not merely a matter of code adaptation; it often necessitates tailored adjustments to the Spark configuration. This is due to the inherent architectural distinctions between how micro-batch and continuous processing modes manage data flows and system resources. Each mode is optimized for different aspects of streaming analytics, with micro-batch providing robustness and fault tolerance and continuous processing focusing on reducing processing time to the minimum. The lowest mean latencies achieved for continuous processing mode was 2 ms using rate source (in contrast to 528 ms in micro-batch mode) and 260 ms using Kafka source (in contrast to 197 ms in micro-batch mode).
Furthermore, continuous processing mode must be carefully managed to balance the inherent trade-offs between latency and throughput. In scenarios where the data inflow exceeds the system's processing capacity, there is a significant risk of increased latency, contradicting the primary goal of this mode. Therefore, continuous mode is not universally superior; its effectiveness is contingent upon the specific characteristics and demands of the workload. Conducting thorough performance evaluations, including stress testing under peak data loads, is essential to ensure that the system remains performant and that latency stays within acceptable bounds.
To fully leverage the potential of Continuous Processing mode in Spark Structured Streaming, developers and system architects need to fine-tune configurations and regularly monitor system performance. This proactive approach ensures that the streaming process remains efficient and aligns with their applications' latency and throughput requirements. Ultimately, the choice between micro-batch and continuous processing modes should be informed by a comprehensive understanding of the trade-offs involved and a strategic assessment of the application’s operational priorities.
Further research could be conducted to reach sub-millisecond latencies in the Kafka source instead of the rate source. The relation between Kafka topic message format (Avro, JSON, Protobuf, etc.) and Continuous Processing latency may be investigated. Also, as the benchmarks were conducted on one machine to eliminate possible latency overheads from the networking side, benchmarks could be tested on standalone clusters. Benchmarking pipelines with user-defined functions (UDFs) and MLlib also could be an interesting topic for extending research.