Everything you need to know about Spark Structured Streaming
From its architecture, event-time processing, stateful processing to how it achieves fault tolerance.
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
Intro
I wrote an article to discuss everything you need to know about Spark for batch processing. Spark is more than that; it is also a reliable stream processing engine. For this week's article, I plan to compare Spark Structured Streaming and Apache Flink. However, I realized that both have numerous aspects that require discussion, so I decided to write a dedicated article about Spark’s stream processing engine first.
That way, I can focus more on the research process and keep the article from being too long. We will first explore its architecture. From there, we move on to explore how they support event time processing, stateful processing, and fault tolerance.
Note: This article won’t talk much about stream processing concepts, such as windowing. I highly recommend reading my previous article, Batch and Stream Processing.
Architecture
Structured Streaming is a stream processing engine built on the Spark SQL engine. Its core design principle is to treat a continuous stream as a subset of bounded data. This approach enables Spark creators to leverage its robust batch engine for stream processing.
Spark cluster
That said, revisiting the way a batch processing application is handled in Spark before we move on is very helpful.
A Spark application consists of:
Driver: This JVM process manages the entire Spark application, from handling user input to distributing tasks to the executors.
Executors: These processes execute tasks that the driver assigns and report their status and results. Each Spark application has its own set of executors.
Anatomy
A Spark application is a cluster of driver and executors. Each application can have multiple jobs. A job represents a series of transformations applied to data. It encompasses the entire workflow from start to finish.
A job is defined by an action and is split into different stages when a transformation requires shuffling data across partitions. Each stage is divided into multiple tasks, which execute processing in parallel across various partitions.
Typical life cycle of an application
Users will define the processing logic for the application. It must include the SparkSession object, serving as the central gateway for interacting with all Spark’s functionalities.
The user then submits a Spark application to the cluster manager and requests the driver resource. When the cluster manager accepts this submission, it initiates the driver process. The driver then asks the cluster to launch executors.
Based on the user’s defined logic, the driver will form the execution plan and start scheduling tasks on the executors. Then, executors physically execute the tasks and send the status to the driver. The process continues until all tasks are processed.
Back to Spark Structured Streaming
When we start a streaming application in Spark Structured Streaming, we create a long-running Spark application. The Driver process stays active continuously, managing the entire streaming query lifecycle. Each stream will have a trigger. This trigger defines when Spark should check for new data. When the trigger fires, the Spark engine does the following:
It queries the source (e.g., asks Kafka “what are the latest offsets?”).
It identifies the new data that has arrived since the last batch (e.g., Kafka offsets 1001 to 5000).
This chunk of new data is conceptualized as a micro-batch. Internally, Spark treats this micro-batch as a small, static DataFrame.
Once Spark has defined the micro-batch as a DataFrame, it applies all the transformations you described in your code (select, withColumn, groupBy, filter, etc.). Spark will form job(s) for each batch. Please remember that transformations in Spark are lazy. Nothing actually executes until an action is called. In Structured Streaming, the action is writing to the sink (e.g., `.writeStream.format(”parquet”).start(”/path/to/sink”)`).
For each micro-batch:
The full query plan (transformations + sink action) is applied to the micro-batch DataFrame.
Logical and physical plans are created.
The driver schedules the tasks for executors based on the physical plan.
The process continues forever (in theory) as each micro-batch is retrieved incrementally. In a real-world application, each micro-batch can contain several files or a range of offsets from sources such as Kafka or Kinesis.
The trigger
To define the frequency of the trigger, the user can choose from the following types:
Default Trigger: No explicit trigger is specified. Spark runs the query as fast as possible. A new micro-batch is started immediately after the previous one completes if new data is available. This type removes the time spent waiting for a scheduled interval.
Fixed-Interval: The query attempts to execute, retrieve, and process the micro-batch at a fixed interval, regardless of when data arrives.
If a batch finishes faster than the interval, Spark waits for the remainder of the time.
If a batch takes longer than the interval, the next batch starts immediately upon the current batch’s completion and doesn’t wait for the next scheduler.
One-Time: The trigger processes a finite amount of data and then stops the streaming query, making them ideal for running streaming logic as a single, batch job.
Available now, micro-batch: It is similar to the one-time trigger; however, it can process the data in multiple batches, depending on the source.
The batch size
In most cases, the primary factor to consider when working with Apache Spark Structured Streaming is the batch size. The user can specify some options to act as a throttle, limiting the maximum amount of data consumed in a single micro-batch
maxOffsetsPerTrigger: It limits the maximum number of records (offsets) to be consumed in one micro-batch for sources like Apache Kafka, Kinesis.
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.