To test and evaluate a heterogeneous stream-processing system consisting of an FPGA-based systemon-chip and a host, we develop a benchmark called SKYSHARK. It uses real-world data from air-trafic control that is publicly available. These data are enhanced for the purpose of the benchmark without changing their characteristics. They are further enriched with aircraft and airport data. We define 14 queries with respect to the particular requirements of our system. They should be useful for other hardware-accelerated platforms as well. A first evaluation has been done using Apache Flink. We envision a great potential because of the flexibility of the approach.
This nourished the idea to use public flight data and utilize the queries that are already in use to analyze and visualize them. Hence, we designed a benchmark called SKYSHARK.
Benchmarks face a significant challenge in obtaining suficient and realistic data for their queries.
Existing benchmarks often rely on synthetic data generated through specialized tools, which may
not accurately represent real-world conditions. Access to authentic production data is limited
due to concerns over data privacy and trade secrets. SKYSHARK ofers a promising solution by
leveraging publicly available real-world data from the aviation domain. This domain provides a
wealth of data as nearly every aircraft continuously broadcasts its position, speed, altitude, and
more. It allows for thorough data collection and evaluation. The data is not encrypted and can
be received using software-defined radios or purpose-built receivers. Commercial providers
collect this type of data from oficial sources like the Federal Aviation Administration (FAA),
Eurocontrol and their own network of ADS-B receivers. Additionally, open-source projects like
OpenSky Network (OSN) [
Recognizing the opportunity presented by this data-rich environment, we created a new
benchmark called SKYSHARK to leverage the real-world data for SPSs. It aims to overcome the
limitations of synthetic data and to provide a more accurate representation of real-world
conditions. While the first implementation has been done in the context of a Master’s Thesis [
Many benchmarks have been proposed and implemented for stream processing. A very good
overview is given in [
The benchmark RIoTBench is much larger than the YSB [
StreamBench [
The approach that is closest to our ideas is DSPBench [
So while very good solutions are available already, we still felt a need to create another benchmark with the following characteristics: We insisted on using real data. Some extrapolations had to be done due to legal issues, but they do not change the characteristics. We have a larger set of queries, and they have been selected from existing applications w. r. t. line-rate processing on dedicated hardware. Queries are given in standard SQL (if possible) to define precise semantics. The benchmark is extensible: Other data can be downloaded, and new queries can be defined. (This may not be suitable for a benchmark, but definitely for testing specific systems.) Since the data are stored, a DMBS (Flight Schedule, Airports, Aircrafts) can also be evaluated. While all benchmarks so far are designed for homogeneous SPSs, either on a single node or distributed, our system is heterogeneous, consisting of one or more SoCs and a host.
It is our hope that other projects on FPGA-based stream processing can also benefit from our
benchmark. There are a number of them; we just cite [
As mentioned earlier, one of our goals is to use data as realistic as possible for our benchmark. Unfortunately, organizations are generally reluctant to share precise data due to concerns over potential data-privacy breaches or loss of trade secrets. In this regard, SKYSHARK emerges as a viable solution by leveraging publicly available real-world data, which can be collected by anyone. This section provides an overview of key concepts and terminology related to Air Trafic Control (ATC). ATC is vital for ensuring safe and eficient air travel.
3.1.1. Radar
Primary Surveillance Radar (PSR) uses radio signals to detect objects by analyzing the delay
and direction of the reflected signals [
Currently, there are several commercial flight-tracking services available, with Flightradar24 2
being the most popular of them. It ofers real-time tracking, access to flight schedules, historical
data, and detailed aircraft information. Other commercial networks like FlightAware3,
RadarBox4, and ADS-B Exchange5 provide similar functionalities. Alongside with the commercial
networks, the OpenSky Network (OSN) [
To facilitate easier access to ADS-B data for researchers, an abstraction called state has been introduced. A state represents an ADS-B message from an aircraft. The following attributes are particularly important for the benchmark: • icao24: Unique 24-bit ICAO identifier. Identifies the aircraft. • time_position: Unix timestamp (seconds) of the last position update. • latitude and longitude: Geographic coordinates of the aircraft’s position. • baro_altitude: Barometric altitude in meters. Can be null. • velocity: Speed of the aircraft, measured in meters per second.
• squawk: Special transponder code, used as ID for ATC or to indicate emergencies.
To facilitate the integration of real-time tracking data with relational (batch) data, we focus on two additional sources of data: a database containing information about all aircrafts, and a database containing information about all airports. It is important to note that the data we utilize for this purpose are publicly available and sourced from open source communities. Aircraft Database: The OSN maintains a comprehensive list of approx. 580,000 aircrafts belonging to 1,800 airlines, as of March 2023. It is important to note that due to the coverage limitations discussed, this list may not be complete. However, since our aircraft-tracking information is sourced from the same database, most states should have a corresponding aircraft entry in the database. The list is periodically exported as a CSV file, which is made available for download on the OSN dataset website.
Airport Database: Another valuable resource is OurAirports7, a community-maintained open-data website that manages a database of over 5,200 airports as of March 2023. This extensive collection encompasses airports of various sizes and types, ranging from major international airports to smaller seaplane or helicopter bases. The airport database can be downloaded in CSV format. It is worth noting that community-maintained databases may not always be up-to-date. However, this is acceptable for our purposes as we do not need absolute accuracy.
The reference dataset that we have created by collecting and processing data from the OSN extends over a two-week period. OSN provides various data-retrieval options, including a REST API, Java and Python packages, and an Impala Shell. Access to the REST API is available both anonymously and as a registered user, with diferent credit-based mechanisms in place to prevent abuse. Anonymous users have a daily budget of 400 credits, registered users have a budget of 4,000 credits, and users with their own ADS-B receiver enjoy a larger budget of 8,000 credits. Within these limitations, we have been able to collect a comprehensive dataset using the SKYSHARK downloader. It allows users to gather a specified amount of data. Depending on the available credits, the downloader calculates the optimal interval between two API requests to achieve evenly spaced states. If users have an account and a daily budget of 4,000 credits, they can make 1,000 API requests per day. Distributing these 1,000 requests evenly over 24 hours would result in an average of approximately one request sent to OSN every 87 seconds. The states are stored as CSV files. The downloader can then convert the CSV file to a desired ifle format, e. g., JSON.
Obtaining airline flight plans without significant financial investment is not feasible due to the high cost associated with commercial services. To overcome this issue, we create our own lfight schedules by querying individual flights using the REST API provided by OSN. We first retrieve the callsigns which consist of an airline abbreviation (e. g., DLH for Lufthansa) and a lfight number (e. g., 101). Private or military aircrafts may not have callsigns or their meaning is unknown. We then use the OSN REST API and the callsigns to retrieve flight information such as origin and destination airports. Not all flights with flight numbers have flights recorded in OSN. We determine flight takeof and landing by monitoring the on_ground field in each state. Changes from true to false indicate takeof, and vice versa indicate landing. By storing and rounding the timestamps of these transitions, we obtain a rough flight schedule that aligns with the recorded flights. It is important to note that this flight schedule is incomplete and not entirely accurate but serves the purpose of the benchmark.
In this chapter, we provide an overview of the general design of the SKYSHARK benchmark. To accomplish this, we will first examine the benchmarking tool. Following that, we will present the benchmark metrics and the 14 SKYSHARK queries. As examples, we will discuss our considerations and constraints in designing two of the queries.
The centerpiece of our benchmark is our custom-developed benchmarking tool. This tool allows to send the collected states to the SPS under test and to receive the resulting data. Additionally, it gathers various metrics that serve the purpose of comparing diferent systems and implementations with each other. In Fig. 1, a schematic representation of our benchmarking tool is shown, and how it can be integrated with an SPS. Our tool provides a set of welldefined interfaces for connecting with the SPS. These interfaces include TCP, UDP, as well as adapters for Apache Kafka3 and ZeroMQ4. When designing the benchmark, we decided not to measure the individual operators of the SPS but rather focus on the system as a whole. This decision was motivated by our intention to create a benchmark for systems that utilize modern hardware, where such operator-level measurements may not be feasible. Considering 3https://kafka.apache.org/ 4https://zeromq.org/ the expected data streams in the area of 10Gbps, measuring the individual operators would impose an additional burden on the system. Generating measurement data at the operator level, which would then need to be processed by our system, would introduce additional CPU, RAM, and potentially network overhead. This additional load would impact the performance of the SPS. The benchmarking tool loads the data from the drive and generates a data stream. Before JSON Report Result
Benchmarking Tool
Evaluation Aggregate Results Compare Timestamps
Stream Generator
Result Writer
TCP {"icao24": "iaa9321", "time_position" : 1687420023, .... sending to the SPS, a hash is computed using the icao24 and the time_position of the state. This key, along with a timestamp, is stored internally. Once the tuple reaches the benchmarking tool again, another hash is computed and stored with a timestamp. The result tuple is then saved to disk. In a process running concurrently with sending and receiving, the latencies of each tuple are calculated and stored for further analysis. Since some queries involve filtering tuples based on certain conditions, we cannot calculate the latency for all tuples. The calculation of latencies is only possible for queries that do not involve blocking operations, as this would disrupt the relationship between input and output. For these queries, only the throughput can be measured.
As mentioned in Chapter 2, there is a variety of metrics that can be used for measurement purposes. For SKYSHARK, we specifically focus on two metrics.
End-to-End Latency: End-to-end latency plays a crucial role in the deployment of SPSs. Depending on the application the latency can have immense impact on the usability of the result. Therefore, we have identified this metric as an important measurement for SKYSHARK. Throughput: Another significant metric in the field of stream processing is throughput. With our benchmarking tool, we can measure it with high precision, as we have full control over the incoming and outgoing tuples. Furthermore, using the aforementioned map where keys and timestamps are stored, we can measure the number of tuples that have been discarded during the process.
Query Focus Test query for implementations JSON decoding/encoding performance Arith. expressions/renaming/rearranging Filter with string equals Filter with Boolean expression (integer) Complex filter with arithmetic expressions Filter with string wildcard Complex projection, floating-point comparison Complex numeric calculations Join Aggregation, Time windows Aggregation, tuple count windows Join with complex calculations
Complex join, updating RDBMS table
In addition to the data and the benchmarking tool, queries play a particularly important role in SKYSHARK. We have closely aligned the design of the queries with real-world aviation and airspace surveillance challenges. A total of 14 queries has been identified. These queries progressively increase in complexity and in demand from the system being measured. We have chosen to code the queries in standard SQL or, if that was impossible, in the extension of SQL developed by Apache Calcite. This extension allows for the definition of stream queries, where no standard is available yet. The goal was to provide a precise and formal description to eliminate misinterpretation. The actual code of the queries can be diferent in a particular SPS. The table in Fig. 2 presents a list of all queries, their intended focus, and an indication of whether latencies can be measured for these queries. A detailed listing of all queries, including their code and intention, can be found on our website5. To further illustrate our design decisions we present two exemplary queries.
Query 9 of the SKYSHARK benchmark (refer to Fig. 3) exemplifies the trade-ofs we made in the query design process. One specific requirement from the ReProVide project is that only basic arithmetic operations can be computed on the FPGA. The purpose of this query is to iflter out aircrafts within a certain radius around the airport. Ideally, the Euclidean distance would be calculated, which requires the use of trigonometric functions. However, due to the FPGA’s limitations in performing these calculations, a decision was made to approximate the distance. The approximation method used assumes that the circumference of the Earth is the 1 −− a i r p o r t _ l a t = 5 0 . 0 3 6 5 2 1 2 −− a i r p o r t _ l o n g = 8 . 5 6 1 2 6 8 3 SELECT STREAM i c a o 2 4 , c a l l s i g n , t i m e _ p o s i t i o n , l a t i t u d e , l o n g i t u d e , b a r o _ a l t i t u d e , 4 ( ( a i r p o r t _ l a t − ( s t a t e _ l a t ) ) ∗ ( a i r p o r t _ l a t − ( s t a t e _ l a t ) ) ) 5 + ( ( a i r p o r t _ l o n g − ( s t a t e _ l o n g ) ) ∗ ( a i r p o r t _ l o n g − ( s t a t e _ l o n g ) ) ) as d i s t a n c e _ s q u a r e d 6 FROM ( SELECT i c a o 2 4 , c a l l s i g n , t i m e _ p o s i t i o n , l a t i t u d e , l o n g i t u d e , b a r o _ a l t i t u d e , 7 ( l a t i t u d e ∗ 1 1 1 . 1 3 9 ) AS s t a t e _ l a t , 8 ( ( 1 0 0 1 . 8 7 9 / 9 ) ∗ ( 0 . 0 1 2 9 2 8 1 ∗ l o n g i t u d e + 1 . 2 ) ∗ l o n g i t u d e ) AS s t a t e _ l o n g , 9 ( a i r p o r t _ l a t i t u d e ∗ 1 1 1 . 1 3 9 ) AS a i r p o r t _ l a t , 10 ( ( 1 0 0 1 . 8 7 9 / 9 ) ∗ ( 0 . 0 1 2 9 2 8 1 ∗ a i r p o r t _ l o n g + 1 . 2 ) ∗ a i r p o r t _ l o n g ) AS a i r p o r t _ l o n g 11 FROM s t a t e s ) as p o s i t i o n 12 WHERE { copy d i s t a n c e _ s q u a r e d from SELECT} < 6 2 5 same everywhere, allowing the cosine calculation to be replaced with a linear function. This approximation introduces a decrease in accuracy at the poles and near the equator. Nonetheless, for our specific purposes, this is acceptable as long as the selected airport is not located in these boundary areas. We believe that these types of calculations are better suited for ofloading to FPGAs or other modern hardware, which are capable of handling more complex computations eficiently.
The second query we take a closer look at is Query 13 (see Fig. 4, “Diversion Airports”). In the
event of an emergency, the pilot needs to know which airports are closest to the aircraft at all
times. These airports can be used for an emergency landing if necessary. In this query, we not
only have the complexity of distance calculation, but also involve joining each incoming tuple
with the airport relation. Additionally, the join condition poses a particularly complex aspect.
The topic of joining data on FPGAs is gaining popularity [
To test the functionality of our benchmarking tool, we have implemented several queries from the benchmark in Apache Flink. The measurements have been performed on a PC with an AMD Ryzen 9 5900X 12-core processor and 32GB DDR4 RAM. Throughout the testing, all queries were able to run at a throughput of 10Gbps. For Query 2, the average latency measured was 100ms, with occasional spikes reaching 180ms or more. Fig. 5 provides a visual representation of the latencies caused by Apache Flink. It is important to emphasize that these measured numbers are preliminary and require further refinement. A thorough evaluation of the feasibility of our benchmark is ongoing work. The hardware with the FPGA should be available within a few weeks.
With SKYSHARK, we have taken the first step in using real-time aircraft-tracking data as a
source for benchmarks. Focusing on SPSs, we were mainly inspired by our project ReProVide and
the FPGA used in it. Our next step is to extensively measure the prototype we developed as part
of ReProVide using our new benchmark. Additionally, we want to complete and measure our
reference implementation in Apache Flink as well, to compare our system with a homogeneous
SPS. Additional queries could be identified, e. g., using match-recognize as proposed in [
This work has been supported by the German Science Foundation (Deutsche Forschungsgemeinschaft, DFG) as a part of SPP 2037 with the grant no. ME 943/9-2. We thank the reviewers for their valuable hints! A description of the benchmark is available at https://skyshark.org.