Data preparation involves transforming, cleaning, and converting raw data into a usable format for further analysis. This process can be time-consuming and resource-intensive. Typically, data to be analyzed is placed in two-dimensional tabular structures called DataFrames. These are the de facto standards for data science and machine learning tasks to store and process large amounts of structured data. Pandas is the most commonly used API for manipulating DataFrames in Python due to its popularity and comprehensive functionality. However, it is important to note that Pandas has some limitations, such as being single-core and non-distributed, which can impact its eficiency and performance for large datasets. Several libraries have been developed to expand the functionality of Pandas by utilizing multi-core and distributed computing capabilities. Therefore, the choice of library can significantly impact the eficiency and performance of the data preparation pipeline, and it is important to consider the specific requirements of the project when selecting a library. In this paper, I will discuss my primary contributions during the first months of my PhD program, which mainly focused on creating an open-source framework to evaluate the performance of seven Python libraries on five datasets of diferent sizes. The primary objective is to identify the best combination of these libraries to create eficient data preparation pipelines that deal with the needs of users who frequently encounter several libraries claiming to perform similar tasks. Finally, I will describe some ideas concerning the future directions of this research.
Data preparation is an important process that involves exploring, combining, cleaning, and
transforming raw data into curated datasets that can be used for various purposes [
One of the most common data structures used for storing and manipulating data in the data
preparation process is the DataFrame. A DataFrame is a two-dimensional data structure that
consists of rows and columns [
Despite its many advantages, Pandas has significant limitations when dealing with Big Data. The main drawbacks of Pandas are: • Memory: Since Pandas loads the full dataset into memory, the size of the dataset that can be analyzed is constrained by the memory capacity of the system. • Speed: Pandas is a single-threaded library by design, so working with large datasets might make it slow. Certain tasks, including grouping and sorting, can take a very long time to complete. • Scalability: Pandas cannot readily grow over diferent nodes or a cluster since they are not optimized to fully exploit distributed or parallel systems.
Over the years, in response to the need to process large amounts of data eficiently, several
libraries have been developed to overcome this limitation. There has been limited research
conducted in the field of optimizing data preparation pipelines in DataFrame systems. Previous
studies have mainly focused on evaluating diferent libraries on data processing tasks. First,
Petersohn et al. [
DFBen focuses on investigating various crucial aspects of these pipelines, enabling researchers and practitioners to make informed decisions. Firstly, the investigation of lazy evaluation and its impact on overall performance is essential, as it ofers advantages such as reduced memory consumption and improved processing eficiency. Secondly, scalability is a critical factor, considering the exponential growth of data volumes. Assessing the libraries’ ability to scale eficiently and utilize computational resources is crucial for real-world implementation. Moreover, the choice of I/O formats in data preparation plays a significant role, as diferent formats have distinct characteristics related to storage eficiency, data compression, and compatibility with downstream processing. Lastly, the setup and configuration requirements of each library significantly influence the ease of adoption and integration into existing pipelines.
In Section 3, I present future research activities, including transitioning to a FaaS (Function-asa-Service) framework and utilizing machine learning models to determine the ideal combination of libraries based on input dataset features. These innovations promise to enhance the eficiency and accuracy of data preparation pipelines.
DFBen is an open-source project that aims to compare various commercial libraries, such as Pandas1, Dask2, Spark3, Modin4, Polars5, Vaex6, and Rapids7, that utilize DataFrames for data preparation tasks. In order to determine how much the libraries have improved on the standard set by Pandas, this project conducts a comparison of the most commonly used functions grouped into four stages of data preparation. To thoroughly assess the libraries’ potential, tests were conducted on a number of distinct datasets with diferent sizes, schemas, and types.
To identify the most efective libraries for constructing data preparation pipelines, I evaluate their performance in four key areas. First, I assess the performance of individual functions used for data preparation. Next, I test libraries in the four key stages of the data preparation pipeline: Input/Output (I/O), Exploratory Data Analysis (EDA), Data Transformation, and Data Cleaning. By looking at the performance of the libraries in each of these areas, I can identify the best ones for each stage of the pipeline. Lastly, I analyze how well libraries perform across the full data preparation pipeline. In addition, I investigate the impact of scalability on the performance of these libraries. This enables me to assess their efectiveness in handling large datasets and complex pipelines.
The project’s open-source nature enables seamless incorporation of additional libraries and preparators through a Library Class that encapsulates the library with the implemented preparators, as well as a Dataset Class that allows for the inclusion of additional datasets in the configuration file. Moreover, the pipelines are managed using a JSON file, enabling the easy addition and modification of existing pipelines. In its current state, DFBen is a helpful tool for anyone looking to compare the many DataFrame librariesavailable and select the one that best ifts their needs and hardware capabilities in terms of performance and scalability.
Data preparation involves multiple stages and is typically performed using preparators, which
are responsible for small-scale preparation steps. In line with Naumann et al.’s (2020) Data
Preparation: A Survey of Commercial Tools [
To ensure optimal pipeline design, I select one of the top three Kaggle8 most voted notebooks, specifically tailored to the pre-selected dataset. This approach leverages the collective expertise of the data science community and ensures that the pipeline design is based on best practices 1https://pandas.pydata.org/. 2https://www.dask.org/. 3https://spark.apache.org/. 4https://modin.readthedocs.io/en/stable/. 5https://pandas.pydata.org/. 6https://vaex.io/docs/. 7https://rapids.ai/. 8https://www.kaggle.com/. and state-of-the-art techniques. To provide an evaluation with diferent levels of granularity, as described in Section 2.3, I organized the preparators into four key stages: • I/O: this stage includes the functions that handle the input and output of data in various formats, such as databases, CSV or JSON files and web APIs; • EDA: here, the functions are grouped that deal with the exploratory analysis of data characteristics for better understanding and detecting any error or anomaly; • Data Transformation: at this stage, functions deal with transformation, such as data normalization, deduplication, categorical data encoding, data aggregation, and other techniques that make the data more suitable for analysis; • Data Cleaning: the final stage involves data cleaning, which may include handling missing or erroneous data, correcting outliers, eliminating outliers, and other techniques that ensure data quality and reliability of analysis results.
The datasets selection for this study, as described in [
As described in Section 2.1, I used 50 preparators for my study. To implement these preparators, I relied on functions defined in the libraries as well as custom functions designed to cover all possible cases. To ensure optimal isolation, repeatability, and eficient management of machine configurations, the code is executed within a Docker 9 container for each library.
The tests are run on 3 machine configurations representing 3 diferent types of use: PC (8 cores-16GB RAM), Workstation (16 cores-64GB RAM) and Server (24 cores-180 GB RAM). In all configurations, the Tesla T4 GPU with 16GB RAM used by Rapids library is included. In this way, users can test the framework in diferent contexts and verify its performance on machines with diferent hardware configurations. Finally, the management of DFBen’s configurations is entrusted to JSON files that can be easily configured by users, who can customize the framework’s settings and the functions to be executed according to their own needs and preferences. 103 ()se m i T e g a r e v A102 load_dataset fil_nan sort
edit cast_columns_types
Method
Pipeline Step Input
EDA data_cleaning
output data_transformation
Step Pipeline Full
Library dask vaex polars pandas pandas20 modin_dask modin_ray spark rapids dask vaex polars pandas
Frpaamndeaws2o0rk modin_dask modin_ray spark rapids
For evaluating the performance, I track the execution time in diferent cases [
Lazy evaluation can lead to more eficient use of resources as unnecessary computations are avoided. When comparing libraries like Pandas, which do not use lazy evaluation, with those that do, it is necessary to calculate the execution time accurately in order to have a fair and objective comparison. In order to do that, the evaluation framework consists of three types of tests: • Core, is responsible for forcing the execution of lazy libraries and evaluating the performance of individual operations. This allows an accurate assessment of the eficiency of each library’s function and the identification of any performance bottlenecks. • Pipeline-Step, executes a data preparation pipeline and calculates the execution time at the conclusion of each stage. This method shows how each library’s function performs at each stage of the pipeline and shows any ineficiencies or room for improvement. • Pipeline-Full, component runs the entire data preparation pipeline. This approach provides a comprehensive evaluation of each library’s performance and allows comparison of the overall eficiency of completing the entire data preparation process.
Figure 1 displays the initial result of the three types of evaluation explained above, which have been obtained by running the tool on the Loan Dataset using a machine configuration of 24 cores and 180 GB of RAM. However, it is important to note that these results are only in the preliminary stage, and the analysis of the data is still ongoing. Further explorations will be needed to provide more detailed information, considering factors beyond eficiency. The efectiveness of the libraries involves various aspects that need to be evaluated. Therefore, it will be crucial to consider the specific requirements of the dataset and employ a refined selection process to identify the libraries that best meet the desired criteria.
During the first few months of my PhD project, I mainly focused on creating an open-source
framework that enables the comparison of seven libraries for the construction of data preparation
pipelines. DFBen provides an opportunity for future research directions. In the next year, I plan
to work on two main research activities:
1. The first one can be enabled by the tool with the automatic selection of the most efective
combination of libraries for constructing pipelines. To address this, the tool will gather
information about the characteristics of the dataset and the required pipeline and, by
taking into account the execution times of individual functions, will be able to suggest
the proper library. This would remove the dificulty of manually selecting the appropriate
one for a given task.
2. The second research activity can be the transition of DFBen to a FaaS framework. One
reason for this is that implementing a data preparation pipeline requires iterating the
steps described in Section 2.1 several times. When resources are limited, a small sample
of the reference dataset may sufice for testing, but this may not always be representative,
and it may not show certain record types that would lead to errors. Frameworks such as
Spark currently dominate this context [
To achieve these objectives, I want to study and test various methods and approaches for creating data transformation pipelines based on FaaS. Additionally, I will explore novel approaches for incorporating machine learning to construct pipelines that combine libraries, thus enhancing their efectiveness and eficiency. By doing so, I believe that this research will make a valuable contribution to the field of data preparation pipelines and help advance the field of data science as a whole. I wish to express my gratitude to my tutor, Prof. Sonia Bergamaschi and my co-tutor, Giovanni Simonini for their guidance and support throughout my PhD journey, as well as to my colleagues at the DBGroup who have been a constant source of encouragement. In addition, I would like to express my sincere appreciation to Leonardo for funding my research grant.