Data confidentiality is a critical issue in the digital age, impacting interactions between users and public services and between scientific computing organizations and Cloud and HPC providers. Performance in parallel computing is essential, yet techniques for establishing Trusted Execution Environments (TEEs) to ensure privacy in remote environments often negatively impact execution time. This paper aims to analyze the performance of a parallel bioinformatics workload for DNA alignment (Bowtie2) executed within the confidential enclaves of Intel SGX processors. The results provide encouraging insights regarding the feasibility of using SGX-based TEEs for parallel computing on large datasets. The findings indicate that, under conditions of high parallelization and with twice as many threads, workloads executed within SGX enclaves perform, on average, 15% faster than non-confidential execution. This empirical demonstration supports the potential of SGX-based TEEs to efectively balance the need for privacy with the demands of high-performance computing.
In recent years, the awareness of the need for privacy has gained significant prominence. In the digital age, where information is predominantly stored and transmitted electronically, concerns regarding the protection of sensitive data have become increasingly prevalent. This confidential information can be extracted and reused without the knowledge or consent of the data owner, posing severe privacy risks. This issue is not confined to the interaction between individuals and digital services; It extends across various fields of scientific computing where data confidentiality is indispensable. Notable examples include bioinformatics, which processes DNA and genomic data; medical research that handles patient health records; epidemiology, particularly highlighted during the recent COVID-19 pandemic; and social sciences that address sensitive topics such as mental health, income levels, and political polarization. Economic considerations also drive the imperative to safeguard sensitive information. For instance, in economics, processing financial data for trading purposes necessitates stringent privacy measures. Similarly, in chemoinformatics, the discovery of drugs and molecular simulations, which possess significant commercial value, require robust data protection to prevent unauthorized access and exploitation.
For these reasons, it is imperative to adopt techniques that protect sensitive data at all stages. In scientific computing, private organizations often lack the computational power to perform their calculations. The simplest and most commonly used solution is outsourcing computation to a remote location by renting the necessary hardware resources. A typical example of this is cloud computing, where resources are allocated on demand, and an ecosystem exists to facilitate the execution of workloads seamlessly. Data protection is typically considered in two primary contexts: at rest (in storage) and in transit (during transmission over the network). However, it is less common to consider the vulnerability of data during computation. Once a program starts executing on a remote machine, such as in cloud computing, there is often no control or protection over the data in the main memory. Confidential computing addresses this issue using trusted hardware to ensure data protection during execution. This approach breaks the chain of trust between the user and the external provider by introducing an additional entity in the trust process, the hardware manufacturer. This indirection step helps safeguard data while it is being processed, enhancing overall data security in outsourced computational environments. Figure 1 illustrates the entities involved and their relationships when a general user utilizes a provider’s remote resources. Without implementing confidential computing, the user transfers the computation to the provider. Even if the sensitive data is encrypted during transmission and on storage, it becomes vulnerable once it is decrypted for execution in the main memory. This exposure occurs because the data is no longer encrypted during processing, making it susceptible to risks in a multitenant environment, where potentially malicious workloads from other users may exist or if the provider is compromised or has malicious intent. In such scenarios, the user has no options; she has to unquestionably trust the provider, which is inherently untrusted. Confidential computing changes this dynamic by breaking the direct trust relationship between the user and the provider. Trusted hardware components, designed by the hardware manufacturer (e.g., CPU or GPU), incorporate specific features that ensure the confidentiality and integrity of the user’s program during execution. This enables the user to establish an indirect trust relationship with the provider. Instead of trusting the provider directly, the user trusts the hardware manufacturer, which in turn supplies trusted components to the provider. This approach ensures that the user’s data remains secure while being processed on the provider’s infrastructure.
The purpose of this paper is to conduct a performance analysis on the use of Intel SGX processors as trusted hardware. The study is performed on an application called Bowtie2, which is a bioinformatics software. Section 2 explains all the necessary background: what Intel SGX CPUs are and how they can be exploited with Gramine and Occlum to facilitate their use. Furthermore, some reasons are given for the choice of Bowtie2 as workload to assess performance. Section 3 discusses related works considering other SGX frameworks besides Gramine and Occlum. In addition, an overview of previous SGX performance studies in the High-Performance Computing (HPC) domain is provided. In Section 4, the configurations implemented to execute Bowtie2 in native and within SGX enclaves are explained. In Section 5, the results of the previously configured environment are illustrated, and finally, in Section 6, conclusions and possible future works are presented.
Intel Software Guard Extensions (SGX) [
allowing them to run entirely within an enclave without significant rewriting. Several frameworks support this method, including Gramine and Occlum Library Operating System (LibOS), which facilitate the execution of legacy applications within enclaves.
Intel SGX has evolved, and the community recognizes two main versions: SGXv1 and SGXv2. These
versions difer primarily in eficiency improvements and enclave size capacities, with SGXv2 supporting
enclaves up to 512GB (against 128MB of SGXv1) and introducing Enclave Dynamic Memory Management
(EDMM) [
Gramine [
One of Gramine’s most notable properties is its support for multiprocessing and related system calls, such as fork, vfork, clone, and execve. This support allows multiprocessing to be handled transparently, much like in non-SGX environments. For example, when a fork occurs, a second enclave is created, and the content is copied using message passing. Before this, a local attestation procedure is conducted between the enclaves, establishing a TLS secure channel for future communications. This method of handling multiprocessing is known as Enclave-Isolated Processes (EIP) (Figure 2a), where each enclave contains an instance of LibOS.
The EIP approach is inherently expensive in terms of execution time. Creating a process within an enclave is costly, and inter-enclave communication requires exchanging encrypted messages over a secure TLS channel. However, despite these disadvantages, the EIP method has significant advantages. The primary purpose of a LibOS with SGX integration is to facilitate the transition of workloads from an unsafe environment to an enclave. By supporting system calls like fork and adopting EIP for multiprocessing, Gramine allows applications that use multiple processes to be deployed quickly, with no additional efort than single-process applications. This ease of deployment is crucial for transitioning existing applications to secure SGX environments.
Occlum [
Bowtie2 1 ([
Sensitive Data Analysis DNA sequence analysis involves highly susceptible data that must be protected, especially in remote environments like cloud providers. Using Bowtie2 helps assess the efectiveness of SGX in safeguarding this data.
Multithreading Performance Bowtie2’s performance can be tuned through multithreading. While using multiple threads typically enhances performance, evaluating this in the context of SGX threads is particularly insightful, as the benefits may not be as straightforward due to the additional overhead and security constraints imposed by SGX.
Besides Gramine and Occlum, there are other technologies whose purpose is to make it easy to run
existing applications inside SGX enclaves:
• Haven [
Performance represents a significant concern in the realm of confidential computing. Although the goal is to achieve privacy, it is crucial not to compromise the execution time in chasing it. The study [18] conducted a performance evaluation using HPC benchmarks within SGX enclaves. The work included a comparison of performance between Gramine and Occlum, even if this comparison is inherently limited due to Occlum’s lack of support for multiprocessing, which is particularly relevant in HPC contexts. To address this limitation, our work focuses on evaluating a single real-world multithreaded workload rather than synthetic benchmarks. This approach ensures a fair comparison between Gramine and Occlum, providing valuable insights into their performance.
Another performance study [19] compares Intel SGX and AMD Secure Encrypted Virtualization (SEV) based-TEEs. Specifically, SCONE is employed to execute on SGX. HPC benchmarks have been used, encompassing traditional scientific computing, machine learning tasks, and graph analytics.
In our further recent work [20], the reference workload focused on the initial two steps of the Next Generation Sequencing (NGS) variant calling pipeline, which has been fully migrated to a cloud-based HPC environment [21]. Specifically, one of these steps involves the execution of Bowtie2 using Gramine.
This section outlines the setup of execution environments for the Bowtie2 DNA alignment bioinformatics workload. The configurations were designed to ensure fairness across diferent LibOSes environments (Gramine and Occlum). Only crucial aspects of the configuration files are presented for each setup. Both LibOSes were established using Dockerfiles, created based on the existing Docker images provided by the respective maintainers. A public GitHub repository2 was established to provide insight into the configurations implemented for running in various environments. However, due to confidentiality concerns, it was not possible to publish the DNA reads input data.
To use Bowtie2 on a native system, it is possible to easily utilize package managers such as Bioconda3, which provides a distribution of bioinformatics software as a channel for the versatile Conda4 package manager. However, in this study, the executables were built directly from the downloaded sources to facilitate fair comparisons between all execution environments (bare-metal, Gramine, and Occlum). In order to run Bowtie2, it is necessary to specify the basename of the index for the reference genome and the two files containing the paired-end reads (short DNA sequences). An example command for performing the alignment against the human hg38 genome is: b o w t i e 2 −S " o u t . sam " −x " H o m o _ s a p i e n s _ a s s e m b l y 3 8 " \ −1 " s a m p l e . r _ 1 _ v a l _ 1 . f q . g z " −2 " s a m p l e . r _ 2 _ v a l _ 2 . f q . g z " \ −p n u m _ o f _ t h r e a d s
In this command, the -x option is used to specify the reference genome. The -S option designates the output file in .sam (Sequence Alignment/Map) format, and the -1 and -2 options are for the compressed paired-end reads in .fq (FASTQ) format. The -p option specifies the number of parallel threads to be used for searching; each thread runs on a diferent core, enabling all threads to find alignments in parallel.
A manifest must be compiled to run an unmodified Linux binary inside an SGX enclave using Gramine. This manifest contains all the configuration information about the LibOS and the SGX enclave. In the Gramine toolchain, the gramine-manifest executable processes a manifest template, which can include Jinja5 syntax for customization. Using this template simplifies the creation of the manifest and allows for more flexible configuration. To streamline the process of creating the manifest required to run Bowtie2 (bow.manifest), a Makefile was written that also includes the recipe below: 2https://github.com/lorenzobrescia/performance-SGX-Bowtie2 3https://bioconda.github.io 4https://docs.conda.io/en/latest/ 5https://jinja.palletsprojects.com bow . m a n i f e s t : m a n i f e s t . t e m p l a t e
gramine − m a n i f e s t − D t h r e a d s = n u m _ o f _ t h r e a d s $ < >$@
As can it be observed from the previous recipe, a manifest.template must be prepared in order to generate bow.manifest. In the template file, all the arguments needed for execution are passed as environment variables in the following Gramine option: l o a d e r . a r g v = [ " / b o w t i e 2 − a l i g n − s " , " − S " , " / o u t . sam " , " − x " , " / H o m o _ s a p i e n s _ a s s e m b l y 3 8 " , " − 1 " , " / s a m p l e . r _ 1 . f q . gz " , " − 2 " , " / s a m p l e . r _ 2 . f q . gz " , " − p " , " { { t h r e a d s } } " ]
The options specified in the manifest.template are self-explanatory in relation to the bare-metal execution of Bowtie2. It is important to note that the bowtie2-align-s binary is run directly, rather than Bowtie2 itself. The latter is a Perl wrapper that selects the appropriate aligner to use. The wrapper is bypassed to simplify the process and ensure a smoother comparison with Occlum. For this reason, the bowtie2-align-s binary is executed directly. Consequently, bowtie2-align-s is set as the LibOS entry point in the manifest.template, meaning it is the code executed immediately after the enclave is ready: l i b o s . e n t r y p o i n t = " / b o w t i e 2 − a l i g n − s "
For handling the EDMM feature, Jinja syntax was used, still within manifest.template. If the environment variable edmm is set to 1, the feature is enabled; otherwise, it is not. This configuration also allows specifying the size of the enclave and the number of threads available inside the enclave. The semantics of these configurations difer depending on whether the EDMM function is enabled. With EDMM enabled, sgx.enclave_size refers to the maximum size the enclave can reach, and sgx.max_threads represents the number of TCS EPCs allocated before execution. If more threads are required during execution, additional TCS pages will be created on demand. If EDMM is disabled, the options are straightforward: sgx.enclave_size sets the fixed size of the enclave, and sgx.max_threads specifies the total number of threads that can be used, both set at the time of enclave creation. The following snippet implements what has just been described: {% i f env . g e t ( ‘ edmm ’ , 0 ) == ‘ 1 ’ %} s g x . edmm_enable = t r u e s g x . e n c l a v e _ s i z e = " m a x _ e n c l a v e _ s i z e " s g x . m a x _ t h r e a d s = n u m b e r _ o f _ p r e a l l o c a t e d _ t h r e a d s {% e l s e %} s g x . edmm_enable = f a l s e s g x . e n c l a v e _ s i z e = " e n c l a v e _ s i z e " s g x . m a x _ t h r e a d s = m a x _ n u m b e r _ o f _ t h r e a d s {% e n d i f %}
Once the bow.manifest is obtained from the Makefile, the SGX manifest ( bow.manifest.sgx) is also created using the Gramine toolchain, and finally, the application is run simply with the command: gramine − s g x bow
To launch a Linux executable inside Occlum, it is necessary to create a workspace that includes the LibOS image that will host the executable inside the enclave. Occlum provides a comprehensive toolchain to facilitate the deployment of this instance. First, the workspace is created using the occlum init command. Subsequently, the file system inside the LibOS must be configured. This configuration is achieved using the copy_bom tool, where an input file bow.yaml specifies that the bowtie2-align-s executable is to be mounted inside the /bin folder. This process ensures the executable is correctly placed within the LibOS image for execution inside the SGX enclave. To achieve what has just been described, the file bow.yaml must contain the following configuration: t a r g e t s : − t a r g e t : / b i n copy : − f i l e s :
− b o w t i e 2 − a l i g n − s } } Next, it is necessary to configure the Occlum.json file, which describes all the characteristics of the SGX enclave. This configuration includes essential information. In cases where EDMM is not active, it is possible to specify the enclave size and the maximum number of threads in this way: " r e s o u r c e _ l i m i t s " : { " u s e r _ s p a c e _ s i z e " : " e n c l a v e _ s i z e " , " m a x _ n u m _ o f _ t h r e a d s " : n u m _ m a x _ o f _ t h r e a d s Instead, the following options should be additionally specified to configure EDMM: " r e s o u r c e _ l i m i t s " : { . . .
" u s e r _ s p a c e _ m a x _ s i z e " : " e n c l a v e _ m a x _ s i z e " , " i n i t _ n u m _ o f _ t h r e a d s " : n u m _ o f _ p r e a l l o c a t e d _ t h r e a d s
Thus, a single Occlum.json file can turn EDMM features on or of. Consequently, two diferent .json configuration files were created to delineate the desired features for the experiments. The occlum build command is used to construct the Occlum SGX enclave and generate its associated file system image according to the specifications in the Occlum.json configuration file. Finally, to run Bowtie2, the following command must be executed, specifying all the necessary options: o c c l u m r u n b o w t i e 2 − a l i g n − s −x " / H o m o _ s a p i e n s _ a s s e m b l y 3 8 " \ −1 " s a m p l e . r _ 1 _ v a l _ 1 . f q . g z " −2 " s a m p l e . r _ 2 _ v a l _ 2 . f q . g z " \ −S " o u t . sam " −p n u m _ o f _ t h r e a d s
To assess the performance of Bowtie2 across various environments, we utilized the configurations detailed in Table 1. The experimental setup involved a machine powered by an Intel Xeon Gold 6346 CPU operating at 3.10 GHz, with an available memory capacity of approximately 400 GB RAM.
Figure 3 illustrates the execution times of Bowtie2 under various configurations for both small and large input sizes. Each experiment was performed 10 times. Since no significant variance or outliers were observed, the mean value was considered representative of the configurations. A small input size refers to aligning approximately 10, 000 reads, while a large input size involves aligning about 3 million reads. As shown in Figure 3a, native execution completes rapidly within seconds for small workloads. However, both LibOSes exhibit poor performance in this scenario, although, as can be noticed, Occlum outperforms Gramine. Furthermore, there is a lack of scalability: increasing the number of threads does not significantly enhance execution time, even in the bare-metal configuration. Enabling EDMM generally leads to a stable and acceptable increase in execution times across most cases, except when Bowtie2 necessitates 32 threads on Gramine. The excessive overhead observed may result from the dynamic management of the TCS enclave pages. As indicated in Table 1, Gramine’s EDMM configuration preallocates 32 threads. Although Bowtie2 operates with exactly 32 threads, Gramine requires at least three additional threads for managing inter-process communication (IPC), asynchronous tasks, and secure TLS communication within the LibOS, and the overhead likely arises from the efort needed to allocate these supplementary threads. These findings discourage the adoption of SGX technologies due to the unacceptable overhead compared to the native case and the absence of scalability. However, it is worth noting that scalability is also lacking in the native case. Consequently, the experiment was repeated with the same configurations detailed in Table 1 but applied to a much larger number of sequences, and the results are depicted in Figure 3b. Some patterns evident in the small input size scenario are also observed here. For instance, in the bare-metal environment, execution times are significantly faster compared to those in the LibOSes, and Gramine’s dynamic thread management severely impacts performance when Bowtie2 uses 32 threads. However, unlike the small input size case, the plot indicates some scalability. All configurations exhibit good scaling, with Occlum performing slightly better than Gramine again.
The critical consideration is justifying using trusted hardware techniques such as Intel SGX. Although SGX provides privacy guarantees, it also significantly increases execution times. In the case illustrated in Figure 3a, the technology appears unfeasible due to the uneven trade-of between overhead and privacy. Conversely, 3b suggests that if the application is parallelizable, good scaling can be achieved even with SGX computations as the number of threads increases. In detail, empirical evidence indicates that running Bowtie2 on bare metal and then re-running the same workload on SGX with twice as many threads often increases performance. This efect is further highlighted in Figure 4, which presents scalability comparison plots. Figure 4a demonstrates the performance gains of bare-metal execution when the number of threads is doubled. Figures 4b and 4c provide comparisons between bare-metal and Gramine, and between bare-metal and Occlum, respectively, under the same conditions. As observed, using SGX doubling threads often results in a performance gain compared to the native case. In some instances, the gain can be substantial; for instance, Occlum shows a 38.96% performance increase when using two threads compared to single-thread in the bare-metal setup. Nevertheless, performance gains are not always achievable, particularly when approaching the scalability limits of the problem. For example, in this bioinformatics workload, the performance gain from 16 to 32 threads is marginal, even in the native case, yielding just a 67% improvement compared to the average 95% increase. Specifically, when comparing 16 native threads to 32 threads in Gramine, there is a performance decrease of 46%, while Occlum shows a decrease of 23% in the same conditions. However, excluding the latter case, SGX with twice as many threads not only eliminates the overhead compared to non-confidential native execution but also achieves, on average, a 15% performance gain.
A final consideration that emerges from the experiments is that Occlum generally outperformed Gramine in terms of execution time and scalability. However, it is essential to note that Gramine supports multiprocess applications, unlike Occlum, which makes Gramine particularly attractive for the portability of legacy workloads. A similar argument applies to the EDMM feature. Although, on average, EDMM increases execution time, it simplifies the configuration of LibOSes by eliminating the need to estimate the memory footprint, thus facilitating the portability of existing applications.
This study provides a foundational analysis of Intel SGX’s performance for parallel executions. Introductory empirical observations obtained in our study ofer essential insights into the feasibility of employing SGX for this kind of execution. The results indicate that doubling the threads almost invariably improves performance compared to an environment without hardware encryption techniques. This scenario is entirely plausible in remote environments managed by external providers, as users typically ofload computations to remote systems due to insuficient local computational resources. In addition, these findings suggest that SGX could efectively mitigate the inherent overhead associated with encryption, thereby preserving privacy at runtime.
Future work may expand this performance analysis in two directions. The first direction involves a deeper exploration of SGX technologies as highlighted in Section 3.1, and a broader examination of other types of hardware that enable the establishment of a TEE, such as AMD SEV or Intel Trust Domain Extensions (TDX). The second direction focuses on analyzing multiprocess applications extensively designed for HPC centers, extending beyond bioinformatics to encompass more general applications. By pursuing these two avenues, future research can provide a more comprehensive understanding of the capabilities and limitations of various hardware-based security technologies in diferent computational environments.
This work was supported by the Spoke 1 “FutureHPC & BigData” of ICSC - Centro Nazionale di Ricerca in High-Performance Computing, Big Data and Quantum Computing, funded by European Union NextGenerationEU. tion on Secret Data, ACM Trans. Comput. Syst. 35 (2018) 533–549. doi:10.1145/3231594. [18] S. Miwa, S. Matsuo, Analyzing the Performance Impact of HPC Workloads with Gramine+SGX on 3rd Generation Xeon Scalable Processors, in: Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23, Association for Computing Machinery, 2023, pp. 1850–1858. doi:10.1145/3624062. 3624267. [19] A. Akram, A. Giannakou, V. Akella, J. Lowe-Power, S. Peisert, Performance Analysis of Scientific Computing Workloads on General Purpose TEEs, in: 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2021, pp. 1066–1076. doi:10.1109/IPDPS49936.2021. 00115. [20] L. Brescia, M. Aldinucci, Secure Generic Remote Workflow Execution with TEEs, in: Proceedings of the 2nd Workshop on Workflows in Distributed Environments, WiDE ’24, Association for Computing Machinery, 2024, pp. 8–13. doi:10.1145/3642978.3652834. [21] A. Mulone, S. Awad, D. Chiarugi, M. Aldinucci, Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment, in: 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), 2023, pp. 1858–1863. doi:10.1109/COMPSAC57700.2023.00288.