For decades, Trusted Computing has tried to anchor trust in the hardware, and the existence of Trusted Platform Modules (TPM) in most modern design is evidence that this approach is now well understood. The default behavior of recent Operating Systems like Windows 11 is even to deny booting if this security feature is absent. But this approach is not sufficient in a modern world where one needs to trust remote platforms. To preserve confidence in security, one needs to limit the trusted computing base (TCB) of a system at a level where an assessment can make sense. Trusted Execution Architecture (TEA) is the result of a partnership with ProvenRun to implement a TCB in Atos servers in a consistent way, from Edge to High Performance Computing. This allows to envision security features based on some common Root-of-Trust known to different platforms, at different scales and levels of interaction. Operating System Smart Cards C&ESAR'22: Computer & Electronics Security Application Rendezvous, Nov. 15-16, 2022, Rennes, France
In 1993, the NSA tried to introduce the Clipper Chip to promote Key Escrowed Encryption [
In this paper, we will bring a quick survey of the state-of-the-art of trust in hardware in section 2. We will discuss the rationale of Atos Trusted Execution Architecture (TEA) and the pros and cons of this software-oriented approach in section 3. We will then detail some aspects of the implementation in section 4. In the end, section 5 will sketch future innovative security in Atos HPC architectures, as allowed by the Atos TEA.
Long before NSA tried to promote its Clipper Chip, the ideas of using small pieces of hardware to secure secrets arose in several places. Several patents were filed around this invention, but the seminal industrialization patent was the creation of the first portable support with both a processor and a
2022 Copyright for this paper by its authors.
memory, allowing the small piece of plastic to cryptographically interact with its environment in an
active way. Embedding the processor and the memory in the same single ship came rapidly after this,
innovating the reign of smart cards. It is worth recalling that Michel Ugon, a French engineer of the
Bull company later acquired by Atos was at the core of these inventions[
The large dissemination of smart cards makes them a primary target of a new class of cryptographic
attacks: if the secret remains in the chip, maybe the way it is used leaks some information on the secret.
A seminal attack of this kind is due to P. Kocher and al. who invented Differential Power Analysis
(DPA) [
Hardware Security Modules (HSM) are another example of devices which were designed to protect
secrets. HSMs usually embeds features to physically protect their internals, and provide tamper
evidence at physical (labels, screws…) and logical levels (logs, alarms…). Certification standards such
as Common Criteria [
As usual, it is worth noting that being certified is not a guarantee of security. Depending on the
security model, a certified HSM can be proven vulnerable to threats which are out of its protective
scope. Interestingly, a recent example proved the need for HSMs to be self-protected against firmware
tampering, not only on their crypto processors, but also on their applicative part [
The Trusted Computing Group (TCG) promotes Trusted Computing concepts across personal
computers around the use of a Trusted Platform Module (TPM). It has now become an international
standard [
It is important to understand that the TPM cannot guarantee the security of the CPU booting process by itself. It must be completed by some bootstrapping process to kick-off the measurements and take their results in account. 2.4.
Following M. Sabt and al. [
The security of a TEE solution results from the hardware technology used, but it depends a lot more
on the usage of this technology at software level. Even if the TEE is fully isolated at hardware level, its
purpose is to exchange data with the normal world and process it in a secured environment. Any
vulnerability in the driver which ensure communication between the two worlds can ruin the overall
security of the TEE [
Other secure chips have been developed in different industries to ensure firmware integrity.
Examples of solution are found in the Set-top-box area where control access system vendors ensure
digital rights management (DRM) on video streams through hardware security features. For instance,
Nagra On-Chip Security (NOCS) “brings the hardware “root of trust” that ensures platform security”
[
The Open Compute Project (OCP) is an organization [
Through its Security project, the objective of the OCP could be summarized as an effort to gather
all previous security technologies like TPM and secure chips in an organized standard able to ensure
secure computing. Also all documentation is shared according to a Creative Commons license [
The overall organization of the OCP is somehow fuzzy, but two parallel approaches are identified
which should eventually converge:
1. The Datacenter Secure Control Module (DC-SCM) specification [
2. The OCP Platform Security Overview [
The main outcome of this last document can be summarized in the excerpted Figure 1. It introduces a new piece of hardware, the Platform Active Root-of-Trust (PA RoT).
The role of this PA RoT, which could be ensured by the DC-SCM, is aligned with the NIST Platform
Firmware Resiliency Guidelines [
The OCP has issued some requirements and recommendations around attestation of system
components [
Because of these requirements, each device must be provided with a set of cryptographic keys: 1. A Unique Device Secret (UDS) which is used to characterize the attester device. 2. A private authentication key unique to each device. The corresponding certificate is allowed for digital signature usage. This key is intended to be immutable and certified by the provisioner. 3. A private signature key unique to each device. The corresponding certificate is allowed for digital signature and content commitment usages. This key is intended to be updated and certified by the device owner.
For the provisioner, the specification also requires some key management infrastructure using HSM to protect: 1. The keys of the Provisioner’s Certificate Authority. 2. The keys of the Updater role. 3. The keys of the Firmware signer role.
To be noted that there is a notion of ownership transfer that implies that the Updater and Firmware signer keys can be changed by the Device owner.
Also, the requirement implies the existence of a root of trust within each attester, able to perform
cryptographic operations, including random number generation with sufficient entropy. References to
NIST publications and FIPS 140-3 [
Once this is set, attester devices must be capable of communicating their authentication and
attestation capabilities to the platform, and platforms must be capable of interrogating potential attester
devices and recording their authentication and attestation capabilities. The two references used to
implement the corresponding protocols are described in DMTF’s SPDM [
When dealing with firmware security, the threat model can drastically impact the level of protection needed. It is indeed a different story to protect the firmware integrity of a server lying in a physically isolated datacenter, or to address the same problem on a smart card which can be easily replaced by a copy. Also of importance is the scope of the intended protection. In our case, and in this paper, we focus on the security of the platform with an agnostic approach of the CPU/GPU components. We aim to ensure some security independently from the existing technologies at OS level. In particular, and as an example, the operating system can still use the TPM when it is present and leverage the CPU technologies to ensure it’s booted in a secured way. This will be further explained in section 4.4. But we want to ensure a certain level of security of the platform even if none of these security features is used. So, let’s first identify the type of attack scenario which one would like to prevent in this context.
The first scenario of attack is basic. If one has physical access to the server, he or she could leverage this access to reprogram the memories of the hardware and have the platform firmware execute unwanted operations. For instance, in the context of an HSM which would protect secret keys, reprogramming the firmware could be a simple way to get a given secret key copied on an external interface, hence compromising it.
Physical RAM access may be assumed as limited in time. If the physical access is possible for days, like during shipment, the attack possibilities in altering firmware are much more important. Components could be replaced which would try to mimic the behavior of the original ones while preserving some backdoor, for instance.
As the firmware will have some critical vulnerabilities discovered, the security objective is to make the platform able to recover from such attack and to avoid its persistency. If such an attack can change every piece of data in the platform memories, then it is pretty clear that the attack can remain, since the platform relies on its memories to boot. The use of some immutable data seems therefore mandatory to ensure security.
The internal threat remains a possibility for any vendor, and whatever the source code tainting approach. The effects are the same in case of an intrusion on the development infrastructure. Controlling the code can mitigate this risk only if these controls are not by-passed. A good way to ensure that the code is controlled at least once is to have it signed with a properly protected cryptographic key. This ensure as well a resilient posture in case of late discovery of some rogue activity. In this case, the rootof-trust remains the cryptographic keys which are used to sign the firmware. 3.2.
From a security perspective, a platform MUST use a Hardware Root-of-Trust (HW RoT). It is the
only way to ensure some protection against software attacks and to achieve a level of resilience. Without
it, any of the above listed attacks, if successful, would achieve a state where trust would be damaged in
an irreversible way. On the other hand, if the Hardware RoT exists, the platform may be rebuilt from
this basis. This is the approach specified by OCP with its Platform Active Root-of-Trust and promoted
by the NIST Platform Firmware Resiliency Guidelines [
But Atos also wanted to limit unknown hardware to increase trust and confidence in the solution.
Even if some of the proposed PA RoT are open source like the Open Titan [
We therefore limited the HW RoT to some public cryptographic key anchored in the silicon, whose
corresponding private keys are handled in an HSM developed by Atos: the Trustway Proteccio
netHSM [
These chains-of-trust will be later detailed in section 4.1. 3.3.
Most of the modern servers have a Baseboard Management Controller (BMC), which is responsible
for the power-on of the main CPUs, and the management of the firmware. From a security perspective,
it is already a piece of the platform you need to trust. And it has already been proven that it can be a
source of weakness for a server [
For the servers developed by Atos, the BMC is hosted in a System-on-Chip (SOC). We decided to leverage this existing hardware and to elevate it as the Platform Active Root-of-Trust for the platform. The Figure 2 modifies the original figure from OCP (see Figure 1) to illustrate the approach.
The server's CPUs are therefore seen as symbiont devices relative to the BMC. The advantage of the approach is that this hardware is mandatory in all servers, as it is the interface to the management infrastructure to power-on the platform or upgrade its firmware. It also plays a key role in the overall integrity of the platform, and its security should be hardened.
ARM
M
B
The current implementation of the Atos BMC is based on Open BMC [
Of course, the use of a non-dedicated hardware for security has some drawbacks. For instance, it is
envisioned to implement a firmware TPM in the Atos BMC. This would allow to add this security
feature in HPC environment where no TPM is usually implemented for physical space reason. But it
cannot be claimed the same level of security, since the TPM chips are usually certified at high level of
security (see for instance [
Allowing the change of cryptographic keys to the platform owner is difficult to ensure while preserving the overall security, since the purpose of anchoring RoT in the hardware is to prevent software-based attacks which could change the keys used for firmware verification. Even if these keys are public, their integrity is of utmost importance to the security objectives.
Signed firmware is used to authenticate firmware before critical functions: 1. CTD: During boot sequence to ensure that the next step of the boot sequence will activate an authenticated binary code. 2. CTU: During updating process to ensure that the firmware image is authenticated before flashing it in RAMs.
These two chains are independent and complementary.
Three types of keys can therefore be identified to verify the signature of a firmware: 1. At the beginning of the boot sequence, to benefit from Hardware root-of-trust (red key in the
Figure 3). 2. During boot sequence where a public key embedded in firmware can be used to pursue the chain-of-trust in a flexible way (orange key in the Figure 3).
Before flashing, where a public key can be used to verify the signature of the firmware payload. The corresponding public key can be stored either in a hardware secured part of the SOC, or in a firmware provided it is protected by another chain-of-trust (see green public key in the Figure 3 as an example).
Except the root-of-trust key secured at hardware level, all public keys used to verify firmware integrity and/or boot chain integrity must be part of a signed firmware. This ensures that the modification of the verification keys is authenticated provided the key store is properly implemented.
It is important to understand that the hardware secure boot is very limited in practice. It only secures the first stage of booting, a little program which cannot exceed a few kilobytes of code (63 K for ARM), because it will be loaded in ARM memory for signature verification. All the other operations of a chainof-trust for detection (CTD) or chain-of-trust for upgrade (CTU) will exceed this limit and will therefore rely on software-based security.
Yet, this also allows ownership transfer provided the customer trusts its vendor, which seems a legitimate hypothesis. Indeed, at least the CTD key is included in the first stage booting code, which ensures the possibility to change it, while preserving the overall security of the scheme. Depending on the needs, CTD and CTU keys may be owned by the vendor or not. Consequently, in theory, the chainof-trust keys can be changed for each owner provided at least one signature is performed involving the hardware RoT key owned by Atos. 4.2.
The root of trust is the initial public key which must be inserted and secured by hardware security measures. The root-of-trust must be immutable once the hardware security measure is in place. Therefore, the corresponding private key is of critical importance in a production environment. Due to the hardware hardening it is not possible to update a root-of-trust key by firmware upgrade. The only allowed operation may be to invalidate a compromised key. It is therefore mandatory to anticipate the compromise of such key by organizational measures and by generating several backup keys that will be injected in the hardware in case of a compromise (see 4.2.2).
For intellectual property reason, we cannot here reproduce the precise way the ARM Secure Boot is implemented. We will therefore just sketch the main points to help understanding how the TEA rootof-trust key is managed.
By default, secure boot is not enabled. This is mandatory in the design process since initialization of the ARM component will need to boot the CPU. The usual chicken-and-egg situation mandates the component to be initially insecure. Several ways to activate secure boot are available. To simplify, let’s say that we have: 1. A reversible way to activate hardware secure booting through jumpers on hardware pins. 2. An irreversible way to activate hardware secure booting through one-time-programmable hardware memories in the SoC.
The first option is intended for development, testing, and qualification. In the end of the production process, the second option will be used to set the component in secure mode.
Once set in secure mode, only a signed first stage can be used to boot the SoC (see Figure 3). The signature will be verified against the keys which have been inserted in the component.
Consequently, it is possible and recommended to introduce the public keys in the ARM core as soon as possible in the factory process. It has no immediate impact and no risk to brick the SoC provided the secure boot is left in reversible mode. It has the advantage to personalize the component early in the process, making it more difficult to tamper with (see Table 1).
For obvious security reason, the hardware secure boot public keys are also injected through OTP memories and cannot be changed once in secure mode. For a given component, the same keys will therefore be in use from day 1 of its production until its end-of-life. If the component is to be used for ten years, the corresponding private keys must be protected during this time. And on such long period, one must anticipate risks such as key loss, key compromise, identity usurpation on the firmware signing chain, etc.
As a first consequence, HSM should always be used to protect hardware secure boot private keys. This seems consistent with the sensitivity of keys which cannot be changed in the field if an incident occurs.
Secondly, the set of hardware secure boot keys should not be limited to one key. It is therefore separated in one production key, and several spare keys. Any of these keys could sign a firmware recognized by the BMC. But the only one in use is the production key. The spare keys are created just in case something weird happens to the production key. But they must be created at the same time because their public part will be injected in production. And the protection of their private part is of utmost importance.
All the keys used to ensure trust in the platform do not deserve the same level of protection. For instance, DEV keys are considered insensitive. They will be deactivated in production and are therefore considered extractable from the HSM. This allows to have some outsourced development without having to give access to the HSM signature mechanisms.
This is obviously not the case for PROD keys which are generated, stored, and used in an Atos
Trustway Proteccio HSM [
Besides, the access to the signature function is controlled: - By logical measures for Chain-of-trust keys. - By the use of a smart card for the HW RoT production key.
- Using a smart card AND the possession of the key backup for the HW Spare production keys. 4.3.
The System-on-Chip (SoC) used in the BMC embeds the TrustZone technology which is part of its
ARM Core [
TeaCore is based on the use of a proven operating system ProvenCore which has been certified by
ANSSI at EAL7 level in a different context [
Together with the HW RoT key anchored in the silicium of the BMC, the TeaCore provides the architecture to implement a full Platform Active Root-of-Trust as proposed by OCP. As of today, the chains of trust for development and upgrade are implemented. Next steps could introduce attestation mechanisms.
TeaCore Secure Services TA TA TA Secure Boot FW upgrade
Normal World TrustZone Secure World Figure 4 - Trusted Execution Environment of the Chains-of-Trust
We have now all the information to understand the full boot sequence of an Atos server implementing the new Trusted Execution Architecture.
For this example, we will use the case of a server based on an Intel CPU implementing the TXT
technology [
Atos TEA do not interfere with this process. It only adds at the beginning a preliminary verification of the IBB. Since the BMC is responsible for powering on the CPU, it will use its TrustZone to perform a signature verification of the IBB and won’t power-on the CPU in case of a signature error. The whole sequence will therefore start from the ARM secure boot sequence of the BMC and ensure firmware integrity through the different existing mechanisms (see Figure 5). The approach would work the same way for another type of CPU.
Now that we have a TEE enabled in our servers, from Edge to Enterprise servers, let’s see the type of application we could envision in a High Performance Computing (HPC) environment. 5.1.
One drawback of the Platform Active Root-of-Trust approach is that the secure component becomes
a single point of failure for the system. This is especially true when comes the definition of the Unique
Device Secret (UDS). As introduced in section 2.6.2, the attestation mechanism assumes the
implementation of some hierarchical certification where each vendor attests the integrity of its product
using some public key infrastructure mechanism. The PA RoT will therefore use all these UDS to
control the authenticity of the platform components with some signature mechanism, and the
verification of the certificates of the public keys. The trusted cryptoprocessor PA RoT is also the
privileged place to store the UDS of the platform itself, allowing it to become attester to the remote
management infrastructure (verifier). But this approach reaches some limits when it comes to big cloud
infrastructure or high-performance computers. As an example, one of the recently deployed Atos HPC
platforms counts 300 000 computing cores shared among roughly 4500 CPUs [
It is not the purpose here to detail the architecture of an HPC installation. Besides, this is an evolving matter. Schematically, an HPC framework will gather nodes of different types exchanging data through an Ethernet or Interconnect fast network. Management Nodes or Rack Management Controllers (RMC) can exist to manage a physical cluster of a hundredth of computing nodes. Each cluster is interconnected with the other clusters to form the overall HPC (see Figure 6).
From an operational point of view, the access to the computing power is devoted to some login nodes, and one of the management software roles is to schedule the job requests submitted to the login nodes to optimize the computing power. Each job will get allocated some computing nodes and storage resources for an amount of time, depending on the pre-requisites of the job request. The whole purpose of the architecture is to avoid latency in messages exchange between the computing nodes and in input/output writing on the storage nodes, while dealing with astronomically high amount of data. In a standard attestation approach, a compute node would have to check the attestation status of the storage node before sending data to it. This would mean data exchange between the nodes consuming the interconnect bandwidth. Even if it may sound marginal, keep in mind that these machines are pushing the specifications at their limits, and are subject to some avalanche effects when unwanted events occur.
On the other hand, all the development framework around HPC has already incorporated error events because the scale of the HPC makes plausible to encounter errors when the machine is in use. Hardware faults, hot swaps, are part of the normal use of an HPC computer. This can be a drawback in a classical platform firmware attestation mechanism, but it can also be turned to our advantage. 5.3.
Under the hypothesis that the Trusted Execution Architecture is implemented, an immediate benefit arises from it. All nodes will get a TEE through their BMC (see Figure 7). Of course, the same would occur if each node would come with a TPM or any form of PA RoT secured chip. But the truth is that adding some secure element in these nodes is not that easy for physical constraints (power alimentation, cooling, space) while a BMC is mandatory anyway. So now, we have this trusted capacity on all our nodes, and we can leverage it.
Like a living body can identify its cells through characteristics determined by the DNA common to all cells, the idea of a local hardware-secured zone keeping some DNA-like secrets shared by all the machine nodes, makes possible for each node of the machine to perform the access controls, without relying on a remote server to determine if a communication is tampered or not. In other words, this generalizes the notion of Unique Device Secret (UDS) to a global platform such as a High-Performance Computer (HPC) or a Cloud-based infrastructure.
Any node of the machine will assume that its counterpart possesses the shared secret. It is therefore possible to encrypt communication under this assumption. If the counterpart fails to decrypt it, this will be treated as a glitch or hardware failure using the normal exception mechanisms of HPC development libraries. 5.4.
The powering-on of an HPC machine is done in several steps. For instance, a rack will be poweredon before its computing blades can be powered-on. There is no guarantee on the order of the poweringon. Some computing blades can be powered-on before another rack is powered-on. The hypothesis is that all nodes will eventually establish a connection through a management network, without guarantee that all connections are feasible. For simplification, we will assume that a Rack Management Controller exists, which can be seen as a BMC dedicated to the management of all the BMCs of a rack. We will also assume that the sequence of initialization starts an RMC before the BMC it manages.
The process must therefore ensure the following properties: - If the machine is powered-on, a new secret must be generated by the first powered on RMC. - If the machine is powered-off, which is an unlikely event, a new secret must be generated on next power-on by the first RMC that will be powered on. - If several RMCs are powered-on in parallel, a negotiation mechanism must converge towards a single secret.
- Any new RMC or BMC that will be powered on must get the secret in a secure way.
Generating a secret in the RMC could be a challenge from a cryptographic perspective, since we do not assume a physical random number generator is available on the BMC. It seems feasible as soon as a reliable random noise generator is available. One needs to avoid the repeatability of the boot process which could lead to the generation of the same secret on each boot. To ensure a proper source of noise, the global entropy must reach at least 256 bits. Fortunately, the RMC is gathering a lot of physical sensors information which can be leveraged to assembly enough noise in the random number generator of the device.
When a new component is added to the management network, it can be of two kinds: - If it is a BMC, it will get the secret from its RMC.
- If it is an RMC, it will negotiate its secret against the other RMC as follows.
One cannot know in advance the order of RMC appearance in the management network. And in the life of the machine, some racks may be shut down for maintenance, then reconnected. One wants the secret to stabilize as soon as possible while preserving the history of the used keys. This is a possible application for blockchain technology and decentralized consensus making.
If RMCa and RMCb have booted and generated their secrets Sa and Sb, one cannot choose among these secrets. But a cryptographic mechanism can take place to establish a common secret Sab. This secret is timestamped in the blockchain and becomes its first block. Two blocks are then added with Sa and Sb.
If RMCc joins later with its secret Sc it will have to adopt the secret Sab. And the block Sc is added to the chain. The block chain length is therefore related to the number of racks whose secrets were changed so far.
If two racks exchange two different block chains with the same initial secret Sab, the block chain is reconciled with the missing blocks.
If the two initial secrets differ, the longer chain will be privileged. The blocks of the shorter chain will be added. If the chains have the same length the chain with the smaller hash will be kept.
If an RMC has to change its secret after negotiation, it has to propagate the new secret to its rack components. The blocks to add in the blockchain indicate the other RMCs to inform of the secret change. 5.5.
To prevent the secrets from being compromised when a computing blade is extracted, the
corresponding secrets are stored on RAM in an encrypted way. The component extraction powers the
component off by design. This ensures that at least a portion of the encrypted key is erased. The
encryption mechanism can therefore guarantee the disappearance of the key if a sufficient portion of
the key is lost. This security mechanism is prone to cold boot attacks [
Before delivering the secret to a new component of the machine, it is of course important to determine if the new component is sane. It is at this step that remote attestation protocol can be used. The UDS at node level makes perfect sense for this as it is inserted at factory time to build security upon trusted remote attestation of a component (see 2.6.2).
This pre-inserted private key should never be exposed outside its security module. Zero-knowledge protocols can use the key to attest the authenticity of the TPM-like feature remotely. This way, a newly inserted component can be checked remotely for sanity before providing the secret. And the private key is needed to decipher the secret, protecting it on first communication.
Based on well-known concepts of Product Security, Atos has implemented a Trusted Execution Architecture (TEA), common to all its servers. The trust in this implementation is founded on: 1. Public cryptographic root-of-trust keys anchored in silicon. 2. Private keys protected by an RGS certified Atos Trustway Proteccio HSM. 3. The well-known ARM TrustZone technology embedded in the existing BMC component of our platforms. 4. The hardened operating system TeaCore developed by ProvenRun on Atos specification and based on their formally proven and EAL7 certified operating system ProvenCore.
This TEA is first used to ensure some Platform Firmware Resiliency through firmware signatures verified at boot time and before any upgrade. Its generalization to all Atos-made platforms makes possible some innovative security features. As an example, we presented an innovative approach to device attestation applicable to High-Performance Computing (HPC) environments which generalizes the notion of Unique Device Secret (UDS) to a global platform such as a HPC or a Cloud-based infrastructure.