11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 A Secure Gateway for Enabling Application Specific Integrated Circuit Design Collaborations Steve Bogol∗ , Paul Brenner∗ , Adam Brinckman∗ , Ewa Deelman‡ , Rafael Ferreira da Silva‡ Sandeep Gupta§ , Jarek Nabrzyski∗ , Soowang Park§ , Damian Perez∗ , Sarah Rucker∗ , Mats Rynge‡ , Ian J. Taylor∗† Karan Vahi‡ , Matt Vander Werf∗ , Sebastian Wyngaard∗ ∗ Center for Research Computing, University of Notre Dame, Notre Dame, IN, USA ‡ Information Sciences Institute, University of Southern California, Marina Del Rey, CA, USA § Department of Electrical Engineering, University of Southern California, Los Angeles, CA, USA † School of Computer Science & Informatics, Cardiff University, Cardiff, UK Abstract—Leading CAD companies are currently developing particular, these provide a private environment for each design virtual secure environments to help lower the barriers for team. In contrast, US Department of Defense (DoD) programs adopting new ASIC design flows. However, such services are pro- such as DARPA CRAFT [2] require sharing of design flows, prietary, lack key features, and present barriers for collaboration and sharing. This paper covers the transition of the CRAFT IP, and best practices across design teams while protecting repository, originally designed as a repository for discovery all information about specific designs being carried out by and documentation of ASIC design flows, to a fully secure individual design teams. Also, the commercial services are environment called the CRAFT Vault hosted in Amazon Gov typically limited to the vendor’s own tools and design services Cloud that allows designers to collaborate and actually implement and hence not optimized for rapid evaluation and adoption of these flows. The Vault is a fully configured environment that is deployed with all the necessary electronic design automation new design flows, especially those that use tools and IP from tools and IP making it easier for end users to implement a multiple vendors. In short, commercially available services are CRAFT Design Flow, augmented with Blockchain functionalities difficult to adapt to serve the needs of programs like CRAFT to provide a non-repudiable audit trail of what happened and that are trying to develop new tools and flows and that are when. interested in building a new community of design teams. Keywords—Collaborative Environments, Design Flows, Chip Design In this paper, we present the CRAFT Secure Vault, an extension to the CRAFT Repository [3] gateway, that provides I. I NTRODUCTION design teams ready access to new design flows, tools, and IP, and an environment that supports the unique combination of A new Application-Specific Integrated Circuit (ASIC) de- rapid learning and quick time to design productivity, security, sign flow often faces high barriers for adoption, even by access control, and community-wide sharing of best practices experienced design teams. New design flows typically require of DoD programs like CRAFT. new tools; Intellectual Property (IP)/licensing considerations of acquiring new tools; installation of new tools and asso- II. R EQUIREMENTS ciated license servers; and the need to ensure that a user’s environment are properly configured and that the new settings The goal of the CRAFT Secure Vault is to provide dis- do not interfere with other installed software. Additionally, tributed design teams a ready-to-use platform that streamlines new design flows typically require new models and libraries, the process which otherwise requires teams to acquire new especially Process Design Kits (PDKs) (which contain all tools and IP from multiple vendors, install and configure these information provided by the semiconductor foundry regarding tools on their computing platform, and learn a new design flow fabricated devices), cell libraries, libraries of larger IP mod- using voluminous user manuals. It also provides a collabora- ules, and scripts for designs. IP/licensing considerations make tive environment for geographically and institutionally diverse all this acquisition time-consuming and expensive, strongly design teams to rapidly learn and evaluate the new flow and deterring even the evaluation of promising new design flows. low time-to-productivity for ASIC design using new flows, Increasingly, members of design teams span different parts of tools, and IP. a large organization or even multiple organizations. Hence, To achieve the above goal, the CRAFT Vault has been coordinating the execution of a new design flow on various designed to support the following use cases: computational resources available to sub-teams is likely to be 1) From the view of the design team, for each design flow time consuming and inefficient. hosted by the Vault, all tools required are installed and To tackle such challenges, leading CAD companies are ready to use, simply upon the completion of an off-the- beginning to develop commercial services, such as Cadence shelf licensing agreement by the team (more ahead). Also, VCAD [1]. However, such services are proprietary, lack key all required PDKs, models, IP libraries, are also pre- features, and present barriers for collaboration and sharing. In installed upon completion of the agreement. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 2) Each hosted design flow is captured precisely and repre- setup, and administration. The high level overview of the sented using interactive views of the design flow for rapid CRAFT Secure Vault architecture is shown in Fig. 1. learning. Also, each design flow and associated tools, In order to lock down access to the CRAFT Secure Vault PDKs, models, and IP libraries are tested thoroughly (Fig. 1-left), participants cannot access it directly, instead by using the flow to design appropriate example Very they connect via a client side gateway using a single secure Large Scale Integration (VLSI) modules (called cores) communication channel [4]. The gateway enforces the traffic for integration into the design of ASICs. The instructions of information to and from the Vault. The architecture of the for each step of a design flow are accompanied by design Vault therefore consists of three core components, namely, a files used/obtained during the design of the example VLSI (1) secure CRAFT gateway, a (2) secure CRAFT Vault, and cores, and these instructions are refined based on the a (3) secure virtual private channel that securely connects the example design experience to make the new design flow two environments. more accessible. From a CRAFT user’s perspective, the result is that they 3) Each design team is provided an environment for collab- are provided seamless access to a remote desktop in a secure orative design which supports multiple design projects, Vault environment from their own desktop. From a technical sharing of best practices with members of the team standpoint however, this remote desktop connection involves and selected best practices with the wider community a secure virtual private connection to an AWS GovCloud [5] of design teams. Further, each design team may bring hosted client side gateway and to a second AWS GovCloud additional tools and IP to the Vault and fork from a hosted virtual private cloud (VPC) environment, and an authorization design flow to modify it for its unique needs. and connection proxy to one of multiple secure Linux (or 4) The tools, PDKs, models, and IP libraries hosted in the Windows) systems to provide remote desktop access to a Vault are governed by off-the-shelf licensing agreements secure Vault that hosts IP, data, and tools for collaboration designed to fit the needs of the range of design teams in ASIC designs. likely to use the Vault. Each design team may customize the licensing agreement by selecting the tools, PDKs, IV. S YSTEM D ESIGN models, IP libraries, and the level of resources for col- In addition to our goal of providing a standalone, functional laboration and computing they wish to option. A single and secure environment for designing chip flows, we also aim licensing agreement will make a customized instance of to capture and provide an audit trail for the whole design Vault available to the user and reduce the barrier to entry process. We decided to leverage Blockchain as a means of by avoiding prolonged delays associated with lengthy hooking the various components into a common distributed licensing, installation, customization, and testing. ledger. The CRAFT Secure Vault consists of the following 5) From the view of the vendors/developers of tools, PDKs, 4 main components, with Fig. 2 illustrating the interaction models, and IP libraries, the Vault guarantees that each between the various pieces. design team can only access the items it has optioned. 1) Blockchain: The data backbone runs on a distributed The Vault is designed to ensure that PDKs, models, ledger with a data store per node for storing sensitive and IP are protected, i.e., cannot be exported outside, data. unless designated by the vendor as open. At the same 2) Secure Vault Servers: A full audit trail is collected for time, each design team is allowed to export reports of each server so we have access to who did what and when. simulations, characterization, and verification of their de- 3) ASIC Design Flows: To support the existing CRAFT sign. Out-of-Vault design improvements, simulation/ver- repository, visualization tools have been ported to the ification, and/or tape-outs (i.e., sending the design to a CRAFT Secure Vault environment. foundry for fabrication of chips) are only permitted if the 4) Collaborative Workflow Tracking: To record transactions design team has separately secured out-of-Vault licenses. on the ledger when users run scripts on certain data for a design flow. III. S YSTEM A RCHITECTURE Our broad goal is to provide a system that is capable of A. Blockchain provisioning a secure environment, called the CRAFT Secure A Blockchain [6] is a distributed ledger that provides a Vault, for a group of CRAFT collaborators that span a number shared write only file system. Data is stored through con- of different teams, each having particular authentication and sensus, meaning that all parties sign off on each transaction. authorization privileges to tools, IP, and design data. A CRAFT Once data is written it cannot be removed by any single Secure Vault is a secure collaborative environment, synergistic entity and hence, it provides a non-repudiable audit trail of to a collaborative project in the CRAFT repository [3], which what happened and when. All Vault implementations sit on gives users direct access to ASIC design tools and data such an underlying Blockchain system, for interaction with engineering capabilities, regardless of geography, so that team the distributed ledger. members can work in a predictable environment that defines Currently, we are using three Ethereum [7] nodes per Vault securely who has access to what. It also avoids the lengthy virtual private network (VPN) deployed in the Vault environ- process of acquiring tools (and their dependencies), IP, system ment. We plan to augment this deployment with additional 2 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 Fig. 1: Architectural overview of the CRAFT Secure Vault. Fig. 2: Overview of the CRAFT Secure Vault Software Component Design. Ethereum nodes one which will be hosted by the University the Vault is to store data locally with the Blockchain only stor- of Notre Dame, and another controlled by DARPA in the near ing the hash of the data (anything used by CRAFT designers future. Addition of more nodes results in them being involved e.g. a design flow, IP, PDK, etc). Hence, it is sensitive material in the consensus, resulting in a stronger consensus backbone, and as such, it is never stored on the Blockchain, it is kept insuring a tamper proof audit chain of events. on the local filesystem in the Vault. But because data maps to The Blockchain creates a cryptographic fingerprint (a hash) hashes, it provides signed non-repudiable proof that an event unique to each block and a consensus protocol, the process happened e.g. design flow has been updated, someone logged by which the nodes in the network agree on a shared history. in, a script was run with this data etc. Copies of actual data are The hash serves as proof that the miner who added the block stored privately on each Vault using a write only filesystem. to the Blockchain did the computational work. It also serves In the Vault, we use Blockchain in two core areas: as a kind of seal, since altering the block would require • Design Flow Visualization and Editing, which consists generating a new hash. Verifying whether or not the hash of a Web GUI for visualizing and editing the design flows matches its block is easy, and once the nodes have done and a backend API that stores design flow version hashes this they update their respective copies of the Blockchain on the Blockchain (distributed ledger) and design flows with the new block. This mechanism makes the Blockchain themselves on a local filesystem (local to the vault). tamper proof, or “immutable”. Since, in CRAFT, each node • Collaborator Workflow Tracking, which consists of a has similar computational capability, it makes it impossible for script wrapping approach and a backend API that stores one party to change the audit trail or events. verification/integrity data on the Blockchain (distributed The general approach for interacting with the Blockchain in ledger) and a local filesystem (local to the vault). 3 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 B. ASIC Design Flows Currently, the CRAFT repository does not provide a mecha- nism to ensure adherence between produced outcomes and the Each CRAFT performer team is developing a user-oriented scripts, tools and input data used to generate them. To address version of their new design flows that can be used by DoD this issue in the Vault, we developed a system to provide the ASIC designers. As such, their representations are specific to necessary generation and verification steps using cryptographic their individual design approaches. Since the CRAFT repos- encryption. The generation step encrypts outcomes, i.e., output itory bridges between the various flows and the target DoD data, based on the scripts, tools, and input data used for design community, we have developed a way for designers generation. The verification step applies then the reverse to capture, document, store, and visualize such flows in a operation, so that a user that would like to make use of an systematic and common methodology. Initially, we conducted output can verify that it adheres to the versions/metadata of several requirements gathering meetings (in-person, telecon- the scripts, tools, and input data used for its generation. ferences, email exchanges, etc.) to create a high-level view of the design flow (no flow specific information was required at V. I MPLEMENTATION this point). Then, we extracted expert knowledge to expand the high-level flow into a complete running example flow, which A. Blockchain Implementation included information about options, flow controls, and data. In order to prepare the CRAFT Secure Vault for operations, These interactions allowed us to iterate towards a common we must generate an identity for the vault server itself. We schema for documenting the flows. then need to generate users in a global way (one single Vault One of the major capabilities that we wanted to demonstrate account) but allow those users to access different vaults in in this representation was the ability for the users to visualize different ways. We can achieve a number of these core goals and edit these flows, and perform flow validation (syntactically by interacting with the ALADDIN framework. and semantically). We decided to describe and formalize the template for the design flows in JSON format, where the ALADDIN and the CRAFT Secure Vault: We have developed schema has three major sections, tools, files, and stages. The a Blockchain service architecture, called ALADDIN (Any tools section describes a list of tools that are used within Ledger, Any Distributed Data using Intelligent Networks), the flow and specifies tool functionality and specific options which incorporates the following services: or configuration parameters. The stage section describes the • A common interface to different distributed ledger individual steps of the flow, and includes its input and output Blockchain systems; files (which may be provided by external vendor tools), a tool • A common interface to different data stores; (referenced by the its unique identification), and flow controls, • A conceptual model, using assets and transactions, that which capture the decision flow based on the tool output. define the relationships between things that need to be We provide the users two options for describing their flows: tracked (assets) and how that tracking occurs (transac- (i) upload a JSON document conformant to the schema— tions) to feed into smart contracts; the repository UI automatically validates and identifies any • A Smart Contract Design Tool to help generate smart con- errors in the document; or (ii) via the use of an interactive tracts automatically from the conceptual model, defined visualization tool, as presented in [3]. using a UI; and • An auto API generator tool that converts the ALADDIN C. Collaborative Workflow Tracking conceptual model to a Node.js REST API to help software integration with the Smart Contract. Many hardware modules (i.e., IP blocks incorporated in a hardware design) must interoperate with other hardware and Defining an ALADDIN application: Each ALADDIN appli- software modules. Across multiple vendors, such interoper- cation is defined using assets and transactions that are written ability is often ensured by adherence to industry standards for a specific Blockchain system, along with data (files) to be (e.g., USB 3.0 for serial ports and DDR4 for process-memory saved in a third party data store. We use the ALADDIN private interfaces). Even when a complex ASIC is designed within Ethereum [7] network adapter for Blockchain, and IPFS [8] a single organization, various sub-teams need to coordinate or MongoDB [9] for the data store. ALADDIN Data Bundles the design to ensure interoperability. Such design coordination wrap data for an asset into an asset Manifest, which specifies typically takes the form of the sub-teams agreeing to adhere data relationships using links to hashes that represent the off- to specific released versions of key IP modules. Since the chain data. We use assets to represent entities, (design flows, CRAFT design flow also includes IP modules created using datasets, or scripts) and transactions to represent updates, generators and other scripts and tools, it is also necessary or usages of those assets. Once assets and transactions are for the design framework to include adherence to specific defined, ALADDIN converts this specification into a format releases of scripts and tools during design and use. Further, that can be used with the Blockchain system by generating: achievement of desired design robustness may also require (1) a smart contract (in Solidity for Ethereum); and (2) a various sub-teams to adhere to specific releases of scripts custom business REST API that is generated for the assets and tools for verification of design’s correctness, including and transactions that the user defined. This allows client adherence to industry/internal standards. applications and SDKs to be written directly against this REST 4 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 Fig. 4: The Design Flow API the ALADDIN framework, which splits the request (a design flow) into the data component (the JSON design flow file) and Fig. 3: A schematic showing how the design flow is stored a hash of the data. The Design Flow itself is stored in the data onto the ledger and data store. store which is local (private) to the vault itself and the hash is recorded on the blockchain to tie this event into the distributed API, which in turn converts the calls into Smart contract calls, ledger. If the JSON design flow changes then the blockchain which in turn writes data to the distributed ledger network. is broken and the system will know that there was an attempt ALADDIN Tooling: Our approach has involved creating some to tamper with the local data. flexible developer-based tooling to help generate backend The API is used to interface with the blockchain for storing Blockchain implementations and APIs, which includes a: hashes of Design Flows. The data, i.e., design flows, are stored • Smart Contract Design Tool that can auto create privately on each vault server, using a write only filesystem Ethereum Solidity Smart Contracts from an asset (design (IPFS). The purpose is to provide a non-repudiable audit of flow) and transaction (change on the design flow) model. each version of a design flow, identifying the author at each • We use the same underlying model in our API Genera- stage. In order to port this to the vault environment, we have tion tool, which can parse a Smart Contract to create a migrated the EmberJS App to an AngularJS/Blockchain API to Swagger OpenAPI specification [10] and then auto gen- work locally in the closed Vault secure environment – since the erate an ExpressJS [11] API in Javascript for interfacing CRAFT repository backend has multiple outgoing connections with the Smart contract and Blockchain. to support its infrastructure. The local App communicates B. Design Flows (locally) via a Blockchain ExpressJS [11] API, generated by our Smart Contract Design and API Generation Tools. The The CRAFT repository Design Flow mechanism is based app provides an editor view and a screen that shows all on an EmberJS App [3], which talks to backend filesystem versions of the design flow. A user can simply click the version for storing the design flows. The repository also performs they want to view and the app will update the JSON flow versioning, so the “time machine” uses this functionality to accordingly. Fig. 4 shows the backend API that interfaces with browse previous versions of the design flow. To enable this the blockchain. This shows two endpoints: one to create a capability in the Vault we have: design flow, which is used to initialize the app with the design • Redesigned the EmberJS App so it talks to a local flow that was imported from the CRAFT repository; and one backend; to update Design Flow that allows revisions of this design flow • Created a new backend capable of storing the different by transacting Ethereum revisions on it. versions of the Design Flows, which also connects to our common distributed ledger backbone, (save or update C. Collaborator Workflow Tracking transactions are saved). These tools enable flow editing For the design flow implementation, we developed the within the Vault system. underlying blockchain implementation and implemented a Here, an asset is a design flow and transactions are updates prototype of a lightweight tool for supporting design flows on this initial design flow. Then, when changes are made within the Vault. In creating the API, we built tooling that to that design flow, transactions on that original design flow allowed us to generically develop smart contracts and build are written along with the changes to it. This allows us to APIs from them automatically, to support extensibility for the create a non-repudiable chain of transactions for such a flow other Blockchain uses we have in CRAFT. representing versions of the design flow made by the users. To track the designer’s collaborative workflow, we use the Fig. 3 shows this process in detail. The CRAFT Secure same tooling to create a Blockchain API for addressing the Vault server, shown on top, hosts the design flow application Data security and integrity assurance task. Since the CRAFT in the form of an EmberJS Web app. The App then talks to design flow includes IP modules created using generators and 5 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 • The Blockchain. To write to the Blockchain, the Node.js script interfaces with the API that was generated to the Ethereum Blockchain for this application. The API is basically a file oriented API that allows the application to specify the filename, its file hash, and other metadata. • A Data Store, which is responsible for the storing of data files that are used by a designer. The CRAFT Blockchain API is based around a Web API Fig. 5: Data security and integrity assurance layered approach. concept. It contains adapters to the Blockchain and it provides interfaces to connect to IPFS [8], which is a lightweight, other scripts and tools, it is also necessary for us to track distributed, versioned, Web-based file system, designed specif- specific releases of scripts and tools that were used during ically for blockchain use. To cope with large design files (> 15 design and verification. In this task therefore, we set out to GBs in some cases), we redesigned the approach to interface design a system that tracks a designer’s use of the underlying through the file system rather than the Web based interface. IP modules, scripts, tools, and data so that we can record the To achieve this, we implemented an adapter that manages file specific versions or releases of scripts and tools that were used usage on the local file system. The scheme, first generates a for verification purposes. Each tracking event is stored on a hash of a file that a designer wants to register. It then checks distributed ledger in AWS GovCloud for transparency, support to see if this file is already being tracked. If so, then a copy is and for audit purposes. Fig. 5 shows the general overview of not stored again, because the hashes already matches a file that the different layers of the approach. has been stored in the file system. If the file has not previously Design Flows: A number of extensions were made to the been tracked then it is copied to the file system and the hash design flow specification to enable realization of the execution code is used as its filename, for future checking. The resulting of the design flow scripts. These modifications included the hash is then passed to the Blockchain adapter to be stored on addition of file system locations specifying the physical paths the Blockchain. For the most part, only the hashes are stored, of the IP, scripts, data, and so on. Essentially, we converted to track usage and so the approach is scalable with respect the passive design flow specification that existed previously to the file space required for tracking. A shortcoming of this and made them actionable, so that they could run within a approach is the time it takes to generate the hash. To generate command line environment. For each stage of the design flow, a SHA2 hash of a 20GB file can take more than two minutes. a Linux shell script is generated to automate (when possible) We are investigating solutions to address this issue currently. the steps involved in the flow generation. VI. C ONCLUSION Flow Scripts: The execution scripts mentioned above are This paper covers the transition of the CRAFT repository, organized into five steps: (1) creation of a blockchain session; originally designed to be a repository for users to discover and (2) registration of the input data as transactions (a transaction document ASIC design flows to a fully secure environment is performed per input file); (3) execution of the flow’s stage called the CRAFT Vault hosted in Amazon Gov Cloud that program (s); (4) registration of the output data produced by allows designers to collaborate and actually implement these the previous step (a transaction is performed per output file); flows. The Vault is a fully configured environment that is and (5) finalizing the session. deployed with all the necessary Electronic design automation Data Interface: To connect the Flow Scripts to the blockchain, tools and IP making it easier for end users to implement a we designed a set of interfaces that could be used to track a CRAFT Design Flow. In addition to the technical challenges, particular design flow session: we had to invest a fair amount of effort to get the legal agree- • Create Session: starts a working session to be tracked. On ments in-place for hosting the tools and providing necessary the blockchain, we create a transaction that identifies a licenses for our users, which re-iterates one of the motivating new thread of transactions. We call this an asset (running reasons for developing the Vault in the first place. design flow) that we wish to track. Once the asset has Acknowledgments. This work was funded by DARPA under contract been registered, transactions can be written against it. #HR0011-16-C-0043 Repository and Workflows for Accelerating Circuit Realization (RACE). • Transaction: records a transaction on the blockchain using the address to the asset, registering a file hash and R EFERENCES metadata about the file being used. The file hash is a [1] “Cadence VCAD,” https://www.cadence.com/content/cadence-www/ fingerprint (i.e. a SHA2 hash) of the contents of the file global/en US/home/services/vcad-services.html. being recorded. [2] “DARPA Circuit Realization at Faster Timescales (CRAFT) program,” • End Session: ends the session being tracked. Once a https://www.darpa.mil/program/circuit-realization-at-faster-timescales. [3] A. Brinckman et al., “Collaborative circuit designs using the craft session has ended, it cannot be reopened, but a new repository,” Future Generation Computer Systems, vol. 94, pp. 841–853, session can be started. 2019. [4] “Ericom,” https://www.ericom.com. Blockchain: The data Interface interacts with: [5] “AWS GovCloud,” https://aws.amazon.com/govcloud-us. 6 11th International Workshop on Science Gateways (IWSG 2019), 12-14 June 2019 [6] M. Swan, Blockchain: Blueprint for a new economy. ” O’Reilly Media, Inc.”, 2015. [7] G. Wood, “Ethereum: A secure decentralised generalised transaction ledger,” Ethereum project yellow paper, vol. 151, pp. 1–32, 2014. [8] J. Benet, “Ipfs-content addressed, versioned, p2p file system,” arXiv preprint arXiv:1407.3561, 2014. [9] “MongoDB,” https://www.mongodb.com. [10] “Swagger OpenAPI Specification,” https://swagger.io/specification. [11] “Express.js,” https://expressjs.com. 7