=Paper=
{{Paper
|id=Vol-2507/170-174-paper-29
|storemode=property
|title=Commissioning the New Detector Interface for the ATLAS Trigger and Readout System
|pdfUrl=https://ceur-ws.org/Vol-2507/170-174-paper-29.pdf
|volume=Vol-2507
|authors=Nikolina Ilic
}}
==Commissioning the New Detector Interface for the ATLAS Trigger and Readout System==
https://ceur-ws.org/Vol-2507/170-174-paper-29.pdf
Proceedings of the 27th International Symposium Nuclear Electronics and Computing (NEC’2019)
Budva, Becici, Montenegro, September 30 – October 4, 2019
COMMISSIONING THE NEW DETECTOR INTERFACE FOR
THE ATLAS TRIGGER AND READOUT SYSTEM
N. Ilic 1,2 on behalf of the ATLAS TDAQ Collaboration
1
Institute of Particle Physics & University of Toronto, King's College Cir, Toronto, ON M5S, Canada
2
Radboud University Nijmegen, Houtlaan 4, 6525 XZ Nijmegen, Netherlands
Email: Nikolina.Ilic@cern.ch
After the current LHC shutdown (2019 - 2021), the ATLAS experiment will be required to operate in
an increasingly harsh collision environment. To maintain physics performance, the ATLAS
experiment will undergo a series of upgrades during the shutdown. A key goal of this upgrade is to
improve the capacity and flexibility of the detector readout system. To this end, the Front-End Link
eXchange (FELIX) system has been developed. FELIX acts as the interface between the data
acquisition; detector control and TTC (Timing, Trigger and Control) systems; and new or updated
trigger and detector front-end electronics. The system functions as an aggregator between custom
serial links from front end ASICs and FPGAs to data collection and processing components via a
commodity switched network. FELIX also forwards the LHC bunch-crossing clock, fixed latency
trigger accepts and resets received from the TTC system to front-end electronics. FELIX uses
commodity server technology in combination with FPGA-based PCIe I/O cards. FELIX servers run a
software routing platform serving data to network clients. These proceedings will cover the design of
FELIX and the results of the installation and commissioning activities for the full system in summer
2019.
Keywords: electronics, readout, upgrade, ATLAS
Nikolina Ilic
Copyright © 2019 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
170
Proceedings of the 27th International Symposium Nuclear Electronics and Computing (NEC’2019)
Budva, Becici, Montenegro, September 30 – October 4, 2019
1. Introduction
The ATLAS detector [1] is one of the two general purpose detectors located at the Large
Hadron Collider (LHC) at CERN. The LHC collides bunches of particles at a rate of 40 MHz. Since
ATLAS is capable of permanently storing data at a rate of 1.5 kHz, it uses a two-level trigger system
to identify which events to retain. The hardware-based Level-1 (L1) trigger reduces the 40 MHz
collision rate to a maximum of 100 kHz trigger rate. The L1 trigger makes its decision on the basis of
coarse sums of calorimeter energy deposits and of coarse transverse momentum measurements of
muon candidates to identify events with interesting signatures. The software-based High-Level Trigger
(HLT) further reduces the L1 rate to 1.5 kHz. The HLT decides which events to retain by using higher
granularity calorimeter and muon detector information, together with inner detector tracking
information.
2. Planned Upgrades
The ATLAS detector is undertaking the Phase I upgrade, during the Long Shutdown (LS2)
from 2019 to 2020 [2]. An upgrade to the Liquid Argon (LAr) electromagnetic calorimeter electronics
will allow for higher granularity energy measurements to be used by the L1 trigger [3]. The small
wheels equipped with muon chambers, installed in the forward direction, will be replaced by New
Small Wheels (NSWs) [4]. In addition to the NSW upgrades, additional resistive plate muon
chambers, called BIS78, will be installed in the transition region between the ATLAS barrel and
endcap. The L1 calorimeter trigger system will also be upgraded to utilize the improved
electromagnetic and muon measurements provided by the LS2 upgrades, resulting in improved
selectivity and background rejection.
The new electromagnetic calorimeter electronics, the NSW muon detectors, BIS78, and L1
calorimeter trigger electronics will require a new infrastructure for reading out data and sending
control information to the electronics. The new approach for the readout and control of the upgraded
systems, shown schematically in Figure 1, allows for a reduction of the number of different types of
custom electronics and protocols. The functionality that is currently provided by dedicated electronics
and firmware will be moved to software running on commodity servers. This is made possible by
advancements in computer technology and a radiation hard GigaBit Transceiver (GBT) link
technology, developed by CERN [7]. The GBT ASIC provide links in both directions. In addition to
transmitting the data from the Front end, GBT links will be used to configure the front-end electronics
and to deliver the Timing, Trigger and Control (TTC) [5] information to them. The multiplexing of
relatively slow links (80, 160 or 320 Mb/s per link) by the standard GBT protocol makes it possible to
transfer different independent data streams via the same physical optical link (raw transfer speed: 4.8
Gb/s). The slow serial electrical links are known as E- links.
The Front-End LInk eXchange (FELIX) will use the GBT protocol over optical links and
transceivers to route data and commands (including TTC) between electronics on the ATLAS detector
and commodity servers. FELIX will also use the GBT protocol to route control data (including TTC),
to nonradiation-hard electronics installed in an underground service area beside the ATLAS detector
(off-detector electronics). For transfers from off-detector electronics towards FELIX, a protocol called
FULL Mode is also provided and consists of simple 8B/10B encoding with a raw transfer speed of 9.6
Gb/s. In contrast to the pre-upgrade system, the functionality of the RODs and of the ROS will be
implemented by software running on commodity servers. The computers containing the Readout
Driver and Readout System (ROD/ROS) functionality are referred to as Software RODs (SW RODs).
The detector-specific data processing, which previously occurred on custom hardware RODs, will be
implemented in customizable Data Handler software applications on the SW RODs. After the upgrade,
commodity servers will continue to be used for the HLT; the control/configuration/monitoring system;
and DCS, these will communicate via a commodity Ethernet network.
FELIX is planned to be detector agnostic and will act as a router between the custom links and
a general commodity network switch. Due to this flexibility, the FELIX system is planned to be used
to read out all the ATLAS subdetectors after the Phase II upgrades [6]. The Phase II upgrade proj ect
171
Proceedings of the 27th International Symposium Nuclear Electronics and Computing (NEC’2019)
Budva, Becici, Montenegro, September 30 – October 4, 2019
with the largest number of optical readout links is the installation of a new tracker, called the Inner
Tracker (ITk) [8,9], whose links will be read out by FELIX.
Figure 1. The upgraded ATLAS readout system. Data is routed by a custom designed FELIX board
and software from the front-end detector electronics onto commercial computers. The commercial
computers contain software to perform the data processing in the Software ROD/ROS, as well as
detector control, monitoring, configuration, High-Level Trigger and Data Storage
3. The FELIX System
3.1. Hardware Production
The Phase I FELIX system will consist of FPGA-based PCIe cards and Network Interface
Cards (NICs) placed inside server PCs. The baseline choice of the FPGA card (Figure 2), known as
the FLX-712, has a 16-lane PCIe Gen3 interface, a Xilinx Kintex Ultrascale XCKU115 FPGA, 8
miniPODs interfacing to 48 bi-directional optical links, a TTC interface and a LEMO connector for a
“BUSY” signal output.
The server PCs that host the FELIX cards run Linux and are housed in a chassis that can be
mounted onto racks located in a shielded cavern separate from the ATLAS detector. Either one or two
FLX-712 cards will be installed per PC, each connecting to either 12, 24 or 48 links, depending on the
needs of the different types of front-end electronics. The server chosen to host the FELIX cards is the
Intel Xeon E5-1660 V4 CPU with Supermicro MBD- X10SRW-F motherboard and 32 GB of memory
in a 2U chassis. The server can be used with either a Dual-port 25 GbE or 100 GbE Mellanox
ConnectX- 5 EN Network Interface Card, depending on the bandwidth needed by the sub-detector.
The server chosen to run the software ROD applications is the Dual Intel(R) Gold 5218 CPU (16
cores), with Supermicro X10SRW-F motherboard and 96 GB of memory.
The first pre-production, consisting of 23 FLX -712 cards has been completed. All cards have
passed extensive testing including: thermal burn in at 85 degrees Celsius, X-ray check of soldering,
check of voltage values, FPGA, FLASH and microcontroller programming, eyescans up to bit error
rates of 10-9 and Bit Error Ratio better than 10-15, 8 hour functionality tests, slow control adapter
tests, trigger & timing tests, BUSY tests, jitter tests. The production of 120 FELIX cards needed for
the Phase I and early commissioning of Phase II systems will be launched in the Fall of 2019.
Figure 2. The FELIX card (FLX-712) showing eight miniPODs and the TTC mezzanine card
with the fan-cooled FPGA
172
Proceedings of the 27th International Symposium Nuclear Electronics and Computing (NEC’2019)
Budva, Becici, Montenegro, September 30 – October 4, 2019
3.2. FELIX Firmware and Protocols
A schematic diagram of the FELIX firmware is shown in Figure 3. The GBT Wrapper
firmware block, receives data from the front end of the ATLAS subdetector, and formats them for use
by the Central Router. The Central Router block maps the GBT data onto the format that corresponds
to individual detector electrical links (E-links). This block of the firmware also demultiplexes the GBT
link into data streams, encodes the data using 8b/10b or HDLC encoding and then forms 1 kbyte
blocks that are read by the DMA. The direction of data flow to the front end is similar. The TTC &
BUSY block of the firmware decodes the trigger data and clock that is received from the LHC and
forwards it to the central router, which then delivers it to the front end detector electronics though the
GBT Wrapper. The same block of firmware propagates BUSY signals from the front end to the
servers. The Wupper PCIe block pushes the data to the memory of the CPU of the server hosting the
FELIX cards. The data is then transferred to the software ROD.
Figure 3. A schematic representation of the FELIX firmware blocks
3.3. FELIX Software
The FelixCore application is a multi-threaded application running on the server PCs of the
FELIX system that controls the FPGA cards via PCIe registers. These are accessible via memory
mapped I/O, set up with the help of a dedicated driver. The control and status of the FPGA card is
handled via registers, while the data flow is handled via DMA. Data is transferred to circular buffers in
the FELIX PC under DMA control after which they are transferred to endpoints on the network. The
contiguous memory needed for the circular buffers is allocated by another driver.
For each E-link, FelixCore reconstructs complete chunks of the data written to the circular
buffers and forwards these to any destination that subscribes to that E-link. The data are forwarded via
the network, using NetIO, a network protocol-agnostic software layer. FelixCore also forwards data,
received from the network by means of NetIO, to the FPGAs.
3.4. Integration with Subdetectors
To facilitate the integration of the readout system with ATLAS subdetectors, various test
stands have been developed. In the test stands, emulated data is sent to the FELIX card, which then
routes it to the software ROD. The emulated data uses realistic traffic patterns, that include random
trigger rates and variable data sizes. Some test stands consist of real ATLAS detector readout chips
sending data to FELIX. The TTC signal is sent by custom electronics timing boards sitting in a VME
crate. A RasberryPi is used to send the trigger commands to the TTC signal that are then routed to the
FELIX. The RasberryPi is used to ensure flexibility of the trigger rate sent, since at the time of the test
stand integration the TTC systems could not be continuously operated at the several hundred 100 kHz
rates needed to stress test the system. These integration and performance tests demonstrated that the
FELIX system performed as expected.
173
Proceedings of the 27th International Symposium Nuclear Electronics and Computing (NEC’2019)
Budva, Becici, Montenegro, September 30 – October 4, 2019
4. Summary
The FELIX system is a detector interface for readout and control, encapsulating functionalities
common to all ATLAS subdetectors. FELIX will be initially deployed for the Phase I upgrades, which
include newer LAr calorimeter electronics, the New Small Wheel muon detectors, additional BIS78
muon chambers, and upgraded L1 calorimeter trigger electronics. The FELIX system will be expanded
to meet the requirements of the Phase II upgrade for the High Luminosity LHC. The FELIX boards
have passed the review for the large scale board production, which will start in Fall 2019.
References
[1] ATLAS Collaboration, The ATLAS Experiment at the CERN Large Hadron Collider, JINST 3
S08003 (2008).
[2] ATLAS Collaboration, Letter of Intent for the Phase-I Upgrade of the ATLAS Experiment, CERN-
LHCC-2011-012 (2011).
[3] ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Phase-I Upgrade Technical Design
Report, CERN-LHCC-2013-017 (2013).
[4] ATLAS Collaboration, New Small Wheel Technical Design Report, CERN- LHCC-2013-006
(2013).
[5] B. G. Taylor, Timing Distribution at the LHC, Proc. 8th Workshop on Electronics for LHC
Experiments, CERN-2002-003 (2002).
[6] ATLAS Collaboration, Technical Design Report for the Phase II Upgrade of the ATLAS Trigger
and Data Acquisition System, CERN-LHCC-2017-020 (2017).
[7] P. Moreira. et al. The GBT, a Proposed Architecture for Multi-Gb/s Data Transmission in High
Energy Physics, 10.5170/CERN-2007-007.332 (2007).
[8] ATLAS Collaboration, Technical Design Report for the ATLAS Inner Tracker Pixel Detector,
CERN-LHCC-2017-021 (2017)
[9] ATLAS Collaboration, Technical Design Report for the ATLAS Inner Tracker Strip Detector,
CERN-LHCC-2017-005 (2017).
174