=Paper= {{Paper |id=Vol-2023/53-60-paper-8 |storemode=property |title=The Trigger Readout Electronics for the Phase-1 Upgrade of the ATLAS Liquid-Argon Calorimeters |pdfUrl=https://ceur-ws.org/Vol-2023/53-60-paper-8.pdf |volume=Vol-2023 |authors=Robert Wolff }} ==The Trigger Readout Electronics for the Phase-1 Upgrade of the ATLAS Liquid-Argon Calorimeters== https://ceur-ws.org/Vol-2023/53-60-paper-8.pdf

Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

THE TRIGGER READOUT ELECTRONICS FOR THE
PHASE-1 UPGRADE OF THE ATLAS LIQUID-ARGON
CALORIMETERS
Robert Wolffa
on behalf of the ATLAS Liquid Argon Calorimeter group
CPPM, Aix-Marseille Université, CNRS/IN2P3, Marseille, France

E-mail: a robert.wolff@cern.ch

The upgrade of the Large Hadron Collider, scheduled for 2019-2020, will increase the instantaneous
luminosity by more than three, hence the ATLAS trigger rates. To cope with this increase, the trigger
signals from the ATLAS Liquid Argon Calorimeter will be rearranged in 34000 so-called super cells
to get a 5 to 10 times finer granularity. This will improve the background rejection performance
through more precise energy measurements and the use of shower shape information to discriminate
electrons, photons and hadronically decaying tau leptons from jets. The new system will process the
super cell signal at 40 MHz and with 12 bit precision. The data will be transmitted at 5.12 Gb/s to the
back-end system using a custom serializer and optical transmitter. To verify full functionality, a
demonstrator set-up has been installed on the ATLAS detector and operated during the LHC Run 2.
This document gives a status on hardware developments towards the final design readout system,
including the performance of the newly developed ASICs. Their radiation tolerance, the performance
of the prototype boards, results of the high-speed link test with the prototypes and the performance of
the demonstrator with collision data are also reported.

Keywords: ATLAS, Calorimeter, LAr, Liquid Argon, Phase-1 upgrade

© 2017 CERN for the benefit of the ATLAS Collaboration.
Reproduction of this article or parts of it is allowed as specified in the CC-BY-4.0 license.

53
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

1. Introduction
1.1. The ATLAS experiment
The ATLAS experiment [1] is a multi-purpose high energy physics experiment at the Large
Hadron Collider (LHC) with the goal to precisely measure the Standard Model of particle physics
(SM) and its extensions. The ATLAS detector consists of sensor layers surrounding the collision
point: an inner detector made of pixel, strip and transition radiation detectors, followed by
electromagnetic and hadronic calorimeters, and then, at the outermost, the muon chambers. Due to a
solenoid and toroidal magnet system of magnetic fields of up to 2 T, momenta of charged particles,
e.g. muons can be precisely measured. More than 80 million channels are read out for each triggered
event. Proton bunches collide in the experiment at a rate of 40 MHz with an instantaneous luminosity
of up to 1.74 × 10−34 cm−2s−1 and on average 25 interactions per bunch crossing (<μ> = 25). The
ATLAS data acquisition is selecting proton-proton collision events using a two-step trigger system.
The first level trigger (L1) selects events based on information from coarse-granularity calorimeters
and muon spectrometers with a limited bandwidth of 100 kHz and a fixed latency of less than 3 μs.
The high level trigger (HLT) further decreases the event rate further down to about 600 Hz to
1.5 kHz using inputs from the whole detector with full granularity.
1.2. The Liquid Argon Calorimeter
The Liquid Argon Calorimeter (LAr) is a sampling calorimeter with liquid argon as the
active medium. Its scheme is shown in Figure 1. In the electromagnetic barrel and end-caps (EMB
and EMEC) it consists of accordion shaped lead absorbers and copper/kapton electrodes. The
hadronic end-caps (HEC) use copper absorbers and copper/kapton electrodes and the forward
calorimeter (FCAL) involves copper absorbers in the electromagnetic section and tungsten absorbers
in the hadronic section. The LAr Calorimeter plays a crucial role in the electron and photon
reconstruction as well as jet identification and missing transverse energy measurement. It consists of
182000 readout channels with 1600 front-end boards (FEBs) and 200 readout driver boards (RODs)
at back-end.

The LAr Calorimeter is currently read out at two stages, which are the regular and the trigger
readout. In the regular readout the calorimeter cell signals are amplified, shaped and sampled in the
FEBs for each bunch-crossing at 40 MHz and digitised and transmitted upon a L1 trigger decision at
100 kHz. The trigger readout uses a reduced granularity by summing signals from calorimeter cells
on the Layer Sum Boards (LSBs) and on Tower Builder Boards (TBB) to form analog trigger tower
signals, which are sent off-detector to the L1 Calorimeter Trigger System (L1Calo) for each bunch-
crossing at 40 MHz.

54
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

1.3. The Phase-1 upgrade
The plan for LHC and high-luminosity LHC (HL-LHC) is presented in Figure 2. The
currently on-going Run 2 will be followed by the long shut-down 2 (LS2) during 2019/2020, in
which the Phase-1 upgrade system will be installed. The Phase-2 upgrade is foreseen for 2024–2026
during LS3.

Figure 2. LHC baseline plan for the next decade and beyond showing the energy of the collisions
(red) and luminosity (green) [2]
The Phase-1 upgrade prepares for a twice as current high instantaneous luminosity of up to
3 × 10−34 cm−2s−1 and <μ> = 80. The lepton trigger rates increase linearly and the forward-jet and
missing transverse energy trigger rates increase exponentially with luminosity and number of
interactions per bunch crossing. The EM total trigger rate has a limited 20 kHz of the 100 kHz global
bandwidth. As increasing the energy thresholds to maintain the limited bandwidth cuts out interesting
physics, it is planned to improve the L1 trigger inputs by a higher granularity in the LAr calorimeter
readout. Therefore the LAr trigger readout electronics are going to be upgraded during LS2.
The Phase-2 upgrade prepares for the HL-LHC with instantaneous luminosity of about
7.5 × 10−34 cm−2s−1 and <μ> = 200. The maximum hardware trigger rate is planned to be increased to
1 MHz. The program foresees the replacement of the current LAr main readout electronics.
1.4. The Phase-1 LAr Calorimeter readout electronics upgrade
The current L1 trigger system uses trigger towers with a 0.1 × 0.1 pseudo-rapidity × polar
angle size in the EMB as input from the LAr Calorimeter. The Phase-1 upgrade of the LAr
Calorimeter readout electronics [3] foresees an increase of granularity by a factor of 5 to 10, resulting
in the so-called super cells. Longitudinal shower information is added by separating the four layers in
the new readout format. This is illustrated for a shower of an electron in Figure 3. New discriminant
variables can improve to distinguish interesting physics objects as electrons and hadronic decaying
tau leptons from jets at the L1 trigger level.

Figure 3. Electromagnetic shower of a simulated electron with a transverse energy of 70 GeV as seen
by the existing L1Calo trigger electronics (left) and by the upgraded trigger electronics (right) [3]
A schematic diagram of the Phase-1 LAr Calorimeter readout electronics upgrade is shown
in Figure 4 with the new components indicated by red outlines and arrows.

55
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

Figure 4. Schematic block diagram of the Phase-1 upgrade LAr trigger readout architecture. The new
components are indicated by the red outlines and arrows [3]
To perform the analog sums to the super cell signals the current LSBs will be replaced. New
Base-planes are needed to keep the compatibility with the existing set-up and to route the new super
cell signals, which brings about ten times more signal lines.
The LAr Trigger Digitizer Boards (LTDBs) receive, digitize and send the super cell signals
to the off-detector back-end system, the so-called LAr Digital Processing System (LDPS). It
calculates the transverse energy of each super cell and various sums of super cells at fixed latency
and sends the results continuously at 40 MHz to the L1Calo feature extractors.
The old analog trigger path using the trigger towers from the TBBs will remain for backup
while commissioning in Run 3. It is planned to be removed at Phase-2 upgrade.

2. The front-end electronics upgrade
2.1. The base-plane, the FEBs and the LSBs
The connectivity, cross-talk and noise performance has been verified for the standard base-
planes for EMB and EMEC with prototypes and the production has started. The development of the
special base-planes for EMEC, HEC and FCAL is progressing. The procurement of common parts
such as ground springs and alignment pins is ongoing. The FEBs will remain unchanged until Phase-
2, but the LSBs will be replaced to comply with super cell signal summing. Their production is
ongoing.
2.2. The LTDB
There are a total of 124 LTDBs reading 34 thousand super cells to be installed. Each LTDB
reads up to 320 super cells. It digitizes analog signals with 12 bits at 40 MHz using custom analog-
to-digital converters (ADCs). The digital signals are transmitted using 40 optical links at 5.12 Gb/s
with custom application-specific integrated circuits (ASICs). It has 5 GigaBit Transceiver (GBT)
serializer-de-serializer links for trigger, timing and control (TTC) signals. Its power distribution
board (PDB) has to comply with the total power consumption of around 125 W.

56
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

The whole system is on the detector and thus, all components need to be radiation-tolerant. It
has full compatibility with the Phase-2 upgrade. The final design has been fixed and prototypes are
being produced.
2.3. The LTDB custom ADCs
Each LTDB requires 80 custom ADCs that continuously process the sampling and
digitization of four super cell signals at 40 MHz. The power dissipation is less than 50 mW per
channel and the latency must be less than 200 ns. The dynamic range is 11.7 bits per sample.
The Nevis13 ADC fulfils all requirements. Its layout is based on a 130 nm IBM CMOS 8RF
of 3.6 mm × 3.6 mm with 72 quad-flat no-leads (QFN) pins. It uses four multiplying digital-to-
analog converters (MDACs) for the most significant bits and one successive approximation ADC
(SAR) for the lower 8 bits. It is tested for radiation up to 10 Mrad which corresponds to 100 times
more than currently expected at HL-LHC.
2.4. The LTDB optical links
Two kinds ASICs for optical links on the LTDB are developed. The serializer (LOCx2) is
based on a 250 nm silicon-on-sapphire technology with a die of 6.0 mm × 3.7 mm and 100 QFN
pins. Its output is at 5.12 Gb/s at a latency of less than 75 ns. It has about 1 W of power consumption.
The laser driver (LOCld) uses the same technology as the LOCx2 on a die of 2.1 mm × 1.1 mm and
40 QFN pins. It is a dual-channel vertical-cavity surface-emitting laser (VCSEL) driver.
Both ASICs have been tested on radiation-tolerance. Only few change in the output eye
diagrams has been observed after about 200 kHz. The wafers are produced and tests on the LTDB
prototypes are ongoing.

3. The back-end electronics upgrade
3.1. The LDPS
The LDPS receives the digital 12-bit data from the front-end system. It calculates the
transverse energy of each super cell and various sums of super cells at fixed latency and transmits the
results at 40 MHz to the L1Calo feature extractors. Additionally it buffers the calculated results for a
readout upon L1 trigger decision for debugging and monitoring. Because the back-end LDPS is off-
detector, there is no need to be radiation-tolerant as the front-end system.
Its main component are about 30 LAr Digital Processing Blades (LDPBs) which read the
outputs of 124 LTDBs. Each LDPB consists of one LAr carrier board (LArC) with four advanced
mezzanine cards (AMCs) and a rear transition module (RTM). It is a custom advanced tele-
communications computing architecture (ATCA) board and uses a Xilinx Virtex7 FPGA. It carries
four AMCs and drives the communication to the FELIX system [4] and the data acquisition for the
data monitoring. The AMCs have 48 input fibers at 5.12 Gb/s and the same amount of output fibers
at 11.2 Gb/s. They are called LATOME, acronym for LAr trigger processing mezzanine.
3.2. The LATOME
The LATOME receives super cell data from the LTDBs at 5.12 Gb/s on up to 48 optical
links. It is responsible for the computation of transverse energies of super cells and sums of super
cells using optimal filtering algorithm (OF) [5] at fixed latency. It assigns bunch crossings by timing
measurement. Finally it sends the data at 11.2 Gb/s on up to 48 optical links to the L1 trigger.
Furthermore it monitors the data and sends report to the DAQ system upon request. In Figure 5 a
picture of the LATOME board prototype is shown.

57
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

Figure 5. Picture of the LATOME board
Its heart is an Intel Arria10 FPGA. First prototypes have been successfully tested and
integrated. The validated features include the optical links up to 11.2 Gb/s, 1 Gb Ethernet and GBT.
The system test is ongoing to check all functionalities and the communication of the LDPB with
other systems.
The firmware of the LATOME is build in a modular way and consists of following parts. The
input stage treats the reception of the ADC data at 40 MHz and aligns the input fibers to the same
time reference (TTC). The configurable remapping matches the input channels to the detector
geometry. The user code computes the transverse energy at the correct bunch crossing time using
OF. The output summing calculates sums of super cell transverse energies for various L1Calo feature
extractors and sends the data to L1 trigger system at 40 MHz. It has a latency of less than 375 ns and
buffers ADC data and energies for at least 2.5 μs, which allows to monitor at the L1 trigger rate of
100 kHz. All LATOME firmware blocks have been developed and are under test. The overall system
test on the hardware is ongoing.

4. The LAr demonstrator
The LAr demonstrator is a pre-prototype of the calorimeter readout of the L1 trigger
processors installed in 2014 during LS1 at the ATLAS experiment. It covers 3.1 % of EMB and is
located in pseudo-rapidity 0 < η < 1.4 and azimuthal angle 1.77 < φ < 2.16. Before its installation no
disturbance to the current system has been verified. It validates the energy reconstruction and bunch-
crossing identification development. The system is successfully calibrated and data from proton-
proton (and heavy-ion) collisions is taken during the on-going Run 2. Two LTDB prototypes provide
the analog summing of calorimeter cell signals to super cells at the rate of 40 MHz. The AMC
prototypes are the so-called ABBA (ATCA board for a baseline of liquid argon acquisition) boards. It
is planned to replace the two LTDBs and the LDPS by final prototypes in early 2018.
4.1. The LTDB prototypes
The two LTDBs handle each up to 320 super cell signals (284 super cells in EMB). The super
cell signals are digitized with a commercial, not radiation-tolerant 12 bit ADC (TI ADS5272). On
one 4.8 Gb/s optical link eight super cell signals are multiplexed. Two prototypes with different
technology have been developed by several institutes. The first one uses an analog mezzanine and a
digital main board and the second one uses a digital mezzanine together with an analog main board.
Both versions are successfully operated during Run 2.
4.2. The ABBA boards
The ABBA board receives the digital signals of up to 320 super cells from one LTDB on up to
48 optical links at 4.8 Gb/s. It stores the ADC super cell data in circular buffers and sends it using the
IP-based protocol IPbus over user datagram protocol (UDP) on a 10 Gb Ethernet upon a L1 trigger
accept in the so-called “monitoring mode”. It contains three Intel (former Altera) Stratix4 FPGAs,
which is shown in Figure 6.

58
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

Figure 6. Picture of the ABBA board
In 2017 three ABBA boards are installed in the ATLAS counting room. The online software
is operational and read out in parallel with the ATLAS default readout since October 2015. It is
integrated in the automated ATLAS data acquisition since November 2016 with still separate data-
flow.
4.3. The LAr demonstrator operation
In calibration electronic pulses are sent by calibration board and measured by the
demonstrator readout. Figure 7 shows ADC pulse measurements and noise level for different super
cells. Good linearity has been observed up to DAC 8000, beyond that analog saturation is expected.
The noise level is well below 1 ADC count for all super cells and consistent with test bench
measurements.

Figure 7. LAr demonstrator calibration pulse shapes for a front layer super cell at φ = 1.82 and
η = 0.01 for different input DAC values (left) and noise level of super cells in ADC counts for
different layers (right) [6]
Since 2015 data is taken with proton-proton and heavy-ion collisions. Therefore a dedicated
topological L1 trigger item is required, which selects EM clusters in the LAr demonstrator region.
The collected data can then be analysed and compared with the events read out by the ATLAS main
readout. In Figure 8 an event display in the LAr demonstrator for a total energy of 124.4 GeV and the
correlation of energy measurements from the LAr demonstrator super cells and sums of calorimeter
cells from the ATLAS main readout are shown. A good agreement is observed between the two
readouts as shown by the linearity.

59
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017

Figure 8. LAr demonstrator proton-proton collision data: Event display of the partial demonstrator
region with a total energy of 124.4 GeV (left) and correlation of energy measurements for one super
cell at φ = 1.91 and η = 0.69 in the middle layer (right) [6]

5. Conclusion
The ATLAS LAr calorimeter electronics will be upgraded during the LS2 (2019-2020). The
trigger path will be digitized at the front-end level with increased granularity. New LTDB (front-end)
and LDPS (back-end) systems have been developed. The digitization and readout of this system will
be done at 40 MHz. Specific radiation tolerant ADCs and optical links have been designed and tested
for the LTDBs. The production will be started in 2018.
A demonstrator system has been installed and successfully run since 2015 for data-taking
from proton-proton and heavy-ion collisions. This gives valuable data to study the filtering algorithm
development for super cell energy measurement. It is planned to replace the LTDB and LDPS with
final prototypes in early 2018 to test the full pre-production readout chain with proton-proton
collision data in 2018. The Phase-1 upgrade is a stepping stone towards the full readout upgrade in
Phase-2.

References
[1] The ATLAS Collaboration. The ATLAS Experiment at the CERN Large Hadron Collider,
JINST 3, S08003 (2008), doi.org/10.1088/1748-0221/3/08/S08003, cds.cern.ch/record/1129811.
[2] G. Apollinari et al. High-Luminosity Large Hadron Collider (HL-LHC). Technical Design Report
V. 0.1, CERN Yellow Reports: Monographs, Vol. 4/2017, CERN-2017-007-M (2017), doi.org/
10.23731/CYRM-2017-004, cds.cern.ch/record/2284929.
[3] M. Aleksa et al. ATLAS Liquid Argon Calorimeter Phase-I Upgrade Technical Design Report,
LHCC 2013, CERN-LHCC-2013-017 (2013), cds.cern.ch/record/1602230.
[4] J. T. Anderson et al. FELIX: a PCIe based high-throughput approach for interfacing front-end and
trigger electronics in the ATLAS Upgrade framework, TWEPP 2016, JINST 11, C12023 (2016),
doi.org/10.1088/1748-0221/11/12/C12023, cds.cern.ch/record/2229597.
[5] W. E. Cleland and E. G. Stern. Signal processing considerations for liquid ionization calorimeters
in a high rate environment, Nucl. Instr. and Meth. Phys. Res., A338 467-497 (1994), doi.org/10.1016/
0168-9002(94)91332-3.
[6] ATLAS Liquid Argon Calorimeter group. Demonstrator for the ATLAS LAr calorimeter Phase I
Trigger Readout Upgrade, twiki.cern.ch/twiki/bin/view/AtlasPublic/LarCaloPublicResultsUpgrade.