Quantum computing in the NISQ era requires powerful tools to reduce the gap between simulations and quantum hardware execution. In this work, we present a machine learning approach for reproducing the noise of a specific quantum device during simulations. The proposed algorithm is meant to be more flexible, in reproducing diferent noise conditions, than standard techniques like randomized benchmarking or heuristic noise models. This model has been tested both with simulation and on real superconducting qubits.
CEUR ceur-ws.org
Noise Intermediate Scale Quantum (NISQ) [
Reinforcement Learning (RL) is a powerful paradigm in machine learning that involves the training of an agent to make optimal decisions in a dynamic environment. It relies on the fundamental concepts of policy and reward functions. The policy determines the behavior of the agent, mapping the states of the environment to actions. The reward function assigns a numeric value to state-action pairs, indicating the immediate desirability or cost associated with them. Training the algorithm involves finding an optimal policy that maximizes the expected longterm cumulative reward. During training, diferent episodes of agent-environment interaction are executed. At the end of each episode the reward is used to update the weights of the policy, which is often approximated by a Neural Network (NN). In recent years diferent optimization methods have been developed to improve RL convergence and stability during training [23]. In our work we have obtained the best results using the proximal policy optimization (PPO) [24]. Noise is a crucial challenge faced in quantum circuits as qubits are susceptible to environment interaction, spontaneous emission or calibration errors. Many mathematical tools have been developed in order to describe quantum noise [25, 26, 27]. In this work we will model quantum noise using the error channels illustrated in the following.
The local depolarizing channel is a simple model for incoherent noise. It is characterized by a parameter that tends to bring the state to the maximally mixed state:
() = (1 − ) + ⋅ /2 The amplitude damping channel models physical processes occurring on the qubits that involve energy dissipation, such us spontaneous emission. Amplitude damping describes a decay process where the state |0⟩ is conserved, while |1⟩ decays to |0⟩ with probability : ( )
|1⟩ = (1 − ) |1⟩ + |0⟩ Single-qubit coherent errors can be represented with rotation gates ( , , ). Although these errors can be corrected once identified, in the NISQ era they are more challenging to address since they accumulate after successive gate executions, leading to a bias in the output of the quantum circuit. In order to obtain a simple modeling of the noise, it is possible to use a technique called randomized benchmarking (RB) [28, 29, 30]. RB allows to estimate the magnitude of the average error of a set of quantum gates. The noise model obtained in this way is unrealistic as all noise sources are projected on the depolarizing channel. However, this technique allows for estimating the average gate fidelity and can be used as a basic benchmark for other, more sophisticated, noise characterization techniques.
All the datasets used in this work are composed of ensembles of random quantum circuits with their relative density matrices (DM). DMs are used as ground truth labels during the training of the algorithm. They are analytically computed in simulation, whereas they are extracted using quantum state tomography [31] when circuits are run on hardware. All the circuits in the (1) (2) datasets have been generated by extracting random gates among a set of native gates. For this work, we have used the native gates set of the Technology Innovation Institute (TII) quantum hardware, composed of the , and gates. In order to train the RL agent it is necessary to translate the quantum circuit formalism to a tensor representation that can be fed to the policy neural network. We will call this array the quantum circuit representation (QCR). In the general case the shape of the QCR is: [, ℎ, _] . The first entrance of the QCR identifies the circuit qubit, while the second specifies the circuit moment. The third entrance, _ , defines the dimension of the latent space used to embed the information about gates and noise channels that act on a specific qubit at a circuit moment. For the training the following steps are performed in this order. A noiseless, quantum circuit representation is given to the agent at the beginning of each episode. For every circuit moment the agent performs an observation of the circuit and takes an action that consists in inserting any combination of the aforementioned noise channels with a chosen parameter. At the end of an episode, the DM of the circuit obtained with this process is computed (generated DM). The generated DM is compared with the real noisy DM, produced by the noise we want to model, and their fidelity is used to compute the final reward. After many episodes the agent should learn, in principle, where to insert noise channels in a non-noisy circuit in order to reconstruct the DM of the real noisy circuit. Once trained, the agent should be able to generalize to new unseen circuits and could be used to perform realistic noisy simulations.
We tested the proposed algorithm for learning both a simulated noise and the real noise of a quantum superconducting chip. In the first case, we have tested the model on simulated circuits of one and three qubits with a custom noise model. This noise model uses all the errors introduced in section 2. In order to train the RL agent, 500 random circuits of depth 7 have been generated: 400 circuits for the training set and 100 for the test set. Figure 1 reports the DM fidelity and trace distance obtained during training for both a single qubit and three qubits circuits. The standard deviation over our train and test sets is represented by error bars. In both cases, the agent is able to learn the simulated noise, no overfit is observed. Convergence is reached after ∼ 4 ⋅ 105 and ∼ 1.5 ⋅ 106 time steps respectively, where the standard deviation of the fidelity begins to shrink. In order to test the generalization properties of the model, we tested our agent on circuits with depth spanning from 3 to 30. Figure 2 reports the comparison with the RB model under diferent metrics. The agent is capable of generalizing to both longer and shorter circuits and always provides more accurate results compared to RB. This means that, as RB considers all the noise sources as depolarizing, our algorithm is able to identify the specific features of the noise.
To test the algorithm on real quantum hardware, we have used a single qubit superconducting transmon [32] chip available at the Technology Innovation Institute (TII) of Abu Dhabi. Figure 3 reports the average DM fidelity and trace distance obtained during the training. Even though the noise of a real quantum chip is far more complex, the agent is still able to learn the noise. We are currently working on testing the algorithm on real three qubits chips.
We presented a RL algorithm to replicate, both, a simulated and a real quantum chip noise, which provided better results than RB. The applications of the proposed algorithm extends beyond noise characterization. By learning the error pattern associated to specific qubit-gate combinations, the model could be used to optimize the transpiling process [33], preferring the execution of gates on less noisy qubits during circuit routing. Moreover, we aim at investigating if a similar, but inverse, strategy could be used to perform error mitigation. The scaling of the model to circuits with many qubits is the current biggest limitation as state tomography is unfeasible for many qubits. A potential solution could be training the model to approximate the statistics of measurement outcomes, or using classical shadow state reconstruction [34, 35]. The neural networks used in this work do not require high computational power, a single policy NN has about 104 parameters. This opens the possibility of training an ensemble of networks that operate in parallel on subsets of qubits from a larger chip. //link.aps.org/doi/10.1103/PhysRevA.86.032324. doi:10.1103/PhysRevA.86.032324. [12] B. Simone, G. Stefano, Convolutional neural network based decoders for surface codes, Quantum Information Processing 22 (2023) 151. URL: https://doi.org/10.1007/ s11128-023-03898-2. doi:10.1007/s11128-023-03898-2. [13] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, S. Lloyd, Quantum machine learning, Nature 549 (2017) 195–202. URL: https://doi.org/10.1038%2Fnature23474. doi:10. 1038/nature23474. [14] J. Choi, J. Kim, A tutorial on quantum approximate optimization algorithm (qaoa): Fundamentals and applications, in: 2019 International Conference on Information and Communication Technology Convergence (ICTC), 2019, pp. 138–142. doi:10.1109/ICTC46691. 2019.8939749. [15] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, P. J. Coles, Variational quantum algorithms, Nature Reviews Physics 3 (2021) 625–644. URL: https://doi.org/10.1038%2Fs42254-021-00348-9. doi:10.1038/s42254-021-00348-9. [16] S. Bordoni, D. Stanev, T. Santantonio, S. Giagu, Long-lived particles anomaly detection with parametrized quantum circuits, Particles 6 (2023) 297–311. URL: https://www.mdpi. com/2571-712X/6/1/16. doi:10.3390/particles6010016. [17] A. Zlokapa, A. Gheorghiu, A deep learning model for noise prediction on near-term quantum devices, 2020. arXiv:2005.10811. [18] K. Georgopoulos, C. Emary, P. Zuliani, Modeling and simulating the noisy behavior of near-term quantum computers, Phys. Rev. A 104 (2021) 062432. URL: https://link.aps.org/ doi/10.1103/PhysRevA.104.062432. doi:10.1103/PhysRevA.104.062432. [19] R. Harper, S. T. Flammia, J. J. Wallman, Eficient learning of quantum noise, Nature Physics 16 (2020) 1184–1188. URL: https://doi.org/10.1038%2Fs41567-020-0992-8. doi:10. 1038/s41567-020-0992-8. [20] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018. [21] L. P. Kaelbling, M. L. Littman, A. W. Moore, Reinforcement learning: A survey, Journal of artificial intelligence research 4 (1996) 237–285. [22] M. A. Wiering, M. Van Otterlo, Reinforcement learning, Adaptation, learning, and optimization 12 (2012) 729. [23] R. Özalp, N. K. Varol, B. Taşci, A. Uçar, A Review of Deep Reinforcement Learning Algorithms and Comparative Results on Inverted Pendulum System, Springer International Publishing, Cham, 2020, pp. 237–256. URL: https://doi.org/10.1007/978-3-030-49724-8_10. doi:10.1007/978-3-030-49724-8_10. [24] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, 2017. arXiv:1707.06347. [25] E. C. G. Sudarshan, P. M. Mathews, J. Rau, Stochastic dynamics of quantum-mechanical systems, Phys. Rev. 121 (1961) 920–924. URL: https://link.aps.org/doi/10.1103/PhysRev.121. 920. doi:10.1103/PhysRev.121.920. [26] M. A. Nielsen, I. L. Chuang, Quantum Computation and Quantum Information: 10th
Anniversary Edition, Cambridge University Press, 2010. doi:10.1017/CBO9780511976667. [27] Y. I. Bogdanov, A. Y. Chernyavskiy, A. S. Holevo, V. F. Lukichev, A. A. Orlikovsky, Mathematical models of quantum noise, in: A. A. Orlikovsky (Ed.), International Conference Micro