<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>QSpark: Towards Reliable Qiskit Code Generation</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Kiana Kheiri</string-name>
          <email>kiana.Kheiri@torontomu.ca</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Aamna Aamir</string-name>
          <email>aamna.aamir@torontomu.ca</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andriy Miranskyy</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Chen Ding</string-name>
          <email>cding@torontomu.ca</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Quantum computing</institution>
          ,
          <addr-line>Qiskit, Large Language Models, Code generation, Reinforcement learning, Quantum</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Toronto Metropolitan University</institution>
          ,
          <addr-line>Toronto</addr-line>
          ,
          <country country="CA">Canada</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2026</year>
      </pub-date>
      <abstract>
        <p>Quantum circuits must be error-resilient, yet LLMs like Granite-20B-Code and StarCoder often output flawed Qiskit code. We fine-tuned the Qwen2.5-Coder-32B model with two RL methods, Group Relative Policy Optimization (GRPO) and Odds-Ratio Preference Optimization (ORPO), using a richly annotated synthetic dataset. On the Qiskit HumanEval benchmark, ORPO reaches 56.29% Pass@1 (≈ +10 pp over Granite-8B-QK) and GRPO hits 49%, both beating all general-purpose baselines; on the original HumanEval they score 65.90% and 63.00%. GRPO performs well on basic tasks (44/78) and excels on intermediate ones (41/68), but neither GRPO nor ORPO solves any of the ifve advanced tasks, highlighting clear gains yet room for progress in AI-assisted quantum programming.</p>
      </abstract>
      <kwd-group>
        <kwd>software engineering</kwd>
        <kwd>The code examples available for training are scarce compared to the vast repositories of classical code</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Imagine writing a quantum teleportation protocol with just a natural language prompt. Although
this might sound futuristic, recent advancements in AI bring us closer to such a reality. Quantum
computing has the potential to solve certain classes of problems faster than classical computing, but
programming quantum computers remains a specialized and challenging task [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Developing correct
and optimized quantum programs requires expertise in quantum mechanics, quantum algorithms,
and software engineering, a combination that is not widespread among developers. Even with
highlevel frameworks like IBM’s Qiskit [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], which provide libraries to design and run quantum circuits,
writing quantum code is often an error-prone process that requires careful handling of quantum-specific
concepts (for example, superposition, entanglement, non-cloning) and resource constraints. As quantum
hardware scales up, the complexity of the software needed to harness it also increases, calling for more
sophisticated development methodologies [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
      </p>
      <p>
        In recent years, researchers have begun to explore how advances in artificial intelligence, particularly
large language models (LLMs), can help make quantum programming more accessible and eficient
(see [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] and [5, Sec. 4.7] for review). Early work in this area, such as Cruz-Benito et al. [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ], demonstrated
that deep learning-based approaches could efectively provide customized assistance during the quantum
coding process, paving the way for more advanced AI-driven tools.
      </p>
      <p>Large language model (LLM) based coding assistants have already transformed classical software
development by providing code autocompletion, generation, and error detection. However, applying
these models to quantum programming presents unique challenges. Quantum programming uses a
distinct set of languages, libraries, and idioms (such as constructing quantum circuits gate by gate) that
difer significantly from classical programming [</p>
      <p>CEUR
Workshop</p>
      <p>
        ISSN1613-0073
in quantum software has been highlighted by the Quantum Software Engineering community [
        <xref ref-type="bibr" rid="ref5 ref7">5, 7</xref>
        ],
which argues that simply porting classical development techniques to the quantum realm is insuficient
due to fundamental diferences in how quantum programs operate.
      </p>
      <p>To bridge this gap, our work proposes a Qiskit-based quantum computing coding assistant, an
AI-driven tool designed to help developers write and refine quantum programs using the Qiskit SDK.
QSpark1 focuses specifically on Qiskit, IBM’s widely used quantum SDK, and supports tasks such as
circuit construction, optimization, and code debugging. We envision a system that can understand
high-level intentions (e.g., “prepare a Bell state” or “optimize this circuit section”) and provide
contextsensitive suggestions or code snippets, much like modern code assistants do for classical languages.
Using a large language model trained on quantum programming data, the assistant can generate Qiskit
code, recommend quantum algorithmic patterns, and catch common mistakes, all within the developer’s
workflow. The goal is to lower the barrier to entry for quantum programming and to accelerate the
development process for experts and beginners alike.</p>
      <p>In this paper, we detail the design of such a Qiskit-based coding assistant, discuss the training and
integration of the underlying LLM model, and evaluate its efectiveness in aiding quantum programmers.
We begin by reviewing related work in two key areas: quantum programming environments and
AIassisted coding tools for quantum software. This overview will contextualize our contributions and
highlight how our approach builds on recent advances. By merging the power of Qiskit and LLMs, we
aim to push the boundaries of developer tools in quantum computing, making quantum programming
not only more eficient but also more accessible to a broader audience.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Literature Review</title>
      <sec id="sec-2-1">
        <title>2.1. Quantum Programming Environments and Tools</title>
        <p>
          Several eforts have been made to create better software environments for developing quantum
applications. IBM’s Qiskit is one of the leading frameworks, providing an open source SDK with tools for
circuit design, simulation, and execution on quantum hardware [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ].
        </p>
        <p>
          In addition to Qiskit, practical resources such as the book by Silva [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ] provide essential hands-on
strategies for programming quantum rigs using Python, the quantum assembly language, and
cloudbased platforms, including IBM QExperience. This resource highlights the inherent challenges in
quantum programming and emphasizes the need for developer-friendly methodologies and tools.
        </p>
        <p>Researchers have also acknowledged that improving quantum software development requires
higherlevel abstractions and more systematic design approaches. For example, Ammermann et al. [9] introduce
a view-based development approach that unifies diverse stakeholder perspectives within a quantum
IDE. This model suggests that future quantum IDEs may provide synchronized views, such as algorithm
logic, circuit layout, and hardware mapping, to better manage the complexity of quantum programs.</p>
        <p>Another notable platform is QuantumPath, developed by Hevia et al. [10], which takes an
engineeringoriented approach to quantum software creation. QuantumPath provides an application lifecycle
management platform for quantum software, supporting developers from algorithm conception through
testing, deployment, and maintenance. Providing an ecosystem of modules and enforcing best practices,
it simplifies the development of hybrid quantum-classical solutions for real-world use.</p>
        <p>These eforts highlight a common theme: quantum software development requires more than just
programming libraries. It also needs robust tools and processes, similar to classical software engineering,
but tailored for the quantum domain. Our proposed Qiskit-based assistant complements these initiatives
by focusing on the coding phase of quantum development. It can be viewed as a plugin designed to
enhance quantum programming environments such as Qiskit, providing intelligent support during the
development process.
1To facilitate reproducibility, we release our full implementation at https://github.com/TMUDeV/QSPARK.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. AI-Assisted Quantum Code Generation</title>
        <p>
          With the growing use of AI in coding-related tasks, it is natural that researchers have started applying
LLMs to quantum programming. Dupuis et al. [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] introduced the Qiskit Code Assistant, an AI model
specifically trained to generate Qiskit code and help quantum programmers. Their research highlighted
the challenges in training a code model for quantum computing, such as understanding quantum gate
operations and circuit semantics, as well as the limited availability of training data. Despite these
challenges, their specialized model outperformed general-purpose code generation models on Qiskit
programming tasks.
        </p>
        <p>
          Similarly, Vishwakarma et al. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ] developed Qiskit HumanEval, a benchmark suite designed to
evaluate how well diferent LLMs can generate correct quantum code. Their results showed that
advanced LLMs, like GPT-style models, can produce executable quantum programs from prompts,
successfully passing many tests in the suite. This finding is encouraging, as it confirms the potential of
AI-assisted quantum coding and provides a benchmark for future improvements.
        </p>
        <p>Beyond Qiskit-focused tools, researchers have also explored broader applications of AI in quantum
algorithm design. For example, Liang et al. [11] examined how LLMs, such as GPT-4, can be used
to suggest viable quantum circuit structures. Their work suggests that AI can play a key role in
supporting the design of quantum architectures when guided appropriately. Similarly, Aragonés-Soria
and Oriol (2024) introduced C4Q, a specialized chatbot that uses pre-trained language models for
user request classification and then utilizes its own engine to generate accurate responses [ 12]. This
approach highlights the potential of AI to streamline the development of quantum algorithms. It also
demonstrates how such tools can make quantum computing more accessible to beginners by simplifying
the learning curve and coding process.</p>
        <p>These studies contribute to a growing consensus that generative AI can be a valuable tool in the
quantum software development process. Our work builds directly on these prior developments. In
particular, we utilize the insights from the Qiskit Code Assistant and the Qiskit HumanEval benchmark
to train and evaluate our coding assistant. While prior models have demonstrated the viability of
quantum code generation, our contribution lies in tightly integrating this model with the Qiskit user
experience by embedding it directly into IDE workflows and extending its capabilities with features
tailored to better support quantum developers. These features include recognizing when a qubit needs
to be measured or reset, suggesting circuit optimizations, and aligning with Qiskit’s latest API.</p>
        <p>
          By positioning QSpark within existing quantum programming environments and AI coding tools,
we aim to advance the support available to quantum computing developers. Ultimately, this work
contributes to the broader goal of quantum software engineering: to bring the productivity and reliability
benefits of modern software development to the field of quantum computing [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ], thereby accelerating
innovation and adoption.
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Methodology</title>
      <p>Our Qiskit-based code assistant is built upon the Qwen2.5-Coder-32B model, a 32-billion-parameter
large language model (LLM) specialized for code generation. It was selected for its strong performance
in both general-purpose programming and domain-specific reasoning. We fine-tune the model using a
curated dataset of Qiskit programs, detailed in the following subsections.</p>
      <sec id="sec-3-1">
        <title>3.1. Generation of Training Data</title>
        <p>To enable robust supervised and reinforcement learning, we construct a high-quality dataset, for training
purposes, comprising 522 Qiskit programming tasks. The data set was created through a multistage
pipeline that includes code retrieval, function extraction, annotation, validation, deduplication, and
formatting. The entire process is automated and designed to ensure consistency, reproducibility, and
broad coverage of real-world quantum programming challenges.</p>
        <p>We start by collecting approximately 10,819 Qiskit-related source code samples from public
repositories. Source files are parsed to extract quantum-relevant functions along with accompanying docstrings
and structural metadata. The extracted functions are filtered for completeness and relevance, and each
is assigned a unique task identifier.</p>
        <p>For each function, a natural language prompt is derived from its docstring or signature. This prompt
is paired with the corresponding canonical implementation, a designated function entry point, and a
dificulty score. The dificulty rating is calculated using a set of code-level features such as circuit depth,
gate complexity, use of measurement or entanglement, and algorithmic structure. This scoring system
enables the construction of a curriculum-aligned dataset that spans tasks ranging from basic quantum
operations to advanced algorithmic workflows (see Table 2).</p>
        <p>To ensure correctness, each solution is automatically validated through simulation-based unit tests.
These tests verify the functional behavior of the quantum circuit, including correct output shape, gate
behavior, and fidelity of simulation results. Tasks that fail validation are excluded from the final dataset
or flagged for manual inspection.</p>
        <p>To improve training diversity and reduce redundancy, we apply structural and semantic deduplication
techniques. Near-duplicate solutions or trivial variants are filtered out using syntactic similarity and
abstract syntax tree (AST) comparisons, ensuring a more diverse set of training signals. To illustrate the
ifltering process, approximately 10,819 raw Qiskit-related functions were initially collected, of which
fewer than 5% passed all validation steps. The majority of rejections were due to incomplete docstrings,
missing test coverage, or trivial circuits (e.g., functions that only returned an empty register). The final
curated set of 522 tasks was therefore deliberately biased toward code that was both executable and
semantically meaningful. Dificulty scores were computed automatically using heuristics: basic tasks
had depth ≤ 3 and no entanglement; intermediate tasks required either measurements or circuit depth
&gt; 3; and dificult tasks contained multi-qubit entanglement or hybrid classical-quantum structures.
Unlike the QHE benchmark, which evaluates generalization, this dataset was designed for training, and
thus prioritizes diversity and coverage across circuit patterns.</p>
        <p>Each finalized task in the dataset consists of:
• A unique identifier,
• A natural language task description,
• A validated Qiskit implementation,
Simple circuits with a few gates, no measurement, no entanglement.</p>
        <p>Circuits with measurements, moderate depth, or basic algorithmic structure.</p>
        <p>Complex circuits involving entanglement, variational algorithms, or multi-step workflows.</p>
        <p>Number of Tasks</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Reinforcement Learning with Preferences</title>
        <p>To further refine the behavior of the model, we employ two independent reinforcement learning
strategies: Group Relative Policy Optimization (GRPO) and Odds-Ratio Preference Optimization (ORPO),
each targeting a diferent aspect of quantum code quality.</p>
        <sec id="sec-3-2-1">
          <title>Odds-Ratio Preference Optimization (ORPO)</title>
          <p>ORPO aligns the model with human-like coding
preferences, focusing on readability and maintainability. It uses pairwise preference data where a
“chosen” response is preferred over a “rejected” one, based on manual review and synthetic annotations [13].</p>
          <p>The ORPO objective increases the likelihood of preferred output while regularizing the divergence
from the original (pre-trained) policy. The loss is defined as
ℒORPO = KL(  ‖ 0) −  log2
  ( ∣̂ )
.</p>
          <p>Here,   is the current policy,  0 is the pre-trained policy,  is the input prompt,  ̂ is the chosen output,
and  is the rejected one. The hyperparameter  controls the strength of the preference signal relative to
the regularization term. The term KL(  ‖ 0)represents the Kullback-Leibler (KL) divergence between
the current policy   and the pre-trained policy  0. For two discrete probability distributions  and  ,
the KL divergence is generally defined as:</p>
          <p>KL( ‖) =</p>
          <p>∑  () log (  ()</p>
          <p>)
()
In this context, it measures how much   deviates from  0, acting as a regularization to prevent the
current policy from straying too far from the original model’s capabilities.</p>
        </sec>
        <sec id="sec-3-2-2">
          <title>ORPO Reward Construction.</title>
          <p>Odds-Ratio Preference Optimization (ORPO) aligns the model with
human-like coding preferences, focusing on readability and maintainability. For each prompt, we
construct a pairwise comparison between a chosen output  ̂ and a rejected output  . The chosen output
is correct, executable, and stylistically aligned with Qiskit best practices. while the rejected output is
either synthetically perturbed or sampled from lower-quality generations.</p>
        </sec>
        <sec id="sec-3-2-3">
          <title>Group Relative Policy Optimization (GRPO)</title>
          <p>GRPO improves execution fidelity by ranking
outputs within a group of candidates generated for each prompt[14]. Each output   is evaluated using
Qiskit v.2.0.0 and Qiskit Aer simulations v. 0.17.1 and assigned a reward  (  ).  represents the number of
(1)
(2)

1
 =1
 =
∑  (  ),  =</p>
          <p>∑( (  ) − ) 2, (  ) =
√
1
 =1
 (  ) −</p>
          <p>The policy is updated using a clipped objective to ensure training stability:</p>
          <p>max(, min( ,  )) bounds   , and  (e.g., 0.1–0.2) sets the range [1 − , 1 +
] . This process guides the model toward producing more executable and resource-eficient quantum
circuits by emphasizing outputs that outperform others in the same generation group.
candidate outputs in a group generated for each prompt. The group mean  and the standard deviation
 are used to compute the normalized advantage:


GRPO Reward Construction. For each prompt, we generate a group of candidate completions.
Each candidate is executed with Qiskit v2.0.0 and Qiskit Aer v0.17.1 simulators. The execution
is scored using three criteria:
1. Unit test pass rate ( 1): fraction of test cases passed (primary correctness signal).
2. Circuit depth penalty ( 2): normalized inverse of circuit depth to reward more eficient solutions.
3. Qubit count penalty ( 3): normalized inverse of the number of qubits used, discouraging wasteful
allocations.</p>
          <p>These are combined into a single scalar reward:</p>
          <p>( ) =  ⋅  1 +  ⋅  2 +  ⋅  3,
with weights  = 0.7 ,  = 0.2 , and  = 0.1 chosen empirically to emphasize correctness while still
encouraging eficiency.</p>
          <p>Within each group, rewards are normalized using the group mean and variance:
(3)
(4)
(5)
(6)
(  ) =
 (  ) − 

1
 =1
,  =
∑  (  ),  =</p>
          <p>∑( (  ) − ) 2,
√</p>
          <p>1
 =1
where  is the number of candidates. This normalization ensures that the rewards are relative: a
candidate only receives a high advantage if it is better than its peers, even if all solutions are weak.
The policy update (Eq. 4) then uses this normalized advantage to push the model towards consistently
producing correct and eficient circuits.</p>
          <p>In practice, this setup allows GRPO to prefer structurally sound and resource-eficient quantum
circuits for simple tasks, while avoiding overfitting to a single absolute scoring heuristic.</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Results and Discussion</title>
      <sec id="sec-4-1">
        <title>4.1. Evaluation Setup</title>
        <p>
          We evaluated our models using the Qiskit HumanEval (QHE) benchmark introduced by Vishwakarma
et al. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ], which extends the original HumanEval benchmark to assess LLMs on quantum programming
tasks. Following the evaluation framework used in that work, we compare our GRPO and ORPO
ifne-tuned models to both general-purpose open-source LLMs and a specialized QHE-tuned baseline,
using the following key metrics:
        </p>
        <p>1. Pass@1 Accuracy: The percentage of completions that pass a unit test on the first attempt.
2. Performance by Dificulty Level: Evaluation of the accuracy of the model in the tasks labeled</p>
        <p>Basic, Intermediate, and Advanced.
3. General-Purpose vs. Domain-Specific Models: A comparison to understand the impact of
domain adaptation on performance.</p>
        <p>Since the original evaluation script was not publicly released and the HumanEval [15] framework was
incompatible with QHE tasks, we implemented a custom benchmarking script customized for the QHE
setting. Although the QHE paper reports 101 tasks, the publicly released dataset contains 151 entries.
This resulted in 78 Basic, 68 Intermediate, and 5 Advanced tasks. This script automatically executes
model completions against the associated unit tests and logs pass/fail outcomes, enabling consistent and
scalable evaluation across all models. Our results are therefore based on a fully automated, reproducible
evaluation pipeline that faithfully adheres to the QHE benchmark structure.</p>
        <p>
          We compared our GRPO and ORPO models with the following strong baseline models.
• General-Purpose Open-Source LLMs: These models, such as CodeLLaMA-34B [16], DeepSeek-33B
[17], StarCoder2-15B [18], and CodeGemma-7B [19], are large language models trained on vast
datasets of general programming code. They serve as a benchmark for how well unspecialized
models perform on quantum programming tasks.
• Granite-8B-Base[20]: This is a general-purpose base model. Its performance helps to understand
the impact of any quantum-specific fine-tuning.
• Granite-8B-QK (QHE-tuned baseline)[
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]: This model is a specialized version of Granite-8B,
finetuned specifically for the Qiskit HumanEval benchmark. It represents the state-of-the-art in
domain-adapted models for Qiskit code generation and provides a direct comparison to our
preference-based fine-tuning approaches.
        </p>
        <p>
          The results for the baseline models are taken from [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ].
        </p>
        <p>Fine-Tuning Hyperparameters: Table 3 summarizes the hyperparameter settings used during
ifne-tuning for both approaches that we ran on the A100 GPU with 80 GB for VRAM.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Results</title>
        <p>We evaluated the performance of our GRPO and ORPO models on the Qiskit HumanEval (QHE)
benchmark, comparing them to several strong general-purpose code LLMs and the QHE-tuned baseline.
Table 4 reports the Pass@1 accuracy on both HumanEval (HE) and QHE under greedy decoding.</p>
        <p>Our models achieve significant improvements over all baseline models in QHE. ORPO achieves
the highest Pass@1 accuracy at 56.29%, outperforming the domain-adapted Granite-8B-QK model by
nearly 10 percentage points. GRPO also performs competitively, achieving 49.00%, and surpasses all
general-purpose models. These results demonstrate the efectiveness of preference-based fine-tuning in
the quantum domain, even compared to models trained specifically for QHE.</p>
        <p>Interestingly, both GRPO and ORPO also show strong generalization on the original HumanEval
benchmark, with Pass@1 scores of 63.00% and 65.90%, respectively, outperforming larger models like
CodeLLaMA-34B and DeepSeek-33B. This suggests that preference optimization not only improves
performance on domain-specific tasks but also may enhance general code generation capabilities.</p>
        <p>To better understand the behavior of the model across the complexity of tasks, Table 5 presents pass
counts grouped by dificulty level. ORPO ranks third in basic-level tasks with 44/78 (slightly behind
Granite-8B-Code-QK at 32/54 and DeepSeek-Coder-33B at 30/54 in terms of completion percentage),
achieves the highest pass count on intermediate tasks (41/68), and outperforms others in total
completions. GRPO performs worse than ORPO but still surpasses many baseline models. Neither model
succeeds on the five advanced tasks, consistent with all other baselines.</p>
        <p>These results ofer complementary insights: GRPO appears to be more efective for simpler structurally
consistent circuits, benefiting from group-level ranking rewards, while ORPO demonstrates stronger
reasoning and robustness on moderately complex tasks due to its fine-grained preference alignment
objective.</p>
        <p>Both models perform on par with or exceed these reference rates, further validating their
practical utility. Overall, the results highlight the strength of preference-driven optimization in quantum
programming and emphasize the importance of evaluating across dificulty levels to capture nuanced
model capabilities.</p>
        <sec id="sec-4-2-1">
          <title>4.2.1. Training Dynamics</title>
          <p>Figure 1 illustrates the training dynamics of our preference-optimized models. The plot on the left
shows the reward trajectory for GRPO, while the plot on the right presents the loss curve for ORPO.</p>
          <p>In the GRPO setup (left), the model is optimized using group-based reinforcement signals derived
from task-specific XML output properties. The observed rewards display high variance throughout
training, a result of simulation-based reward assignment and the stochastic nature of quantum program
outputs. Despite fluctuations, the trend demonstrates that the model is able to consistently explore and
exploit high-reward completions. Importantly, this noisy, yet bounded, reward signal is characteristic
of preference-driven reinforcement learning in sparse-reward domains.</p>
          <p>The ORPO training curve (right) exhibits more stable and gradual convergence. The pairwise ranking
loss steadily decreases as the model learns to align its output with preference-labeled completions. The
initial sharp drop is followed by continued fine-tuning and refinement, reflecting efective optimization
using contrastive supervision. Huang et al. [21] provide theoretical insights into pairwise learning for
ranking, supporting our observation of a smooth convergence process under such loss formulations.
This steady progression contrasts with the volatility of GRPO and highlights the complementary
strengths of the two methods. GRPO encourages exploration and robustness through output diversity,
while ORPO guides the model toward aligning with desired behavior patterns.</p>
          <p>Together, these training signals validate the design of our preference learning pipeline. GRPO
encourages structural diversity and correctness in simpler tasks, while ORPO promotes nuanced
alignment and interpretability in more complex scenarios.</p>
        </sec>
      </sec>
      <sec id="sec-4-3">
        <title>4.3. Discussion</title>
        <p>Our updated evaluation highlights the efectiveness of reinforcement learning with preferences, through
GRPO and ORPO, for quantum code generation. Both models substantially outperform general-purpose
LLMs on the Qiskit HumanEval (QHE) benchmark under greedy decoding. ORPO achieves the highest
QHE pass@1 score at 56.29%, surpassing even the domain-specific Granite-8B-Code-QK baseline
(46.53%), while GRPO also delivers a strong 49.00%. These improvements were achieved without explicit
supervised instruction tuning or access to the original QHE fine-tuning scripts, emphasizing the strength
of our preference optimization pipeline. At the task dificulty level, ORPO consistently outperforms
GRPO.</p>
        <p>
          Interestingly, even the benchmark reference implementation, executed via the latest version of Qiskit,
only achieves 69/78 on Basic, 63/68 on Intermediate, and 2/5 on Advanced tasks. This highlights potential
fragility and version sensitivity in quantum execution environments. It also raises a critical point: our
models were tested under realistic run-time conditions and still matched or exceeded these reference
pass counts, reinforcing their practical applicability. Compared to previous work by Vishwakarma et
al. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ], our results are competitive despite several evaluation challenges. Although their paper reports
101 tasks, the public release contained 151 files. Furthermore, their evaluation script was not released.
To address this, we wrote our benchmark evaluation script and validated completions using the unit
tests provided, ensuring consistent and reproducible results.
        </p>
        <p>Finally, the persistent failure across all models, including ours and the benchmark baseline, on the five
advanced tasks (0/5) underscores the dificulty of complex quantum reasoning. These results suggest
that success in advanced quantum programming likely requires novel strategies, potentially involving
curriculum learning, richer supervision signals, or domain-specific memory mechanisms.</p>
        <p>Our findings validate reinforcement learning with preferences as a promising direction for quantum
LLMs. They also highlight the urgent need for standardized, version-controlled benchmarks and tooling
in quantum code generation research.</p>
      </sec>
      <sec id="sec-4-4">
        <title>4.4. Challenges and Ethical Considerations</title>
        <p>Although the integration of AI into quantum computing ofers significant promise, it also introduces
important challenges, both technical and ethical, that must be carefully considered [22].</p>
        <p>Lim et al. [23] and Hernandez and Patel [24] highlight the dual role of generative AI in education,
portraying it as both a transformative enabler and a potential threat to traditional learning paradigms.
This perspective aligns with our work, where reinforcement learning is used not to replace human
quantum programmers but to augment their workflows. Our use of preference optimization explicitly
reflects this balance: the goal is to guide AI-generated code toward human-aligned styles, best practices,
and interpretable solutions, rather than generate opaque or overly optimized outputs that lack usability.</p>
        <p>Ahmadi [25] explores the convergence of quantum computing and artificial intelligence, emphasizing
the revolutionary potential of this union in fields such as cryptography and optimization. However, he
also underscores several concerns, ranging from algorithmic reliability to ethical deployment, which
are directly relevant to our work. In particular, issues like execution fidelity, qubit resource constraints,
and reproducibility are amplified in the quantum domain, where small errors in AI-generated code can
lead to significant deviations in output. Our manual validation of test cases, necessitated by the lack of
standardized evaluation tools, further reflects the importance of transparency and accountability in
quantum AI development.</p>
        <p>Looking ahead, we advocate for the development of community-driven benchmarks, shared evaluation
pipelines, and stronger documentation practices. These are critical steps not only for reproducibility
but also for building trustworthy AI systems that can be safely and ethically deployed in quantum
research and education.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusion and Future Work</title>
      <p>In this work, we present a Qiskit-based quantum code assistant built on the Qwen2.5-Coder-32B
model, fine-tuned using reinforcement learning with preferences. By introducing Group Relative
Policy Optimization (GRPO) and Odds-Ratio Preference Optimization (ORPO), we explore how
domainaligned feedback can improve quantum code generation beyond conventional supervised fine-tuning.
Our models demonstrate competitive performance on the Qiskit HumanEval benchmark, particularly
excelling at Basic and Intermediate tasks, where they outperform several general-purpose LLMs. These
results underscore the promise of preference-based optimization for aligning large language models
with quantum programming best practices.</p>
      <p>However, this work also presents important challenges. We encountered inconsistencies in benchmark
releases, missing evaluation scripts, and had to manually run and validate test cases, which afected
reproducibility. Furthermore, none of the evaluated models, including ours, succeeded in the
Advancedlevel tasks, pointing to the need for better instruction tuning, longer-horizon reasoning, and deeper
integration with quantum hardware constraints. Although our results demonstrate the potential
for preference-based optimization for quantum code generation, there are several limitations to be
acknowledged. First, the training data remains relatively small compared to classical code datasets,
which may restrict generalization to novel quantum tasks. Second, due to the absence of an oficial
evaluation script and inconsistencies in the published benchmark, we relied on manual validation for
scoring, introducing potential subjectivity, and making direct comparisons to other models less precise.</p>
      <p>In future work, our aim is to:
• Integrate GRPO and ORPO into a unified reward framework [ 26].
• Develop sampling-based decoding strategies that align with human-in-the-loop workflows.
• Broaden the data set to encompass a wider range of quantum use cases, including error correction,
hybrid quantum-classical algorithms, and hardware-specific optimizations.</p>
      <p>In addition, we intend to work with a more comprehensive and clearly defined benchmark and
develop a robust, automated evaluation pipeline to support consistent testing and comparison across
models. We also advocate for the open release of standard evaluation tools to support fair benchmarking
and collaborative development in quantum LLM research.</p>
      <p>By addressing these open challenges, we hope to push the boundaries of AI-assisted quantum
programming, making it more accessible, reliable, and aligned with human intent.</p>
    </sec>
    <sec id="sec-6">
      <title>6. Acknowledgements</title>
      <p>This work is partially sponsored by Natural Science and Engineering Research Council of Canada
(grants # 2020-04760 and RGPIN-2022-03886).</p>
    </sec>
    <sec id="sec-7">
      <title>Declaration on Generative AI</title>
      <p>During the preparation of this work, the authors used ChatGPT in order to: Grammar and spelling
check, paraphrase, and reword. After using this tool, the authors reviewed and edited the content as
needed and take full responsibility for the content of the publication.
[9] J. Ammermann, W. Mauerer, I. Schaefer, Towards view-based development of quantum software,
2024. URL: https://arxiv.org/abs/2406.18363. arXiv:2406.18363.
[10] J. L. Hevia, G. Peterssen, M. Piattini, Quantumpath: A quantum software development platform,</p>
      <p>Software: Practice and Experience 52 (2022) 1517–1530.
[11] Z. Liang, J. Cheng, R. Yang, H. Ren, Z. Song, D. Wu, X. Qian, T. Li, Y. Shi, Unleashing the
potential of llms for quantum computing: A study in quantum architecture design, arXiv preprint
arXiv:2307.08191 (2023).
[12] Y. Aragonés-Soria, M. Oriol, C4q: A chatbot for quantum, in: Proceedings of the 5th ACM/IEEE
International Workshop on Quantum Software Engineering, Q-SE 2024, Association for Computing
Machinery, New York, NY, USA, 2024, p. 29–36. URL: https://doi.org/10.1145/3643667.3648222.
doi:10.1145/3643667.3648222.
[13] J. Hong, N. Lee, J. Thorne, Orpo: Monolithic preference optimization without reference model,
2024. URL: https://arxiv.org/abs/2403.07691. arXiv:2403.07691.
[14] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, D. Guo,
Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL:
https://arxiv.org/abs/2402.03300. arXiv:2402.03300.
[15] M. C. et al., Evaluating large language models trained on code, 2021. URL: https://arxiv.org/abs/
2107.03374. arXiv:2107.03374.
[16] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez,
J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A.
Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, G. Synnaeve, Code llama:
Open foundation models for code, 2024. URL: https://arxiv.org/abs/2308.12950. arXiv:2308.12950.
[17] DeepSeek-AI, Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,
2025. URL: https://arxiv.org/abs/2501.12948. arXiv:2501.12948.
[18] R. L. et al, Starcoder: may the source be with you!, 2023. URL: https://arxiv.org/abs/2305.06161.</p>
      <p>arXiv:2305.06161.
[19] C. Team, Codegemma: Open code models based on gemma, 2024. URL: https://arxiv.org/abs/2406.</p>
      <p>11409. arXiv:2406.11409.
[20] M. Mishra, M. Stallone, G. Zhang, Y. Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran,
S. Singh, M. Sethi, X.-H. Dang, P. Li, K.-L. Wu, S. Zawad, A. Coleman, M. White, M. Lewis,
R. Pavuluri, Y. Koyfman, B. Lublinsky, M. de Bayser, I. Abdelaziz, K. Basu, M. Agarwal, Y. Zhou,
C. Johnson, A. Goyal, H. Patel, Y. Shah, P. Zerfos, H. Ludwig, A. Munawar, M. Crouse, P. Kapanipathi,
S. Salaria, B. Calio, S. Wen, S. Seelam, B. Belgodere, C. Fonseca, A. Singhee, N. Desai, D. D. Cox,
R. Puri, R. Panda, Granite code models: A family of open foundation models for code intelligence,
2024. URL: https://arxiv.org/abs/2405.04324. arXiv:2405.04324.
[21] S. Huang, J. Zhou, H. Feng, D.-X. Zhou, Generalization analysis of pairwise learning for ranking
with deep neural networks, Neural Computation 35 (2023) 1135–1158. doi:10.1162/neco_a_01585.
[22] D. B. Rawat, C. Bajracharya, The intersection of quantum computing, ai, and cybersecurity:
Challenges and opportunities, in: 2024 IEEE 6th International Conference on Trust, Privacy
and Security in Intelligent Systems, and Applications (TPS-ISA), 2024, pp. 176–181. doi:10.1109/
TPS-ISA62245.2024.00029.
[23] W. M. Lim, A. Gunasekara, J. L. Pallant, J. I. Pallant, E. Pechenkina, Generative ai and the future
of education, The International Journal of Management Education (2023). URL: https://www.
sciencedirect.com/science/article/pii/S1472811723000289. doi:10.1016/j.ijme.2023.100790.
[24] K. Hernandez, T. Patel, Enhancing early quantum computing education with quantumaied:
Bridging the educational gap, in: Proceedings of the ACM SIGCSE, ACM, 2025, p. 1755. URL:
https://dl.acm.org/doi/10.1145/3641555.3705028.
[25] A. Ahmadi, Quantum computing and artificial intelligence: The synergy of two revolutionary
technologies, American Journal of Engineering and Science (AJES) (2023). URL: https://ajesjournal.
org/index.php/ajes/article/view/4118. doi:10.51983/ajes-2023.12.2.4118.
[26] J. Dai, M. O. Gluzman, Queueing network controls via deep reinforcement learning, ArXiv
abs/2008.01644 (2020). doi:10.1287/stsy.2021.0081.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>A.</given-names>
            <surname>Ajagekar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Humble</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>You</surname>
          </string-name>
          ,
          <article-title>Quantum computing based hybrid solution strategies for largescale discrete-continuous optimization problems</article-title>
          ,
          <source>Computers &amp; Chemical Engineering</source>
          <volume>132</volume>
          (
          <year>2019</year>
          )
          <article-title>106630</article-title>
          . doi:
          <volume>10</volume>
          .1016/j.compchemeng.
          <year>2019</year>
          .
          <volume>106630</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>A.</given-names>
            <surname>Javadi-Abhari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Treinish</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Krsulich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. J.</given-names>
            <surname>Wood</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Lishman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Gacon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Martiel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. D.</given-names>
            <surname>Nation</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. S.</given-names>
            <surname>Bishop</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. W.</given-names>
            <surname>Cross</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. R.</given-names>
            <surname>Johnson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Gambetta</surname>
          </string-name>
          ,
          <article-title>Quantum computing with qiskit, arXiv preprint (</article-title>
          <year>2024</year>
          ). URL: https://arxiv.org/abs/2405.08810. doi:
          <volume>10</volume>
          .48550/arXiv.2405.08810. arXiv:
          <volume>2405</volume>
          .
          <fpage>08810</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>N.</given-names>
            <surname>Dupuis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Buratti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Vishwakarma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. V.</given-names>
            <surname>Forrat</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kremer</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Faro</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Puri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Cruz-Benito</surname>
          </string-name>
          ,
          <article-title>Qiskit code assistant: Training llms for generating quantum computing code</article-title>
          ,
          <source>in: Proceedings of the IEEE LLM-Aided Design Workshop (LAD)</source>
          ,
          <year>2024</year>
          . URL: https://arxiv.org/abs/2405.19495. doi:
          <volume>10</volume>
          .1109/LAD62341.
          <year>2024</year>
          .
          <volume>10691762</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>S.</given-names>
            <surname>Vishwakarma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Harkins</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Golecha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V. S.</given-names>
            <surname>Bajpe</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Dupuis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Buratti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kremer</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Faro</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Puri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Cruz-Benito</surname>
          </string-name>
          ,
          <article-title>Qiskit humaneval: An evaluation benchmark for quantum code generative models</article-title>
          ,
          <source>in: Proceedings of the IEEE International Conference on Quantum Computing and Engineering (QCE)</source>
          ,
          <year>2024</year>
          , pp.
          <fpage>1169</fpage>
          -
          <lpage>1176</lpage>
          . doi:
          <volume>10</volume>
          .1109/QCE57912.
          <year>2024</year>
          .
          <volume>00001</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Murillo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Garcia-Alonso</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Moguel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Barzen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Leymann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ali</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Yue</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Arcaini</surname>
          </string-name>
          , R. PérezCastillo, I.
          <string-name>
            <surname>García-Rodríguez de Guzmán</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Piattini</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Ruiz-Cortés</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Brogi</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Zhao</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Miranskyy</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Wimmer</surname>
          </string-name>
          ,
          <article-title>Quantum software engineering: Roadmap and challenges ahead</article-title>
          ,
          <source>ACM Transactions on Software Engineering and Methodology</source>
          <volume>34</volume>
          (
          <year>2025</year>
          ). doi:
          <volume>10</volume>
          .1145/3712002.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>J.</given-names>
            <surname>Cruz-Benito</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Faro</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Martín-Fernández</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Therón</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F. J.</given-names>
            <surname>García-Peñalvo</surname>
          </string-name>
          ,
          <article-title>A deep-learning-based proposal to aid users in quantum computing programming</article-title>
          ,
          <source>in: International Conference on Learning and Collaboration Technologies</source>
          , Springer, Springer International Publishing,
          <year>2018</year>
          , pp.
          <fpage>421</fpage>
          -
          <lpage>430</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -91152-6_
          <fpage>32</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>J.</given-names>
            <surname>Bausch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Leditzky</surname>
          </string-name>
          ,
          <article-title>Quantum codes from neural networks</article-title>
          ,
          <source>New Journal of Physics</source>
          <volume>22</volume>
          (
          <year>2018</year>
          ). doi:
          <volume>10</volume>
          .1088/
          <fpage>1367</fpage>
          -2630/ab6cdd.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>V.</given-names>
            <surname>Silva</surname>
          </string-name>
          ,
          <article-title>Practical quantum computing for developers: programming quantum rigs in the cloud using Python, quantum assembly language and IBM QExperience</article-title>
          , Apress,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>