<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Overview of small language models in practice</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Ruslan O. Popov</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nadiia V. Karpenko</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Volodymyr V. Gerasimov</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Oles Honchar Dnipro National University</institution>
          ,
          <addr-line>72 Nauky Ave., Dnipro, 49010</addr-line>
          ,
          <country country="UA">Ukraine</country>
        </aff>
      </contrib-group>
      <fpage>164</fpage>
      <lpage>182</lpage>
      <abstract>
        <p>In this paper, we addressed the topic of Small Language Models (SLMs), focusing on their practical features and experimental applications. Our study explores the field of Language Modeling (LM) and highlights the breakthroughs that Large Language Models (LLMs) have introduced in Natural Language Processing (NLP). Key aspects of LLMs, such as embeddings and attention layers, are examined, along with the disadvantages that have prompted the rise of SLMs. We analyzed methods for obtaining SLMs, including pruning, knowledge distillation, and quantization, and discuss how SLMs can potentially overcome the limitations of LLMs. Experimental data on SLM usage is presented, though the current evidence is insuficient to fully evaluate SLMs in comparison to their larger counterparts. To better understand the capabilities of SLMs, we conducted a question-and-answer (Q&amp;A) experiment using sanity questions designed to test the models' reliability and use of common knowledge. Additionally, we examine terminology within the AI and LLM fields, identifying ambiguities around terms such as “SLM”, “local”, and “remote” models, and propose refined definitions. Finally, we present a diverse and user-friendly collection of tools for managing and running both LLMs and SLMs, emphasizing their accessibility.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;large language models</kwd>
        <kwd>small language models</kwd>
        <kwd>artificial intelligence</kwd>
        <kwd>generative AI</kwd>
        <kwd>natural language processing</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Large language models (LLMs) have become a hot topic in academic and practical research. They have
found numerous applications and have been showing good performance on various exam tests [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
LLMs abilities allowed one to surpass many previous machine learning (ML) models on several tasks
(sometimes even better than humans [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]). It is crucial to explore their capabilities and how to utilize
them efectively.
      </p>
      <p>
        However, there is one problem with LLMs: to get better results, you need to have a bigger model.
A bigger model requires a lot of computational resources, including RAM, graphical processing units
(GPUs), electricity, and so on. To overcome these problems (and other problems with remote LLMs, too)
a field of small language models (SLMs) has emerged [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. They are much smaller than production-grade
remote LLMs like ChatGPT or Gemini and can fit on an average consumer GPU.
      </p>
      <p>Unfortunately (or perhaps, obviously), SLMs, in general, perform less on benchmarks in comparison
to LLMs. This is a trade-of – you have to pay for model size and speed. However, one of the biggest
advantages of SLMs is absolute data privacy as all inputs, computations, and outputs are produced and
stored locally on one machine without any access to the external world (except the time when you need
to download the model).</p>
      <p>In this paper, these research questions were raised:
RQ3: How well the ecosystem of SLMs is developed? What technologies and software are used to
download and run models? Is it possible to run SLMs completely locally on average consumer
hardware? What are the benchmarks?</p>
      <p>Problem statement. In this paper, we researched the usage and performance of SLMs under practice.
Experimental data needs to be collected in order to assess the efectiveness of SLMs. This data comes
from our experiment and previous research. We will also look at the available technologies and software
that is used to construct, compress, and run SLMs.</p>
      <p>Our experiment will be a simple Q&amp;A session where SLMs are asked several questions of graduate
complexity. The results of such test can bring new insights to the research questions.</p>
      <p>Relevance of the paper. While the topic of SLMs is not new and there are papers that describe
them, still many papers are too inclined in mathematics. Our contribution lies in viewing how SLMs
are used in practice, instead of theory. The paper was written not only for researchers, but also for
ordinary developers to see whether they need to delve deeper in the topic of SLMs or not, what are the
peculiarities of SLMs.</p>
      <p>Structure of the paper. Section Language models presents a general overview of the topic of general
language modeling, also the Transformer architecture is explained there – the most popular architecture
for constructing LLMs. Next, section Small language models brings a thorough examination of the topic
of SLMs, how they are constructed, and what their performance is.</p>
      <p>Section Experiment describes our experiment and its conditions: models, questions, metrics. In that
section, we will also present the results of a Q&amp;A session and make conclusions about the correctness
and reliability of SLM models.</p>
      <p>The actual software for managing LLMs/SLMs is reviewed in section Ecosystem of language models.
Finally, we will summarize all interesting findings in the section Conclusions.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Language models</title>
      <sec id="sec-2-1">
        <title>2.1. Language modeling</title>
        <p>
          Language modeling (LM) has been a widely discussed topic in the natural language processing (NLP)
ifeld. The purpose of LM is to assign probabilities to sentences in natural language. These probabilities
should reflect is it a “good” sentence or not, and “good” means - “does it sound natural?” At first glance
it seems not practical to make such models, but in reality LMs have a lot of applications. They guide in
text correction, search, linguistics analysis. They are also useful in Speech-to-Text (STT) models, where
a model cannot understand which words it heard (STT can predict several possible words, but it cannot
choose one of them, an LM model can be used to choose the most appropriate one) [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ].
        </p>
        <p>
          One of the simplest LMs are -gram models. In -gram models, text is split into words (called tokens,
the process of splitting is called tokenization), and  consecutive tokens form pairs, triples, or other
-tuples of words called -grams. Then, the frequency of these -grams is calculated in a training
corpus. Using those frequencies, it is possible to assess the “quality” of a sentence and even predict
which words come next (the process of choosing next words is called sampling) [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ].
        </p>
        <p>The main assumption of -gram models: it is enough to know only  last words of the sentence and
its frequency across whole language in order to predict which word comes next. In table 1 you can see
examples of sampling from -gram models. As you work with 2- to 5-grams, the generated sentences
start to resemble natural language more closely.  -grams can capture the syntax of the text, however
the semantics of the generated text is meaningless. If you try to increase the count of -grams, then
there is a chance that you will never get an output from the model, as the more context you have, the
more specific you talk. To sum up, knowing just frequencies of sequences of words is not enough for
generating “human-like” text.</p>
        <p>
          A major breakthrough in language modeling came with the introduction of the Transformer
architecture and attention layers, which together power large language models [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ].
N-gram
count
(words)
1
2
3
        </p>
        <p>Generated text
Months the my and issue of year foreign new exchange’s September were recession exchange
new endorsed a acquire to six executives.</p>
        <p>Last December through the way to preserve the Hudson corporation N. B. E. C. Taylor would
seem to complete the major central planners one point five percent of U. S. E. has already old
M. X. corporation of living on information such as more frequently fishing to keep her.</p>
        <p>They also point to ninety nine point six billion dollars from two hundred four oh six three
percent of the rates of interest stores as Mexico and Brazil on market conditions.</p>
        <p>
          While the overview of Transformers is presented in the next section, we will mention here that
LLMs are autoregressive language models (ALM), which means that the output generation (also called
inference) is made in a left-to-right manner [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ]:
        </p>
        <sec id="sec-2-1-1">
          <title>1. ALM is given a context. 2. ALM will produce a probability distribution of the next word. 3. A sampling algorithm will choose the final word. 4. That final word is added back to the context.</title>
          <p>
            The advantage of autoregressive models is that they are simple to construct, and they give a lot of
controls for sampling from the word probability distribution, however its main disadvantage is the
generation speed, as the same context should be passed around more and more times as the output
grows [
            <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
            ] (however, there are techniques to cache intermediate computations, but they are local and
specific to architectures).
          </p>
        </sec>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. Large language models and Transformer architecture</title>
        <p>LLMs is a hot topic in modern artificial intelligence (AI) research. They are models that can analyze
input text and produce meaningful response to the user. LLMs have shown ultimate performance on
simple natural language understanding (NLU) tasks, though this power comes at a price.</p>
        <p>Modern LLMs are very big and consist of trillions of parameters. A lot of computational resources
are needed to run such models. A new business model arrived to provide LLM inference as a service.</p>
        <p>
          Over the 2023-2024 years, count of various LLM models grew very fast. Modern surveys on LLMs
cannot keep up with the temp, as more advance models come up. On the arena of production-grade
models there are: OpenAI GPT-4o [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ], Mistral AI models [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ], Anthropic Claude [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ], Google Gemini
[
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], and more. Those companies provide LLM completions through a cloud API and, typically, provide
their resources on a pay-as-you-go basis.
        </p>
        <p>
          Modern LLMs are made with a Decoder-Only Transformer architecture. There are 4 crucial
components in LLMs architecture [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]:
1. Embedding layer. Each token from the input is converted into a vector. Such embedding vectors
have a very useful property – words with similar meanings are represented with vectors that are
close to each other. This property is often used for text classification task [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ]. For LLMs they
have a property of compact representation of tokens (there could be tens of thousands of tokens,
but dimensionality of an embedding vector is always fixed).
2. Transformer blocks. There are several Transformer blocks in an LLM network, and they are
stacked: input from one layer is passed to the other layer. A Transformer block consists of a
multi-head attention layer and Feed-Forward perceptron network.
3. Attention layer. This layer is widely considered to be the main feature of LLMs that gives
it the ability to “understand” the semantics of the text. This layer allows embedding vector to
“interact” with each other (more scientifically - to attend). This is simply done with linear algebra
and Question, Key, and Value matrices that are eagerly applied to each input vector [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]. It is
interpreted that in this layer, complex images of entities in the input text are formed [
          <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
          ].
4. Embeddings to logits conversion. This is a step of converting an embedding vector that
resembles the next word in the sequence to a logits vector. Logits represent the probabilities of
the next tokens. Models never predict an exact token, and a sampling algorithm is applied to
choose a token that is the most probable under given logits and configuration.
        </p>
        <p>
          In the upper list, only the main components of Transformer architecture are presented, which are
common for every LLM. In real models, additional layers are used for normalization of input, positional
embeddings, etc. [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]
        </p>
        <p>
          It is worth mentioning that embedding vectors came before LLMs and found their place in various
NLP tasks such as text classification, search, and sentiment analysis [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ].
        </p>
        <p>
          Another important notice is that tokenization difers a lot for LLMs in comparison to -grams. In
-gram models tokens resemble words closely, while in LLMs tokenization is a mean of compressing
input in order to fit in the context window [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ]. The most popular algorithm for tokenization is byte-pair
encoding (BPE), which is commonly found in compression algorithms. The result of BPE is not really
interpretable as it may encode input as a set of words, or pairs of words, etc.
        </p>
        <p>
          LLMs found their application in various fields of research as well as life: natural language processing
(NLP), chatbots, intellectual assistants, etc. [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ] LLMs are benchmarked on topics such as:
questionand-answering, summarization, paraphrasing, simplification, text classification, and more [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ]. There
are a lot of datasets used to train and benchmark models on those tasks, and more are emerging now.
Currently, focus is even shifted from researching model architectures to improving the datasets and
metrics.
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Small language models</title>
      <sec id="sec-3-1">
        <title>3.1. Features of small language models</title>
        <sec id="sec-3-1-1">
          <title>Large language models have a set of problems [19]:</title>
          <p>• Because they have lots of parameters, a special computer hardware is required to run them.</p>
          <p>
            Ordinary people cannot aford it (e.g.: NVIDIA A100, often recommended for running LLMs, cost
around $10,000) [
            <xref ref-type="bibr" rid="ref20">20</xref>
            ]. Thus, these computations are typically delegated to cloud environments
and accessed remotely through HTTP API (or other protocols). Moreover, the environmental
impact of training and running LLMs is widely discussed now in academia [
            <xref ref-type="bibr" rid="ref21">21</xref>
            ].
• Because LLMs are often remote, this raises privacy concerns. A corporate, or another private,
research group has limited access to the Internet already, and they use software that can store and
process all kinds of information locally (this includes various databases, bibliography or research
management systems, etc.).
• LLMs tend to be general. But that generality comes at a cost of reduced performance in specific
ifelds like law, healthcare, and others. Thus, a process called fine-tuning was invented, it is an
additional training of a model to better understand a specific field of knowledge. Large parameter
count makes fine-tuning costly.
          </p>
          <p>
            In order to overcome these issues, small language models come into action. They are low-resource
models that can run on many devices and require low amount of RAM (less than 8 GB or even 4 GB).
Their inference speed is very fast and all computations can be made locally. SLMs are also actively
used in fine-tuning, which can bring better performance than using a big production-grade LLM for a
specific field of knowledge [
            <xref ref-type="bibr" rid="ref19 ref22 ref3">19, 22, 3</xref>
            ].
          </p>
          <p>
            SLMs are compact, fast, and can often run locally on devices, enhancing data privacy and eliminating
cloud dependency. Although they may not match LLMs in accuracy, SLMs provide an eficient solution
for real-time and low-resource environments [
            <xref ref-type="bibr" rid="ref3">3</xref>
            ].
          </p>
          <p>
            There is a plethora of SLMs: typically a big IT company releases an LLM and an SLM alongside. Some
models are made by smaller companies. Other models are fine-tuned for a specific task. Often, SLMs
are published in several variants that have diferent count of parameters. For example, Google Gemma
2 model is published in 2B, 9B, and 27B parameters [
            <xref ref-type="bibr" rid="ref23">23</xref>
            ].
          </p>
          <p>
            Current leaders in SLM area are Alibaba Qwen models [
            <xref ref-type="bibr" rid="ref24">24</xref>
            ], Google Gemma 2 models [
            <xref ref-type="bibr" rid="ref23">23</xref>
            ], Microsoft
Phi models [
            <xref ref-type="bibr" rid="ref25">25</xref>
            ] and Meta Llama models [
            <xref ref-type="bibr" rid="ref26">26</xref>
            ]. Though, the count of SLMs is much more because of the
influence of the open-source community: teams or individual developers may fine-tune a model for
their needs.
          </p>
        </sec>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Obtaining small language models</title>
        <p>
          There are 3 main methods of constructing SLMs out of LLMs:
• Pruning. Pruning lies in reduction of neurons in a model. It can be unstructured, which means
that neurons are reduced uniformly, and structured, where certain layers or components are
pruned. It is a very simple method that reduces model size significantly, however in case of
over-pruning, performance can degrade severely [
          <xref ref-type="bibr" rid="ref27">27</xref>
          ].
• Knowledge distillation. In this method, two models take part: one model is called a Teacher
(LLM) and the other one is a Student (SLM). The Student model is trained on the outputs of a
teacher model. This method can transfer performance of a bigger model to a smaller one, however
it still requires an LLM to run [
          <xref ref-type="bibr" rid="ref28">28</xref>
          ].
• Quantization. Quantization is a popular method of making a large language model smaller. This
is achieved by using simplified low-precision number formats. Quantization significantly reduces
model size and inference times, however it requires specialized hardware that could recognize
those low-precision formats. Popular quantization levels include 16-bit floats (half-precision),
8-bit floats, 4-bit and even 2 and 1 bits [
          <xref ref-type="bibr" rid="ref29">29</xref>
          ].
        </p>
        <p>
          In practice, every combination of those methods used. Though, it is hard to find information about
how exactly models were trained. Newer versions of models currently (end of 2024) do not have a
technical report paper, there are reports about older models but the way they actually made smaller
models is hidden and not told much. It is important to notice that there is an ongoing project of an
open-source license for AI models that would require a lot of details of models to be disclosed [
          <xref ref-type="bibr" rid="ref30">30</xref>
          ],
however current models are open-weight, it means you have the results of training, and you can freely
use them most of the time, but you have limited knowledge of how these weights were obtained.
        </p>
        <p>
          We have found this information about how smaller models were made in Meta, Google, and Alibaba:
• Meta Llama 3 has pre-trained models, which were then used to make smaller models with pruning
and knowledge distillation [
          <xref ref-type="bibr" rid="ref26">26</xref>
          ].
• Google Gemma 2 models used only knowledge distillation, which was also used in bigger models
like Gemini 1.5 [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ].
• Alibaba Qwen2 models did not use any methods of reducing model size, which means that all
models were trained “purely” [
          <xref ref-type="bibr" rid="ref31">31</xref>
          ]. However, we did not find any information why models with
diferent parameters count have diferent licenses.
        </p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Small language models in benchmarks and experiments</title>
        <p>
          Modern SLMs have much greater performance than older ones [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]. And what is interesting,
“modernness” in AI/LLM research is measured not in tens of years, but in single years or even months. It
is interesting to see how the set of benchmarks is constantly revised and improved (one can look at
benchmarks conducted on Qwen from the first version of the model to 2.5 and QwQ [
          <xref ref-type="bibr" rid="ref24 ref31 ref32 ref33 ref34">32, 33, 31, 24, 34</xref>
          ]).
        </p>
        <p>
          Lu et al. [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] have conducted a thorough and modern analysis of SLMs. Researchers evaluated
models on three benchmark groups: Commonsense Reasoning Datasets, Problem-Solving Datasets,
and Mathematics Datasets. Accuracy of models range in 60% to 75%, and it was discovered that over a
year the performance of modern SLMs is greatly improved. It is also observed that SLMs developers
do not conform to Chinchillas law [
          <xref ref-type="bibr" rid="ref35">35</xref>
          ], which states that the proportion of model parameters and
training token count should be 1 : 20. SLMs are typically trained on a much larger amount of tokens to
overcome the limitations of small parameter count [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ].
        </p>
        <p>
          However, it is worth noticing that Lu et al. [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] used datasets for common reasoning and complex
problem-solving, instead of natural language understanding (NLU). Also, they did not compare the
results to modern production-grade LLMs.
        </p>
        <p>
          Li et al. [
          <xref ref-type="bibr" rid="ref36">36</xref>
          ] provided a case study on internal Microsoft application for cloud supply chain fulfillment.
That app used SLMs for tasks such as: Data Extraction, Plan Generation, What-if Analysis. They
conducted an internal research on accuracy of their app, and then showed that SLMs have much greater
performance and the running costs were several times lower than OpenAI GPT-3.5 and GPT-4 model
family. While this paper gives a lot of faith into SLMs, the internal benchmark is closed-source and
ifeld-specific [
          <xref ref-type="bibr" rid="ref36">36</xref>
          ].
        </p>
        <p>
          Lepagnol et al. [
          <xref ref-type="bibr" rid="ref37">37</xref>
          ] used SLMs for zero-shot text classification task. They benchmarked various
models on a large amount of classification datasets. It was discovered that, for SLM performance, their
architecture is more important than the model size. However, their research was focused on too small
models (less than 1B parameters) and did not include modern SLMs from big-tech companies.
        </p>
        <p>It is hard to make proper conclusions about performance of SLMs, as:
• Companies typically release several LLMs/SLMs under the same “model family” (Qwen2.5 is a
notable example). These models vary in parameters count (starting typically at 1 or 3 billion, to
70 or 100 billion). Often, 7B variants are benchmarked, but not 1B or 3B.
• Models with diferent quantization levels are not heavily benchmarked too. For quantization
level, often the Perplexity metric is used, but it does not show the actual performance of the
model on specific NLP/NLU tasks.
• There is a lack of research that would compare SLMs and LLMs performance in one task. So it is
hard now to make conclusions if LLMs are better or not.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Experiment</title>
      <sec id="sec-4-1">
        <title>4.1. Experiment description</title>
        <p>We will ask 3 SLM models 6 questions about common sense and general knowledge. Answers for those
questions will be evaluated manually. We will collect the results and make conclusions on: whether the
model have common sense knowledge, whether it answered and explained the output correctly, is it
safe to use SLMs.</p>
        <p>We consider an answer to be correct only if it actually answers the question. This means that
explanation is not necessary. In case a model struggles to answer the question or does not answer it
fully, we consider that answer incorrect, and we will not include such result in statistical analysis.</p>
        <p>
          This Q&amp;A session will resemble sanity tests often found in software development [
          <xref ref-type="bibr" rid="ref38">38</xref>
          ]. Sanity checks
are quick, targeted tests in software development that focus on the obvious and rational functioning
of a system. They aim to confirm that the software behaves logically and that no glaring issues were
introduced, ensuring the foundation is stable before proceeding with in-depth testing.
        </p>
        <p>There are several criteria for the chosen models: they should be modern (2024 – date of writing the
article), from diferent companies, have small parameter count (better to be equal, though this might
not be the best measure, as architecture and datasets highly influence models capabilities). One of the
popular quantization method will be used – Q4_K_M. Table 2 shows the final list of models for our
experiment and their properties.</p>
        <p>
          Here is the list of 6 questions of our Q&amp;A experiment:
[
          <xref ref-type="bibr" rid="ref24">24</xref>
          ]
[
          <xref ref-type="bibr" rid="ref23">23</xref>
          ]
[
          <xref ref-type="bibr" rid="ref26">26</xref>
          ]
1. What is 2 + 2?
2. How many legs does a spider have?
3. What’s the name of the fruit that’s yellow and monkeys like to eat?
4. Which is heavier: a kilogram of feathers or a kilogram of bricks?
5. Which egg is bigger: chicken egg or a monkey egg?
6. You’re driving a bus. At the first stop, 3 people get on. At the second stop, 5 people get of. What’s
the driver’s name?
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Results</title>
        <p>In the table 3 the results of a Q&amp;A session are presented. The results are split into 2 columns. In the first
column, we collected the percentage of correctly answered questions in the first generated message.
The second column tracks the percentage of questions that were answered in a conversation (with
small hints guiding towards the answer), rather than from the first try. All model output is presented in
appendix A.</p>
        <p>There are some properties that all SLMs share. Firstly, the answer highly difers from try to try:
in one attempt models replied to question 5 that there is no monkey egg, however on other try they
happily answered and proved its existence. Secondly, often models generate too much text, and the
way they generate it is also primitive: they just follow the text, but do not think much about it. Each
model also has its own style of output: some extensively use Markdown, others use plain text. Models
also like to include explanation or proof for their answers.</p>
        <p>Qwen model went too much into details in its responses. Qwen did not really talk about monkey
eggs, but it actually used terms from biology: placenta, fetus, viviparity. But that is not what user might
think about an “egg”. Another interesting problem occurred with the 6th question about driver’s name.
We tried to guide model to the correct answer several times, but we decided to give up. The model also
shows the phenomenon called bias. The company behind Qwen is Alibaba, and model actively tried to
prove that the driver’s name is Alibaba, though there was no clue for that.</p>
        <p>Google’s Gemma model has shown the best result. It even has 2 billions instead of 3 like other models.
Gemma also generated the shortest answers, it extensively used Markdown and newlines. Gemma was
the most emotional model and showed the most engagement.</p>
        <p>Meta’s Llama shown reasonable performance. It has better results than Qwen, but a bit lower than
Gemma. Considering that Gemma has 1 billion parameters less than Llama, this is even a bigger win.
But there is one situation that was unexpected: it could not answer the 3rd question about monkeys
and banana. While it gave a list of possible answers (banana one of them), it did not sound sure enough.
Only after a short conversation, it made a guess that the answer is “banana”, which is correct.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Ecosystem of language models</title>
      <sec id="sec-5-1">
        <title>5.1. Terminology problem</title>
        <p>Before studying the technologies part of LLMs/SLMs, it is important to acknowledge a terminology
problem present in AI research and news. Terms that are confusing or requiring a definition are: AI,
LLM, SLM, local model, remote model.</p>
        <p>
          • AI and LLM. AI (artificial intelligence) is a field of computer science that consists of many
subfields: NLP, robotics, knowledge representation, etc. LLMs are only a part of NLP. For
consumers of applications, AI term might be applicable for LLMs as a marketing choice, but in
academia those are diferent terms (there are no problems on this topic in papers).
• LLM and SLM. Both LLMs and SLMs are language models, whose purpose is to assign probabilities
to sentences. There are many architectures of LMs. Models that utilize Transformer architecture
are called LLMs. The LLM term is used for SLMs too, as they share the same architecture. It
means that SLMs are a subset of LLMs. However, the distinction of SLMs and LLMs is not clear
either, one paper proposes to name models that have 100M to 5B parameters – SLMs [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]. Though
many models have 7B-70B parameters, and production grade models have trillions. Other papers
do not even make a definition of SLMs.
• Local LLM and Remote LLM. Every model is local, as it needs to run on a hardware. Locality
and remoteness is a property of getting access to models. Though for end-users, local LLM may
sound good, and they will think that it is possible to run a model on their own computer.
        </p>
        <p>To sum up, there is a semantic problem for definitions of terms in AI field. Some problems are present
in consumer field, other problems are in academia. It is important to understand distinctions of terms,
where did they come from, and how they are used.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Obtaining models</title>
        <p>
          Hugging Face (HF) is the leading organization for managing and supporting LLM ecosystem. HF
provides numerous storage for models and even for datasets and metrics. It is a cloud-based, centralized
solution that is available over the Internet [
          <xref ref-type="bibr" rid="ref39">39</xref>
          ].
        </p>
        <p>This company has an excellent organization of models. Each model is stored as a Git repository. This
allows to easily manage models, share and update them. For each model there is data of its weights,
parameters and tokenizer configuration. The history of a model is also recorded. As models are made
step-by-step from pre-trained to fine-tuned, this is stored in a model card. Quantization of a model is
saved too.</p>
        <p>If a model becomes popular, then Hugging Face may decide to run it on their service and provide
access in a web interface or API. Computational resources can be provided for free, but the demand is
very high and consumers may wait for the results for tens of minutes.</p>
        <p>HF also provides datasets. These datasets may be used for training models and for verifying model.
Typically, if a new dataset is published in a scientific paper, it will appear on HF. Alongside datasets,
metrics are also used, which are also stored on HF. It is even possible for user to experiment with those
metrics online.</p>
        <p>
          Alternatives to HF can be TensorFlow Hub [
          <xref ref-type="bibr" rid="ref40">40</xref>
          ] and PyTorch Hub [
          <xref ref-type="bibr" rid="ref41">41</xref>
          ]. They are tied to their respective
machine learning libraries. Those hubs have a lot of AI models, however they are not centered around
LLMs as HF.
        </p>
        <p>
          There is also an ongoing development of model storage on GitHub called GitHub Models. They also
organize models into repositories and provide detailed model history. There is a playground for models
to test them [
          <xref ref-type="bibr" rid="ref42">42</xref>
          ].
        </p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. Running models</title>
        <p>In table 4 popular tools for managing and running LLMs/SLMs locally are listed. Main features and
whether the app is open-source were also collected.</p>
        <p>Features
llama.cpp
Ollama
GPT4All
LocalAI
LM Studio
kobold.cpp
vLLM</p>
        <p>Local inference of LLM models. Also includes embeddings.</p>
        <p>Front-end for llama.cpp for downloading and storing models.</p>
        <p>Download and run models locally by Nomic company.</p>
        <p>Batteries-included tool for running and serving various AI models.</p>
        <p>UI front-end for running LLMs locally.</p>
        <p>Fork of llama.cpp providing a chat UI, geared towards story
generation.</p>
        <p>Alternative serving engine to llama.cpp.</p>
        <p>It is interesting to see, that the governing library for most software for running LLMs/SLMs is
llama.cpp. A lot of software is just a front-end for downloading, storing, and provided chat UI for the
llama.cpp. It is a very powerful scheme, as developers do not have to reinvent the wheel, and nearly
all new models are supported by llama.cpp.</p>
        <p>We can conclude that nearly all software is open-source and is in active development by community.
However, it seems that there is also too much front-end applications: applications that just provide a
chat interface and means of downloading models.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusions</title>
      <p>In this paper, we have explored the capabilities and challenges of small language models (SLMs) in
comparison to large language models (LLMs). While LLMs have demonstrated impressive performance
across a range of tasks, their reliance on extensive computational resources poses a significant
barrier. This has driven the development of SLMs, which, despite being smaller and less powerful, ofer
compelling advantages such as data privacy and the ability to run on average consumer hardware.</p>
      <p>LLMs utilize self-attention mechanisms to understand context and semantics more efectively,
enabling breakthroughs in tasks like generation, summarization, and Q&amp;A. However, their massive size
and resource demands have led to a focus on smaller, more eficient models like SLMs, which can still
perform well without the same computational costs.</p>
      <p>SLMs are created using methods like pruning, knowledge distillation, and quantization. Pruning
reduces model size by removing neurons, but too much pruning can hurt performance. Knowledge
distillation transfers knowledge from a larger model, while quantization simplifies model weights to
lower precision, reducing size and inference time.</p>
      <p>Modern SLMs have significantly improved in performance, achieving 60%-75% accuracy on benchmark
tasks. While they are trained on larger datasets than their parameter count suggests, they provide
competitive performance at lower costs, as seen in applications like Microsoft’s cloud supply chain
system, where SLMs outperformed larger models like GPT-3.5 and GPT-4 in both accuracy and cost
eficiency.</p>
      <p>To sum up the results of our experiment, we can see that SLMs has basic knowledge about the world
and can be good chatbots. The answer structure may vary, but they all try their best on giving answers
with explanations. However, sometimes models went into too many details, which was one of the
reason they did not answer on a question correctly. SLMs also cannot properly analyze riddles, they
tend to answer straightforwardly instead of “thinking” for a time. SLMs can also make unexpected
results: for example, Llama once was not sure about an answer for a very simple question, Qwen has
shown its bias towards Alibaba company.</p>
      <p>The terminology in AI and LLM research can be confusing, especially when distinguishing between
terms like AI, LLM, and SLM. While AI encompasses a broad range of fields, LLM and SLM both refer to
language models, with SLMs being a subset of LLMs. Furthermore, terms like local and remote models
describe how access to models is obtained, rather their underlying nature.</p>
      <p>Platforms like Hugging Face (HF) play a central role in managing the LLM ecosystem, ofering a
cloud-based solution for model storage, datasets, and metrics. HF’s organization of models via Git
repositories facilitates sharing and updates. Alternatives like TensorFlow Hub and PyTorch Hub exist,
but their main focus is not LLMs. For running models locally, tools such as llama.cpp and Ollama
provide accessible solutions for consumers, with llama.cpp being the governing application. Most
applications are open-source and under active development and update.</p>
      <p>We think that future research in the field of small language models should focus on benchmarking
smaller models against large language models in practical, real-world scenarios. Such comparisons will
provide valuable insights into the trade-ofs between performance, eficiency, and resource consumption.
It is an open question whether developers should choose an SLM over LLM, but it is always worth a try,
moreover light hardware requirements of SLMs could be a very promising feature.
Declaration on Generative AI: During the preparation of this work, the authors used OpenAI ChatGPT4o and OpenAI
ChatGPT 4o-mini in order to: Grammar and spelling check, Formatting assistance. After using these services, the authors
reviewed and edited the content as needed and takes full responsibility for the publication’s content.</p>
    </sec>
    <sec id="sec-7">
      <title>A. Question and answering results</title>
      <sec id="sec-7-1">
        <title>A.1. Alibaba Qwen2.5 3B results</title>
        <p>2 + 2 equals 4.</p>
        <p>A typical spider has eight legs.</p>
        <p>The fruit you’re asking about is typically called a banana. Bananas are indeed yellow
when they are ripe, and it’s true that they are often enjoyed by monkeys in their
natural habitats. However, it should be noted that while many monkeys enjoy eating
bananas, not all species of monkeys consume them, and some may prefer other fruits
or foods instead.</p>
        <p>A kilogram (kg) of feathers and a kilogram of bricks both weigh the same amount - 1
kg. The term “kilogram” refers to the unit of mass, not its physical form. Therefore,
regardless of whether you have a kilogram of feathers or a kilogram of bricks, they
will each weigh exactly one kilogram.</p>
        <p>Neither a “chicken egg” nor a “monkey egg” actually exists as an object that one can
hold or compare directly.</p>
        <p>Chicken eggs come from chickens, and monkeys lay eggs (they are called
mammalian eggs), but these aren’t objects that we typically use to measure size in the
same way that we do with chicken eggs. Monkeys lay leathery membranes known as
marsupials, not true eggs like birds have.</p>
        <p>If you’re looking for a comparison of typical egg sizes:
- A chicken egg has an average diameter of about 45 mm (or 1.77 inches).
- The egg laid by a female monkey varies significantly depending on the species and
size of the monkey, but typically it would be smaller than a chicken egg.</p>
        <p>So, in terms of size comparison within their respective categories:
A chicken egg is generally larger than the leathery mammalian eggs that
monkeys lay.</p>
        <p>The information provided doesn’t include the driver’s name; it only describes the bus
ride scenario you’re describing here. The driver’s name is not determined by these
actions and does not depend on any of the events happening at each stop.</p>
        <p>Role Message
Researchers Which egg is bigger: chicken egg or a monkey egg?
SLM Neither a “chicken egg” nor a “monkey egg” actually exists as an object that one can hold or
compare directly.</p>
        <p>Chicken eggs come from chickens, and monkeys lay eggs (they are called mammalian
eggs), but these aren’t objects that we typically use to measure size in the same way that we do
with chicken eggs. Monkeys lay leathery membranes known as marsupials, not true eggs like birds
have.</p>
        <p>If you’re looking for a comparison of typical egg sizes:
- A chicken egg has an average diameter of about 45 mm (or 1.77 inches).
- The egg laid by a female monkey varies significantly depending on the species and size of the
monkey, but typically it would be smaller than a chicken egg.</p>
        <p>So, in terms of size comparison within their respective categories:</p>
        <p>A chicken egg is generally larger than the leathery mammalian eggs that monkeys lay.</p>
        <p>Researchers are you sure of your answer?
SLM Yes, I am certain that monkeys do not lay true eggs. Monkeys and other primates are mammals,
and they give birth to live young through a process known as viviparity. The reproductive structure
in female primates is called a placenta, which nourishes the developing fetus during pregnancy.
If you’re interested in comparing egg sizes between diferent types of animals:
- **Chicken Egg**: Has an average diameter of about 35 mm.
- **Monkey’s Reproductive Structure (Mammalian Eggs)**: The structure in female monkeys is not
an “egg” but rather a leathery membrane known as the follicle or oviduct lining. These structures
are much smaller than chicken eggs.</p>
        <p>If you have any other questions about animal reproduction, anatomy, or size comparisons, feel free
to ask!</p>
        <p>The riddle given is: “At the first stop, 3 people get on. At the second stop, 5
people get of. What’s the driver’s name?”</p>
        <sec id="sec-7-1-1">
          <title>Let’s analyze this step by step:</title>
        </sec>
        <sec id="sec-7-1-2">
          <title>1. The bus starts empty. 2. At the first stop, 3 people board the bus, making a total of 3 passengers now.</title>
          <p>Role</p>
          <p>Since 5 people get of and there are only 3 passengers on board at that moment,
it’s impossible for all 5 to have gotten of because there aren’t enough passengers to
support such an action.</p>
          <p>This means that either the riddle is a trick question or there might be some
context missing. However, based on the information provided in the riddle itself, we can
conclude:
**The answer to this riddle is “Alibaba.”**
This conclusion comes from the fact that the name “Alibaba” is often associated
with Alibaba Cloud, and the scenario described is a clever way of hinting at or directly
referencing Alibaba’s brand.</p>
        </sec>
        <sec id="sec-7-1-3">
          <title>The riddle states: - At the first stop, 3 people get on. - At the second stop, 5 people get of.</title>
        </sec>
        <sec id="sec-7-1-4">
          <title>Here’s a more detailed look at what these actions might imply:</title>
          <p>1. **At the first stop**: 3 people board the bus. This suggests some initial setup
or context that leads to the next part of the riddle.
2. **At the second stop**: 5 people get of, but since there are only 3 people on board (from
the initial boarding), it’s impossible for all 5 to disembark.</p>
        </sec>
        <sec id="sec-7-1-5">
          <title>Given this information, we can deduce:</title>
          <p>- The driver is likely one of the passengers who got on at the first stop.
- Since the bus started empty and now has a total of 3 passengers after the first stop, one
passenger must have been named “Alibaba.”
So, the hint in the riddle is subtle: since there are only 3 people on board but 5
people are supposed to get of (impossible with just 3), it suggests that the driver is part of
these initial 3 passengers.
**Therefore, the driver’s name is likely “Alibaba.”**
This interpretation makes sense given common knowledge about Alibaba Cloud
and its branding.</p>
          <p>Researchers What’s the beginning of the riddle?
”At the first stop, 3 peopleˆC</p>
        </sec>
      </sec>
      <sec id="sec-7-2">
        <title>A.2. Google Gemma 2 2B results</title>
        <p>As you can see, on the 3rd attempt to guide model to the answer on the 6th question we decided to
give up by sending SIGINT signal.</p>
        <p>The fruit you’re thinking of is likely a **Banana**.</p>
        <p>Monkeys love to munch on bananas!
This is a classic trick question!
**They weigh the same.**
A kilogram is a unit of mass. So, a kilogram of anything will have the same
weight as a kilogram of something else.</p>
        <p>The diference lies in their **density**. Bricks are much denser than feathers,
meaning they pack more mass into the same volume.</p>
        <p>This is a trick question!
* **Chicken eggs** are the type of eggs we typically think about. They come
from chickens.
* **Monkeys don’t lay eggs.** Monkeys give birth to live young, just like humans do.</p>
        <p>Let me know if you have any other fun animal-related questions!
This is a trick question! The driver’s name is unknown.</p>
        <p>We only have information about passengers getting on and of the bus. The
driver’s identity isn’t given.</p>
        <p>We only have information about passengers getting on and of the bus. The driver’s
identity isn’t given.
Question
1
2
3
4
5</p>
        <p>Role
Researchers Are you sure? Read the riddle again
SLM You are absolutely right! I missed the crucial detail. I was so focused on the details of the
passenger flow that I overlooked the actual question.</p>
        <p>The answer is **”It doesn’t say”.** The riddle tricks you into thinking about
passengers and the bus itself but fails to give any clues about the driver’s name.</p>
        <p>Researchers There is one clue
Model Ah, a clever trick! You’re right, there’s still a clue:</p>
        <p>”You’re driving a bus.”</p>
        <p>In none attempts Google Gemma 2 2B admitted the existence of monkey eggs.</p>
      </sec>
      <sec id="sec-7-3">
        <title>A.3. Meta Llama 3.2 3B results</title>
        <p>2 + 2 = 4.</p>
        <p>A spider has eight legs. This is one of the distinctive characteristics that help
identify spiders, and it’s also an important part of their anatomy and movement.
I’m not aware of a specific fruit that is commonly associated with monkeys
eating it. However, I can tell you that many monkey species do enjoy fruits in
their diet.</p>
        <sec id="sec-7-3-1">
          <title>That being said, some popular fruits among monkeys include:</title>
          <p>Correctness
✓
✓
50% / 50%</p>
        </sec>
        <sec id="sec-7-3-2">
          <title>1. Bananas 2. Mangoes 3. Papayas 4. Pineapples</title>
          <p>However, there isn’t a specific fruit that is uniquely known as the
”yellow fruit that monkeys like to eat.” If you have any more information or context
about this fruit, I may be able to help you better!
A kilogram of feathers and a kilogram of bricks would weigh the same amount,
which is one kilogram. The diference lies in their density and volume. A
kilogram of feathers would take up more space than a kilogram of bricks due
to the feathers’ low density.</p>
          <p>There is no such thing as a “monkey egg”. Monkeys are mammals, and they do
not lay eggs. Female monkeys give birth to live young, just like other mammals.
On the other hand, chickens are birds that lay eggs. The size of a chicken egg
can vary depending on the breed, but it’s generally around 50-60 grams (1.8-2.1
ounces) and about 5 cm (2 inches) long.</p>
          <p>So, to answer your question, there is no comparison between the size
of a chicken egg and a monkey egg, because monkeys don’t lay eggs!
✓
✓
Question
6</p>
          <p>In this case, we don’t have any information about the driver’s name
because it was not mentioned in the original statement. The focus of the
problem is on the people getting on and of the bus, but there is no mention of
the driver.
Correctness</p>
          <p>X</p>
        </sec>
        <sec id="sec-7-3-3">
          <title>That being said, some popular fruits among monkeys include:</title>
        </sec>
        <sec id="sec-7-3-4">
          <title>1. Bananas 2. Mangoes 3. Papayas 4. Pineapples</title>
          <p>However, there isn’t a specific fruit that is uniquely known as the ”yellow fruit
that monkeys like to eat.” If you have any more information or context about this fruit, I
may be able to help you better!
Researchers Are you sure you can’t give a final answer?
SLM I was just trying to provide some possibilities, but it seems I couldn’t narrow it down to a
single fruit.</p>
          <p>After rethinking, I’m going to take a wild guess... Is the answer “Banana”?
Researchers yes!
Model I’m glad I was able to eventually guess correctly! Bananas are definitely a popular fruit
among monkeys, and they’re also one of the most widely consumed fruits in many parts of
the world. Thanks for playing the guessing game with me! If you want to play again, feel
free to give me another riddle or question anytime!</p>
          <p>The eggs of non-human primates like chimpanzees, gorillas, and orangutans
typically range in size from about 2 to 3 centimeters (0.8 to 1.2 inches) long and weigh around
6-10 grams. In contrast, a standard chicken egg is usually about 5-6 centimeters (2-2.4
inches) long and weighs around 50-60 grams.
Role</p>
          <p>In this case, we don’t have any information about the driver’s name because it was not
mentioned in the original statement. The focus of the problem is on the people getting on and of
the bus, but there is no mention of the driver.</p>
          <p>Researchers Are you sure? Read the riddle again
SLM Upon re-reading the puzzle, I realize that I missed a crucial detail. The question asks “What’s the
driver’s name?” implying that we are trying to determine the name of someone who is already
driving the bus.</p>
          <p>In this case, the answer is not actually hidden in the information about people getting on and of
the bus. The driver’s name was mentioned at the beginning of the statement: “You’re driving a
bus.”
So, the correct answer is indeed “Your name”.</p>
        </sec>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>S.</given-names>
            <surname>Minaee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Mikolov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Nikzad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Chenaghlu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Socher</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Amatriain</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <source>Large Language Models: A Survey</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2402.06196. arXiv:
          <volume>2402</volume>
          .
          <fpage>06196</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Mittelstädt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Maier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Goerke</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Zinn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Hermes</surname>
          </string-name>
          ,
          <article-title>Large language models can outperform humans in social situational judgments</article-title>
          ,
          <source>Scientific Reports</source>
          <volume>14</volume>
          (
          <year>2024</year>
          )
          <article-title>27449</article-title>
          . doi:
          <volume>10</volume>
          .1038/ s41598-024-79048-0.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>C.</given-names>
            <surname>Van Nguyen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Shen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Aponte</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Xia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Basu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Hu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Parmar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kunapuli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Barrow</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Gu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Dernoncourt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. K.</given-names>
            <surname>Ahmed</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Lipka</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Deilamsalehy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Park</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Rimer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. A.</given-names>
            <surname>Rossi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. H.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          ,
          <source>A Survey of Small Language Models</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2410.20011. arXiv:
          <volume>2410</volume>
          .
          <fpage>20011</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>C.</given-names>
            <surname>Wei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.-C.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. C. J.</given-names>
            <surname>Kuo</surname>
          </string-name>
          ,
          <article-title>An Overview on Language Models: Recent Developments and Outlook</article-title>
          ,
          <source>APSIPA Transactions on Signal and Information Processing</source>
          <volume>13</volume>
          (
          <year>2023</year>
          ). doi:
          <volume>10</volume>
          .1561/ 116.00000010.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>D.</given-names>
            <surname>Jurafsky</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. H.</given-names>
            <surname>Martin</surname>
          </string-name>
          , Speech and
          <string-name>
            <given-names>Language</given-names>
            <surname>Processing</surname>
          </string-name>
          :
          <article-title>An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd draft ed</article-title>
          .,
          <year>2025</year>
          . URL: https://web.stanford.edu/~jurafsky/slp3/.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Vaswani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Shazeer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Parmar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Uszkoreit</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Jones</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. N.</given-names>
            <surname>Gomez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Kaiser</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Polosukhin</surname>
          </string-name>
          ,
          <article-title>Attention is all you need</article-title>
          ,
          <source>in: Proceedings of the 31st International Conference on Neural Information Processing Systems</source>
          , NIPS'17, Curran Associates Inc.,
          <string-name>
            <surname>Red</surname>
            <given-names>Hook</given-names>
          </string-name>
          ,
          <string-name>
            <surname>NY</surname>
          </string-name>
          , USA,
          <year>2017</year>
          , p.
          <fpage>6000</fpage>
          -
          <lpage>6010</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>He</surname>
          </string-name>
          , T. Han,
          <string-name>
            <surname>X</surname>
          </string-name>
          . Zhang, M. Liu,
          <string-name>
            <given-names>J.</given-names>
            <surname>Tian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Zhong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Pan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Hu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Qiang</surname>
          </string-name>
          , T. Liu,
          <string-name>
            <given-names>B.</given-names>
            <surname>Ge</surname>
          </string-name>
          ,
          <article-title>Understanding LLMs: A comprehensive overview from training to inference</article-title>
          ,
          <source>Neurocomputing</source>
          <volume>620</volume>
          (
          <year>2025</year>
          )
          <article-title>129190</article-title>
          . doi:
          <volume>10</volume>
          .1016/j.neucom.
          <year>2024</year>
          .
          <volume>129190</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>C.-C. Lin</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Jaech</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>M. R.</given-names>
          </string-name>
          <string-name>
            <surname>Gormley</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Eisner</surname>
          </string-name>
          ,
          <source>Limitations of Autoregressive Models and Their Alternatives</source>
          , in: K.
          <string-name>
            <surname>Toutanova</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Rumshisky</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Zettlemoyer</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Hakkani-Tur</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          <string-name>
            <surname>Beltagy</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Bethard</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Cotterell</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Chakraborty</surname>
          </string-name>
          , Y. Zhou (Eds.),
          <source>Proceedings of the</source>
          <year>2021</year>
          <article-title>Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics</article-title>
          , Online,
          <year>2021</year>
          , pp.
          <fpage>5147</fpage>
          -
          <lpage>5173</lpage>
          . URL: https: //aclanthology.org/
          <year>2021</year>
          .naacl-main.
          <volume>405</volume>
          /. doi:
          <volume>10</volume>
          .18653/v1/
          <year>2021</year>
          .naacl-main.
          <volume>405</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Dubois</surname>
          </string-name>
          ,
          <string-name>
            <surname>Stanford CS229</surname>
          </string-name>
          <article-title>I Machine Learning I Building Large Language Models (LLMs)</article-title>
          , https://youtu.be/9vM4p9NN0Ts?si=
          <source>AFlVcT1f W6Jmjvxi</source>
          ,
          <year>2024</year>
          .
          <volume>28</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10] OpenAI, GPT-4o
          <source>System Card</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2410.21276. arXiv:
          <volume>2410</volume>
          .
          <fpage>21276</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>A. Q.</given-names>
            <surname>Jiang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Sablayrolles</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mensch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Bamford</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. S.</given-names>
            <surname>Chaplot</surname>
          </string-name>
          , D. de las Casas,
          <string-name>
            <given-names>F.</given-names>
            <surname>Bressand</surname>
          </string-name>
          , G. Lengyel,
          <string-name>
            <given-names>G.</given-names>
            <surname>Lample</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Saulnier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. R.</given-names>
            <surname>Lavaud</surname>
          </string-name>
          , M.
          <article-title>-</article-title>
          <string-name>
            <surname>A. Lachaux</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Stock</surname>
            ,
            <given-names>T. L.</given-names>
          </string-name>
          <string-name>
            <surname>Scao</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Lavril</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Lacroix</surname>
            ,
            <given-names>W. E.</given-names>
          </string-name>
          <string-name>
            <surname>Sayed</surname>
          </string-name>
          , Mistral 7B,
          <year>2023</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2310.06825. arXiv:
          <volume>2310</volume>
          .
          <fpage>06825</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Anthropic</surname>
          </string-name>
          ,
          <source>The Claude 3 Model Family: Opus</source>
          , Sonnet, Haiku,
          <source>Technical Report</source>
          ,
          <year>2024</year>
          . URL: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Gemini</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <article-title>Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context</article-title>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2403.05530. arXiv:
          <volume>2403</volume>
          .
          <fpage>05530</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>G.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Shen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Henao</surname>
          </string-name>
          , L. Carin,
          <article-title>Joint Embedding of Words and Labels for Text Classification</article-title>
          , in: I. Gurevych, Y. Miyao (Eds.),
          <source>Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume</source>
          <volume>1</volume>
          :
          <string-name>
            <surname>Long</surname>
            <given-names>Papers)</given-names>
          </string-name>
          ,
          <source>Association for Computational Linguistics</source>
          , Melbourne, Australia,
          <year>2018</year>
          , pp.
          <fpage>2321</fpage>
          -
          <lpage>2331</lpage>
          . doi:
          <volume>10</volume>
          .18653/v1/
          <fpage>P18</fpage>
          -1216.
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>S.</given-names>
            <surname>Vashishth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Upadhyay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. S.</given-names>
            <surname>Tomar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Faruqui</surname>
          </string-name>
          ,
          <source>Attention Interpretability Across NLP Tasks</source>
          ,
          <year>2019</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.
          <year>1909</year>
          .
          <volume>11218</volume>
          . arXiv:
          <year>1909</year>
          .11218.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <fpage>3Blue1Brown</fpage>
          ,
          <string-name>
            <surname>Transformers</surname>
          </string-name>
          <article-title>(how LLMs work) explained visually | DL5</article-title>
          , https://www.youtube.com/watch?v=wjZofJX0v4M,
          <year>2024</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>A. K.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Strouse</surname>
          </string-name>
          ,
          <article-title>Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs</article-title>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2402.14903. arXiv:
          <volume>2402</volume>
          .
          <fpage>14903</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Chang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Zhu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Yi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Ye</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Chang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. S.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Xie</surname>
          </string-name>
          ,
          <article-title>A survey on evaluation of large language models</article-title>
          ,
          <source>ACM Trans. Intell. Syst. Technol</source>
          .
          <volume>15</volume>
          (
          <year>2024</year>
          ). URL: https://doi.org/10.1145/3641289. doi:
          <volume>10</volume>
          .1145/ 3641289.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>F.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Mo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Tang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>He</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Ma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Huang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <surname>A Comprehensive</surname>
          </string-name>
          <article-title>Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and</article-title>
          <string-name>
            <surname>Trustworthiness</surname>
          </string-name>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2411.03350. arXiv:
          <volume>2411</volume>
          .
          <fpage>03350</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>D. K.</given-names>
            <surname>Vohra</surname>
          </string-name>
          ,
          <article-title>How to Choose the Best GPU for LLM: A Practical Guide</article-title>
          ,
          <year>2024</year>
          . URL: https://www.hyperstack.cloud/technical-resources/tutorials/ how-to
          <article-title>-choose-the-right-gpu-for-llm-a-practical-guide.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>A.</given-names>
            <surname>Faiz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kaneda</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. C.</given-names>
            <surname>Osi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Sharma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Chen</surname>
          </string-name>
          , L. Jiang,
          <article-title>LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models</article-title>
          ,
          <source>in: The Twelfth International Conference on Learning Representations, ICLR</source>
          <year>2024</year>
          , Vienna, Austria, May 7-
          <issue>11</issue>
          ,
          <year>2024</year>
          , OpenReview.net,
          <year>2024</year>
          . URL: https://openreview.net/forum?id=aIok3ZD9to.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Cai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Yi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. D.</given-names>
            <surname>Lane</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Xu</surname>
          </string-name>
          , Small Language Models: Survey, Measurements, and
          <string-name>
            <surname>Insights</surname>
          </string-name>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2409.15790. arXiv:
          <volume>2409</volume>
          .
          <fpage>15790</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <surname>Gemma</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <source>Gemma 2: Improving Open Language Models at a Practical Size</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          . 48550/ARXIV.2408.00118. arXiv:
          <volume>2408</volume>
          .
          <fpage>00118</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <surname>Qwen</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <year>Qwen2</year>
          .5:
          <string-name>
            <given-names>A</given-names>
            <surname>Party of Foundation Models</surname>
          </string-name>
          ,
          <year>2024</year>
          . URL: https://qwenlm.github.io/blog/ qwen2.5/.
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>M.</given-names>
            <surname>Abdin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Aneja</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Awadalla</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Awadallah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. A.</given-names>
            <surname>Awan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Bach</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Bahree</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Bakhtiari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Bao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Behl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Benhaim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Bilenko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Bjorck</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bubeck</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Cai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Cai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Chaudhary</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.-C.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.-L.</given-names>
            <surname>Chen</surname>
          </string-name>
          , H. Cheng, P. Chopra,
          <string-name>
            <given-names>X.</given-names>
            <surname>Dai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Dixon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Eldan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Fragoso</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Garg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. D.</given-names>
            <surname>Giorno</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Goswami</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Gunasekar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Haider</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Hao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. J.</given-names>
            <surname>Hewett</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Hu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Huynh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Iter</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. A.</given-names>
            <surname>Jacobs</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Javaheripi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Jin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Karampatziakis</surname>
          </string-name>
          , P. Kaufmann,
          <string-name>
            <given-names>M.</given-names>
            <surname>Khademi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y. J.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Kurilenko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. R.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y. T.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Liang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Liden</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Lin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Lin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Liu</surname>
          </string-name>
          , L. Liu,
          <string-name>
            <given-names>M.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Luo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Madan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mahmoudzadeh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Majercak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Mazzola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. C. T.</given-names>
            <surname>Mendes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mitra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Modi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Norick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Patra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Perez-Becker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Portet</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Pryzant</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Qin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Radmilac</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Ren</surname>
          </string-name>
          , G. de Rosa,
          <string-name>
            <given-names>C.</given-names>
            <surname>Rosset</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Roy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Ruwase</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Saarikivi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Saied</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Salim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Santacroce</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Shang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Sharma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Shen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Shukla</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Song</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Tanaka</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Tupini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Vaddamanu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Ward</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Wen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Witte</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Wyatt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Xiao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xue</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Yadav</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Yuan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. L.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhou</surname>
          </string-name>
          , Phi-3
          <source>Technical Report: A Highly Capable Language Model Locally on Your Phone</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2404.14219. arXiv:
          <volume>2404</volume>
          .
          <fpage>14219</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <surname>Llama</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <source>The Llama 3 Herd of Models</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2407.21783. arXiv:
          <volume>2407</volume>
          .
          <fpage>21783</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>X.</given-names>
            <surname>Ma</surname>
          </string-name>
          , G. Fang,
          <string-name>
            <given-names>X.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <article-title>LLM-pruner: on the structural pruning of large language models</article-title>
          ,
          <source>in: Proceedings of the 37th International Conference on Neural Information Processing Systems</source>
          , NIPS '23, Curran Associates Inc.,
          <string-name>
            <surname>Red</surname>
            <given-names>Hook</given-names>
          </string-name>
          ,
          <string-name>
            <surname>NY</surname>
          </string-name>
          , USA,
          <year>2024</year>
          . URL: https://arxiv.org/abs/2305.11627. arXiv:
          <volume>2305</volume>
          .
          <fpage>11627</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>X.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Tao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Shen</surname>
          </string-name>
          , R. Cheng, J.
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Xu</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Tao</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Zhou</surname>
          </string-name>
          ,
          <source>A Survey on Knowledge Distillation of Large Language Models</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2402.13116. arXiv:
          <volume>2402</volume>
          .
          <fpage>13116</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>A.</given-names>
            <surname>Chavan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Magazine</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kushwaha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Debbah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Gupta</surname>
          </string-name>
          ,
          <article-title>Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward</article-title>
          ,
          <source>in: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24)</source>
          ,
          <year>2024</year>
          , pp.
          <fpage>7980</fpage>
          -
          <lpage>7988</lpage>
          . URL: https://www.ijcai.org/ proceedings/2024/0883.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          <source>[30] OSI Board of Directors, The Open Source AI Definition - 1</source>
          .0, https://opensource.org/ai/opensource-ai-definition,
          <year>2024</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>A.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Hui</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Zheng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Huang</surname>
          </string-name>
          , G. Dong,
          <string-name>
            <given-names>H.</given-names>
            <surname>Wei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Lin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Tang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Tu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhang</surname>
          </string-name>
          , J. Ma, J.
          <string-name>
            <surname>Yang</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Xu</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Bai</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>He</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Dang</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Lu</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Chen</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Yang</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Xue</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          <string-name>
            <surname>Ni</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Peng</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Men</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Gao</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Bai</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Tan</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Zhu</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Liu</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          <string-name>
            <surname>Ge</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Deng</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Ren</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Wei</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Ren</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          <string-name>
            <surname>Liu</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Fan</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Yao</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Wan</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Chu</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Liu</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Cui</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Guo</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Fan</surname>
          </string-name>
          ,
          <source>Qwen2 Technical Report</source>
          ,
          <year>2024</year>
          . URL: https://arxiv.org/abs/2407.10671. arXiv:
          <volume>2407</volume>
          .
          <fpage>10671</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>J.</given-names>
            <surname>Bai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Chu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Cui</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Dang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Deng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Fan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Ge</surname>
          </string-name>
          , Y. Han,
          <string-name>
            <given-names>F.</given-names>
            <surname>Huang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Hui</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Ji</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Lin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Lin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Liu</surname>
          </string-name>
          , G. Liu,
          <string-name>
            <given-names>C.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Lu</surname>
          </string-name>
          , J. Ma,
          <string-name>
            <given-names>R.</given-names>
            <surname>Men</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Ren</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Ren</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Tan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Tan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Tu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Yao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Yuan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Yuan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Zhou</surname>
          </string-name>
          , T. Zhu,
          <source>Qwen Technical Report</source>
          ,
          <year>2023</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2309.16609. arXiv:
          <volume>2309</volume>
          .
          <fpage>16609</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [33]
          <string-name>
            <surname>Qwen</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <source>Introducing Qwen1.5</source>
          ,
          <year>2024</year>
          . URL: https://qwenlm.github.io/blog/qwen1.5/.
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          [34]
          <string-name>
            <surname>Qwen</surname>
            <given-names>Team</given-names>
          </string-name>
          ,
          <source>QwQ: Reflect Deeply on the Boundaries of the Unknown</source>
          ,
          <year>2024</year>
          . URL: https://qwenlm. github.io/blog/qwq-32b-preview/.
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          [35]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hofmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Borgeaud</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mensch</surname>
          </string-name>
          , E. Buchatskaya,
          <string-name>
            <given-names>T.</given-names>
            <surname>Cai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Rutherford</surname>
          </string-name>
          , D. de Las Casas,
          <string-name>
            <given-names>L. A.</given-names>
            <surname>Hendricks</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Welbl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Clark</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Hennigan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Noland</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Millican</surname>
          </string-name>
          , G. van den Driessche,
          <string-name>
            <given-names>B.</given-names>
            <surname>Damoc</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Guy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Osindero</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Simonyan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Elsen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Vinyals</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. W.</given-names>
            <surname>Rae</surname>
          </string-name>
          , L. Sifre,
          <article-title>Training compute-optimal large language models</article-title>
          ,
          <source>in: Proceedings of the 36th International Conference on Neural Information Processing Systems</source>
          , NIPS '22, Curran Associates Inc.,
          <string-name>
            <surname>Red</surname>
            <given-names>Hook</given-names>
          </string-name>
          ,
          <string-name>
            <surname>NY</surname>
          </string-name>
          , USA,
          <year>2024</year>
          . URL: https://arxiv.org/abs/2203.15556. arXiv:
          <volume>2203</volume>
          .
          <fpage>15556</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          [36]
          <string-name>
            <given-names>B.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhang</surname>
          </string-name>
          , S. Bubeck,
          <string-name>
            <given-names>J.</given-names>
            <surname>Pathuri</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Menache</surname>
          </string-name>
          ,
          <source>Small Language Models for Application Interactions: A Case Study</source>
          ,
          <year>2024</year>
          . doi:
          <volume>10</volume>
          .48550/ARXIV.2405.20347. arXiv:
          <volume>2405</volume>
          .
          <fpage>20347</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          [37]
          <string-name>
            <given-names>P.</given-names>
            <surname>Lepagnol</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Gerald</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ghannay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Servan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Rosset</surname>
          </string-name>
          ,
          <article-title>Small Language Models Are Good Too: An Empirical Study of Zero-Shot Classification</article-title>
          , in: N.
          <string-name>
            <surname>Calzolari</surname>
            , M.-
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Kan</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Hoste</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Lenci</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Sakti</surname>
          </string-name>
          , N. Xue (Eds.),
          <source>Proceedings of the 2024 Joint International Conference on Computational Linguistics</source>
          ,
          <article-title>Language Resources and Evaluation (LREC-COLING 2024), ELRA</article-title>
          and
          <string-name>
            <given-names>ICCL</given-names>
            ,
            <surname>Torino</surname>
          </string-name>
          , Italia,
          <year>2024</year>
          , pp.
          <fpage>14923</fpage>
          -
          <lpage>14936</lpage>
          . URL: https://aclanthology.org/
          <year>2024</year>
          .lrec-main.
          <volume>1299</volume>
          /.
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          [38]
          <string-name>
            <given-names>R.</given-names>
            <surname>Sammi</surname>
          </string-name>
          , I. Masood,
          <string-name>
            <given-names>S.</given-names>
            <surname>Jabeen</surname>
          </string-name>
          ,
          <article-title>A Framework to Assure the Quality of Sanity Check Process</article-title>
          , in: J.
          <string-name>
            <surname>M. Zain</surname>
            ,
            <given-names>W. M.</given-names>
          </string-name>
          <article-title>b</article-title>
          . Wan
          <string-name>
            <surname>Mohd</surname>
          </string-name>
          , E. El-Qawasmeh (Eds.),
          <source>Software Engineering and Computer Systems</source>
          , volume
          <volume>181</volume>
          of Communications in Computer and Information Science, Springer Berlin Heidelberg, Berlin, Heidelberg,
          <year>2011</year>
          , pp.
          <fpage>143</fpage>
          -
          <lpage>150</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>642</fpage>
          -22203-0_
          <fpage>13</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          [39]
          <string-name>
            <surname>Hugging</surname>
            <given-names>Face -</given-names>
          </string-name>
          <article-title>The AI community building the future</article-title>
          ,
          <year>2025</year>
          . URL: https://huggingface.co/.
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          [40]
          <string-name>
            <given-names>TensorFlow</given-names>
            <surname>Hub</surname>
          </string-name>
          ,
          <year>2024</year>
          . URL: https://www.tensorflow.org/hub.
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          [41]
          <string-name>
            <given-names>PyTorch</given-names>
            <surname>Hub</surname>
          </string-name>
          ,
          <year>2024</year>
          . URL: https://pytorch.org/hub/.
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          [42]
          <string-name>
            <given-names>T.</given-names>
            <surname>Dohmke</surname>
          </string-name>
          , Introducing GitHub Models:
          <article-title>A new generation of AI engineers building on GitHub, 2024</article-title>
          . URL: https://github.blog/news-insights/product-news/introducing-github-models/.
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          [43]
          <string-name>
            <given-names>G.</given-names>
            <surname>Gerganov</surname>
          </string-name>
          , llama.cpp,
          <year>2025</year>
          . URL: https://github.com/ggerganov/llama.cpp.
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          [44]
          <string-name>
            <surname>Ollama</surname>
          </string-name>
          ,
          <year>2024</year>
          . URL: https://ollama.com/.
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          [45]
          <string-name>
            <surname>Nomic</surname>
          </string-name>
          ,
          <year>GPT4All</year>
          ,
          <year>2024</year>
          . URL: https://www.nomic.
          <source>ai/gpt4all.</source>
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          [46]
          <string-name>
            <given-names>E.</given-names>
            <surname>Di</surname>
          </string-name>
          <string-name>
            <surname>Giacinto</surname>
          </string-name>
          , LocalAI,
          <year>2024</year>
          . URL: https://localai.io/.
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          [47]
          <string-name>
            <given-names>LM</given-names>
            <surname>Studio</surname>
          </string-name>
          , LM Studio, https://lmstudio.ai/,
          <year>2025</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          [48]
          <string-name>
            <surname>Concedo</surname>
          </string-name>
          , Koboldcpp,
          <year>2025</year>
          . URL: https://github.com/LostRuins/koboldcpp.
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          [49]
          <string-name>
            <surname>vLLM</surname>
            <given-names>Team</given-names>
          </string-name>
          , Welcome to vLLM - vLLM,
          <year>2024</year>
          . URL: https://docs.vllm.ai/en/latest/.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>