In recent years, generative AI has made remarkable strides in algorithmic innovation, enhancing zeroshot generalization for downstream tasks. However, challenges of accessibility and extensibility hinder their use as foundation models, primarily due to scale dependence. This article investigates the potential of sustainability and programmability principles in architectural design to create a more context-aware and user-focused AI ecosystem, promoting the responsible development of foundation models.
Artificial intelligence (AI) has made significant progress in recent years, particularly in the area
of generative modeling [
However, training on web-scale corpora with limited filtering can lead to unforeseen failure
modes and a lack of semantic context, which can raise ethical concerns such as bias and
discrimination.[
While the impressive computational scale-up achieved by these models is undeniably an engineering feat, it is not a sustainable approach in the long run due to its significant energy consumption and carbon footprint. Scaling up comes with drawbacks that go beyond environmental concerns and also afect accessibility and extensibility. The sheer scale of these models makes them impractical for inference on standard hardware, thus limiting their accessibility to researchers and developers. Moreover, the reliance on scale undermines their efectiveness as foundation models, as they encounter various challenges including limited control over output, brittleness, domain dependence, ineficiencies in training, and a lack of customizability for users.
In this paper, we delve into the potential of sustainability and programmability principles in architectural design to shape the future of foundation models. In Section 2, we focus on improving the accessibility of these models by addressing their computational complexity and exploring sustainable alternatives for internal components. We examine techniques for updating AI models in real-time without retraining to address unforeseen failure modes and adapt to changing definitions of ethics and propriety.
In Section 3, our focus shifts to the crucial aspect of extensibility in the design of foundation models. We explore how programmability plays a vital role in enabling users to customize and exert control over these models. Additionally, we delve into innovative approaches that facilitate context-length generalization. Moreover, we emphasize the importance of tailoring AI systems to cater to individual needs and values, thus fostering a more personalized and adaptive experience.
Finally, in Section 4, we summarize our findings and discuss the implications of our work. We hope to provide valuable insights into designing and deploying such models and help guide future research and development eforts in this area.
Bommasani Et al. [
As the models increase in scale, the optimization demands a proportional increase in data.
To meet this need curation processes have seen a similar shift from their traditionally
humanled nature to heuristics-based automated filtering. To overcome the prohibitive cost of label
generation, most foundation models adopt self-supervised learning (SSL), which formulates the
pretraining task as a property of raw unannotated data such as next-word prediction or masked
reconstruction. [
An unfortunate side efect of automated curation based on web crawling (which is further
augmented by the use of SSL) is that imperfections in curation heuristics lead the resultant
model to exhibit biases, show memorization of private data, vulnerability to adversarial assaults.
[
The fundamental constituent of all modern foundation models is the transformer block based
on the idea of self-attention [
We will go through each of the components of the transformer block and discuss the possible alternatives to make them more sustainable. The original formulation of the transformer is presented with an encoder-decoder architecture. Modern variants such as GPTs focus on decoder-only architecture. This simplification allows for straightforward implementations and removes the overhead of architectural choices such as early or late fusion, number of context injections, etc. However, the encoder-decoder architecture allows caching of the context. This is particularly useful when downstream applications can request multiple generations based on the same input.
It is commonly thought that attention is the bottleneck in transformer-based models.
Numerous papers have attempted to reduce the computational complexity of the attention mechanism
by providing approximations to the full attention matrix at the cost of a slight decrease in
accuracy. [
Flash Attention [
It has been found that the efect of quadratic cost for attention minimizes with scale as it
only adds () FLOPs per token per layer, whereas matrix multiplications (in the form of
linear layers) add (2)[
In an orthogonal direction, Jaegle Et al. [
[
Training large models incur a significant cost in terms of the environment, finances, and time. Unfortunately, the current foundation models require periodic retraining lest they become outdated in their predictions. Knowledge about world events and new inventions is an important indirect context that contributes to the quality of the generated text.
Based on the scale, even the cost and resources required for retraining can be significant. Consumers often resort to non-robust methods such as prompt tuning for incorporating specific knowledge to avoid fine-tuning. However, since the context length for most models is limited, prompt tuning is only efective in limited cases. Instilling expert knowledge in trained models has been an area of active research.
Boregeaud Et al. showed that the quality of generation can be improved with additional
context obtained from retrieval [
This method (RETRO) can be used with models trained from scratch as well as be retrofitted after the initial training has been done. While RETRO hasn’t yet been widely adopted it provides a promising direction for service providers to perform hotfixes to the model, an essential feature in mature products.
Combining multiple modalities, such as text, images, audio, and video, can improve the performance and generalization ability of large AI models. This is because multimodal data provides more comprehensive information, is more robust to errors in any individual modality, and better reflects the complexity of real-world situations. For example, if an image is blurry or distorted, the accompanying text may still provide enough information to accurately identify the object.
Networks that use a common structure to process independent modalities have been shown
to have synergistic efects on generalization. While current state-of-the-art models for computer
vision are trained to predict a fixed set of categories, in real-world scenarios, the categories are
often not known in advance and the model must be able to adapt to new categories as well as
provide a null prediction when the category difers from the known set. Because the cardinality
of text is far larger than the categories in classification datasets, models that share a common
representation space can be applied in a zero-shot manner [
To enable adaptation to new modalities, it is crucial to consider tokenization as part of the
architectural design process. Tokenization refers to the process of breaking down the input
modality into smaller units, such as words or subword units for text, to facilitate processing
by transformer models. One approach that has shown promise is byte-based tokenization [
In this section, we explore various aspects of programmability that empower users with customization, control, and alignment with their individual needs and values. We discuss the concept of empowering users through customization and control, highlighting difusion models and pluggable control as efective techniques. Additionally, we delve into two innovative approaches for context length generalization, namely ALiBi, and RoPe, which utilize advanced positional embeddings. Lastly, we address the significance of aligning AI systems with individual needs and values, highlighting the adaptation capabilities of LoRA. By examining these aspects, we aim to shed light on the potential of programmability to enhance the user experience and enable AI systems to better meet the diverse requirements of individuals.
A new class of methods has rapidly captured the image generation landscape, called difusion
models. [
While initially, difusion models were costly to train and obtain generations from at high
resolution, recent work has shifted difusion to be an internal component of a larger network
and work with latent representations [
Difusion-LM [
A new technique, introduced on top of difusion models but applicable to most generative
models is inversion [
Context-length generalization is crucial in transformer models as it enables the ability to process and understand larger contexts in downstream applications such as chatbots and summarization. Transformer models are typically trained on sequences of up to 4k tokens, which limits their capacity to handle longer contexts. Since transformers are permutation invariant, the choice of positional embeddings plays a significant role in determining their ability to generalize across diferent context lengths [31].
ALiBi [32] adopts a diferent strategy. It introduces a penalty proportional to the distance between query-key attention scores, rather than using positional embeddings. This biasing mechanism enables ALiBi to efectively handle longer contexts by discouraging excessive attention to distant tokens and promoting a better focus on relevant information within the given context. Rotary positional embeddings (presented in [33]) have been shown to handle larger contexts by changing their base at inference time [34].
Fine-tuning large models has traditionally been a challenging task in AI research. The complexity and size of these models often make fine-tuning computationally expensive and resourceintensive. Moreover, the need for large-scale labeled datasets for fine-tuning can be a bottleneck, as collecting and annotating such datasets is time-consuming and costly. However, the introduction of LoRA [35] (low-rank adaption) has brought significant simplification to the process of ifne-tuning large models. LoRA and its subsequent extension [ 36] utilizes low-rank decomposition to eficiently update the weights of the pre-trained model during fine-tuning. This approach reduces the computational requirements and memory footprint, making it more feasible to ifne-tune large models on limited resources.
In conclusion, integrating sustainability and programmability principles into the architectural design of foundation models holds the potential to address the challenges of accessibility and extensibility, fostering a context-aware and user-focused AI ecosystem. By prioritizing responsible development, we can empower users with customization and improve eficiency. It is crucial for researchers and practitioners to embrace these principles and collaboratively design foundation models that promote a more inclusive and responsible AI future.
This research was made possible thanks to the generous support of the SIMAR GROUP s.r.l., Monte Urano (FM, Italy) and NextGenerationEU, which provided a Ph.D. scholarship to the lead author. arXiv preprint arXiv:2208.01618 (2022). [31] R. Csordás, K. Irie, J. Schmidhuber, The devil is in the detail: Simple tricks improve systematic generalization of transformers, arXiv preprint arXiv:2108.12284 (2021). [32] O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, arXiv preprint arXiv:2108.12409 (2021). [33] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, arXiv preprint arXiv:2104.09864 (2021). [34] G. Georgi, llama.cpp, https://github.com/ggerganov/llama.cpp, 2023. [35] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.09685 (2021). [36] T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Eficient finetuning of quantized llms, arXiv preprint arXiv:2305.14314 (2023).