The popularity of artificial intelligence has led to an increasing need for machine learning models tailored to specific needs. AutoML aims to automate the process of creating effective machine-learning solutions, but current systems need to be more versatile to meet the demand. While more flexible and extensible heterogeneous systems have overcome many limitations of traditional AutoML systems, they lack accessibility due to their programmatic interfaces. We propose a research project to address this issue to develop a heterogeneous AutoML system that can produce optimal machine-learning pipelines using a natural language interface.
Machine learning has expanded rapidly, presenting researchers and practitioners with many new algorithms and data sets. However, selecting the most suitable strategy for a given issue has become increasingly complex, requiring extensive experimentation and technical expertise. AutoML has emerged as a solution to this problem by providing powerful tools to search through large machine-learning pipelines [1]. Nevertheless, the range of possible techniques for natural language processing is vast, making it hard to combine and compare different algorithms. Thus, AutoML algorithms must agree on a standard protocol for sharing outputs as inputs for any other algorithm.
To achieve the primary goal of AutoML, the systems must have interfaces that are easy to use for those with limited computer science and machine learning knowledge. Furthermore, the systems should have strong generalization capabilities to create tools that can be utilized in various scenarios and produce machine learning models that can be applied to a wide range of applications. However, current AutoML systems focus on a specific set of algorithms, often tailored to a library or toolkit [2,3,4]. This reduces their ability to explore various algorithms from different domains and find optimal solutions to complex, multifaceted problems. In contrast, Heterogeneous AutoML systems generate learning solutions by mixing techniques from different domains [5]. However, they do not provide natural interfaces for novice users in programming or AutoML, and their execution requires preparation of the environment, the definition of the problem in appropriate terms, and the provision of data in specific formats [2,3,4,6,7,8,9].
Using natural language as a user interface can significantly enhance the accessibility and user-friendliness of AutoML. Large Language Models (LLMs) are trendy for their ability to process raw text and effectively identify patterns and connections in data [10,11]. However, these models can sometimes generate incorrect responses or need help with inference tasks [12]. Recent studies suggest that incorporating external knowledge significantly improves the performance of LLMs [13].
Researchers are exploring combining these techniques to address the limitations of both AutoML and LLMs. Shen et al. [14] employed an LLM to process queries and generate learning task planning using pre-trained models. This approach has successfully integrated image, audio, and text prediction, classification, and processing capabilities into a single system. However, it does not focus on optimizing model selection or hyperparameter optimization. It does not produce tuned learning models in standalone programs or allow for the export of inference power to arbitrary environments.
The research project consists of producing a Heterogeneous AutoML system that integrates natural language processing as its primary interface. The ultimate goal is to design a tool to generate optimal machine learning models that are flexible and adaptable to different contexts and heterogeneous situations. This leads to our Main Research Question: "In what way can we integrate Natural Language into a Heterogeneous AutoML process?".
Most AutoML systems only use limited algorithms specific to a particular library or toolkit. This limitation hinders their ability to solve complex problems by exploring various algorithms from different areas. On the other hand, Heterogeneous AutoML systems combine techniques from multiple domains to create better learning solutions. However, they require a user-friendly interface for those new to programming or AutoML. This setup involves defining the problem correctly, providing data in specific formats, and setting up the environment.
Table 1 contrasts several existing AutoML systems with the system proposed in this research regarding their capabilities of dealing with heterogeneous scenarios. This evaluation focused on their capacity to handle diverse scenarios encompassing multiple algorithms. It is worth noting that this assessment was solely based on their ability to handle heterogeneous algorithms without consideration for their overall performance, capacity, or applicability.
AutoML systems vary in capabilities and limitations, depending on the specific learning libraries they are built on. Some, like Auto-Sklearn [3], Auto-Weka [4], and Auto-Keras [2], are restricted to using Scikit-learn [15], Weka [16], and Keras [17], respectively. Other systems, such as RECIPE [9] and Hyperopt [7], can incorporate algorithms from different libraries but require a concrete implementation. TPOT [6] and ML-Plan [8] provide a more flexible approach, combining technologies from multiple learning libraries to create concrete implementations of learning pipelines. AutoML systems are mostly focused on supervised learning, but some offer the potential to integrate unsupervised learning functionality, like Hyperopt.
Multiple ML problems Probabilistic Extensible Automatic Discovery Multiobjective Distributed Deployable Pipelines
Table 1 Comparison of several existing AutoML systems' capabilities to deal with heterogeneous machine learning problems. Entries marked with ≈ indicate that the system design enables the given capability, but we have no record of its implementation.
Systems like Auto-Sklearn and Auto-Keras benefit from a unified underlying API, while modularly designed systems such as ML-Plan allow the addition and modification of algorithms. Several systems allow balancing different objectives or metrics during optimization, which is relevant in multiple development and research scenarios. For example, Auto-Keras and ML-Plan use a weighted sum approach to combine multiple evaluation metrics into a single objective function. These systems allow you to specify both the metrics and the weights assigned to each metric, which allows you to control their relative importance in the optimization process.
Hyperopt, ML-Plan, Auto-WEKA, and Auto-Sklearn include mechanisms to distribute search processes and resources among multiple computers, optimizing search time and generating learning pipelines more quickly. To meet the goal of AutoML, system solutions should be easily usable in arbitrary environments and applicable as portable machine learning algorithms outside the AutoML context. Systems such as Hyperopt, TPOT, and Auto-Sklearn allow exporting the best pipeline found during the search process as a Python script, while Auto-WEKA can export to JAVA [18] code. Auto-Keras allows exporting the models in various formats, including TensorFlow [19], PyTorch [20], and Keras [17].
Using probabilistic models to describe the space of possible pipelines is another interesting feature of AutoML systems. AutoML systems based on Bayesian optimization build an internal representation of the space of possible algorithm pipelines, which can be interpreted as assigning a probability distribution to each particular pipeline. This feature describes the algorithm pipeline space and allows researchers to gather additional information by analyzing which regions have higher or lower probabilities of generating effective Pipeline components and allow researchers to gather additional information.
Previous research addressed the limitations of AutoML systems. Estevez-Velarde et al. [21] introduced the concept of Heterogeneous AutoML, a more general formulation of the AutoML Problem. Additionally, they introduced AutoGOAL, a flexible and efficient system for heterogeneous AutoML implemented in Python. With AutoGOAL, users can describe a specific machine problem, input and output requirements, and a set of objectives. The system then automatically finds the best pipeline of algorithms from various libraries, including Scikit-learn [15], NLTK [22], Keras [17], and Gensim [23]. It is also customizable, allowing users to add and integrate new algorithms into the existing pipelines. AutoGOAL uses a Pareto Front approach to multiobjective optimization and an optimization process based on probabilistic grammatical evolution for context-free grammar [24].
AutoGOAL has achieved state-of-the-art performance against other AutoML systems and has been able to solve machine-learning tasks outside of supervised learning. Additionally, it can build complex pipelines targeting difficult NLP tasks like Named Entity Recognition, being able to connect algorithms of different natures. This research project will use AutoGOAL as a baseline for its capabilities regarding Heterogeneous AutoML.
According to Estevez-Velarde et al. [21], the Heterogeneous AutoML problem's space of all possible pipelines can be represented as a 𝐺 𝐴 graph. This graph consists of nodes representing each known algorithm 𝑎 𝑖 , and edges exist between all pairs 𝑎 𝑖 , 𝑎 𝑗 such that their corresponding output and input types are compatible. Given a machine-learning task defined as a function that transforms an input type into an output type, we can build a specific search space graph 𝐺 ′ 𝐴 that only models valid pipelines. To extract 𝐺 ′ 𝐴 , we must introduce two additional nodes to 𝐺 𝐴 : Input and Output. These nodes are then connected to all algorithms capable of producing the desired output from the specific input. By identifying any path in 𝐺 ′ 𝐴 that connects the Input and Output nodes, we can obtain a pipeline that addresses the machine-learning problem at hand.
A suitable computational implementation of this process requires solving the following problems:
1. Defining each algorithm and their respective input and output, such that it is computationally feasible to determine if two algorithms can be connected and construct the graph. 2. Designing an optimization strategy that can effectively search in the space of all pipelines, algorithms, and their hyperparameters, given restricted computational resources.
AutoGOAL utilizes a set of Semantic Type objects to implement this compatibility function. Each semantic data type is a Python class that belongs to a hierarchy in which object inheritance directly represents the relation for type compatibility (e.g., Word can also be interpreted as Text, a more general type). The data types have a semantic interpretation beyond their underlying computational structure. For example, a string in computational terms can either be a Document, a Sentence, or a Word. Each algorithm is implemented as a class with a run(input: Tin) -> Tout method that performs the corresponding processing, potentially wrapping an underlying implementation from a machine learning library. Each algorithm's input and output types are specified using Semantic Types and represented by the Tin and Tout annotations.
While this method for computing compatibility has advantages, it is rigid and difficult to maintain. Due to the closed nature of the type system, precise type definitions must be matched for any new algorithms identified and added to the AutoGOAL search space. Also, because adding new Semantic Types does not automatically update current algorithm annotations, users need to check on every existing algorithm to identify which should be annotated accordingly. Moreover, this mechanism assumes a tree-like structure of type compatibility when there might be more complex relationships (e.g., a Stem might also be considered a Word, albeit these two types do not inherit from each other). This leads to an interesting question: "Can we model algorithm compatibility more openly?".
Recent proposals suggest using natural language to store information describing algorithms. Shen et al. [14] uses Jsons, mainly text-based, to store information about pre-trained models. They parse natural language prompts into multiple tasks that are matched with suitable algorithms using an LLM. However, this tool does not address the AutoML problem, as it does not optimize model selection or hyperparameter configurations for algorithms. In contrast, Zhang et al. [25] aims to develop an AutoML system called AutoML-GPT, which uses LLMs to train models on datasets with user inputs and descriptions automatically. The LLMs serve as an automatic training system to establish connections with models and process inputs.
In this research, we propose the integration of description cards based on text data for algorithms in AutoGOAL. By leveraging the power of LLMs, we can match algorithms based on their description. This method adds a vital factor of generalization that might solve the previous limitations of AutoGOAL. Moreover, by substituting the current Semantic Type system with natural language, we open the tool's interface to accept user text prompts.
To achieve our main objective, we must address various questions within the proposal:
1. Which LLM should we use? 2. What language should we support in the interface?
To answer the first question, we must conduct experiments to determine which LLM will be most suitable for our project. One idea is to use two different LLMs that are each fine-tuned for specific purposes. For example, one LLM can help determine the compatibility between algorithms used during the optimization process. At the same time, the other can identify problem definitions (input and output types and objective functions) out of natural language for better user interaction with the system.
For the second question, we aim to develop an interface not tailored to a specific language to make it fully inclusive. However, the performance of LLMs can vary significantly in different languages due to differences in available training data. Therefore, we will compare the performance of multilingual models against specific-language models in our objective tasks before deciding which approach to follow.
Finally, our main objective is to extend existing tools, specifically AutoGOAL, to saturate the definition of Heterogeneous AutoML, thus achieving more flexibility and integrating more tasks seamlessly. We can achieve this by using the compatibility function to discover algorithms that were once bound to a specific machine-learning problem but can be part of the solution of another one. The diversity and amount of algorithms determine the limit of tasks we can solve in the system.
The proposed system has valuable scientific, economic, and social implications. It can enhance our understanding of artificial intelligence and apply it to robotics and process automation. Economically, it can speed up the development of applications and decrease the cost and time required for building machine learning solutions.
From a social standpoint, an AutoML system based on natural language can improve the accessibility and ease of use of machine learning, especially in critical areas such as healthcare and education. Furthermore, automating the building of learning models can lessen the need for human intervention in repetitive and monotonous tasks, thus reducing the carbon footprint associated with computer system operations.
To evaluate the potential of our proposed system, we plan to develop a benchmark that incorporates challenging tasks from multiple domains such as vision, NLP, audio, and others. We will perform ablation studies to comprehend the significance of different LLMs and optimization strategies in the overall performance of our system. In addition, to make the evaluation more comprehensive, we will compare our new system with its previous version and other stateof-the-art AutoML systems. By providing more flexibility, we aim to test the capabilities of our system against human adversaries in various challenges. Furthermore, we will explore the possibility of integrating human feedback into the learning process, which can provide valuable insights and lead to further improvement.
The purpose of this publication is to present the research framework for a thesis that aims to investigate the intersection between AutoML and Large Language Models (LLMs). Our objective is to improve AutoML systems, making them more accessible, user-friendly, and versatile. In order to achieve this goal, we will begin by examining the current state of the art in this field. Subsequently, we will develop corresponding description cards for each algorithm available in AutoGOAL and also include new algorithms, such as pre-trained models from Hugging Face along with their respective cards. The next step will be integrating an LLM to model the compatibility function between algorithms, thereby enabling a natural language interface for user interaction. Ultimately, our aim is to pave the way for more inclusive and efficient machine learning applications in various domains.