ARCLIGHT is an AI fusion system that leverages Large Language Models, perception learning, knowledge graphs, and human guidance to describe high-level concept instances with lower-level attributes and afordances. By combining structured models and unsupervised exploration, ARCLIGHT discovers attributes and afordances in both known and unknown objects, entities, or activities. This enables automated novelty detection, curation of a symbolic knowledge graph, and a dialogue agent that asks discriminating questions. The system's perception component utilizes Bayesian models to recognize unknown and novel concepts, flag regions of high epistemic uncertainty, and update the knowledge graph based on user interactions. ARCLIGHT can potentially improve human-machine collaboration and advance artificial intelligence in various fields.
The Automated Clustering and Curriculum Learning Guided by Human Training (ARCLIGHT) system is
able to learn high-level concept instances (objects, entities, activities) with lower-level concept instances
of attributes (features) and afordances (capabilities). ARCLIGHT discovers attributes and afordances in
both known and unknown objects, entities, and activities (see Figure 1 for the knowledge representation)
through a combination of structured models and unsupervised exploration. The structured models
allow for automated novelty detection (e.g., this entity appears to be a dinosaur, but it is purple) while
the unsupervised attribute exploration (through masking for unknown objects and more nuanced
saliency maps and uncertainty attribution for known objects) allows discovery of defining features (e.g.,
cars have wheels). Both approaches allow curation of a symbolic knowledge graph (adding attributes
to objects) as well as enabling a Large Language Model-based dialogue agent to ask discriminating
questions. Our approach is to build on and extend the Whyis [
https://tw.rpi.edu (J. McCusker); https://tw.rpi.edu (D. L. McGuinness); https://lacailab.cogsci.rpi.edu (T. Strzalkowski);
CEUR
ceur-ws.org
schema:hasPart schema:ImageSegment
schema:about schema:VideoObject
has part
subclass of
Entity subclass of has participant has input has target has output has product has attribute
exists at schema:hasPart
has agent
wd:activity schema:Clip schema:about the knowledge graph to reason about an unknown object, and can update the knowledge graph directly based on user interactions.
ARCLIGHT is implemented as a multiagent system using the Whyis knowledge graph framework’s
autonomous inference architecture [
In Figure 2, we show a high-level diagram of the diferent services and types of agents in the system interacting via what is in efect a publish-subscribe model. Since each agent is looking for specific graph patterns using SPARQL queries, the agents are modular. As a result, multiple agents can be prototyped, evaluated, and deployed in parallel as desired without disrupting the overall system. 1https://jena.apache.org/documentation/fuseki2/ 2.1. Media Analysis Agents ARCLIGHT uses agents for each perceptual task, which includes classification and simultaneous uncertainty quantification for objects, activities, attributes, and afordances in incoming multi-modal data. Additionally, insight agents perform post-hoc analyses on the classification agents and their output to provide saliency maps, uncertainty attribution maps, and decompose known and unknown objects, which enables eficient resolution of novelty from oracles and stored knowledge.
The perception agents adopt an open-world detection framework, which separates the task of entity
localization (left side of Figure 2) from classification (right side of Figure 2), enabling the detection of
unknown objects. For localization in the full system, the perception agents will use a common region
proposal network (RPN) combining a baseline RPN similar to He et al [
The design of the Object Classification Agent focuses on evaluating the use of uncertainty
quantification and attribution for novel object detection in a SOTA deformable detection transformer model
architecture, similar to work from Zohar et al [
The Attribute Classification Agent employs feature extractors based on the model and dataset
presented by Ramanathan et al [
Interaction between ARCLIGHT and human users (e.g., analysts, teachers, or evaluators) happens
through a multimodal, instruction-tuned, tool-aware dialogue agent. The dialogue agent is responsible
for initiating conversation with users to facilitate active learning when uncertainty arises from the
system’s perception. It is also responsible for responding to user-initiated conversation to aid in typical
analytic tasks including question answering and reasoning over the media presented by the user. All
discussion is mediated using the Activity Streams [
At the core of the dialogue system is the instruction-tuned Large Language Model (LLM) Llama 3
70b [
When an agent initiates a conversation with a user following the ingestion of media with high uncertainty, the ARCLIGHT dialogue agent selects actions that update Whyis with new knowledge obtained from the interaction. The LLM is responsible for locating the appropriate labels, descriptions, and other relevant information and crafting the call to a Whyis API to update the ontology accordingly.
The ARCLIGHT system is a knowledge graph-centric AI fusion system that allows users to easily upload media that can be analyzed by a suite of perception, classification, and dialogue agents, creating knowledge graph fragments to describe each depicted scene. This graph-augmented knowledge to is used to drive discussion through an instruction-tuned LLM that knows how to query the graph and extract relevant knowledge to provide suitable responses to user dialogue. Further, the media, dialogue, and image knowledge are all represented within the knowledge graph, allowing for comprehensive explanations of any analysis and tracing of any given source of information. Our hope is that this kind of modular system can serve as a model for neuro-symbolic learning using perception, language, and knowledge in a meaningful way.