Large Language Models (LLMs) are increasingly used for tasks involving Knowledge Graphs (KGs), whose evaluation typically focuses on accuracy and output correctness. We propose a complementary task characterization approach using three complexity frameworks from cognitive psychology. Applying this to the LLM-KG-Bench framework, we highlight value distributions, identify underrepresented demands and motivate richer interpretation and diversity for benchmark evaluation tasks.
Understanding the dificulty and structure of tasks often requires going beyond surface-level features.
In cognitive science and educational research, several frameworks have been developed to describe
the complexity of tasks based on the type of knowledge involved and the mental operations required.
One of the most well-known is Bloom’s Taxonomy [
These frameworks form the basis for our task characterization approach, which we apply to
LLM-KGBench as an illustration. The LLM-KG-Bench framework was developed to address the lack of scalable
evaluation tools for LLMs targeting KG tasks such as RDF serialization, SPARQL query generation and
structured extraction. The framework supports a wide range of tasks with built-in correction cycles and
output validation, emphasizing automated, reproducible evaluation across a broad selection of models.
Here, we provide a short overview and description of the task groups used in LLM-KG-Bench, for more
details, see [
Extract facts from a textual fact sheet and create a KG [
Identify the node with the most incoming edges [
Correct a syntactically invalid RDF graph [
Given a small KG and a SPARQL SELECT query, return the respective
result set for the query [
Return the result set answering a given textual question on a given KG
(withot a SPARQL SELECT query) [
Given a KG and its description, construct a SPARQL SELECT query
corresponding to a given natural language query [
Given a SPARQL SELECT query with syntax errors, return a corrected
query [
RdfConnectionExplainStatic Find the shortest connection between two nodes in an RDF graph [
Generate small Turtle KGs satisfying given requirements [
To understand what kinds of abilities and operations are required by benchmarking tasks, we apply structured characterization criteria drawn from the three established frameworks, as introduced above. While we adopt terminology from cognitive psychology, we do not claim that LLMs engage in these processes in a human sense. Rather, we assess the extent to which their outputs reflect behavior consistent with such operations. Table 1 provides an overview of all possible values across the three frameworks. In the following, we describe the interpretation and assignment criteria for each value.
The task depends primarily on “mechanically” recalling facts or definitions without further processing.
The task requires interpreting given information, structures or queries without fundamentally transforming or generating new representations.
A known procedure or pattern must be correctly executed, such as retrieval or following syntactic rules.
The task demands recognizing or decomposing relationships between data, especially when multiple elements or steps must be coordinated.
A task involves judging the correctness, relevance or quality of a result.
The task involves generating new content, such as generating queries or data structures.
Factual Task execution success depends on recalling or recognizing specific terminology, syntax elements or concrete information.
Conceptual Structural or relational understanding is necessary, such as schema structure, data models or logical organization.
Procedural The task requires a correct application of known methods, routines or transformation steps.
Metacognitive Awareness and control over one’s strategies and thinking processes, such as selecting appropriate approaches, planning task execution or monitoring correctness, which might be relevant for more complex or interactive settings.
Low
The task involves interpreting or manipulating individual binary relations or isolated, simple structures with minimal dependencies.
Multiple relations and entities must be processed simultaneously, such as coordinating several triples or variables in a query.
The task involves multiple interrelated entities or nested dependencies that must be simultaneously considered, often requiring more abstract or hierarchical reasoning. 3.1. Application to Benchmark Tasks The assigned values for each task are displayed in Table 2. Note that not all values across the frameworks are represented, as the current set of tasks does not span the full theoretical space. The assigned values represent the minimal operational and structural requirements. Some variability in the relational complexity dimension is certainly possible given a prompt that requires more complex operations.
We can observe several recurring value combinations. Most tasks fall into a characterization combining Understand and Apply as cognitive processes with Conceptual and Procedural knowledge dimensions and a Low level of relational complexity. This reflects the prevalence of tasks requiring interpretation and rule application without substantial structural coordination. Tasks involving generation (FactExtractStatic, TurtleSampleGeneration and Text2SparqlList) are naturally the only ones annotated with the Create process. Among them, only the RDF-based generation tasks are assigned Medium relational complexity, reflecting the need to coordinate multiple entities and their relationships when constructing a graph. In contrast, SPARQL generation tasks tend to result in single triple pattern and are thereby assigned Low relational complexity.
A consistent, expected dependency can be observed between some processes and knowledge types. Factual and Conceptual knowledge always coincide with Understand, as interpreting a meaning inherently involves factual or structural knowledge. On the other hand, Procedural knowledge always coincides with Apply, Analyze or Create, since carrying out a certain procedure by definition requires knowing the necessary steps. In the current task set, Factual and Conceptual knowledge do not co-occur, distinguishing between surface-level terminology and deeper structural comprehension. Similarly, Apply, Analyze and Create do not co-occur, as they all describe mutually exclusive operations that either follow a procedure, decompose a structure, or generate new ones.
In this paper we proposed a task characterization that provides a complementary perspective on benchmark design and evaluation beyond accuracy metrics, inspired by theories of cognitive complexity. This can guide the creation of more balanced and targeted benchmarks by ensuring diversity across the diferent dimensions. Moreover, it enables identification of potential blind spots in model behavior for tasks that require similar processing.
We demonstrate how to assign the characterization values on a set of evaluation tasks from the LLMKG-Bench framework. Several values do not appear in the task set due to current design preferences, but that does not imply that such dimensions are irrelevant or unassignable. In cognitive processes, we note Remember which would describe a task that asks for reproduction of terminology or exact syntax, e.g., listing reserved SPARQL keywords from memory, while Evaluate would require making judgments between alternative options, e.g., selecting the most eficient query. One knowledge dimension was not assigned, Metacognitive knowledge, which might be tackled by tasks that require justification, such as explaining the reasoning behind a generated query. Finally, High relational complexity would emerge in tasks requiring coordination of more than two entity roles simultaneously, like multi-dimensional event data or nested dependencies. This suggests a direction for extending task design to capture a broader range of structural demands.
The proposed framework could be applied to other benchmarks in the semantic web and beyond, allowing for cross-benchmark comparisons of task complexity profiles. It could also be integrated into such evaluation pipelines, helping understand the types of processes the models succeed or struggle with. In turn, this could support more systematic error analysis, design, and task selection.
This work was partially supported by grants from the German Federal Ministry of Education and Research (BMBF) to the projects ScaleTrust (16DTM312D) and KupferDigital2 (13XP5230L).
The authors have not employed any Generative AI tools.