The cultural heritage of theatrical dance involves diverse sources requiring complex multi-modal approaches. Since manual analysis methods are labor-intensive and so limited to few data samples, we here discuss the use of the DanXe framework, which combines diferent AI paradigms for comprehensive dance material analysis and visualization. However, DanXe lacks models and datasets specific to dance domains. To address this, we propose a human-in-the-loop (HITL) extension to the DanXe to accelerate multi-modal data labeling through semi-automatic, high-quality data labeling. This approach aims to create detailed datasets providing humans with a set of user-friendly and efective tools for advancing multi-modal dance analysis and optimizing AI methodologies for dance heritage documentation. To this date, we designed a novel middleware that allows us to adapt data generated from visual Deep Learning (DL) models within DanXe to visual annotation tools, to empower domain experts with a user-friendly tool to preserve all the components included in the choreographic creation, enriching the process of metadata creation.
CEUR ceur-ws.org
The cultural heritage of theatrical dance consists of a multitude of sources, both tangible and
intangible. These sources are diverse by nature and type, location, and preservation methods,
creating a complex constellation that requires a diverse set of skills to be efectively enhanced [
International Workshop on Artificial Intelligence and Creativity (CREAI), co-located with ECAI 2024 †These authors contributed equally.
Due to this challenge, we have attempted to envision a framework that allows for an integrated
approach to theatrical dance’s documentary assets, using the artistic and cultural legacy of
dancer Rudolf Nureyev (1938-1993) as a case study [
Such approaches exhibit three main limits: (a) even being an expert, the process of analyzing
such data by hand is time-consuming; (b) the outcomes of such analyses would be hard to
organize and visualize in an efective way (e.g., discover correlations); (c) it would prevent
discovering semantical knowledge that could be only found by adopting a multi-modal analytical
approach on a vast amount of data [
In such a line, a recent work introduced a unified multi-modal analysis tool, DanXe, an
Extended Artificial Intelligence framework that blends (i) AI algorithms for digitization and
automated analysis of both tangible and intangible materials, with the goal of crafting a digital
replica of dance cultural heritage, and (ii) XR solutions for immersive visualization of the
derived insights. This framework introduces a novel space for the concurrent analysis of all
elements that define the essence of dance. [
Tools like DanXe can be employed to digitalize dance heritage and at the same time accelerate
the labeling of multi-modal dance data, which can be used to train multi-modal models, that
can be employed to improve heritage preservation and analysis. Nevertheless, the integration
of human experts is required to ensure the quality of generated data and provide novel and
connected knowledge to those. For this reason, we here propose an extension of the DanXe
framework to inject a human-in-the-loop (HITL) component that leverages the initial AI-inferred
annotations as a foundation, enabling a semi-automatic approach to provide high-quality labels.
This approach aims to facilitate the creation of richer and more accurate datasets to support the
optimization of future heritage preservation models. We here contextualize such an approach
for a multi-modal dance data annotation process, considering the specific case of choreography,
where there is a lack of datasets that capture fine-grained labels of specific dance moves, often
focusing on the general style [
We here provide a detailed overview of the materials and methodologies employed in our study. We begin with the Video Dataset subsection, which describes the collection and characteristics of the video data used for analysis. Following this, the AI Augmented Human Annotator subsection outlines the HITL approach that leverages an AI Dance toolbox to enhance human annotation eficiency and accuracy. Finally, the Visual Annotation Tool Integration Middleware subsection discusses the middleware designed and implemented to seamlessly integrate the synthetic AI-generated annotations in a visual annotation tool, facilitating a cohesive and streamlined workflow for annotation domain experts. Each subsection aims to elucidate the integral components and techniques critical to our research process.
The dataset was created using materials from the case study, which were originally recorded on iflm, distributed in cinemas and on VHS, and later digitized. The original recording format often sufered from wear and tear (e.g., film damage, darkening). Additionally, the original intended use, designed for cinemas or home video viewing, included video direction elements such as close-ups, zooms, and fade-ins/outs. These elements often cover movements and are not ideal for a comprehensive recording of the performance. The process of selecting a video for building the initial basic dataset was therefore inevitably influenced by the need for well-lit footage, the highest possible definition, and minimal directorial interventions. To further reduce noise (e.g. background dancers, extras), it was decided to analyze a solo performance: Nureyev’s adagio of Prince Siegfried in Swan Lake (Act I). In the analysis of this adagio, we’ve focused on the initial 20 seconds of choreography. Here, the dancer transitions from a static pose, embodying their character without movement, to a sequence of steps performed in place. These steps showcase a range of volumes and heights, adding depth and dimension to the performance.
Considering our main use case, choreographic-related data, various labels, and information can
be inferred, including music, dance styles, individual dance moves, and background descriptions.
Some of this information could be inferred with a high degree of accuracy by modern DL
approaches, like the ones introduced in the DanXe platform [
This annotation layer assumes that all synthesized AI label data are stored in a local database after their inference. A data adaptation middleware ingests and transforms the various data formats inferred by diferent AI models, ensuring compatibility with the visual annotation tool at hand. This setup enables human annotators to use the tool to correct and add new labels on top of the existing information. Subsequently, the updated annotations are re-adapted and stored in the database, following the inverse chain of processes. This iterative approach facilitates the eficient enhancement of dance video annotations, leveraging both AI and human expertise. To implement such a framework, a fundamental step amounts to defining a smart middleware able to bridge diferent file formats and data structures coming from AI models and make them interpretable from diferent visual annotation tools. For this reason, in the following, we will describe the general architecture of the middleware we defined to ingest and adapt annotations coming from diferent AI tools to visual annotation tools.
In response to the growing complexity of data formats and interpretation regarding diferent tasks (e.g. Human Pose Estimation), a middleware solution has been developed to foster interoperability between diverse AI models and various visual annotation tools. This middleware serves as a bridge, facilitating seamless communication and data exchange between diferent components of the annotation pipeline. Its architecture is reported in Figure 2. Implementing standardized interfaces and protocols, enables the integration of multiple deep learning models, each specializing in diferent aspects of visual analysis, such as pose estimation or object detection. Simultaneously, the middleware performs a conversion of the ingested data respecting a range of diferent visual annotation tools interfaces, providing a unified platform for annotators to interact with and refine the output of these models. Through this interoperability, the annotation process is augmented, ofering annotators the flexibility to leverage the strengths of diferent models while at the same time having user-friendly interfaces. Moreover, by automating certain aspects of annotation and providing semi-automatic functionalities, the middleware accelerates the annotation workflow, significantly reducing the time and efort required to generate high-quality annotated datasets.
We concretely applied our introduced methodology to accelerate single dance moves annotation from a multi-modal perspective (i.e., linking human pose estimation and single dance moves). To the best of our knowledge, this is the first attempt to do so through a custom-defined middleware and semi-automatic approach.
In particular, we took as a use case choreographical human pose estimation by using the AlphaPose models 1 that were included in the DanXe pipeline. AlphaPose allows to extract and 1https://github.com/MVIG-SJTU/AlphaPose track multi-person poses, codified into 17 body key points when used in the model trained on the COCO dataset [17]. In our case, we applied it for a variation from Swan Lake performed by Rudolf Nureyev in 1967. The human pose estimation extracted from AlphaPose is stored in a JSON file which contains one record per each frame where a person was detected. Some visual representations of the inferred key points are reported in Figure 3 while an example of the resulting JSON file is provided in Listing 1.
Listing 1: Human pose estimation JSON data generated by AlphaPose on a single image. 1 { 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
In this example, there was only one person identified (category ID 1), indicating the human ID within the considered frame. The key points array contains precise x and y coordinates along with confidence scores for various body joints, exemplified by the first key point positioned at (311.995, 307.967). We do not include the confidence score provided for each key point inferred for description simplicity. As mentioned, those are 17 key points, corresponding to the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. Each key point serves as a precise indicator of a specific body part’s location within the image frame. The overall confidence in the pose estimation is quantified by a score of 3.001. Also, information related to the bounding box enclosing the detected human figure is inferred but was not reported for simplicity.
Starting from this representation, we then considered adapting it for our target visual annotation tool Vidat 2, which could be exploited. Vidat is a high-quality video annotation tool for computer vision and machine learning applications that is simple and eficient to use for 2https://github.com/anucvml/vidat a non-expert and supports multiple annotation types including temporal segments, object bounding boxes, semantic and instance regions, and human pose (skeleton). Moreover, it is completely data-driven: all the data can be stored and loaded by encoding them in a predefined key-value structure (i.e., a JSON file). Our goal was to load the annotated data from AlphaPose in a format readable by Vidat. However, the Vidat skeleton structure description does not take into account the elbow data. This means that we first filtered out the data per each detection and then re-adapt the remaining information to match the reading structure of the Vidat tool. The resulting JSON is reported in Listing 2.
Listing 2: JSON representation of video annotations and configurations. 1 { 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 ..., "objectAnnotationListMap": {}, "regionAnnotationListMap": {}, "actionAnnotationList": [], "skeletonAnnotationListMap": { "0": [{ ..., "pointList": [ { "id": 0, "name": "nose",
"x": 312.0, "y": 308.0}, { "id": 1, "name": "left eye",
"x": 312.0, "y": 304.0}, { "id": 2, "name": "right eye",
"x": 315.0, "y": 305.0}, ...
], "centerX": 315.67, "centerY": 346.13} ] }, "config": { "objectLabelData": [...], "actionLabelData": [...], "skeletonTypeData": [...] }}}
The provided JSON encapsulates metadata crucial for video annotation and analysis. Within its structure, key parameters such as video dimensions, frame rate, and duration are outlined, essential for Vidat temporal analysis and processing (not reported in the example for simplicity). The inclusion of keyframe listings ofers strategic markers for video segmentation and analysis, facilitating eficient data handling. Furthermore, the presence of object and region annotation maps anticipates future expansion into object detection and spatial characterization. The delineation of action annotation lists underscores the intention to annotate dynamic data. Particularly noteworthy is the skeleton annotation list, which furnishes detailed skeletal representations. The configuration segment provides an extensive catalog of object and action label data, coupled with skeleton-type specifications, forming the cornerstone for semantic understanding and classification in video content.
Finally, since this JSON is aligned with the original video frames, it can be loaded into the Vidat visual annotation tool. Our dance domain expert used the inferred human key-point labels to add new dance move labels, supported by the already-generated dance poses. Each label corresponds to a name (e.g., arabesque) and a time interval, representing the duration of the step execution. After completing the label descriptions, the next step would normally amount to the labeling of human movement frame by frame, but those were already labeled automatically generated, so the domain expert only corrected minor interpolations or mismatches between the skeleton and the video image. Finally, the dance domain experts annotated dance moves linked with one or more inferred poses. The resulting JSON can be stored at any moment, and will now include both skeleton and dance move label data. The outputs of such a process are visually reported in Figure 4.
The introduction of the DanXe framework represents a significant leap forward in digitizing and analyzing dance heritage materials, ofering promising capabilities for the automatic annotation of archive videos. Supported by human oversight and augmented by XR technologies, the proposed multi-modal, semi-automatic annotation framework signifies a substantial advancement in cultural heritage conservation, especially in cases involving intangible heritage alongside tangible assets. Given the unique nature of the analyzed case study (that of an archival collection related to a dancer’s legacy), the annotations cannot be limited to just recognizing steps but must also allow for tracking props, stage settings, and performers involved. This would enable the preservation of all scenic components that contributed to a choreographic creation, ensuring better preservation, facilitating restaging processes, and enriching the process of metadata creation, which is typically limited to principal performers or even just the choreographer. Providing a tool that can support the work of scholars and archivists, without replacing their expertise but leveraging it to validate semi-automatic acquisitions, not only represents a valuable contribution in expediting their work but also enriches the metadata associated with archival sources, thus enabling user research. This approach promises to generate richer, more accurate datasets, ultimately fostering a deeper understanding and appreciation of the art form.
This work was partly funded by: (i) the PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - “FAIR - Future Artificial Intelligence Research” - Spoke 8 “Pervasive AI”, funded by the European Commission under the NextGeneration EU program. mixed quantitative-qualitative analyses, in: Proceedings of the 2022 ACM Conference on Information Technology for Social Good, 2022, pp. 161–166. [17] H.-S. Fang, J. Li, H. Tang, C. Xu, H. Zhu, Y. Xiu, Y.-L. Li, C. Lu, Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time, IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).