Psychomotor learning develops our bodies in organized patterns with the help of environmental signals. With modern sensor arrays, we can acquire multimodal data to compare activities with stored reference models of body motions. To do this on a large scale in an eficient way, we need cloud-based infrastructures for the storage, processing and visualization of psychomotor learning analytics data. In this paper, we propose a conceptual sensor stream processing pipeline for this purpose. The solution is based on the so-called MLOps approach, a variation of the successful DevOps model for open source software engineering for large-scale machine learning solutions based on standard components. This processing pipeline will facilitate the multimodal analysis of many training scenarios collaboratively.
Multimodal stream data ingestion Multimodal sensor applications generate vast amounts
of heterogeneous and multidimensional data with unique properties [
Multimodal data processing pipelines The usage of processing pipelines for multimodal
ML data has been discussed and compared in several publications [
ML artifact management A versioning system for ML models such as DVC 3 provides crucial tracking and managing of changes to the ML pipeline. A system like TorchServe 4 can then be used to publish and serve the diferent versions of the ML models. While diferent stages of the MLOps pipeline can be supported by separate tools, an integrated platform like OpenML 5 can provide all the benefits in a central place.
1https://kafka.apache.org 2https://www.influxdata.com 3https://dvc.org/ 4https://github.com/pytorch/serve 5https://openml.org
practice that aims to reliably [
A multimodal sensor network produces continuous data streams with various sampling frequencies, which should be processed in near real-time to ofer learning analytics and learner feedback [26]. Figure 1 gives an overview of the proposed MPC infrastructure, which is structured in three phases: data ingestion (phase 1), machine learning (phase 2) and data exploitation (phase 3). This contribution is based on the MPC architecture [27].
Feature pipeline (phase 1) - Multimodal stream data ingestion First, a data ingestion system (1.1) should be used to provide scalable communication between the sensor devices and the MPC. Annotating the raw sensor data is done in the next step of the pipeline (1.2) to provide the labels necessary for the ML feature detection and decision-making. To allow repeatable ML training, the data from multiple concurrent source streams need to be aggregated and stored with some versioning mechanism. A TSDB provides mechanisms for versioned data storage and analysis (1.3) and should thus be used for data ingestion and leverage.
ML pipeline, continuous training (phase 2) Next, we use a Federated ML approach for model building and evaluation (2.1) [28]. Trained and evaluated ML models need to be versioned (2.2). Once a new version of a model is checked into the storage system, it needs to be deployed and made available (2.3). After training and serving the ML model, the deployed pipeline needs to be monitored and continuously evaluated for the model performance (2.4). The measures used for this evaluation need to be defined by domain experts and calculated automatically in the context of the project depending on the specific ML use case. Alternatively, a MLOps solution (2.X) can provide an integrated solution for model building, versioning and serving.
If a drop in accuracy or a suficiently large amount of new input data (e.g. a new expert recording) is detected through pipeline monitoring (2.4), a retraining of the ML pipeline (phase 2) can be triggered as described by continuous training. For example, an ML algorithm could be trained to diferentiate whether the feedback given by the system correlates with positive learner progress and change the feedback mechanism if it does not.
Data exploitation & feedback (phase 3) With a properly trained and deployed ML model, a frontend application can then use it to provide feedback to the learner (3.1), either directly in near real-time through interactive AR/VR elements, or long-term performance reports and other gamification elements. The usage of explainable AI principles could provide an additional layer of context to the decisions made by the ML algorithm which would benefit the domain experts and learners in gaining insights into the feedback mechanism. This would not only reduce potential bias embedded in the ML model but also enhance the trust of the end-users in the system leading to better learning results. Finally, new multimodal sensor data is generated through learner interaction with the system (3.2) leading to a new MPC pipeline cycle with further data ingestion.
Legend 1 2 3 Data storage & versioning
1.3 Time-Series DB InfluxDB
Continuous training ML performance monitoring
2.4 Pipeline monitoring
Evidently, ...
Learner progress Data generation Multimodal sensors
3.2 Offline tool Cloud tool
Data flow 1 / 1.1 nPuhmasbee/rstep
Pipeline flow Pipeline cycle Stream data ingestion
Data collection Apache Kafka
1.1 Model building & evaluation 2.1 Federated ML PyTorch, ...
Data annotation Data annotation PSI, VIT, ... 1.2
Model storage & versioning 2.2 Model versioning Git, DVC, ...
Model deployment & serving Model serving TorchServe, ...
2.3
OR Integrated MLOps Platform 2.X ClearML, MLReef, Kubeflow, MLFlow, ...
Learner feedback Direct feedback AR/VR, Reports, ... 3.1 Pipeline phase Pipeline step
At diferent stages of the proposed pipeline, diferent stakeholders are involved in decisionmaking, execution and practice. The stakeholder groups involved in the project range from domain experts to end-users, including pedagogical specialists, data science engineers, coaches, teachers and students. Recording and annotation of training data and raw sensor readings are performed by sports scientists due to their expertise in the psychomotor skills to be taught using the proposed pipeline.
The intricacies of the ML training and evaluation need to be executed by experienced ML experts, while the result of the “data exploitation & feedback” phase will mostly benefit coaches, teachers and learners. For efective interaction with the toolset used in the MPC, each stakeholder group needs to be provided with a diferent set of interfaces. Sports scientists need a simple way to feed their data into the system and a powerful annotation engine to add meaning to the multimodal data stream. Application developers and designers can make use of the data stored in the TSDB together with the trained and deployed ML models to create immersive learning environments and provide learner feedback. Since the proposed pipeline will be running its toolset in a cloud with powerful hardware, ML experts and researchers will have access to computing resources to process the TSD, so they can focus on model training and development. The versioning in the MLOps pipeline enables coaches, teachers and students not to require extensive ML expertise to make use of the best-performing version of the ML models. The proposed cloud pipeline enables the stakeholders to collaborate on diferent aspects of the immersive learning environment, from data ingestion and annotation over model training to interaction with the learners.
This contribution presents a ML-based sensor data processing pipeline to facilitate direct and long-term learner feedback in psychomotor education. Pipeline automation through MLOps is critical in ensuring reproducibility, helping to reduce manual infrastructuring inherent with ML development. The usage of XAI to support the ML process is beneficial in facilitating trust and providing individualized feedback for multimodal psychomotor learning scenarios. Future work includes further implementation and evaluation of the proposed framework in a practical environment. We will focus on an end-user-oriented MLOps execution and evaluation.
We thank Aleksandra Nekhviadovich for their help in the study. This work was funded by the German Federal Ministry of Education and Research (BMBF) within the project “MILKI-PSY”6 under the project id 16DHB4015.