The development methods are based on tools such as FastAPI, HTML, and Python, which guarantees efficient processing and analysis of biological data. The results of the work include algorithms and software implementation of a web application for self-testing for COVID-19 and a mobile application for the Android platform. The system is characterized by high accuracy and efficiency in determining test results, which makes it potentially useful for a wide range of medical research and improving biological sample collection systems.
Automated collection of biological samples becomes particularly relevant in pandemic and epidemic scenarios, particularly during the spread of COVID-19. Fast and effective detection of infections and diseases is a key task for global health. The successful implementation of this task directly affects the quality of medical diagnostics, allowing for timely detection of diseases and effective treatment. Automating the sample collection process can significantly improve accuracy and reduce the risk of errors, which is critical for medicine.
The use of computer vision and machine learning technologies, such as MediaPipe Face Mesh and YOLOv5, not only automates the sample collection process but also ensures high accuracy and reliability of the results. This is especially important for the swabbing procedure, which plays a key role in the diagnosis of diseases. Automation of this process not only ensures patient safety and comfort but also significantly reduces the time of medical intervention.
In the global context, automation and the introduction of artificial intelligence are important trends in medicine. These technologies are changing approaches to treatment and diagnostics, making them more efficient and accessible. The main challenge remains improving the accuracy of detectors and integrating the latest technologies with medical practice. In this context, the introduction of new algorithms and methods for collecting biological samples is of considerable practical value.
The purpose of this work is to develop an innovative approach to the automated collection of biological samples using advanced technologies that will increase the efficiency of diagnostics and improve the quality of medical research.
This study takes a comprehensive approach that combines technological and scientific
methods to develop and improve an automated biological sample collection system [
One example of automated biological sample collection is the use of an algorithm to detect
a smear in an image. This algorithm utilizes advanced technologies such as MediaPipe Face
Mesh [
Many works focus on the development of algorithms for detecting and classifying
biological objects in images, including smears. The use of deep learning methods, such as
convolutional neural networks (CNNs), has shown high accuracy in identifying various
elements in images, providing a basis for further research. But there are also challenges. For
example, the problem of determining the correctness of biological sample collection remains
relevant. Some works consider techniques for determining the angles and position of the face,
which can affect the detection accuracy. Studies [
The MediaPipe Face Mesh library was used to detect landmarks face and nose elements [
Methods for creating and comparing polygons to assess the correctness of swab insertion into the nostril: 1. Data collection and processing. This includes collecting images from different angles, and manual labeling of key elements (face, nose, tip of the swab). 2. Analysis and selection of algorithms. Evaluation. MediaPipe Face Mesh and YOLOv5 for face and swipe detection, and optimization of image processing through OpenCV image processing via OpenCV. 3. Development and testing. Implementation of algorithms for searching face and swipe detection in real-time, validation using real samples. 4. Integration and optimization. Combining algorithms into a system, optimization for stable operation. 5. Validation and testing. Comparison of results with alternative methods, and testing of the demo application. 6. Evaluation of the contribution. Analysis of the impact of algorithms on the process of collecting on the process of collecting biospecimens. 7. Ethical aspects. Ensuring the confidentiality of patient data of patients and compliance with ethical standards.
This methodology covers all stages, from data collection to testing and implementation of the developed algorithms in medical practice.
Methods and technical solutions for the development of an automated biological sample collection system include the use of MediaPipe Face Mesh and YOLOv5 technologies for face and swab identification (data labeling was performed using Label Studio). Image optimization was performed using OpenCV, and the Shapely library was used to work with geometric objects. These solutions ensure the high accuracy of the system. The interface for user interaction is based on FastAPI (Fig. 1).
MediaPipe Face Mesh was used to effectively detect the face and nose elements. It has the
main advantage of being able to accurately determine the shape and location of the face in the
image, taking into account its various aspects. This technology is able to detect even subtle
details such as moving facial lines or nose contours[
Swipe detection with YOLOv5: The YOLOv5 model was used to accurately determine the position and contour of the swab in the images, which ensured fast and efficient object detection. Image optimization with OpenCV: OpenCV helped to improve swab tip detection through Canny and HoughLinesP functions, which allowed to detect the contours and direction of the swab, increasing detection accuracy. Polygon processing with Shapely: The Shapely library was used to create and analyze polygons, identifying the nose elements and the swab tip area, which improved interaction with geometric objects. Integration and efficiency: The algorithms were integrated into a single system where MediaPipe Face Mesh was used to determine the face and key points. Optimization of system performance and stability ensured reliable real-time operation. Image preprocessing focused on the white lines of the swab. Interaction with the user via FastAPI: FastAPI provided the development of an intuitive web interface for presenting results and interacting with the system.
The initial stage of the algorithm involves image analysis to determine the position and orientation of the swab (Fig. 1). For this purpose, YOLOv5 is used to efficiently detect the swab object in the photo. The YOLOv5 model was trained on an extended dataset that included different positions and states of the swab.
The YOLOv5 model is used to detect the swap object, which was previously trained on an extended dataset.
The YOLO (You Only Look Once) algorithm provides an efficient and accurate location and outline of the swab in the image (Figure 2).
Image preparation and processing: The image is processed to improve the accuracy of swab detection. For this purpose, the OpenCV library is used to optimize and reduce noise. The Canny and HoughLinesP methods are used to determine the contours and direction of the swab (Fig. 2).
Integration of swap detection results: The results of the swab detection are combined with the subsequent stages of the algorithm. The coordinates and orientation of the swipe are used to determine its position in space. Interaction with MediaPipe Face Mesh: The results of swipe detection are integrated with data from MediaPipe Face Mesh to accurately determine the relationship between the swipe and the face in the image. Analysis and filtering of results: The results are analyzed to detect errors and correct unacceptable variants, for example, by checking the shape and size of the swipe. An integrated algorithm provides high accuracy in swab detection and positioning, which is critical for automated biological sample collection.
MediaPipe Face Mesh is used to accurately detect the face and nasal features. Facial landmarks are used to determine the direction of gaze and the area of the nose where the tip of the swab should be located (Figure 3). Based on the swab detection and facial landmarks, the position and orientation of the swab relative to the face and nose are accurately determined. OpenCV is used to optimize image processing and determine the exact angle of the swab.
All components of the algorithm were integrated into a single system and tested on a variety of images and scenarios. Testing included trying the algorithm on images with different lighting, angles, and backgrounds. The details of the image pre-filtering to detect the swab line by emphasizing the white elements are shown in Figure 4.
The results obtained in the process of detecting and collecting biological samples indicate
the high accuracy of the developed algorithm. The performance analysis indicates that the
algorithm is effective even in real-time, even with limited computing resources. Tests of the
algorithm on a variety of images, including variable illumination, different angles of
inclination, and a variety of backgrounds, confirm its stability and reliability in different
conditions [
The development of the algorithm was an important step in automating the collection of biological samples, characterized by high efficiency due to extensive data and powerful tools. YOLOv5, MediaPipe Face Mesh, OpenCV, and Shapely technologies allowed us to create a system for accurately determining the swab's position in real-time, which is critical for medical applications.
These systems have the potential not only to combat COVID-19, but also for diagnostics, health monitoring, and scientific research. The success of the project opens up opportunities for further improvement of the algorithms, adaptation to different imaging conditions, and expansion of functionality.
The project serves as a basis for future research in medical technology and deep learning, which can significantly improve diagnostics and monitoring in modern medicine. Our work demonstrates the potential of using deep learning and computer vision in medicine, and we hope that this will contribute to the development of automated and accurate biological sample collection systems.