Workplace accidents are a critical problem that causes many deaths, injuries, and financial losses. Given this impact on a large amount of people, it is important to proactively find unsafe environments where injuries could occur by detecting the use of personal protective equipment (PPE) and identifying unsafe activities. Thus, we propose an industrial workplace alerting and monitoring platform to detect PPE use and classify unsafe activity in group settings involving multiple humans and objects over a long period of time. Our proposed method is the first to analyze prolonged actions involving multiple people or objects. It benefits from combining pose estimation with PPE detection in one platform. Additionally, we propose the first open source annotated data set with video data from industrial workplaces annotated with the action classifications and detected PPE. The proposed system can be implemented within the surveillance cameras already present in industrial settings, making it a practical and efective solution. If accepted, we hope to present this work as a short paper or as a poster.
Every year, almost two million people die from work- Object detection is a technique commonly used in comrelated accidents or diseases [1], and another 317 million puter vision for a variety of problems. Significant prior sufer from work-related injuries [2]. The economic cost work has been done to use object detection to detect the of these illnesses, injuries, and deaths is nearly three use of PPE [6-9]. [6, 7] use convolutional neural nettrillion dollars annually [3]. Beyond the direct toll on works (CNNs) to detect the use of PPE (hard hat, vest, afected workers and medical expenses, workplace in- and shoes), while [8] can detect PPE (hard hat and vest) juries lead to loss of productivity for all involved in the but also analyzes the safety level using the present PPE. accident, property damage, and ultimately worsening If either a hard hat or vest is missing, then the output morale among all workers [4]. Accounting for 39 percent states the missing PPE. [9] considers PPE detection in the of work related injuries, overexertion is the most com- context of COVID-19 and detects PPE such as a face mask, mon cause of these injuries [5]. Overexertion consists of face shield, and gloves. However, these prior methods lifting, lower, and other repetitive actions, which are all have not been combined with activity detection methergonomic issues. Another 31 percent of workplace in- ods to create a platform-based approach. Our proposed juries comes from contact with other objects [5], making platform-based approach would allow industrial workthe use of personal protective equipment (PPE) critical. places to have one system integrated within their cameras
In this work, we propose an AI-powered industrial in order to monitor their workplace. workplace alerting and monitoring platform in order to Furthermore, pose estimation is widely used to study improve workplace safety. We will detect personal pro- human action within a specific context. For analyzing tective equipment (PPE) compliance along with unsafe unsafe activity within workplaces, previous work has activity within the workplace using pose estimation. Our focused on using pose estimation to monitor workplaces work will take a platform approach, allowing for easy im- for ergonomic safety. [10] use deep learning to estimate plementation within the CCTV cameras already present the risk factor involved in lifting boxes, but cannot be within many industrial workplaces. Finally, we will re- generalized for more actions and does not account for lease an open-source annotated dataset containing video the time taken per action. [11] use machine learning (randata from industrial workplaces annotated with the ac- dom forest classifier) to classify activity in workplaces tion classifications and detected PPE. using sensors located on the foot but cannot determine when a set of actions are deemed to be unsafe. [12] use multiple machine learning algorithms to classify actions, but uses a time threshold to determine when a set of actions is unsafe. This does not account for human-object interactions and for the dynamic nature of time thresholds (lifting a heavy box for a period of time is diferent from lifting a light box for the same period of time). [13] can ergonomically analyze the actions of a single human but cannot diefrentiate between people and recognize human-object interactions. [14] determine the risk factor of a human’s pose but only looks at a single frame rather than a period of time. Our proposed method is the first to generalize unsafe activity detection to multiple people and/or objects over an extended period of time using pose estimation and action classification.
Finally, there is no open-source dataset on industrial workplace actions, so we propose the first open-source annotated dataset containing images of industrial workers along with their classified action and detected PPE. 3. Methods label and their actions will be assigned to them. A report will be generated detailing each human’s action and the time taken per action, and this information can be used to determine whether a given set of actions is repetitive and can be deemed as unsafe. To detect human-object interactions, such as a human lifting a box or driving a forklift, we will use a human-object interaction detection method. We will experiment with multiple methods described in [19-21]. This will allow the platform to detect human-object interactions in addition to the specific action itself, which can be used to determine the risk level of a set of actions. After estimating the time taken and repetitive count for a set of actions per person, the data is then used to evaluate the postures using the Occupational Safety and Health Administration’s guidelines [22].
When a person’s actions are deemed as unsafe, an alert is sent to prevent potential future injuries.
We will annotate and release an open-source dataset based on real industrial workplace actions. We have collected raw data from video footage from sources such as Youtube, Shutterstock, Pixabay, and other web-based sources which contain videos of factories. After compiling the raw data, we have split the video into individual frames to find specific actions at a given time. We are in the process of annotating and labeling the dataset using Labelbox [15] with the correct action classification. The action will be annotated as slight bend, extended bend, random moves (small moves between actions), stand, or walk. Additionally, we will provide an annotation with respect to a given object, such as a box when such objects are present. In the case of multiple persons in a frame, we will assign the labels to the individual persons and de- 4. Conclusion and Future Work note their specific actions. Figure 1 shows two annotated frames from the proposed dataset. In this paper, we proposed a platform to improve work
Our platform will combine PPE detection with unsafe place safety. In addition, we are preparing an open source activity detection to create a unified AI-based system dataset to help train future models. Our method is the that can be easily integrated within cameras in industrial first to generalize the detection of unsafe activities to workplaces. To perform PPE detection, we will use a scenarios that contain multiple people and objects over pose estimation approach rather than an object detection a period of time. This can be done when multiple people approach, as used in [16]. This is motivated by the fact and objects are within the same frame, and we will also that, for the detection of PPE specifically, it is important account for tasks involving multiple people for the same to localize diferent body parts to know which protective task. Additionally, our method is the first to combine equipment should be present. For example, a hard hat PPE detection and unsafe activity detection into a single should be worn on the head, so the head must first be platform that can be implemented in CCTV cameras to localized using pose estimation to determine whether a monitor industrial workplaces in real time. hard hat is present. A similar technique can be used for Our future work will focus on deploying the platform protective vest detection and shoe detection. within the CCTV cameras which are already present in
In order to detect unsafe activity, we will use pose esti- factories today. Our platform does not require any spemation to first classify individual poses. This can be done cialized hardware other than a generic camera, making using OpenPose [17], which identifies a skeleton of 18 it a low-cost and practical solution to be implemented points at various key points that make up a human and within industrial settings. Our proposal demonstrates will use those points to determine the action of the hu- the feasibility of using an AI-based platform to improve man through the distances and angles between the points. workplace safety in industrial settings, potentially saving Using a multiple-target tracking (MTT) method as de- many lives. scribed in [18], each human in the frame will receive a References [15] "Labelbox | Software platform for building AI [1] World Health Organization. "WHO/ILO: Almost 2 with unstructured data." N.p., n.d. Web. 13 Sep. 2022 Million People Die from Work-Related Causes Each Year." https://labelbox.com (2021). [16] Xiong, Ruoxin, and Pingbo Tang. “Pose [2] Woolford, Marta Helen, et al. "Missed opportuni- Guided Anchoring for Detecting Proper Use ties to prevent workplace injuries and fatalities." New of Personal Protective Equipment.” Automasolutions: a journal of environmental and occupational tion in Construction, vol. 130, 2021, p. 103828., health policy 27.1 (2017): 16-27. https://doi.org/10.1016/j.autcon.2021.103828.
[3] Safety+Health oficial magazine of National Safety [17] Cao, Zhe, et al. "Realtime multi-person 2d pose Council of United State of America for Congress and estimation using part afinity fields." Proceedings of the Expo dated 06th September 2017 on Global cost of work- IEEE conference on computer vision and pattern recogrelated injuries and deaths totals almost USD $ 3 trillion. nition. 2017.
[4] DIRECT AND INDIRECT COSTS OF AC- [18] Kamkar S, Ghezloo F, Moghaddam HA, Borji CIDENTS. Course 700 - Introduction to Safety A, Lashgari R. Multiple-target tracking in human Management. (n.d.). Retrieved September 4, 2022, from and machine vision. PLoS Comput Biol. 2020 Apr https://www.oshatrain.org/courses/pages/700costs.html 9;16(4):e1007698. doi: 10.1371/journal.pcbi.1007698.
[5] “Three Leading Causes of Workplace Injury PMID: 32271746; PMCID: PMC7144962. and How to Prevent Them - South Shore Ortho- [19] Ulutan, Oytun, A. S. M. Iftekhar, and Bangalore S. pedics.” South Shore Orthopedics, 15 Oct. 2021, Manjunath. "Vsgnet: Spatial attention network for detectsouthshoreorthopedics.com/three-leading-causes-of- ing human object interactions using graph convolutions." workplace-injury-and-how-to-prevent-them. Proceedings of the IEEE/CVF conference on computer [6] Nath, Nipun D., Amir H. Behzadan, and Stephanie vision and pattern recognition. 2020. G. Paal. "Deep learning for site safety: Real-time detec- [20] Gkioxari, Georgia, et al. "Detecting and recogniztion of personal protective equipment." Automation in ing human-object interactions." Proceedings of the IEEE Construction 112 (2020): 103085. conference on computer vision and pattern recognition.
[7] Balakreshnan, Balamurugan, et al. "PPE compli- 2018. ance detection using artificial intelligence in learning [21] Kim, Bumsoo, et al. "Hotr: End-to-end humanfactories." Procedia Manufacturing 45 (2020): 277-282. object interaction detection with transformers." Proceed[8] Delhi, Venkata Santosh Kumar, R. Sankarlal, and ings of the IEEE/CVF Conference on Computer Vision Albert Thomas. "Detection of personal protective equip- and Pattern Recognition. 2021. ment (PPE) compliance on construction site using com- [22] “Department of Labor Logo United States puter vision based deep learning techniques." Frontiers Department of Labor.” Ergonomics - Identify Probin Built Environment 6 (2020): 136. lems | Occupational Safety and Health Adminis[9] Protik, Adban Akib, Amzad Hossain Rafi, and Shah- tration, https://www.osha.gov/ergonomics/identifynewaz Siddique. "Real-time Personal Protective Equip- problems#risk-factors. ment (PPE) Detection Using YOLOv4 and TensorFlow." 2021 IEEE Region 10 Symposium (TENSYMP). IEEE, 2021.
[10] Liu, Ya, et al. "AI-based framework for risk estimation in workplace." Aggression and Violent Behavior (2021): 101616.
[11] Fridolfsson, Jonatan, et al. "Workplace activity classification from shoe-based movement sensors." BMC biomedical engineering 2.1 (2020): 1-8.
[12] Ray, Soumitry J., and Jochen Teizer. "Real-time construction worker posture analysis for ergonomics training." Advanced Engineering Informatics 26.2 (2012): 439-455.
[13] Martin, Chris C., et al. "A real-time ergonomic monitoring system using the Microsoft Kinect." 2012 IEEE Systems and Information Engineering Design Symposium. IEEE, 2012.
[14] Paudel, Prabesh & Choi, Kyoung-Ho. (2020).
A Deep-Learning Based Worker’s Pose Estimation. 10.1007/978-981-15-4818-5_10.