Proposals for Improving the Assessment of Medical Device Software in Thailand

Abstract

The registration of Medical Device Software (MDS) and Software as a Medical Device (SaMD) with the Food and Drug Administration (FDA) is a prerequisite before entering the market. The FDA relies on several international standards as regulatory benchmarks to ensure the quality of MDS. Key components of this regulatory framework include IEC 62340 [1], ISO 14971 [2], and ISO 13485 [3]. Our experience assessing MDS in Thailand highlighted common challenges manufacturers face during software evaluation. Notably, clause 6 (Software Maintenance Process) and clause 8 (Software Configuration Management Process) demonstrate the highest rates of evaluation failure. Clause 7 (Software Risk Management Process) and clause 9 (Software Problem Resolution Process) closely follow, ranking as the secondhighest areas of concern regarding evaluation failures. The primary factor contributing to these software evaluation challenges is a deficiency in knowledge and understanding of IEC 62304 [1]. To address this issue, we propose a solution in the form of a chatbot designed to assist manufacturers in comprehending and generating IEC 62304-compliant documents. medical device software, software as a medical device, Thais' medical device software assessor, medical device software obstacle, medical device software quality assessment.01 PCWrEooUrckResehdoinpgs ISSNc1e6u1r-3w-0s0.o7r3g

1. Introduction

In Thailand, Medical Device Software (MDS) and Software as a Medical Device (SaMD) are required to register with the Thailand Food and Drug Administration (Thai FDA) [ 4 ]. The Thai FDA has established criteria aligned with international standards, including mainly ISO/IEC 62304:2006 - "Medical device software - Software life cycle processes" (IEC 62304) [ 1 ], ISO 14971:2019 - "Medical devices - Application of risk management to medical devices" (ISO 14971) [ 3 ], ISO 13485:2016 - "Medical devices - Quality management systems - Requirements for regulatory purposes" (ISO 13485) [ 3 ], and IEC 60601-1 clause 14 (IEC 60601), which pertains to Programmable Electrical Medical Systems (PEMS) for medical electrical devices [ 5 ].These standards outline the processes, activities, and configuration tasks that form a holistic framework for developing MDS.

IEC 62304 [ 1 ] encompasses six processes outlined in clauses 4 to 9. Each clause specifies a breakdown into sub-clauses, activities, and tasks. These subclauses are interconnected with other clauses and sub

All MDS must undergo testing in adherence to these standards, following the stipulations of the Thai FDA [ 6 ] requirements. The procedure for registering MDS with the Thai FDA is detailed in Section 1.1 of the Regulatory Framework for Medical Device Software in Thailand. Additionally, section 1.2 describes the MDS Software Quality Assurance and Assessment ecosystem, providing a comprehensive overview of the processes and standards involved in ensuring the quality, safety, and regulatory compliance of MDS in the Thai context.

1.1. Regulatory Framework for Medical Device Software in Thailand

The oversight of MDS falls under the scope of the Medical Device Control Division (MDCD) of the Thai FDA [ 6 ]. Thai FDA [ 6 ] relies on the Health and Science Authority (HSA) of Singapore [ 7 ]. The Thai FDA [ 6 ] mandates a two-step process for registering MDS and other medical devices. In the first step, known as "Establishment Licensing," the medical device manufacturers must provide business registration documents, complete request forms, and submit other relevant government documents. This step aims to verify the manufacturer credentials, enabling oversight of the quality of medical devices by restricting importation locations for production and storage. The second step, "Product Registration," necessitates manufacturers to submit comprehensive documentation about the medical device. This includes details such as the device description, intended use, indications, instructions for use, storage conditions, shelf life, contraindications, warnings, precautions, potential adverse effects, alternative therapy options, materials used, product specifications, and the production development flow chart. The submission must align with the essential principles of safety and performance of the medical device as stipulated by the ASEAN Medical Device Directive, EU regulations, Singapore standards, and other applicable guidelines.

Moreover, the submission should summarize verification and validation, incorporating pre-clinical studies, clinical evidence, test reports, clinical evaluation reports, and clinical data. Additionally, the marketing history and safety declaration template documentation must be included. The inclusion of risk management processes that comply with ISO 14971, such as the risk plan, risk control measures, and the risk report, is imperative. A valid certificate of compliance with ISO 13485 or GMP for medical devices and ISO 9001 should be part of the submission. Lastly, the package should also contain a declaration of conformity and a letter of authorization.

Manufacturers must submit documentation to the Thai FDA's E-Submission system to adhere to the product registration process. The submitted documents will be meticulously examined and evaluated in alignment with the risk classification of the medical device to verify compliance with regulatory standards. In the event of uncertainties or the need for additional information, the Thai FDA communicates with the manufacturer. This interaction serves the purpose of seeking clarification and ensuring that all requisite details are accurately furnished. Following a successful review and approval, the Thai FDA issued a certificate for the medical device. The type of certificate, whether “listed”, “notified”, or “licensed”, depends on the risk classification assigned to the device. Subsequently, the issued certification allows the manufacturer to gain authorization to manufacture or import the medical device in Thailand [ 8 ].

1.2. Medical Device Software Quality

Assurance and Assessment Ecosystem

Medical Device Software Quality refers to the comprehensive set of characteristics, standards, and processes established to ensure that software integrated into medical devices meets predefined quality criteria. This commitment encompasses various elements to guarantee the software's safety, effectiveness, and reliability.

The Medical Device Software Assessment Ecosystem functions as a holistic framework, orchestrating crucial processes, adhering to standards, involving stakeholders, and utilizing tools to evaluate software integrated into medical devices' quality, safety, and regulatory compliance. This complex ecosystem, which is based on established standards such as IEC 62304 [ 1 ], ISO 13485 [ 3 ], and ISO 14971 [ 2 ], provides a solid foundation for assessment processes. Quality Management Systems (QMS) are pivotal, overseeing the entire software development life cycle and ensuring meticulous documentation and training. The ecosystem integrates robust risk management processes, verification and validation (V&V) activities, and configuration management, as well as change control procedures. Internal and external audit mechanisms gauge adherence to quality standards, while post-market surveillance mechanisms monitor the real-world performance of the software.

Before the release of the MDS to the market, the manufacturer developed the medical device in compliance with established standards. Subsequently, the documentation is forwarded to a testing laboratory for verification and validation according to the IEC 62304 [ 1 ] standards. The resulting test report, upon release, is utilized for submission to either the Certification Body (CB) or the Regulatory Body (RB). Once the MDS has successfully undergone registration procedures from the Thai FDA [ 6 ], the manufacturer is then authorized to release the MDS to the market to the consumer.

The MDS assessment approach thoroughly examines the MDS development processes through documentation examination. This process involves a detailed analysis of the system's internal components, ensuring a systematic and comprehensive testing process and other elements of the software life cycle mentioned in Section 1. However, manufacturers who fail to provide the mentioned elements or only partially offer them may be required to request alterations to add information to the document. Manufacturers who do not give any information would fail the testing outcome.

The assessor evaluates three key components of the documents: completeness, accuracy, and consistency of the submitted information. These criteria ensure that the documentation adequately reflects the development processes and meets IEC 62304 [ 1 ] in the assessment approach.

The assessment report, also known as the test report, guarantees standard compliance during software development through product release phases. This verifies the safety of the system and confirms the proper functioning of the MDS. The guarantee emphasizes that the development process has been rigorous and thorough, ensuring that the MDS meets user requirements and is fit for use. Additionally, it assures compliance with specified standards of accuracy, safety, and regulatory requirements.

The comprehensive MDS quality has highlighted the complete regulatory frameworks. The essential components of quality assurance and assessment ecosystems require delving into the intricate development process, testing, and regulatory compliance.

This work aims to gain insights into robust processes and frameworks that govern the development and deployment of MDS, contributing to the broader landscape of healthcare technology.

2. Literature Review

The literature encompasses a diverse range of topics related to regulatory compliance and software evaluation of MDS. Literature delves into the regulation’s framework compliance to physical medical devices and MDS in the EU. Furthermore, another piece of literature investigates the evaluation of MDS by the Australian Therapeutic Goods Authority (TGA) [ 9 ], emphasizing standards including IEC 62304 [ 1 ], ISO 14971 [ 2 ], and ISO 13485 [ 3 ].

The study published by Granlund et al. [ 10 ] extensively explores medical devices under the CE mark [ 11 ] and the European Commission (EU) [ 12 ], focusing on the regulatory frameworks and challenges associated with their evaluation and development. The research highlights the reliance on the Council Directive 93/42/EEC on Medical Devices (MDD) [ 13 ] and MEDDEV [ 14 ] documents for a standardized application within the CE mark [ 11 ] and EU [ 12 ]. The challenge organizations face in developing MDS to meet the regulatory requirements of medical devices is that there is no distinction between physical medical devices and standalone software criteria, by classifying both as medical devices.

The paper highlights several regulatory requirement mismatches between physical medical devices and standalone MDS, such as the design change approval process, the use of public cloud computing platforms, the regulation of artificial intelligence and machine learning, and the implementation of a quality management system. The authors emphasize the need for a more streamlined software development and certification process and precise AI/ML-driven systems guidelines. They also suggest that smaller manufacturers could benefit from cooperation or partnerships to navigate the complexities of regulatory compliance in the cloud computing environment.

Ceross and Bergmann's [ 15 ] study focuses on recalls and adverse events associated with Software as a Medical Device (SaMD) in Australia. SaMD is distinguished from medical devices with software, and data is collected from three Australian Therapeutic Goods Authority (TGA) [ 9 ] databases. The analysis reveals over ninety cases of recall and adverse events for SaMD, with fewer than thirty cases for medical devices with software. The study identifies challenges in risk evaluation associated with SaMD, citing limited regulatory vocabulary for software defects as a key obstacle. The need for regulatory vocabulary support for software developers during early-stage research and development is emphasized, and integration into computer science courses is proposed.

This literature exposes regulatory challenges across various SaMD types stemming from misinterpretation and a lack of guidance. Existing regulatory requirements do not adequately support diverse SaMD categories, including post-market development procedures and AI MDS. Identifying these challenges underscores the necessity for a tool to assist manufacturers in overcoming significant obstacles in MDS development.

3. Medical Device Software

Evaluation

The Software Quality Testing Laboratory (SQUAT) [ 16 ] is Thailand's first software testing laboratory certified with TIS 17025 (ISO/IEC 17025 [ 17 ]) (Certificate No.: 19T016/0793) by the Thai Industrial Standards Institute (TISI) [ 18 ], under the Ministry of Industry. Operating under the Software Engineering and Product Testing Section (SEPT) at the National Electronics and Computer Technology Center (NECTEC) [ 19 ], SQUAT is dedicated to verifying system performance by following the criteria outlined in IEC 62304.

Having conducted many MDS evaluations at SQUAT, the challenges encountered while evaluating MDS became evident. Twenty-three MDS evaluation cases from different manufacturers were analyzed, comprising twenty systems classified as Software Safety Class A and three systems classified as Software Safety Class B. The evaluation results, categorized into Pass and Fail for each IEC 62304 [ 1 ] clause, revealed that six out of twenty-three manufacturers achieved a fully-passed result. At the same time, the remaining seventeen had a failed outcome.

Further analysis indicated a predominant trend of more failed results than passed in each IEC 62304 [ 1 ] clause across all cases. Notably, only clause 4 had a higher pass rate, with twelve cases passing and eleven failing. However, clause 7 has the second highest pass rate, with ten cases passing and thirteen failing. The other four clauses resulted in a majority of failed assessments. Clauses 5 and 9 had eight passed cases and fifteen failed cases. Meanwhile, clauses 6 and 8 showed similar patterns of seven passed and 16 failed results.

In clause 4, the documentation lacks details regarding the decomposition of the software system into software items. Moreover, when a software item is further decomposed into additional software items, these inherit the software safety classification of the original software item (or software system) unless the manufacturer provides a rationale for classifying them differently. Additionally, the rationale should elucidate how the new software items are separated to warrant distinct classification. Suppose the software safety class of a newly created software item differs from the class of the software item from which it was decomposed. In that case, the manufacturer must document the safety class of each software item. Furthermore, there is often an absence of information regarding the identification of legacy software, the rationale for its use, and the risk management associated with legacy software.

In clause 5, specifically under sub-clause 5.1 (Software Development Planning), the deliverables, which encompass documentation of activities and tasks, often fall short of achieving the intended goals. The planning related to software configuration and change management, including software configuration items, system integration, verification and validation, risk management, and the software development life cycle, exhibits a high incidence of failure. In sub-clause 5.2 (Software Requirement Analysis), manufacturers frequently fail to identify all software requirements, such as functional and capability requirements, software system inputs and outputs, interfaces with other systems, software-driven alarms, warnings, and operator messages, security requirements, user interface requirements implemented by software, data definition and database requirements, installation and acceptance requirements at the operation and maintenance site, requirements related to methods of operation and maintenance, IT-network aspects, user maintenance requirements, and regulatory requirements.

Moreover, in sub-clause 5.7 (Software System Testing), there is a failure to provide documentation in uniformity with a) reference to test case procedures showing required actions and expected results, b) the test result (pass/fail and a list of anomalies); c) the version of software tested; d) relevant hardware and software test configurations; e) relevant test tools; f) date tested; and g) the identity of the person responsible for executing the test and recording the test results. Lastly, in sub-clause 5.8 (Software Release for Utilization at a System Level), the manufacturer must establish procedures to ensure the released MDS can be reliably delivered without corruption or unauthorized change. These procedures should address the production and handling of MDS media, including replication, media labeling, packaging, protection, storage, and delivery, as appropriate.

In clause 6 (Software Maintenance Process), there is a deficiency in having a software maintenance plan to conduct activities and tasks related to receiving, documenting, evaluating, resolving, and tracking. The usage of the software problem resolution process for analyzing and resolving issues that arise after the release of the MDS is often not adequately addressed. In sub-clause 6.2 (Problem and Modification Analysis), the documentation and evaluation of feedback to ascertain the existence of a problem in a released MDS is either not generated or inconsistently conducted. Additionally, there is a lack of effective implementation of the software problem resolution process to generate problem reports. Consequently, the evaluation and approval of change requests based on the problem reports fail to be addressed appropriately. As a result, there is a failure to identify the approved change requests that impact the released MDS.

In sub-clause 6.3 (Modification Implementation), the manufacturer must modify the instructions outlined in clause 5. Additionally, the release of modifications must align with the provisions specified in 5.8 (Software Release for Utilization at a System Level), but these requirements are frequently not fulfilled.

In clause 7 (Software Risk Management Process), there is a failure to maintain the risk management of software changes under sub-clause 7.4. The manufacturer must identify hazardous situations, conduct risk analysis, and implement software risk control measures corresponding to those situations. This ensures an evaluation of potential hazards that may arise following software changes.

In clause 8 (Software Configuration Management Process), most cases fail in sub-clause 8.2 (Change Control). Manufacturers must identify and perform any activities that need to be repeated due to the change, including changes to the software safety classification of software systems and software items. However, manufacturers often fail to verify the change, neglecting to repeat any verification invalidated by the change and failing to account for 5.7 (Software System Testing) and 9.7. Additionally, in sub-clause 8.3, most manufacturers fail to retain retrievable records of the history of controlled configuration items.

For clause 9 (Software Problem Resolution Process), most manufacturers failed to identify and present the process for problem reporting, investigating, and evaluating emerging problems and communicating the problem's existence to relevant parties, as appropriate. The manufacturer approves and implements all change requests, ensuring adherence to the requirements of the change control process. Furthermore, the manufacturer maintains records of problem reports and their resolution, including verification, and updates the risk management file as appropriate. Additionally, the manufacturer analyzes to detect trends in problem reports. Conducting testing, retesting, or regression testing of software items and systems after changes is essential. The manufacturer is required to include the following elements in the test documentation: a) test results, b) anomalies found, c) the version of software tested, d) relevant hardware and software test configurations, e) relevant test tools, f) date of the test, and g) identification of the tester.

The obstacles that resulted in unsuccessful MDS evaluations primarily stemmed from language translation issues and a limited understanding of the interconnected nature of Software Engineering and IEC 62304 [ 1 ]. These challenges led to incomplete document submissions, generating uncertainty about the necessary content inclusion. Additionally, manufacturers, mainly with an engineering background, encountered difficulties comprehending the standard's contextual nuances. Lastly, adherence to IEC 62304 [ 1 ] guidelines faced constraints due to copyright limitations.

The challenges identified in the MDS evaluation process underscore the critical need for targeted solutions to enhance understanding, compliance, and effective documentation, particularly in adherence to IEC 62304 [ 1 ]. The issues identified, such as language translation complexities, limited comprehension of software engineering principles, and constraints related to copyright, highlight the intricate landscape that manufacturers navigate during the evaluation process.

4. Experience-based Solution

Based on experience, various solutions, including short course training (onsite training), information on websites, and other technologies, have been explored to address the challenges highlighted in the preceding section.

Short course training emerges as a promising solution, offering instructors who elucidate the nature and ecosystem of IEC 62034 [ 1 ]. The exercises conducted during these courses prove beneficial in helping trainees grasp the concepts and context of IEC 62304 [ 1 ]. However, the associated costs of short course training can be prohibitively high, and the inflexible location and schedule may pose challenges for trainees. While hiring a consultant is an effective solution, its affordability remains a concern for manufacturers. Alternatively, numerous websites provide information and explanations on IEC 62304 [ 1 ] but lack a structured outline or instructions on applying the standards and producing required documents.

A potential solution lies in the utilization of chatbots. These AI-powered tools offer a simple, quick, and flexible means of assisting manufacturers in creating IEC 62304 [ 1 ] documentation. Embedding chatbots into websites or instant messaging software can offer support for IEC 62304 [ 1 ] knowledge. The recent release of ChatGPT [ 20 ] provides an opportunity, although developing a similar chatbot poses challenges.

This chatbot can be divided into two parts: one for learning user-entered keywords and sentences and another for understanding the regulatory framework, including IEC 62304 [ 1 ]. This involves training the bot to fetch essential template links and files for users. The chosen technology for this endeavor is Botpress [ 21 ], primarily because of its compatibility with WordPress websites, enabling seamless chatbot integration into an existing platform.

The Botpress [ 21 ] architecture for addressing inquiries related to the IEC 62304 [ 1 ] standard is structured to provide an intelligent and adaptable chatbot experience. Users interact with the system via a user interface connected to the Botpress Core [ 21 ]. The Integration with Generative AI [ 22 ], exemplified by models like GPT-3 [ 23 ], enhances language understanding and facilitates content generation. User inputs undergo Natural Language Processing (NLP) [ 24 ] to identify intent and context, directing queries to the IEC 62304 Query Handler, which interprets and retrieves relevant information from the knowledge base. External resources are accessed through connectors, and an AI training interface ensures ongoing knowledge base updates. The architecture incorporates security measures, logging, analytics tools for user interactions, multi-channel support, and a continuous improvement module that collects feedback for iterative enhancement. The workflow of Botpress [ 21 ] is illustrated in Figure 5.

In conclusion, exploring solutions based on a range of experiences emphasizes the potential of chatbots and generative AI to address challenges in comprehending and applying IEC 62304 standards [ 1 ].

5. Conclusion

In conclusion, the presented guidelines for improving MDS assessment in Thailand offer a comprehensive framework to enhance MDS's quality, safety, and regulatory compliance. The importance of adhering to international standards, such as IEC 62304 [ 1 ], ISO 13485 [ 3 ], and ISO 14971 [ 2 ], has been underscored throughout the guidelines, emphasizing the need for a robust quality management system.

Incorporating innovative solutions, including integrating chatbots using technologies like Botpress [ 21 ], showcases a forward-looking approach to addressing challenges in understanding and implementing complex standards. By leveraging AI-driven tools, manufacturers can benefit from quick, flexible, and accessible support in creating IEC 62304 [ 1 ] documentation, ultimately contributing to streamlined processes and improved compliance.

Furthermore, the guidelines advocate for a continuous improvement mindset, focusing on ongoing training, user feedback, and data analysis to adapt to evolving standards and industry best practices. The emphasis on multi-channel support, security measures, and the incorporation of generative AI highlights a commitment to creating a comprehensive and user-friendly ecosystem for MDS assessment.

Overall, these guidelines provide a roadmap for manufacturers, assessors, and regulatory bodies in Thailand to navigate the intricate landscape of MDS assessment, fostering a culture of quality, innovation, and regulatory adherence in the rapidly advancing field of healthcare technology.

References

[1] ISO. "IEC 62304:2006/Amd 1:2015 Medical device software - Software life cycle processes - Amendment 1 ." https://www.iso.org/standard/64686.html.

[2] ISO . "ISO 14971:2019 Medical devices - Application of risk management to medical devices." https://www .iso.org/standard/72704.html.

[3] ISO. "ISO 13485 : 2016 . " https://www .iso.org/ iso-13485 - medical-devices.html.

[4] "FDA THAI: Food and Drug Administration, Thailand." https://en.fda.moph.go.th/entrepreneursmedical-devices/category/how-to-apply-forpermission-on-medical-devices/.

[5] "IEC 60601-1 : 2005 + AMD1:2012+AMD2:2020 CSV | IEC Webstore." https://webstore .iec.ch/ publication/67497.

[6] "Thai FDA" https://en .fda. moph.go.th/ home.

[7] "Health Sciences Authority (HSA)." https://www .hsa.gov. sg (accessed 2022 - 08 -23).

[8] "FDA THAI: Food and Drug Administration, Thailand." [Online] . https://en.fda.moph.go. th/entrepreneurs-medical-devices/category/how-to-apply-for-permission-on-medical-devices/.

[9] T. G. Administration . "Therapeutic Goods Administration (TGA) | Australian Government Department of Health." https://www .tga.gov.au/

[10] T. Granlund , T. Mikkonen , and V. Stirbu , "On medical device software CE compliance and conformity assessment," in 2020 IEEE International Conference on software architecture companion ( ICSA-C) , 2020 : IEEE, pp. 185 - 191 .

[11] "CE marking," https://single-market-economy.ec.europa.eu/single-market/ce-marking_en.

[12] "European Commission, official website," 2023 /11/15/ 2023. https://commission.europa.eu/index_en.

[13] CONSIL , ( 1993 , 1993 /06/14/). OJ L 169, Council Directive 93/42/EEC of 14 June 1993 concerning medical devices.

[14] "MEDDEV Guidance List - Download ," in Medical Device Regulation, ed.

[15] A. Ceross and J. Bergmann , "Evaluating the presence of software-as-a-medical-device in the Australian therapeutic goods register," Prosthesis , vol. 3 , no. 3 , pp. 221 - 228 , 2021 2021.

[16] "Software Quality Testing Laboratory (SQUAT)." https://www .squat.in.th/.

[17] "ISO - ISO/IEC 17025 - Testing and calibration laboratories," ISO , 2021 /01/26/ 2021 . [Online]. Available: https://www.iso.org/ISO-IEC-17025 - testing-and -calibration-laboratories .html.

[18] "Thai Industrial Standards Institute (TISI)." https://www .tisi.go.th/home/en.

[19] "Home - NECTEC: National Electronics and Computer Technology Center," ed, 2007 .

[20] "Introducing ChatGPT," . https://openai.com/ blog/chatgpt.

[21] "Botpress | the Generative AI platform for ChatGPT Chatbots," https://botpress.com/.

[22] "Generative AI" , Generative AI . https://generativeai.net.

[23] "GPT-3," in Wikipedia, ed, 2023 .

[24] "Natural language processing," in Wikipedia, ed, 2023 .