Within the lifecycle of a medical device, Post-Market Surveillance (PMS) refers to the proactive, systematic and periodic monitoring of adverse events and other findings related to medical devices on the market, aiming to ensure the safety of patients, healthcare professionals and other users. PMS and associated activities, like Vigilance, are required by law and part of the obligatory risk management for medical products where regulatory requirements to be met are, for instance, defined by the Medical Device Regulation. Information to be monitored includes adverse event reports, clinical studies, scientific publications and other reports from any relevant internal or external data source. This requires the collection of heterogeneous data from various data sources in order to perform qualitative and quantitative analysis. In this work, we propose an initial version of a standard process with emphasis on data collection and analysis of data relevant for PMS and associated activities within the medical device lifecycle which is based on experience from regulatory specialists in this field. In addition, the challenges and potential for AI-support within these processes are discussed.
Post-Market Surveillance (PMS) is statutory for European medical device manufacturers in order
to ensure the safety of patients, users and third parties [
Post-Market Surveillance, Vigilance and other associated processes mandatory by MDR have in common that dispersed data from various sources has to be collected, reviewed and analysed. Searching for relevant data has to follow a standardized procedure and all activities must be documented. Using the relevant data, diferent reports are created and sent to notified bodies 1. As the MDR mandates the inclusion of any relevant data, there are several main data sources of interest. Data is unstructured multilingual texts; they come from unsolicited sources (e.g., spontaneous reports from healthcare professions and patients; screenings of global and local literature; screenings of internet/digital media, internal complaints, etc.) or solicited sources (e.g., derived from organised data collection system such as clinical trials, non-interventional studies, registries). For clinical studies, Pubmed2 and Medline3 are predominantly used. Adverse events are mainly included within country specific databases, for instance FDA Maude 4 in the USA, BfArM5 in Germany and Swissmed6 in Switzerland. Almost every country has its own database, typically in the country’s language. For Europe, the deployment of the EUDAMED7 database is planned, which is meant to collect all adverse events occurring in all the European countries and provide them in all European languages.
While there are many regulations and guidelines to follow, to the best of our knowledge, there are no standard processes in the form of a process model for such activities. For instance, there is no standard procedure that supports regulatory specialists in obtaining insights that can be used for writing a conformity report. From our experience, this makes it hard for manufacturers to follow the MDR and conform to all requirements imposed thereby. Furthermore, PMS and Vigilance are time-consuming tasks which are often not supported by information systems. Finding relevant documents and reports is a challenge due to the distribution of the data in diferent databases in diferent languages, diferent data formats and the huge amount of reports that are potentially of interest. Thus, there is a lot of manual work to be done in order to collect, review and analyse relevant information.
In this paper, we propose a scheme for performing such processes in a regulatory environment.
1In the EU they are designated in each country by the member states[
The remainder of this paper is structured as follows. In the next Section, we will introduce the scheme for PMS and the associated processes. In Section 3, we will present challenges and possibilities to give AI-support within the processes. Section 4 closes the paper with a discussion and conclusion.
How PMS and similar processes are performed varies from company to company, and product to product. However, we observed similar sub-procedures how users proceed. In this section, we will first introduce our methodology before introducing the runs we have observed in practice and the scheme that is derived from the runs.
We started by collecting regulatory requirements for PMS and similar processes such as laws,
regulations and standards, e.g., proposed by MDR. Next, we conducted several interviews with
regulatory specialists of three international manufacturers of medical devices that are based
in Germany to find out how they proceed, e.g., in PMS. The companies difer in size and have
devices of diferent classes on the market. From the interviews we obtained several sample runs
[
We use Heraklit [
Within the interviews with the regulatory specialists we collected and summarized challenges they face. To make their work easier and more eficient, we added possibilities how to support them with methods based on artificial intelligence.
Afterwards, we presented the runs and the scheme to the regulatory specialists and interviewed them again to evaluate its correctness and usefulness. The specialists agreed that the runs are plausible and the scheme covers the processes they perform. Furthermore, they found no other problems with the models. We concluded that the scheme has no obvious errors and ifts the procedure of how PMS and similar processes are performed in practice. Although the scheme has no errors according to the specialists, this does not guarantee that a report created when following the scheme will be approved by the notified bodies in the end, as the approval depends on the reports content.
In the lifecycle of a medical device there is a pre-market and a post-market phase [
Once the device is on the market, the data collection continues. The requirements are
described in detail in Annex III of the MDR [
In the following, we will show three sample runs that demonstrate what actions are performed within the lifecycle of a medical device and what data objects are created. Note that these are just examples of what could have happened. Many other ways are possible and plausible. Diferent colors indicate which module a action is part of.
Figure 1 shows an exemplary run of the actions performed in the lifecycle of a medical device i.e., what activities have been performed and what objects are created. First, Data Collection is performed to obtain the information, i.e., the search results used for writing the initial conformity report. With the conformity report, the device can be released to the market. Eventually, an assignment indicates to perform the Data Collection again, which is triggered by action Collect Data. The search results are used to update the conformity report and create a PMS report. At this point the collected information can also be used to create other reports required by MDR like a trending report or PSUR. The same procedure of Data Collection and associated actions is done again after a certain time. This particular run ends after two iterations.
In contrast to the previous one, the run in Figure 2 shows the situation where a non-conformity was detected while the device is on the market. In such an event, the device may be retired which requires an update of the initial conformity report with information from the second data collection. Afterwards, a notification is sent and the device is retired from the market.
Figure 3 shows the situation where insights gathered during data collection have to be communicated to notified bodies and users. This may be required if a problem has been detected that is not as serious as to retire the device but requires to send a Field Safety Notice to all users to prevent the problem in the future.
The previous three runs have shown some of the behaviour that can happen in the lifecycle of a
medical device. The runs can be structured into Heraklit modules to capsule their behaviour
and create a level of abstraction which follows the idea of reference modelling [
The interactions are as follows. If, for instance, a PMS is due, module Assignments sends an Assignment to module Data Collection which starts the procedure to collect relevant data. Devices in Market contains all devices in market and those that are to be put on the market. All relevant data obtained in Data Collection is passed to module Additional Actions as Search Results where additional analysis are performed, and reports, such as PMS-report or action plans, are created. Notices are sent back to Devices in Market to trigger actions like sending a Field Safety Notice to the notified bodies, removing the device from the market or notifying users of the medical device. If a device has been retired, it is moved to module Devices out of Market.
The actions to actually collect data, assess it for relevance and create reports are performed in the modules Data Collection and Additional Actions which we will focus on now. Figure 5 shows the inner of the module Data Collection which is again divided into two modules shown in Figure 6. While in Prepare, the search strategy is defined, the actual search and assessment of relevant reports is performed in module Perform. Both modules contain sub-modules again. In submodule Prepare, a decision is made whether the assignment requires to perform an initial data collection for a new product or for a known device. If it is an initial data collection, a new search profile will be created in module Initial Search. If a search strategy exists for this medical device, module Reoccurring Search is used to find the corresponding search profile and update it (if required). Module Start Search combines both situations as well as the situation where the search strategy is passed back from an unsuccessful search (indicated by Modified Search Strategy) and starts the data collection in module Perform. The submodule Perform consists of modules to retrieve and evaluate reports. Using the search profile, reports are retrieved in module Query Data Sources (Figure 8). Next, the query results are evaluated in module Assess Query Results (Figure 9). In the end, all relevant information are collected and passed as search results to module Additional Actions (Figure 7). In case the search strategy does not give the desired result, it can be revised in module Modify Search Strategy which also triggers to start the whole module Perform from scratch using a Modified Search Strategy .
An example how the inner of a module looks like is given in Figure 7 which shows module Additional Actions. The signature on the left defines the data and its structure. For instance, search results F is a tuple of the relevant search results, the not relevant search results and the search strategy used to query the databases. Using the search results gained in module Data Collection several actions are performed in this module. For instance, an initial conformity report (denoted as c) can be created based on the results, an existing one updated, a PMS report created or other preventive actions can be triggered. All actions produce a notice (denoted as n) that is returned to module Devices in Market.
In the following, we present challenges regulatory experts face and propose suitable techniques how AI-support for such situations can be given on two modules, Query Data Sources and Assess Query Results. Red transitions indicate that support can be given.
Once the search profile is defined, reports from diferent databases have to be retrieved. There are diverse data sources with diferent access options and data schemes. While some data sources can only be accessed manually and do not provide a search function, others ofer the possibility to retrieve reports via API using keywords and filters. For example, the BfArM database is a mandatory data source for PMS in Germany containing warnings and notes from medical device manufacturers. However, the portal does neither ofer the possibility to query the reports via API nor to search for specific reports. Instead, all released documents can only be viewed chronologically. Furthermore, documents need to be downloaded manually as PDFs in a non-standardized format. On the other side, Medline ofers an extensive API that allows users to query the database directly using keywords and filters returning the reports in a machine-readable and structured format (XML). Figure 8 shows the module Query Data sources in detail. The module contains the mentioned steps to retrieve reports from relevant data sources and prepare them for further analysis. As retrieving the reports from the data sources is a repetitive and time consuming task, we present AI-techniques supporting this part.
Depending on the type of data source, data can either be queried via API resulting in
machinereadable reports (denoted as r), via click-bots that query the database as a human would do or
manually, yielding PDFs (denoted as p). Automatically querying via API or click-bots gives a
list or set of reports or PDFs (denoted as sr or sp) which are element-wise put into the place
Query Results using the elm function [
Automatically querying BfArM can be done using a simple click-bot which downloads all
PDFs. Optical Character Recognition (OCR) and techniques from document understanding are
applied to extract the text of PDFs in a structured format [
When searching in databases, often more reports are retrieved than are actually relevant. For instance, Medline contains mainly clinical studies which can contain adverse events. Unfortunately, Medline does not index adverse events and ofers no possibility to filter by them. However, finding the relevant reports or information within reports is a crucial step. The next module deals with the assessment of the reports retrieved by the module Query Data Sources. For a report to be relevant, there mainly are two qualitative reasons as mentioned by the regulatory specialists: • The report contains an unknown problem.
• The report contains a known problem but with an unknown cause.
Depending on the type of search being conducted, there might be other reasons why a report is relevant. Furthermore, quantitative and statistical information from the query can also be a piece of relevant information, e.g., if a rising number of known or unknown problems with a certain medical device over a period can be noted. This can be found by investigating the collection of search results.
Regulatory specialists tend to reduce the number of retrieved reports by experimenting
with diferent search strings and using filter until an appropriate number of reports is reached.
Afterwards, each report is examined to collect the relevant information within the report. Based
on the relevant information within the report, each report is classified into diferent categories.
Basically, a report is either relevant or not relevant. However, as MDR requires to justify why
reports are classified as relevant or not relevant, some companies use finer-grained classification
schemes. For instance, they classify scientific publications according to their level of evidence,
i.e. their quality. The level of evidence is then used as one parameter for justifying if a report is
relevant or not. For example, for clinical trials, which are a type of scientific publications, the
evidence is determined based on the number of patients, interventions taken and the outcome.
This information is referred to as PICO-Parameters [
Module Assess Query Results in Figure 9 shows how reports are assessed. In case the user ifnds that the search strategy used produces not the desired result, e.g. if too many reports have been retrieved, the search strategy used is passed back to module Modify Search Strategy. Similarly, the strategy is passed back if the relevance of the reports after the assessment is too low. This starts the querying part again.
Sorting the reports The retrieved reports are typically not ordered by their relevance with respect to a specific company. Given the reports retrieved from the query, the reports should be ordered so that the most relevant reports are shown first and the least relevant ones at the end. This happens in Sort by Relevance. It takes all reports r and adds a relevance score to be used for sorting the reports.
The assessment of relevance with respect to the query can be done automatically using
methods like inverse term frequency and field length (TF-IDF). One popular TF-IDF function is
BM25 [
Extracting relevant information Finding and extracting the relevant information is an
important step before being able to classify the reports. Certain entities of interest can be
extracted using techniques such as Named-Entity-Recognition [
Report classification By using the extracted information in the reports, it can be assessed and classified according to the individual classification scheme of a company. While the AI can support the user by proposing a classification, the actual decision should always be made by the human. Thus, Classify reports is not colored completely red. Using information from recent assessments, we have seen that we are able to train a ML-algorithm to propose a classification to the user.
Retrieval Given a specific report which was ranked as relevant, the user might be interested in reports that are very similar to that one. For this, all reports are compared against the report of interest regarding a certain metric such as the cosine distance from features produced, e.g., by TF-IDF obtaining a set of new reports that can be assessed. This allows the user to quickly ifnd reports that are of specific interest. Those reports can also be retrieved from data sources that were not queried originally.
Customized AI Given the continuous nature of PMS within the lifecycle of a medical device, historic data from previous assessments is likely to be available, i.e. data which reports and passages are relevant and which are irrelevant for a given medical product from a certain company. Using this information, the information extraction, relevance assessment and retrieval can be customized to the specific needs of a customer or company.
In this work, we have proposed a way for structuring and modelling the activities supporting the safety and conformity of medical devices as a Heraklit scheme. The processes and the scheme are based and approved in interviews with regulatory specialists. Furthermore, AI-support along the process has been shown.
While the scheme supports how three companies perform those processes, we cannot claim completeness to support all possible ways. Furthermore, the processes have yet not been used for creating reports that are accepted by notified bodies. Furthermore, a more extensive evaluation would be required to further assess the scheme’s usefulness, correctness and conformity, also with respect to other companies and other regulations.
However, the scheme gives an overview of relevant activities and details of important parts. Due to the modularization, it can be modified to specific needs or requirements.
In the future, we plan to use the scheme for developing information systems that support humans in PMS, Vigilance and related activities. Whenever possible, AI-solutions will be added that take over the repetitive parts of this work, i.e., help in collection and analysing reports and support in finding the relevant information within the large amount of information available. Special attention will be payed to regulatory requirements. Furthermore, we will evaluate the applicability and usefulness of the information system, which is based on the scheme, in supporting regulatory specialists in their daily work. This also involves an evaluation if the reports created using the tool will be accepted by notified bodies.
Part of this work has been done in the project SmartVigilance (FKZ: 01IS20028C) with financial support by the Federal Ministry of Education and Research (BMBF). We would like to thank pfm medical ag (https://www.pfmmedical.de/) and AkknaTek (https://www.akknatek.com/) for their support. biomedical information retrieval, in: Proceedings of the 2018 World Wide Web Conference, 2018, pp. 77–86. [11] B. Mitra, N. Craswell, Neural models for information retrieval, arXiv preprint arXiv:1705.01509 (2017). [12] I. J. Marshall, A. Noel-Storr, J. Kuiper, J. Thomas, B. C. Wallace, Machine learning for identifying randomized controlled trials: An evaluation and practitioner’s guide, Research Synthesis Methods 9 (2018) 602–614. doi:https://doi.org/10.1002/jrsm.1287. [13] W. Yang, H. Zhang, J. Lin, Simple applications of BERT for ad hoc document retrieval, arXiv preprint arXiv:1903.10972 (2019).