Periodic Reporting for period 2 - CORE-MD (Coordinating Research and Evidence for Medical Devices)
Período documentado: 2022-10-01 hasta 2024-03-31
CORE-MD is a CSA co-led by two major medical professional associations in Europe (ESC and EFORT). It is unique in bringing together medical experts from universities and clinical societies with national regulatory agencies of EU member states, notified bodies, HTA agencies, patients’ organisations, and public health institutes, all to consider how best to evaluate high-risk medical devices (MDs) that are implanted into patients.
A new EU law concerning MDs was passed in 2017, although it has not yet been fully implemented. It established that high-risk MDs need to undergo more thorough clinical investigations before they are approved, but it does not recommend which types of investigations are best for which devices. New MDs are constantly being developed, some devices are used infrequently for example in children and patients with rare diseases, and for some devices such as AI-enabled systems there are no clinical standards to date. The European regulatory system for approving MDs faces the pressing need to adapt to such challenges with the aim to provide safe and effective MDs for patients and clear guidance for manufacturers to undergo the conformity assessment and post-market surveillance processes.
The main objective of this project was thus to develop expert recommendations that regulatory authorities can use to set European guidance and standards for the clinical investigation of high-risk MDs.
A new EU law concerning MDs was passed in 2017, although it has not yet been fully implemented. It established that high-risk MDs need to undergo more thorough clinical investigations before they are approved, but it does not recommend which types of investigations are best for which devices. New MDs are constantly being developed, some devices are used infrequently for example in children and patients with rare diseases, and for some devices such as AI-enabled systems there are no clinical standards to date. The European regulatory system for approving MDs faces the pressing need to adapt to such challenges with the aim to provide safe and effective MDs for patients and clear guidance for manufacturers to undergo the conformity assessment and post-market surveillance processes.
The main objective of this project was thus to develop expert recommendations that regulatory authorities can use to set European guidance and standards for the clinical investigation of high-risk MDs.
Despite it being essential for healthcare professionals to practise evidence-based medicine, there has been no requirement for transparency of the clinical evidence that has been presented by manufacturers for their devices to be approved in the EU. The first task of CORE-MD, therefore, has been to review and summarise what study methods have been used and what results are available about MDs that are commonly implanted into patients. These systematic evaluations of the published clinical literature for the selected high-risk MDs in the fields of cardiovascular, orthopaedic and diabetes care indicate that the clinical evidence publicly available prior to market release (CE-mark) is generally insufficient. It is characterized by small studies with short follow-up times and methodological heterogeneity. This highlights the need for better guidance documents throughout the different medical fields.
On the other hand, for certain devices, approvals obtained without large and robust clinical studies can be balanced by collecting detailed information about devices after they have been implanted. Medical associations collect significant amounts of information in well-conducted registries which may be very useful for manufacturers, notified bodies, and regulatory authorities, but their results are not yet used routinely. CORE-MD has thus reviewed device registries in Europe, which showed that they are not yet designed to provide the results needed by regulators.
The consortium has also reviewed international standards for clinical investigations to benchmark existing approaches already in use in other jurisdictions, and investigated specific questions such as how AI-enabled MDs can be evaluated.
Lastly, research studies were undertaken to evaluate the experience of innovators in obtaining advice about the design of early clinical studies, and the experience of notified bodies in issuing certificates of conformity with restrictions.
This high-quality research work has generated a large set of relevant results that have been widely disseminated. Notably, leveraging the participation in the consortium of three competent authorities and several stakeholders regularly invited to attend the MDCG meetings as well as its Working Groups, a continuous and fruitful exchange of views with the EC DG SANTE relevant unit and the national regulators, has been maintained.
On the other hand, for certain devices, approvals obtained without large and robust clinical studies can be balanced by collecting detailed information about devices after they have been implanted. Medical associations collect significant amounts of information in well-conducted registries which may be very useful for manufacturers, notified bodies, and regulatory authorities, but their results are not yet used routinely. CORE-MD has thus reviewed device registries in Europe, which showed that they are not yet designed to provide the results needed by regulators.
The consortium has also reviewed international standards for clinical investigations to benchmark existing approaches already in use in other jurisdictions, and investigated specific questions such as how AI-enabled MDs can be evaluated.
Lastly, research studies were undertaken to evaluate the experience of innovators in obtaining advice about the design of early clinical studies, and the experience of notified bodies in issuing certificates of conformity with restrictions.
This high-quality research work has generated a large set of relevant results that have been widely disseminated. Notably, leveraging the participation in the consortium of three competent authorities and several stakeholders regularly invited to attend the MDCG meetings as well as its Working Groups, a continuous and fruitful exchange of views with the EC DG SANTE relevant unit and the national regulators, has been maintained.
Besides the findings emerged from the systematic reviews and stakeholder consultations, as well as the open datasets made available on Zenodo (https://zenodo.org/communities/core-md/) CORE-MD produced a set of relevant exploitable results (tools, frameworks, and recommendations) which constitute its tangible legacy to the European regulatory science community, and are briefly listed below.
Methods and tools:
- AI MDSW (MD software) risk-scoring system: this is a simple point-scoring system that can be used to estimate the overall risks associated with the use of a machine-learning algorithm or other AI MDSW, in order to direct the level/depth of the pre- and post-release requirements for clinical evaluation. From a clinical perspective three parts of evidence are required, namely, valid clinical association, technical performance, and clinical performance, and for each CORE-MD has proposed scores from 1 to 3, relevant to the characteristics of the AI tool and its application, where lower values are associated with lower risk. Further details are available in D2.4 and in the related publication (under revision).
- Statistical tool: Due to the high-risk nature of many implantable MDs and to the absence of specific guidance about sample sizes or minimum cumulative follow-up required, limited sample sizes are common. This implies substantial uncertainty of the resulting risk estimates. Our tool (https://core-md.shinyapps.io/RiskCalculator) gives insight into the relation between sample size and the implications for the level of risk that is accepted.
- Decision framework: Post-market surveillance (PMS) is essential for monitoring the performance of high-risk MDs; no implantable device can be guaranteed to be completely free of risks over time. PMS is the responsibility of manufacturers, but the best quality data are collected by medical professional associations which conduct registries. It is anticipated that EU regulators will consider the recommendations prepared by CORE-MD in its ‘Decision framework’ (D3.1) as the basis for developing a checklist for recognizing which registries and other sources of post-market clinical follow-up can be used for regulatory purposes.
- Web-scraping tool: The CORE-MD PMS Support Tool (D3.2) has been designed to be useful to members of Expert Panels, regulators, and evaluators in notified bodies, whenever they wish to find out if problems have been reported with a particular device or type of device. That information will not be available in EUDAMED until it has been fully operational for some years, since it will not include any historical data. The Tool may be useful also for manufacturers to summarize the ‘state of the art’ relevant to a certain device.
Recommendations:
- Clinical study design recommendations: Recommendations for the conduct of registry-based randomised controlled trials of high-risk MDs, have been exposed in D4.3 and submitted for publication; and the recommendations for clinical trials, as a ‘hierarchy’, are being prepared for publication. Specific recommendations have also been provided for paediatric MDs (D2.5 and respective publications).
- Education roadmap and recommendations: The main aim of the roadmap is to present the training needs of stakeholders, namely notified bodies, regulators and clinicians, to develop appropriate educational objectives, and to provide respective recommendations, including a proposal for a regulatory science curriculum. The recommendations are gathered in D4.1 and the respective publication ad were shared with members of the EMA who have been planning educational programmes for experts in the EU.
Methods and tools:
- AI MDSW (MD software) risk-scoring system: this is a simple point-scoring system that can be used to estimate the overall risks associated with the use of a machine-learning algorithm or other AI MDSW, in order to direct the level/depth of the pre- and post-release requirements for clinical evaluation. From a clinical perspective three parts of evidence are required, namely, valid clinical association, technical performance, and clinical performance, and for each CORE-MD has proposed scores from 1 to 3, relevant to the characteristics of the AI tool and its application, where lower values are associated with lower risk. Further details are available in D2.4 and in the related publication (under revision).
- Statistical tool: Due to the high-risk nature of many implantable MDs and to the absence of specific guidance about sample sizes or minimum cumulative follow-up required, limited sample sizes are common. This implies substantial uncertainty of the resulting risk estimates. Our tool (https://core-md.shinyapps.io/RiskCalculator) gives insight into the relation between sample size and the implications for the level of risk that is accepted.
- Decision framework: Post-market surveillance (PMS) is essential for monitoring the performance of high-risk MDs; no implantable device can be guaranteed to be completely free of risks over time. PMS is the responsibility of manufacturers, but the best quality data are collected by medical professional associations which conduct registries. It is anticipated that EU regulators will consider the recommendations prepared by CORE-MD in its ‘Decision framework’ (D3.1) as the basis for developing a checklist for recognizing which registries and other sources of post-market clinical follow-up can be used for regulatory purposes.
- Web-scraping tool: The CORE-MD PMS Support Tool (D3.2) has been designed to be useful to members of Expert Panels, regulators, and evaluators in notified bodies, whenever they wish to find out if problems have been reported with a particular device or type of device. That information will not be available in EUDAMED until it has been fully operational for some years, since it will not include any historical data. The Tool may be useful also for manufacturers to summarize the ‘state of the art’ relevant to a certain device.
Recommendations:
- Clinical study design recommendations: Recommendations for the conduct of registry-based randomised controlled trials of high-risk MDs, have been exposed in D4.3 and submitted for publication; and the recommendations for clinical trials, as a ‘hierarchy’, are being prepared for publication. Specific recommendations have also been provided for paediatric MDs (D2.5 and respective publications).
- Education roadmap and recommendations: The main aim of the roadmap is to present the training needs of stakeholders, namely notified bodies, regulators and clinicians, to develop appropriate educational objectives, and to provide respective recommendations, including a proposal for a regulatory science curriculum. The recommendations are gathered in D4.1 and the respective publication ad were shared with members of the EMA who have been planning educational programmes for experts in the EU.