10 Things You Should Be Doing to Verify Your Testing Labs' Data
How strong is your independent data verification program?
This guide is available in full for paid subscribers. If you’re not already a paid subscriber, you can upgrade here.
As reported in a new letter to industry, which we recently wrote about here, the FDA has identified an increase in fraudulent and unreliable laboratory testing data in premarket submissions.
The agency reminded sponsors of device studies and manufacturers of devices to carefully evaluate the third parties they engage to conduct performance testing and to independently verify all testing results before submitting them to the FDA.
In our first issue on the subject, we identified a list of questions to bring to your testing lab in light of this alert. This issue is a companion piece to that — stepping back to dig into the specific activities firms should be engaging in to double-check their testing labs’ data.
Before we jump in though, it’s helpful to review the FDA’s expectations here.
A brief look at the FDA’s expectations for independent data verification
To provide specific guidance and expectations related to the verification of third-party lab data for medical devices, the FDA has published several resources:
21 CFR §820.50 Purchasing Controls: This section of Part 820 requires manufacturers to ensure that all purchased or otherwise received product and services conform to specified requirements. It mandates that manufacturers evaluate suppliers, contractors, and consultants based on the ability to meet specified requirements, including quality requirements. The regulation requires establishing and maintaining procedures to ensure that the specified requirements are met. These procedures must include, where possible, an evaluation of the supplier's ability to meet defined requirements and the assessment of the quality of supplied products and services.
This section is directly relevant to the selection and oversight of third-party labs, as it emphasizes the need for medical device manufacturers to have controls in place to ensure that external parties (including labs) meet all specified requirements, including those related to the quality of testing and data integrity. The emphasis in §820.50 is on ensuring that manufacturers have a systematic approach to evaluating and selecting suppliers (which would include third-party laboratories), defining the type and extent of control to be exercised based on the evaluation, and maintaining records of these evaluations and any necessary actions arising from the evaluation.
Manufacturers are expected to have documented procedures for these purchasing controls, which should cover the selection and monitoring of suppliers to ensure they comply with the manufacturer’s requirements. This includes ensuring that third-party labs conducting testing or validation studies on behalf of the manufacturer meet all necessary quality and regulatory requirements, thereby ensuring the reliability and integrity of the data they provide.
Design Control Guidance For Medical Device Manufacturers: This guidance provides comprehensive instructions on implementing design controls throughout the development process of medical devices. The guidance outlines the importance of carefully selecting and managing suppliers and third-party labs contributing to the design and development process. This includes ensuring these partners meet all regulatory, quality, and technical requirements.
It stresses the need for clear communication of design and development needs, verification and validation requirements, and quality expectations. Manufacturers are advised to conduct audits and assessments of suppliers and third-party labs to ensure ongoing compliance and capability.
Use of International Standard ISO 10993-1, 'Biological evaluation of medical devices - Part 1: Evaluation and testing within a risk management process’: ISO 10993-1 integrates biological evaluation within a risk management process, emphasizing that the assessment of a medical device's biological safety starts with the identification and analysis of potential hazards. It aligns with the broader principles of ISO 14971, "Application of risk management to medical devices," guiding manufacturers in identifying risks, estimating their significance, evaluating risk acceptability, and controlling these risks.
This approach requires rigorous testing and data analysis procedures, often requiring collaboration with third-party laboratories specializing in biocompatibility testing. The standard specifies that the selection of tests for evaluating the biological effects of a device should be based on a thorough understanding of the device's material composition, its intended use, and the nature of the body contact (e.g., duration, exposure). This implies that manufacturers must work closely with third-party labs to ensure that the chosen tests are not only appropriate but also validated for the specific context of the device under evaluation.
A critical aspect of the guidance is the emphasis on using validated test methods to ensure the reliability and reproducibility of results. For manufacturers relying on third-party labs for biocompatibility testing, it is crucial to verify that the labs use test methods that have been validated according to recognized standards. This includes ensuring that the lab is capable of performing the tests in compliance with ISO 10993-1 requirements and any relevant FDA guidance.
Adequate documentation of the biological evaluation process, including test plans, methodologies, results, and conclusions, is vital. Manufacturers are expected to critically review and interpret the data obtained from third-party labs, integrating it into the device's risk management file and technical documentation. This comprehensive documentation is crucial for regulatory submissions and for demonstrating compliance with both ISO 10993-1 and FDA expectations.
The guidance also underscores the importance of reliable lab practices and the quality of testing. Manufacturers must ensure that third-party labs adhere to high-quality standards, such as GLP regulations, where applicable. This involves conducting audits or assessments of the lab's qualifications, procedures, and quality control measures to ensure that the biological evaluation of the device is conducted accurately and reliably.
FDA Recognized Consensus Standards: The FDA maintains a database of recognized consensus standards that can be applied to medical device testing and development. This includes standards for lab testing practices, data integrity, and more. For example, standards under ISO 17025 (General requirements for the competence of testing and calibration laboratories) are often recognized and can guide the verification of third-party lab competencies.
Accreditation Scheme for Conformity Assessment (ASCA) Pilot Program: Although more specific to the accreditation of testing labs, this program outlines the FDA’s approach to third-party reviews and the criteria labs should meet, which can indirectly inform expectations for third-party data verification.
Generally speaking, we see a few key pillars of regulatory expectation here:
Due Diligence and Vendor Qualification: The FDA expects device firms should conduct thorough due diligence when selecting third-party labs. This includes verifying the lab's qualifications, credentials, and history of compliance with applicable regulations and standards. The FDA expects sponsors to choose labs that have demonstrated competence and reliability in testing medical devices.
Verification of Test Methods and Results: Sponsors are expected to independently verify the test methods used by third-party labs to ensure they are appropriate for the specific device and testing requirements. This involves reviewing the lab's method validation data and, if necessary, conducting or requesting independent validation of the test methods. Additionally, firms should verify the test results through independent analysis or by cross-referencing with known benchmarks or internal data.
Data Integrity and Transparency: Sponsors are responsible for ensuring that the data submitted from third-party labs are complete, accurate, and untampered. This includes maintaining a transparent and traceable chain of custody for data and samples, implementing checks for data manipulation or falsification, and ensuring that any electronic data submissions comply with requirements for electronic records.
Documentation and Record Keeping: Comprehensive documentation of all aspects of third-party testing is expected. This includes detailed records of the testing protocol, raw data, final reports, communications with the lab, and any actions taken to verify the data. The FDA expects sponsors to maintain these records in a manner that they are readily available for review during inspections or audits.
Reporting and Addressing Discrepancies: Expectation: If discrepancies or issues are identified in the third-party lab data, sponsors are expected to thoroughly investigate, document, and address these issues before submitting the data to the FDA. This may involve re-testing, using alternative testing methods, or engaging additional third-party labs for verification.
Use of Accredited Labs and Standards” While not always mandatory, the FDA encourages the use of labs accredited under recognized schemes, such as the ASCA, which provides a level of assurance in the lab's testing capabilities and adherence to quality standards. However, the FDA also emphasizes that accreditation does not replace the sponsor's responsibility to ensure data accuracy and integrity.
Ongoing Monitoring and Quality Assurance: Sponsors are expected to implement ongoing monitoring and quality assurance practices to ensure that the third-party labs continue to meet the required standards throughout the device development lifecycle. This includes periodic audits of the labs and re-evaluation of their qualifications and performance.
With these requirements and expectations in mind, let’s explore a few of the specific ways sponsors should be independently verifying lab data.
1. Conduct an independent statistical analysis
Reanalyze a subset of the raw data provided by the third-party lab using independent statisticians or in-house statistical software. This can help verify the accuracy of the reported results and identify any discrepancies or anomalies in the data analysis.
How to do it: Select a random sample of raw data for independent analysis. Determine the size of the sample based on statistical significance and resource availability. Then, use in-house or external statisticians to reanalyze the data using the same and different statistical methods. Compare these results with those provided by the third-party lab to identify discrepancies.
Make sure the independent analysis accounts for the potential variability in biomedical data, which often has inherent noise. Always use robust statistical methods that can handle outliers and non-normal distributions.
Example: A medtech firm receives cardiovascular performance data from a lab. An independent statistician reanalyzes the raw heart rate variability data using both the original statistical method and a non-linear analysis method. The reanalysis reveals inconsistencies in the reported improvements in variability, prompting a review of the lab's analysis protocols.
2. Conduct a benchmark comparison
Compare the test results with historical data from similar devices or industry-standard benchmarks. Assess whether the results align with expected outcomes based on known device characteristics and published data for comparable technologies.
How to do it: Identify industry benchmarks or historical data from similar devices as references. Look to academic literature, regulatory submissions, or internal data from past projects. Compare the third-party lab's results with these benchmarks to check for consistency. Significant deviations should be flagged for further investigation.