【法規工具文】Quick-guide on Software Pre-Cert Program (5) — Real World Performance

【導讀】本文最初用英文寫作，目的是保留FDA原意。而本版的目的為工具用文章，故用中文註釋以便快速抓到重點，但細節部分還是建議細讀英文說明

理論上，任何醫療產品，上市後一定要監督產品是否在功效性，安全性是否如預期的合規，如果沒有，則需作調整。這其實在ISO的上市後監督（PMS），以及目前的MDR，IVDR的PMCF(Post Market Clinical Follow Up)都是在提這個觀念。只是在之前會落實的廠商實在不多，都是應付稽核為主。

這次FDA的Pre-Cert希望給廠商開發產品的彈性，但也要求廠商把基本工作做好，這個Real World Performace Analytics(真實世界表現分析) 就是這個觀念。

其核心的觀念及應用，在Fig 2有很清楚的說明。讀者也可先看Fig 2，了解每節的關係，再從前面逐步了解，也會有全盤的了解。

During the Excellence Appraisal, all organizations would demonstrate the capability to collect and analyze post-launch Real World Performance (RWP) data, whether by instrumenting their SaMD products to generate needed data, or by leveraging alternative data sources.

Therefore, FDA expects that these organizations will consistently collect and analyze post-launch data related to the safety, effectiveness, and performance of the products they manufacture in order to inform their decision-making related to product or process improvements.

Real world Performance Analytics (RWPA) framework is introduced for achieve above-mentioned goals.

The real-world performance (RWP) component of the Software Pre-Cert Program is designed to use these readily available data analytics to verify ongoing excellence following precertification, identify emerging safety and cybersecurity risks, provide critical feedback to the other components of the Program, and support the appropriate use of postmarket data in clinical evidence generation.

【導讀】RWPA Framework（真實世界表現分析)架構,是本章的核心，其有三個區塊，RWHA，UXA，PPA。 其評量方式可參考Table 1, Table 2, Table 3. 可發現還是走COQE的原則來評估。

此外，這個架構是執行真實世界表現分析的核心，在執行上，需要有計畫，計畫的核心就是根據這個架構。

RWP Analytics (RWPA) framework

RWPA are defined as systematic computational analyses of data relevant to the safety, effectiveness, and performance of a SaMD product in real-world settings marketed by a precertified organization.

FDA anticipates that not only data from appropriately instrumented SaMD products may be generated, collected, and analyzed efficiently, but also real-world data from device registries, well-structured data commons, and other electronic health information sources, including patient registries and the National Evaluation System for health Technology (NEST) currently under development.

The FDA considers RWPA to encompass at least three types of analyses, RWHA, UXA, and PPA shown in Fig 1.

Fig 1 illustrates the domains and subdomains of RWPA under Pre-Cert 1.0. FDA anticipates that precertified organizations would identify specific data analytics aligned to each of these domains and subdomains of the proposed RWPA framework based on the intended use, functionality, and risk category of the SaMD product.

Real-World Health Analytics (RWHA) are defined as analyses of real-world clinical outputs and outcomes related to the intended use of the SaMD product.

RWHA can inform changes to the intended use of a SaMD product, support expanded functionalities and use in broader target populations, and identify emerging safety issues in postmarket use. While all medical device manufacturers are required to report clinical adverse events related to their products, excellent organizations would leverage the ability of appropriately designed SaMD products to collect and analyze clinical safety data, in order to identify and address issues in a more timely manner. For lower-risk products, RWHA may be collected from sources including user complaints, search analytics, and product-level monitoring of human factors measures, such as use errors. Depending on the claim, FDA anticipates that precertified organizations with higher-risk products may proactively seek out external sources of safety and effectiveness data through activities, such as participation in registries, partnerships with healthcare systems, or utilization of data commons or other structured postmarket data collection efforts.

Table 1 is the example how RWHA is evaluated via CQOE or Excellenc principles.

【導讀】使用者經驗（UX）分析，一直產品評估的重要一環。但就一般醫材所強調的使用性皆與安全性有關。就本架構列入本項，個人的看法認為在於，鼓勵好的使用者經驗，才會讓使用者主動回饋資訊，以更多元的角度回饋產品的問題。

行筆至此，本Program不單單是是法規的要求，可看出如何用法規要求的方式，引導廠商開發安全有效，甚至是商業成功的產品。

User Experience Analytics (UXA) are defined as analyses of user experience outputs related to the real-world use of a SaMD product.

UXA monitoring facilitates timely identification and correction of user issues and enables improvements to the utilization and effectiveness of the software. Depending upon the intended use of the software, UXA may be collected from a variety of sources. Products instrumented to collect UXA may passively monitor measures including use patterns, download rates, and user retention. For metrics related to product satisfaction or user complaints, organizations may use proactive mechanisms of soliciting or incentivizing user feedback. To ensure that feedback is representative of the full range of users, excellent organizations would also actively seek UXA from diverse sources, which may include social media platforms, search analytics, or other third-party online networks.

Table 2 is the example how UXA is evaluated via CQOE or Excellence principles.

Product Performance Analytics (PPA) are defined as analyses of outputs and outcomes demonstrating the real-world accuracy, reliability, and security of a SaMD product.

PPA monitoring enables excellent organizations to address software bugs and security vulnerabilities through timely patches and product updates. FDA expects that excellent organizations would instrument SaMD products to track product performance measures related to the reliability and availability of the device. As described for UXA, proactive surveillance may be needed to identify product defects and track time to resolution

Table 3 is the example how PPA is evaluated via CQOE or Excellence principles.

【導讀】這篇的邏輯就是先介紹前面的架構，然後本節提的就是根據這個架構所訂的計畫，計畫訂好了，就是為下一節鋪路，也就是執行

RWPA Collection Plan

The RWPA framework for post-launch product monitoring is intended to provide robust evidence supporting the determination of safety and effectiveness of a particular SaMD product from a precertified organization, while retaining sufficient flexibility to accommodate the full range of organizations capable of demonstrating excellence in digital health.

For all products, a RWPA plan should include:

• Proposed RWP data elements to be collected;
• Intended frequency of data collection;
• Intended data structure and format; and
• Commit and stretch goals for each proposed data element.

For purposes of further optimizing the RWPA framework, FDA intends to review submitted RWPA plans and to work with individual Pilot Participants to refine needed data elements in the 2019 test phase。

【導讀】前面提了計畫，這節提的是執行，執行的目的有4個，Verification of ongoing excellence， Early identification and remediation of safety and security vulnerabilities， Addressing premarket uncertainty，以及Generation of adequate clinical evidence to support label expansion.

其無非就是確認性能，功效是和標的，如果發現有任何問題，也可儘快改善

RWPA Monitoring

The purpose for which RWPA is being monitored and analyzed influences the type of data to be collected and the duration of data collection. Precertified organizations would use best practices of monitoring and analyzing product-specific data to demonstrate their ongoing commitments to organizational excellence, and to identify and address any safety or security issues in a timely manner.

• Verification of ongoing excellence. Verification that precertified organizations are identifying, tracking, analyzing, and responding to measures related to the safety and effectiveness of SaMD products postmarket enables FDA to have continued confidence in the excellence of the organization. Robust reporting and increased transparency in postmarket data collection on the part of precertified organizations may also enable FDA to explore opportunities to optimize existing postmarket obligations for precertified organizations, including reporting and inspection requirements.
• Early identification and remediation of safety and security vulnerabilities. Ongoing monitoring of product-specific RWPA enables organizations to respond rapidly to emerging issues, including safety and security vulnerabilities. Regular access to RWPA would increase FDA familiarity with the types of data being collected and any potential signals being monitored by precertified organizations. The Agency believes that this increased familiarity would facilitate collaborative engagement between FDA and precertified organizations when products require modifications or updates. Collaborative engagement aimed at rapidly addressing any postmarket safety or security concerns, in turn, may reduce the need for compliance actions and streamline review of any required product modifications.
• Addressing premarket uncertainty. In certain circumstances, the review team may determine that postmarket data should be utilized as a special control. RWPA used for this purpose will be collected over a defined time period and should be structured in collaboration with the review team to ensure that the evidence generated is appropriate, reliable, and scientifically valid.
• Generation of adequate clinical evidence to support label expansion. Manufacturers may introduce a SaMD product to market with limited functionality at first, intending to expand its functionalities and associated claims over time. FDA anticipates that continuous collection and analysis of RWPA would support precertified manufacturers in understanding postmarket performance and in improving the product over time. Where postmarket RWPA provide evidence of superior real-world performance as compared to premarket data, FDA would work with the precertified manufacturer to modify claims and labeling to reflect actual performance characteristics. Real-world performance data may identify issues that the manufacturer must address, as appropriate.

Note

Fig 2 is the summarized concept regarding Real World Performance Analytic. RWPA collection plan is based on the RWPA framework. Then RWPA monitoring should demonstrate Verification of ongoing excellence, Early identification & remediation of safety and security, Addressing premarket uncertainty, and Generation of adequate clinical evidence to support label expansion.

The appraisal via CQOE is conducted according to the RWPA framework including RWHA, UXA, and PPA.

【導讀】以下就是本章的重點節錄，說明架構，計畫，執行，及最後評鑑結果的關係。